Perspectives on the fusion of image and non-image data

Increasingly, multi-sensor systems are being developed to collect, process, and disseminate image and non-image data. Applications include homeland security, monitoring of facilities, and military situation assessment. Fusion of image and non-image data has traditionally been performed with extensive human-in-the-loop involvement. Typically the image data are used as the "fundamental" data source with non-image data simply overlaid on the image data, or conversely the non-image data are treated as fundamental, and the image data are used to confirm the identity of observed entities. This paper discusses the problem of multi-sensor fusion and argues that new techniques are emerging that allows fusion of image and non-image data at multiple levels of inference from the "raw" data level, to the feature level, decision-level, and knowledge level.