Perspectives on visualization and virtual world technologies for multi-sensor data fusion

Rapid advances in visualization technology and virtual world tools provide opportunities for improvements in multisensor data fusion. These technologies can re-engage the human user in the fusion process, improving multi-analyst collaboration, enhancing data understanding by engaging the analystpsilas visual pattern recognition capabilities, and providing new mechanisms for hypothesis generation and understanding. The virtual world environments can leverage gaming concepts to provide rich story-telling capabilities. Much like the traditional use of cases or logical templates for target identification or event/activity detection, gaming concepts involving characterization of characters and world views can assist the formulation and evaluation of hypotheses for non-traditional targets. As new requirements emerge for fusion systems to support asymmetric warfare and non-traditional operations, these technologies become increasingly important. This paper provides a perspective on these concepts and argues for a systematic theory-driven approach to explore these enhancements to data fusion, grounded in human-in-the-loop experiments.

1 Introduction

In 2000, Hall, Hall and Tate [1] proposed the inclusion of the Human Computer Interface as Level five of the Joint Directors of Laboratory (JDL) Data Fusion Process Model (Figure 1). The aim of the introduction of a level 5 process was to highlight the need to explicitly consider the interaction between automated data fusion processes and a human-in-the-loop analyst/decision-maker. The level-5 process ([2], [3]) is considered by its proponents as a human-side analog to the sensor-side level 0 process that had previously been introduced by Steinberg et al [4]. That is, level 0 processing was originally introduced to recognize the increasing evolution of smart sensors having embedded processing such as pre-detection fusion, advanced signal and image processing and pattern recognition. In effect, the level 0 process enhanced sensor oriented functions that affected the other traditional levels (1–4) of fusion processing. Using a similar analogy, Hall et al [1] argued that advances in human computer visualization tools, cognitive aids, and collaboration tools should be explicitly considered as advances that affect the human side of fusion processing and their effect on the traditional levels of fusion processing. Figure 1: JDL Data Fusion Model with Level 5 [3]