Systemic test and evaluation of a hard+soft information fusion framework: Challenges and current approaches

The area of hard+soft fusion is a relatively new topic within the information fusion community. One research effort which has confronted the subject of hard+soft fusion is the Multi-disciplinary University Research Initiative (MURI) titled “Unified Research on Network-based Hard+Soft Information Fusion”. Developed on this program is a fully integrated research prototype hard+soft fusion system in which raw hard and soft data are processed through hard sensor processing algorithms, natural language understanding processes, common referencing, alignment, association and situation assessment fusion processes. The MURI program is currently in its 5th (and last) year. During years 1 through 4, the MURI team dealt with the research issues in developing a baseline hard+soft fusion system, while identifying a number of design alternatives for each of the framework processing elements. For example, within natural language understanding different stemmers or ontologies could be utilized. The mathematical nature of hard or physical sensor processing and data association involved design choices about numerous parameters which affect the solution quality and solution quality/runtime tradeoff. While traditional experimental or training approaches may be used in assessing these processes in isolation, the nature and dependencies of hard+soft fusion require a systemic approach in which the integrated performance of framework components are understood. In this paper we describe the design of a test and evaluation framework for systemic error trail analysis and parametric optimization of hard+soft fusion framework sub-processes. We will discuss the performance metrics utilized including notions of “system optimality,” issues in defining the parametric space (design variants), cross-process error tracking methodologies and discuss some initial results. The presented system results are based on the Synthetic Counterinsurgency (SYNCOIN) dataset which is a dataset developed within the program and utilized for training and system optimization. Future work, including plans for the validation of experimental results will also be discussed.