Test and evaluation of data association algorithms in hard+soft data fusion

The information gathered by sources during counter insurgency (COIN) operations can be classified into two types. The data gathered by humans and recorded in the textual format such as field reports is referred to as soft data. Data gathered with the help of physical sensors such as video cameras, LIDAR, acoustic sensors, etc. is referred to as hard data. To process this information, various hard and soft processing techniques are used, which convert these data into relational graphs. Many times this hard and soft data contains duplicate references of the same entities, events and relationships, caused by multiple sources reporting on the same entity, event or relationship or due to successive reports on an entity, event or relationship. The role of data association is to identify these duplicate references across the different observations and merge them into fused (cumulative) evidence. This cumulative evidence will contain more information about the real world than offered by any single observation. We want the cumulative evidence to describe the real world as accurately as possible, so as to draw satisfactory conclusions on the state of the real world. This calls for development of an objective strategy for evaluating the performance of data association processes. In this paper, we describe in detail the testing and evaluation strategy developed for this task. This strategy was deployed for the evaluation of three different data association algorithms on the Sunni Criminal (SUN) Thread of the Synthetic Counterinsurgency (SYNCOIN) dataset. The SUN thread consists of 114 soft messages and 13 hard messages. This test and evaluation strategy can also help in selecting the best data association algorithm to deploy based on the different properties of the input dataset and processing time permissible.