Challenges for IF performance evaluation in practice

This study presents a literature analysis of 52 publications on the topic of information fusion performance evaluation. The aim is to identify whether information fusion researchers are aware of the problematic nature of performance evaluation in practice, which is substantially different from performance evaluation in test environments. As expected, only 6% of the publications do actually report on studies conducted in practice, and therefore more such studies are needed. However, 58% of the publications do critically reflect on the problematic nature of performance evaluation in practice. As ground truth is lacking in practice, there is a need for (a set of) performance measures which are both comprehensive and adaptable to domain or situation context and changing circumstances over time. More studies need to address the overall question how multiple performance measures can function in concert and how exactly can be judged whether adaptation has been an improvement or not.