A Survey of Multimodal Sensor Fusion for Passive RF and EO Information Integration

Integrating information collected by different types of sensors observing the same or related phenomenon can lead to more accurate and robust decision making. The purpose of this article is to review sensor fusion approaches to achieve passive radio frequency (RF) and electro-optical (EO) sensor fusion and to present the proposed fusion of EO/RF neural network (FERNN). While research has been conducted to integrate complementary data collected by EO and RF modalities, the processing of RF data usually applies traditional features, such as Doppler. This article explores the viability of using the histogram of I/Q (in-phase and quadrature) data for the purposes of augmenting the detection accuracy that EO input alone is incapable of achieving. Specifically, by processing the histogram of I/Q data via deep learning and enhancing feature input for neural network fusion. Using the simulated data from the Digital Imaging and Remote Sensing Image Generation dataset, FERNN can achieve 95% accuracy in vehicle detection and scenario categorization, which is a 23% improvement over the accuracy achieved by a stand-alone EO sensor.