Aircraft Marshaling Signals Dataset of FMCW Radar and Event-Based Camera for Sensor Fusion

The advent of neural networks capable of learning salient features from radar data has expanded the breadth of radar applications, often as an alternative sensor or a complementary modality to camera vision. Gesture recognition for command control is the most commonly explored application. Nevertheless, more suitable benchmarking datasets are needed to assess and compare the merits of the different proposed solutions. Furthermore, most current publicly available radar datasets used in gesture recognition provide little diversity, do not provide access to raw ADC data, and are not significantly challenging. To address these shortcomings, we created and made available a new dataset that combines two synchronized modalities: radar and dynamic vision camera of 10 aircraft marshaling signals at several distances and angles, recorded from 13 people. Moreover, we propose a sparse encoding of the time domain (ADC) signals that achieve a dramatic data rate reduction (>76%) while retaining the efficacy of the downstream FFT processing (<2% accuracy loss on recognition tasks). Finally, we demonstrate early sensor fusion results based on compressed radar data encoding in range-Doppler maps with dynamic vision data. This approach achieves higher accuracy than either modality alone.