Deep Combination of Radar With Optical Data for Gesture Recognition: Role of Attention in Fusion Architectures

Multimodal time-series classification is an important aspect of human gesture recognition, in which limitations of individual sensors can be overcome by combining data from multiple modalities. In a deep-learning pipeline, the attention mechanism further allows for a selective, contextual concentration on relevant features. However, while the standard attention mechanism is an effective tool when working with natural language processing (NLP), it is not ideal when working with temporally- or spatially-sparse multimodal data. In this article, we present a novel attention mechanism, multimodal attention preconditioning (MMAP). We first demonstrate that MMAP outperforms regular attention for the task of classification of modalities involving temporal and spatial sparsity and second investigate the impact of attention in the fusion of radar and optical data for gesture recognition via three specific modalities: dense spatiotemporal optical data, spatially-sparse/temporally-dense kinematic data, and sparse spatiotemporal radar data. We explore the effect of attention on early, intermediate, and late fusion (LF) architectures and compare eight different pipelines in terms of accuracy and their ability to preserve detection accuracy when modalities are missing. Results highlight fundamental differences between late and intermediate attention (IA) mechanisms with respect to the fusion of radar and optical data.