Multimodal Deep Learning Empowered Millimeter-Wave Beam Prediction

Traditional millimeter-wave beam selection or prediction algorithms typically rely on beam scanning measurements at the transceivers, incurring substantial training overhead and exhibiting limited adaptability in diverse environments. Recent efforts have aimed to mitigate these challenges by incorporating sensing information, thereby reducing or eliminating the need for extensive beam training. However, existing works predominantly concentrate on exploiting a single sensing modality and often overlook the potential benefits of utilizing historical sensing information. In this paper, we introduce an intelligent beam prediction framework that leverages a deep integration of multimodal sensing data, encompassing GPS, camera, radar, and LiDAR data. The design proposed involves the application of customized deep neural networks to extract features from camera, radar, and LiDAR data. These extracted features, combined with user position and selected beam index, are concatenated to form an aggregated feature vector at each time instance. Subsequently, a time series of these concatenated feature vectors is utilized to exploit temporal correlation for beam prediction through a dedicated long short-term memory network module. Numerical simulations confirm the effectiveness of the proposed design and its superiority over several considered state-of-the-art baselines.