Multi-Modal Sensing and Communication for V2V Beam Tracking via Camera and GPS Fusion

Beam alignment is a critical challenge in vehicle-to-vehicle (V2V) communications, particularly at millimeter-wave (mmWave) frequencies, where massive multi-input multi-output (mMIMO) systems rely on high directivity to mitigate propagation losses. The highly dynamic nature of V2V scenarios, characterized by short channel coherence times, complicates channel state information (CSI) acquisition and increases the overhead associated with traditional beam training processes. To address these challenges, we propose a transformer-based beam tracking framework that leverages sensory data, including GPS and camera inputs, to enable efficient and proactive beam alignment without extensive training overhead. GPS data is transformed into relative coordinates, while features including relative velocity and orientation are extracted from this data. Moreover, camera images provide complementary contextual information about the surrounding environment, enhancing the accuracy of beam predictions. The model predicts beam states up to 500 ms ahead using predefined beamforming codebooks, facilitating both precise beam selection and effective blockage mitigation. Trained on the DeepSense V2V dataset, our method demonstrates a reduction of up to 7 dB in power loss and a 26 % improvement in top-5 accuracy compared to a linear interpolation baseline, highlighting the potential of GPS and camera fusion for advancing beam alignment in dynamic V2V mmWave networks.