Multimodal Visual Image Based User Association and Beamforming Using Graph Neural Networks

This paper proposes an approach that leverages multimodal data by integrating visual images with radio frequency (RF) pilots to optimize user association and beamforming in a downlink wireless cellular network under a max-min fairness criterion. Traditional methods typically optimize wireless system parameters based on channel state information (CSI). However, obtaining accurate CSI requires extensive pilot transmissions, which lead to increased overhead and latency. Moreover, the optimization of user association and beamforming is a discrete and non-convex optimization problem, which is challenging to solve analytically. In this paper, we propose to incorporate visual camera data in addition to the RF pilots to perform the joint optimization of user association and beamforming. The visual image data help enhance channel awareness, thereby reducing the dependency on extensive pilot transmissions for system optimization. We employ a learning-based approach based on using first a detection neural network that estimates user locations from images, and subsequently two graph neural networks (GNNs) that extract features for system optimization based on the location information and the received pilots, respectively. Then, a multimodal GNN is constructed to integrate the features for the joint optimization user association and beamforming. Simulation results demonstrate that the proposed method achieves superior performance, while having low computational complexity and being interpretable and generalizable, making it an effective solution as compared to traditional methods based only on RF pilots.