Federated Reinforcement Learning with Knowledge Transfer for Network Selection in Hybrid WiFi-VLC Networks

The dramatic growth of mobile data demand may impose a heavy traffic burden in indoor environments such that conventional radio frequency wireless networks might not be enough to ensure a high quality of service (QoS). Applications such as texting, 4k video streaming, and virtual reality have substantial differences in terms of data rate requirements. Visible light communication (VLC) has emerged recently as a complementary technology in heterogeneous networks, as it supports a high data rate and uses an unlicensed spectrum. In hybrid WiFi-VLC networks, users can be assigned to WiFi or VLC access points (AP) in downlink to maximize the minimum user data rate. In this paper, we propose an intelligent network selection that is capable of reducing network complexity and enhancing users’ QoS. A federated Q-learning (FQL) algorithm is proposed, where each VLC AP performs local Q-learning and updates the global model at the WiFi AP. A new local reward function is proposed to maximize user satisfaction with limited shared information. Furthermore, a deep neural network (DNN) is proposed to reduce the complexity of the FQL. The output of the DNN is adjusted to assign some users directly to WiFi or VLC and create an initial policy for the FQL. To quantify the improvements achieved, the proposed federated Q-learning assisted by knowledge transfer (FQL-KT) is compared against state-of-the-art benchmark approaches, such as centralized Q-learning (CQL), FQL, signal strength strategy (SSS), and exhaustive search. Numerical simulation results show that the proposed FQL-KT significantly outperformed the other approaches in terms of convergence speed, maximizing the minimum user data rate, fairness, and minimizing the outage probability.