Self-Optimizing Data Offloading in Mobile Heterogeneous Radio-Optical Networks: A Deep Reinforcement Learning Approach

In addition to the exploration of more spectrum at high-frequency bands, next-generation wireless networks will witness an intelligent convergence of radio frequency (RF) and non-RF links such as optical and visible light communication. Optical attocell (OAC) networks provide an additional layer to RF-based wireless networks with gigabit-per-second data transmission rate and centimeter-level location accuracy. However, the directionality, line-of-sight constraints, as well as strong sensitivity to the location and orientation of user terminals challenge the stringent requirements for throughput and latency. In this article, we consider mobile heterogeneous networks (HetNets) incorporating indoor OAC with femtocells and macrocells to provide a low-cost and energy-efficient solution. The HetNets solution satisfies diverse service requirements in terms of user-experienced data rate, mobility, latency, accuracy, and security in the Internet of Things. To support seamless connectivity and optimal resource allocation in the proposed HetNets with mobility awareness, handover in dynamic environments needs to be addressed efficiently. Incorporating rich environmental parameters into such a decision making problem facilitates the self-optimization process, but extensively expands the state space. To achieve a fast convergence speed, a deep reinforcement learning approach is proposed to optimize the handover parameters (e.g., time-to-trigger and hysteresis margin). This is a model-free and off-policy reinforcement setting that trains and employs a deep neural network to predict future rewards for successions of states and actions. Thus, the optimal parameters are obtained by selecting the best actions to take. Through numerical simulation and performance analysis, we discover the gain from enriching the state space and the adaptability of the system to dynamic environments.