Cell-Free Networking for Integrated Data and Energy Transfer: Digital Twin Based Double Parameterized DQN for Energy Sustainability

Cell-free networking enables full cooperation among distributed access points (APs). This paper focuses on reducing the long-term energy consumption of a cell-free network in the downlink integrated data and energy transfer (IDET) for achieving energy sustainability. The resultant design includes both the AP classification on a large time-scale and the beamforming of the APs on a small time-scale in order to simultaneously satisfy the IDET requirements of data users and energy users. For dealing with binary integer actions (AP classification) and continuous actions (beamforming) together, we innovatively propose a stable double parameterized deep-Q-network (DP-DQN), which can be enhanced by a digital twin (DT) running in the intelligent core processor (ICP) so as to achieve faster and more stable convergence. Therefore, the cell-free network may avoid suffering from performance fluctuation during the training process. The simulation results demonstrate that our DP-DQN exceeds in convergence compared to other benchmarks while guaranteeing an optimal solution.