A Unified Federated Deep Q Learning Caching Scheme for Scalable Collaborative Edge Networks

Edge caching-enabled networks can efficiently alleviate data traffic and improve quality of service. However, effectively adapting to users’ heterogeneous requests and coordinating among multiple edge servers remains a challenge. In this paper, we address the collaborative cache update and request delivery problem in an edge caching system, aiming to minimize the long-term average system cost under uncertainties of users’ heterogeneous demands and dynamic content popularity. To overcome the curse of dimensionality, we decompose the formulated problem into two subproblems: the coordinated proactive cache updating and local request processing. Next, we propose a unified federated deep Q learning (DQL) caching scheme to tackle and coordinate these two subproblems. Particularly, our scheme features a scalable DQL approach with a two-phase action selection procedure to learn the heterogeneous user requests across distributed servers in an online manner. Furthermore, we develop a federated learning (FL)-empowered training process to improve coordination among multiple servers, in which a Thompson sampling (TS)-based algorithm is introduced for smart server selection. We evaluate the performance of our proposed caching scheme in both small-scale and large-scale scenarios through comprehensive experiments, which highlights the advantages of the proposed scheme in terms of caching performance, scalability and robustness.