Efficient Content Replacement in Wireless Content Delivery Network with Cooperative Caching

Wireless content delivery networks (WCDNs) have received attention as a promising solution to reduce the network congestion caused by rapidly growing demands for mobile content. The amount of reduced congestion is intuitively proportional to the hit ratio in a WCDN. Cooperation among cache servers is strongly required to maximize the hit ratio in a WCDN where each cache server is equipped with a small-size cache storage space. In this paper, we address a content replacement problem that deals with how to manage contents in a limited cache storage space in a reactive manner to cope with a dynamic content demand over time. As a new challenge, we apply reinforcement learning, which is Q-learning, to the content replacement problem in a WCDN with coooperative caching. We model the content replacement problem as a Markov Decision Process (MDP) and finally propose an efficient content replacement strategy to maximize the hit ratio based on a multi-agent Q-learning scheme. Simulation results exhibit that the proposed strategy contributes to achieving better content delivery performance in delay due to a higher hit ratio, compared to typical existing schemes of least recently used (LRU) and least frequently used (LFU).