Deep Reinforcement Learning for Dynamic Spectrum Access: Convergence Analysis and System Design

In dynamic spectrum access (DSA) networks, secondary users (SUs) need to opportunistically access primary users’ (PUs) radio spectrum without causing significant interference. Since the SU-PU interaction is limited, deep reinforcement learning has been introduced to help SUs conduct spectrum access. Specifically, deep recurrent Q network (DRQN) has been utilized in DSA networks for SUs to aggregate information from recent experiences to make spectrum access decisions. DRQN is notorious for its sample efficiency since it needs a rather large number of training samples to tune its parameters which is a computationally demanding task. Deep echo state network (DEQN) has been introduced for DSA networks to address the sample efficiency issue of DRQN. In this work, we compare the convergence of DRQN and DEQN by comparing the upper bounds we obtain on their covering number, a notion of richness. Furthermore, we introduce a method to determine the right hyper-parameters for DEQN, providing system design guidance for DEQN-based DSA networks. Extensive performance evaluation confirms that DEQN-based DSA strategy is the superior choice with regard to computational power while outperforming DRQN-based ones.