A Markovian approach to spectrum access in cooperative spectrum sharing networks

In cooperative spectrum sharing networks, dynamic spectrum access is regraded as an important and challenging issue. In this paper, we consider a non-real-payoff cooperative spectrum sharing between one primary user (PU) and one secondary user (SU) over multiple time slots where the channel conditions are changeable due to the channels fading. However, the real-payoff means is that the cooperative spectrum sharing between the PU and the SU is over a time slot where the channel conditions are constant. There exists three access means: the PU's access including the direct and the relay links, and the SU's access. We propose the access decision algorithm considering the access reward and the switching cost for the non-real-payoff means. Herein, the access reward depends on the system's capacity based on the links' channel conditions and the switching cost represents the overheads such as feedbacks and signallings in the communication networks. The access decision problem is formulated as a constrained Markov decision process (CMDP). The objective is to maximize the overall reward with the access cost constraints including the users' energy constraints and the PU's quality of service (QoS) constraints. The value iteration and Q-learning algorithms are used to determine the optimal access policy.