User Clustering and Resource Allocation in Hybrid NOMA-OMA Systems Under Nakagami-m Fading

In this paper, we tackle the problem of optimizing user clustering, power, and resource (time slot or bandwidth) allocation in the downlink of a hybrid non-orthogonal multiple access (NOMA)-orthogonal multiple access (OMA) system. In such a system, users are organized into several clusters under one of the following scenarios: (1) fixed cluster size, (2) fixed number of clusters, and (3) variable number of clusters and variable cluster size. A power domain NOMA (PD-NOMA) scheme is used in each cluster, while OMA is employed for allocating resources to different clusters. The goal is to maximize the minimum success probability (which is equivalent to minimizing the maximum outage probability) among all users to guarantee fairness. We prove that at the optimal solution, all users have the same success probability, which is called the common success probability (CSP). Then, we propose an efficient algorithm for finding the optimal CSP and cluster resource allocation factors simultaneously. The optimal power allocation factors and the optimal decoding order of users in each cluster are then derived in closed-form expressions based on the obtained optimal CSP. Simulation results show considerable performance gains by the proposed scheme, compared to existing schemes in terms of fairness, the minimum success probability of users, and the sum throughput.