Outage Probability and Ergodic Capacity of User Clustering and Beamforming MIMO-NOMA Relay System With Imperfect CSI Over Nakagami-$m$ Fading Channels

Although the theory of nonorthogonal multiple access (NOMA) has been widely investigated, so far, there is a lack of applying beamforming into NOMA systems. In this article, we propose a multiple-input and multiple-output (MIMO)-NOMA system, where all users in the whole system are divided into multiple clusters to reduce the inner interference. Next, the superpositioned signals at base station are beamed to the relays, and then, the relays communicate with the users through superposition coding in the power domain. We derive the exact expressions of the outage probability of each user and the ergodic capacity (EC) of each cluster in both cases, i.e., perfect successive interference cancellation (SIC) and imperfect SIC, taking into account the Nakagami-m fading channels and imperfect channel state information (CSI). We also compare the outage performance and the EC of our proposed MIMO-NOMA relay system with an orthogonal multiple access (OMA) relay system. Monte Carlo simulations are used validate the derived mathematical expressions. Numerical results show that the outage performance and the EC of each user and the whole system are reduced remarkably due to the imperfect CSI but get better as the fading severity is less. Moreover, the EC of our proposed MIMO-NOMA relay system is higher than that of the OMA relay system.