Design of Improper Constellations for Optimal Data Rates in Downlink NOMA Systems

This paper addresses the achievable rate improvement for the downlink two-user non-orthogonal multiple access (NOMA) system, in the context of imperfect successive interference cancellation (SIC), by means of improper signaling techniques. We investigate a basic scenario where the strong user transmits data from a proper constellation, subject to a minimum rate constraint, while the weak user adopts an improper signaling scheme. For rigor, the data rates are formulated in terms of the circularity coefficient of the improper constellation, under residual interference due to imperfect SIC. We find that such a scheme always increases the achievable rate of the strong user while the weak user may also benefit when the channel-to-noise ratios (CNRs) of the two users satisfy certain mild conditions. Moreover, analytical expressions for data rates are provided based on the power budget of the weak user, which allows us to find the optimal circularity coefficient for the best improper constellation mapping. Simulations on the downlink NOMA system support the analysis.