Multi-User Rate-Splitting Approach for Enhanced Private Stream Interference Management with Error and Power Bound Analysis

Rate-splitting multiple access (RSMA) has emerged as a promising solution for managing multi-user interference in wireless communication systems. However, a significant challenge persists in mitigating high interference during private stream detection in conventional RSMA and quadrature-RSMA (Q-RSMA), particularly in multi-user scenarios. To address this issue, this work proposes a rate-splitting technique called private stream interference-management RSMA (P-RSMA), which leverages coordinate interleaving among combinations of private streams to reduce the number of interfering symbols. To analyze and compare P-RSMA with RSMA and Q-RSMA, a constellation-based analytical framework is presented to derive symbol error rate (SER) expressions for multi-user P-RSMA, RSMA, and Q-RSMA schemes under the single-input single-output (SISO) setting, considering both perfect and imperfect detection events of the common stream. A detailed power-bound analysis is also performed to resolve the constellation ambiguity problem in all considered schemes for arbitrary modulation orders of the common and private streams, emphasizing the strategic importance of adapting modulation orders for different symbols. Closed-form uncoded and coded average SER (ASER) expressions are then derived under Rayleigh fading channels with channel ordering. This work further incorporates a diversity order evaluation for the constellation-based framework to characterize asymptotic performance and benchmark against SINR- based methods. Additionally, the analytical framework is extended to the multiple-input multiple-output (MIMO) setting, and results are compared with those from the SISO scenario to demonstrate the performance gains and practical relevance of the proposed P-RSMA scheme. Numerical results demonstrate the superiority of P-RSMA over RSMA and Q-RSMA, which are corroborated through Monte Carlo simulations.