Optimal Modulation Order and Power Allocation for Rate-Splitting Multiple Access Scheme

In the era of 6G, there is a growing need for a more integrated approach to effective interference management (IM). Rate-splitting multiple access (RSMA) has emerged as a promising IM solution by partitioning each user's data into a single composite common stream (CS) and private streams (PSs). This partitioning enables flexible modulation order (MO) assignments for either the CS or PSs while maintaining the targeted information bits per user per transmission (IBPU). Each assignment is expected to have a distinct optimal power allocation (PA) between the CS and PSs, with certain configurations anticipated to outperform other available choices. However, to the best of our knowledge, prior work has not comprehensively investigated the full range of MO combinations for each stream. This paper addresses these gaps by employing a constellation-based analytical framework to derive the precise symbol error rate (SER) for a two-user RSMA scheme, wherein each user's CS and PSs utilize square quadrature amplitude modulation (QAM). The analysis considers both perfect and imperfect detection scenarios for the CS. Additionally, a power constraint analysis is conducted across arbitrary MOs to effectively address constellation ambiguity. The average SER expressions are derived in closed form under ordered Rayleigh fading channels and are validated through Monte Carlo simulations. Furthermore, an investigation is conducted to identify the optimal MOs and PAs for the CS and PSs to minimize error rates. Finally, optimal MO values are determined for both odd and even IBPU scenarios.