Design and Analysis of Rotated-QAM Based Probabilistic Shaping Scheme for Rayleigh Fading Channels

Probabilistic shaping (PS) combined with bipolar amplitude shift keying modulation is a promising transmission scheme. However, such 1-dimensional PS scheme inevitably suffers shaping loss in Rayleigh fading channels due to the lack of signal diversity. In this paper, we design a rotated quadrature amplitude modulation based PS strategy. With the discrete inputs, the ergodic constellation constrained (CC) capacity is derived based on the mutual information conditioned on full channel state information available at the receiver, which is found to be mutually influenced by both the probability mass function (PMF) and the rotation angle (RA) of the input signals. Furthermore, the closed-form cutoff rate is also derived to simplify the analysis, which is the lower bound of the mutual information with finite decoding complexity. Correspondingly, the optimizations of RA and PMF are both considered for maximizing the ergodic CC capacity and the cutoff rate, respectively, where the single parameter RA is firstly obtained by the exhaustive search. And then, the optimal PMFs of the subproblem for the fixed RAs and the fixed constellation scaling factor are obtained. Furthermore, with this non-equiprobable transmission scheme, the closed-form pairwise error probability is derived for the first time, which also validates the superiority of the proposed PS scheme in comparison with the uniform schemes with elaborate rotations.