Rate-optimal communication under nonlinear Gaussian noise via constellation shaping

In the traditional model of wireless communications systems, additive white Gaussian noise (AWGN) is assumed to be linear. However, in an increasingly important class of emerging communication systems — e.g. device-to-device (D2D), full-duplex (FD) and low-cost Internet-of-things (IoT) communication systems — the nonlinearity caused by factors such as device proximity (D2D), residual self-interference (FD) and imperfect power amplification (IoT) can no longer be neglected. A possible mechanism to improve the performance of such systems is to optimize the transmit constellation utilized, which is known in the literature as constellation shaping. With that in mind, we propose a probabilistic constellation shaping scheme to maximize the achievable rates of communication systems affected by nonlinear AWGN. To this end, we derive the analytical expression of the mutual information (MI) of such nonlinear additive white Gaussian noise (AWGN) systems with arbitrary modulation, and maximize the latter by numerically optimizing the corresponding channel input distribution. The result is a semi-analytical scheme (with analytical objective optimized numerically) which are shown to outperform systems employing conventional (uniformly distributed) constellations.