Design of Asymptotically Optimal Improper Constellations With Hexagonal Packing

This paper addresses the problem of designing asymptotically optimal improper constellations with a given circularity coefficient (correlation coefficient between the constellation and its complex conjugate). The designed constellations are optimal in the sense that, at high signal-to-noise-ratio (SNR) and for a large number of symbols, yield the lowest probability of error under an average power constraint for additive white Gaussian noise channels. As the number of symbols grows, the optimal constellation is the intersection of the hexagonal lattice with an ellipse whose eccentricity determines the circularity coefficient. Based on this asymptotic result, we propose an algorithm to design finite improper constellations. The proposed constellations provide significant SNR gains with respect to previous improper designs, which were generated through a widely linear transformation of a standard M-ary quadrature amplitude modulation constellation. As an application example, we study the use of these improper constellations by a secondary user in an underlay cognitive radio network.