Design criterion and construction methods for partially coherent multiple antenna constellations

We consider multiple-antenna communication systems in Rayleigh fading channel, where the transmitter does not know the channel coefficients and the receiver has only an estimate of them. We further assume that the transmitter and receiver know the statistics of the estimation error. We refer to this system as partially coherent system, for which we derive the expressions for the optimal detector and study the constellation design problem. Since the Chernoff bound on pairwise error probability of the partially coherent systems appears to be intractable, we use Stein's Lemma to propose a design criterion based on Kullback-Leibler (KL) distance between conditional distributions. Using the KL-based design criterion, we construct constellations for multiple-antenna systems which can be demodulated in the presence of channel estimation errors. The proposed constellations are multi-level, with multi-dimensional spherical constellations at each level. We show that these new constellations provide significant performance improvement over the conventional single-antenna PSK and QAM constellations, and multiple-antenna techniques such as Bell Lab's Space-Time (BLAST) architecture and orthogonal transmit diversity (OTD) schemes, when the estimation variance is comparable to the reciprocal of the signal-to-noise ratio.