Joint Transceiver, Data Streams, and User Ordering Optimization for Nonlinear Multiuser MIMO Systems

We consider nonlinear signal processing algorithms for downlink multiple-input multiple-output (MIMO) systems with multiple-antenna users. The design goal is to improve bit error rate (BER) performance by jointly optimizing the transceiver, data streams, and user ordering, under given total transmit power. We consider a general global objective function, whose elements are Schur-convex functions of the mean square error (MSE) of each user. With nonlinear Tomlinson-Harashima precoding combined with block successive zero-forcing precoding at the transmitter, we show that the optimal nonlinear transceiver leads to favorable diagonal and parallel structures for all users with the general global performance metric. The number of data streams of each user is allowed to be an arbitrary number no more than the rank of the effective channel. We then provide closed-form expressions of the transceiver matrices and optimal power allocation of each user, and analytically characterize the optimal number of data streams of each user for the minimax and average BER metrics. The user ordering is also optimized to further improve the system performance. We also analytically investigate the impact of channel spatial correlation on the performance of our scheme and illustrate why our proposed adaptive strategy can mitigate the performance degradation caused by channel spatial correlation. The superiority of our proposed framework is demonstrated through numerical results.