A Novel Nonlinear Joint Transmitter-Receiver Processing Algorithm for the Downlink of Multiuser MIMO Systems

This paper focuses on signal processing algorithms for the downlink of multiuser multiple-input multiple-output (MIMO) systems with multiple-antenna mobiles. A novel nonlinear joint transmitter-receiver processing algorithm is proposed based on the zero-forcing (ZF) criterion. In this algorithm, nonlinear Tomlinson-Harashima precoding (THP) is applied at the base station, whereas linear receiver processing and modulo operation are applied at each mobile. It is first shown that the proposed algorithm effectively decomposes the multiuser MIMO channel into parallel independent single-user MIMO channels, and then, the performance of each mobile can be separately optimized. Subsequently, closed-form expressions for the transmitter and receiver processing matrices are derived to optimize the asymptotic bit error rate (BER) of each mobile. When used on the downlink of multiuser MIMO systems with multiple-antenna mobiles, this algorithm achieves significantly better performance than the ZFcriterion-based nonlinear preprocessing algorithm designed for the multiuser MIMO systems with single-antenna mobiles, because it effectively utilizes the processing capabilities of the mobiles. Moreover, the proposed algorithm achieves a much higher sum capacity at a high signal-to-noise ratio (SNR) than the known block diagonalization technique due to the effective application of the nonlinear preprocessing at the transmitter. When the proposed algorithm is applied, it is found that better system performance can be achieved by suitably ordering the channel matrices of different mobiles, and a combined optimal diversity and best-first (CODBF) ordering method is proposed to perform the ordering. Simulation is used to show the advantages of the proposed algorithm and the CODBF ordering method.