Improved Tomlinson-Harashima precoding for the downlink of multi-user MIMO systems

By pre-equalizing the interlayer interference at the transmitter, the Tomlinson-Harashima precoding (THP) algorithm provides a solution for the downlink of multi-user multiple-input multiple-output (MIMO) systems, in which, because of the absence of cooperation among mobiles, receiver-based layer-separation algorithms cannot be implemented. In this paper, a new THP algorithm using the minimum mean-square error (MMSE) criterion is developed for the downlink of multi-user MIMO systems. This algorithm, called the MMSE THP algorithm, provides significant performance improvement compared to the zero-forcing (ZF) THP algorithm discussed in the literature. Moreover, ordering of the rows of the channel matrix is found to be important to the THP system performance, and the way in which this ordering affects the system structure is presented. An ordering lemma that generalizes the conditions, under which the "best-first" ordering is optimal, is introduced.With this lemma it is proved that the "best-first" ordering method achieves the optimal order in the minimax noise-variance sense for the ZF THP and the optimal order in the minimax error-variance sense for the MMSE THP. Simulation results show the performance advantage attained by the MMSE THP and the optimal ordering.