Linear MMSE Transceiver Design in Amplify-and-Forward MIMO Relay Systems

We consider the precoding problem in an amplify-and-forward (AF) multiple-input-multiple-output (MIMO) relay system in which multiple antennas are equipped at the source, the relay, and the destination. Most existing methods for this problem only consider the design of the relay precoder, and some even ignore the direct link. In this paper, we consider a joint source/relay precoder design problem, taking both the direct and the relay links into account. Using a minimum-mean-square-error (MMSE) criterion, we first formulate the problem as a constrained optimization problem. However, it is found that the mean square error (MSE) is a highly nonlinear function of the precoders, and a direct optimization is difficult to conduct. We then design the precoders to diagonalize the MSE matrix in the cost function. To do that, we pose certain structural constraints on the precoders and derive an MSE upper bound. It turns out that minimization with respect to this bound becomes simple and straightforward. Using the standard Lagrange technique, we can finally obtain the solution with an iterative water-filling method. Simulation results show that the proposed method, with an additional precoder, outperforms the existing methods, in terms of either the MSE or the bit error rate (BER).