Transceiver Optimization for Two-Hop AF MIMO Relay Systems With DFE Receiver and Direct Link

In this paper, we consider precoding and receiving matrices optimization for a two-hop amplify-and-forward (AF) multiple-input multiple-output (MIMO) relay system with a decision feedback equalizer (DFE) at the destination node in the presence of the direct source-destination link. By adopting the minimum mean-squared error (MMSE) criterion, we develop two new transceiver design algorithms for such a system. The first one employs an iterative procedure to design the source, relay, feed-forward, and feedback matrices. The second algorithm is a non-iterative suboptimal approach which decomposes the optimization problem into two tractable subproblems and obtains the source and relay precoding matrices by solving the two subproblems sequentially. Simulation results validate the better MSE and bit-error-rate (BER) performance of the proposed algorithms and show that the non-iterative suboptimal method has a negligible performance loss when the ratio of the source node transmission power to the relay node transmission power is small. In addition, the computational complexity analysis suggests that the second algorithm and one iteration of the first algorithm have the same order of complexity. As the first algorithm typically converges within a few iterations, both proposed algorithms exhibit a low complexity order.