Joint source and relay matrices optimization for interference MIMO relay systems

In this paper, we investigate the transceiver design for an amplify-and-forward interference multiple-input multiple-output (MIMO) relay communication system. The minimum mean-squared error (MMSE) of the signal waveform estimation is chosen as the design criterion to optimize the source, relay, and receiver matrices for interference suppression. An iterative algorithm is proposed to solve the nonconvex source, relay, and receiver optimization problem. Simulation results demonstrate that the proposed algorithm outperforms the existing technique in terms of both MSE and bit-error-rate.