Joint Transceiver Beamforming Design for End-to-End Optimization in Full-Duplex MIMO Relay System With Self-Interference

In this letter, a novel transceiver beamforming design is proposed for improving the end-to-end performance of a full-duplex amplify-and-forward relay system. Different from previous schemes based on nulling out residual self-interference (RSI), our proposed schemes can capture a balance between useful signal improvement and RSI suppression for further achievable rate improvement. To solve the nonconvex optimization problem, an iterative algorithm with proven convergence is developed. Specifically, at each iteration, an optimal solution derived by reformulating the optimization problem as a fractional program problem and an efficient sub-optimal closed-form solution are presented, respectively. The simulation results show that the proposed design outperforms the traditional schemes in all signal-to-noise ratio regime.