Optimal Design of Spatial Source-and-Relay Matrices for a Non-Regenerative Two-Way MIMO Relay System

This paper considers a system where two users exchange information via a non-regenerative half-duplex two-way MIMO relay and each of the two users and the relay is equipped with multiple antennas. We study the design of the spatial source covariance matrices (or source matrices) at the two users and the spatial transformation matrix (or relay matrix) at the relay to maximize the achievable weighted sum rate of the system. The source matrices and the relay matrix are optimized alternately until convergence. If the relay matrix is given, we show that the optimal design of the source matrices (for uniformly weighted sum rate) follows a generalized water filling (GWF) algorithm. If the source matrices are given, we show two search algorithms to optimize the relay matrix. The first algorithm is a hybrid gradient method which adaptively switches between the (steepest) gradient descent and the Newton's search. The second is an iterative weighted minimum mean square error (WMMSE) method which alternately refines the MMSE equalizers at the users and the relay matrix. We compare the convergence behaviors of the two algorithms and demonstrate their advantage over prior algorithms. We also show an optimal structure of the relay matrix, which is useful to reduce the search complexity.