Cognitive Beamforming in Underlay Two-Way Relay Networks With Multiantenna Terminals

This paper studies an underlay cognitive network consisting of a two-way amplify-and-forward (AF) relay and two multiantenna terminals (SU
₁
and SU
₂
). Despite enhanced spectral efficiency and spectrum utilization, the underlay network is limited by low power transmissions and short coverage owing to secondary-to-primary (S2P) and primary-to-secondary (P2S) interference. To alleviate these, we consider beamforming at SU
₁
and SU
₂
. However, concurrent bidirectional transmissions with the two-way relay complicates beamforming and power allocation. Nevertheless, we use the performance criterion of maximizing the worse received signal-to-interference-and-noise ratio (SINR) at SU
₁
and SU
₂
. The resulting maximization problem for the optimal beamforming vectors and power allocation is a nonconvex quadratically constraint quadratic program (QCQP), which is NP-hard. Thus, we develop an iterative bisection search, but determining its feasibility at each iteration is still a nonconvex NP-hard QCQP. We thus generate two equivalent interference minimization problems, which we solve by semidefinite relaxation (SDR). Simulation results show that our proposed optimal design improves SINR by as much as 20 dB. We also propose suboptimal maximal-ratio-transmission (MRT) and zero-forcing beamforming and maximal-ratio-transmission (ZFB-MRT), and develop their optimal power allocations. Importantly, the performance loss due to these suboptimal strategies is modest (e.g., as low as 1 dB for ZFB-MRT with optimal power allocation).