Joint Transmit Beamforming and Antenna Selection Design for MISO-NOMA Systems Based on Improper Gaussian Signaling

In this paper, we consider the joint antenna selection (AS) and beamforming (BF) vector design problem for a downlink two-user power-domain non-orthogonal multiple access (PD-NOMA) system considering improper Gaussian signaling (IGS). We formulate an optimization problem concerned about the max-min rate between users with hardware constraints such as the total power constraint (TPC), per-antenna power constraint (PAPC), and the number of activated antennas along with the conditions for successful successive interference cancellation (SIC) that are proved in existing works. To deal with the log-determinant objective function and binary optimization, we propose a solution that combines successive concave quadratic programming (SCQP) and the exact penalty approach to obtain suboptimal results by iterations. As shown in simulation results, IGS can prevail over its proper Gaussian signaling (PGS) counterpart in terms of achievable sum rate (ASR) particularly in high transmit signal-to-noise ratio (SNR) scenarios.