Achieving Capacity Gains in Practical Full-Duplex Massive MIMO Systems: A Multi-Objective Optimization Approach Using Hybrid Beamforming

This paper presents a novel approach to full-duplex (FD) massive multiple-input multiple-output (mMIMO) systems using hybrid beamforming (HBF) architecture, enabling simultaneous uplink (UL) and downlink (DL) transmission within the same frequency band. The proposed solution aims to mitigate strong self-interference (SI) and maximize the total achievable rate based on over-the-air (OTA) measurements of the SI channel. Our objective is to leverage the spatial degrees of freedom (DoF) in mMIMO systems to enhance FD capacity without the need for expensive analog SI-cancellation circuitry. To address this challenging issue, we employ a sub-array configuration for transmit and receive antennas at the base station (BS) and design the RF stages using non-orthogonal beamforming (NOBF) in both UL and DL user directions. Additionally, sub-array selection (SAS) is utilized to identify the optimal Tx-Rx antenna pair. To solve the non-convex multi-objective optimization problem (MOOP), we propose a swarm intelligence-based algorithmic solution to determine the optimal perturbations in user directions jointly with Tx-Rx sub-array indices while satisfying directivity degradation constraints. The illustrative results show that the proposed NOBF scheme with SAS can achieve an SI suppression of −78 dB. Furthermore, in FD mMIMO systems, this approach can effectively double the capacity compared to half-duplex (HD) transmissions.