Performance analysis of inter-cell interference reduction using AOA-based beamforming in Mm wave massive MIMO systems

Millimeter Wave Massive MIMO is considered as one of the main features in the upcoming fifth generation (5G) systems. Inter-cell interference which is represented by pilot contamination in Massive MIMO is less critical at mm wave frequencies due to mm waves' characteristics, however it still has side effects on such system, which cannot be ignored. In this paper, we focus on pilot contamination effects by analyzing the uplink transmission for the mm wave Massive MIMO system that employing hundreds of antenna arrays at Base Station (BS) and multiple antennas at each Mobile Station (MS). Through the uplink training, the channel estimation is used to detect the signals transmitted from a plurality of MSs in their respective cell. By taking the correlation between the channel estimate and the interference from other cells, which is represented as pilot contamination interference into consideration, we propose a 2D-unitary estimation of the signal parameters through rotational invariance techniques (U-ESPRIT) algorithm that utilizes the steering vectors of the uniform rectangular array (URA) to form a basis of the spatial space that can distinguish the desired signal from interfered signal without any change of pilot construction of training signals. The partial space for hybrid beamforming is being selected in accordance with Angle of Arrival (AOA) information, to improve the received average sum rate in the cell. We derive an analytical approximations of the exact and lower bound on the achievable rate. Consequently, the system performance has been evaluated numerically. Simulation results show that our proposal achieved low computational complexity and can achieve better performance in mitigating the majority of inter-cell interference and outperform other prominent techniques that are based on pilot identity only.