Limited-Resolution Hybrid Analog–Digital Precoding Design Using MIMO Detection Methods

While fully-digital precoding achieves superior performance in massive multiple-input multiple-output (MIMO) systems, it comes with significant drawbacks in terms of computational complexity and power consumption, particularly when operating at millimeter-wave (mmWave) frequencies and beyond. Hybrid analog-digital architectures address this by reducing radio frequency (RF) chains while maintaining performance in sparse multipath environments. However, most hybrid precoder designs assume ideal, infinite-resolution analog phase shifters, which cannot be implemented in real systems. Another practical constraint is the limited fronthaul capacity between the baseband processor and array, implying that each entry of the digital precoder must be picked from a finite set of quantization labels. This paper proposes novel designs for the limited-resolution analog and digital precoders by exploiting two well-known MIMO symbol detection algorithms, namely sphere decoding (SD) and expectation propagation (EP). Unlike prior works that rely on heuristic or sub-optimal designs for the low-resolution hybrid precoder, the proposed transformative MIMO detection-inspired designs are able to achieve optimal and near-optimal solutions. The main objective is to minimize the Euclidean distance between the optimal fully-digital precoder and the hybrid precoder to minimize the degradation caused by the finite resolution of the analog and digital precoders. Taking an alternating optimization approach, we first apply the SD method to find the precoders in each iteration optimally. Then, we apply the lower-complexity EP method which finds a near-optimal solution at a reduced computational cost. The effectiveness of the proposed designs is validated through extensive numerical simulations, which demonstrate that both SD-based and EP-based hybrid precoding schemes significantly outperform widely-used sub-optimal approaches.