Probabilistic Examination of Least Squares Error in Low-Bitwidth Cholesky Decomposition

In this paper, we propose a new approach to justify a round-off error impact on the accuracy of the linear least squares (LS) solution using Cholesky decomposition. This decomposition is widely employed to inverse a matrix in the linear detector of the Multi-User multi-antenna receiver. The proposed stochastic bound is much closer to actual errors than other numerical bounds. It was tested with a half-precision format and validated in realistic scenarios. Experimental results demonstrate our approach predicts errors very close to those achieved by simulations. The proposed approach can be employed to analyze the resulting round-off error in many other applications.