Success Probability of the Babai Estimators for Box-Constrained Integer Linear Models

In many applications including communications, one may encounter a linear model where the parameter vector x̂ is an integer vector in a box. To estimate x̂, a typical method is to solve a box-constrained integer least squares problem. However, due to its high complexity, the box-constrained Babai integer point x
^BB
is commonly used as a suboptimal solution. In this paper, we first derive formulas for the success probability P
^BB
of x
^BB
and the success probability POB of the ordinary Babai integer point x
^OB
when x̂ is uniformly distributed over the constraint box. Some properties of P
^BB
and P
^OB
and the relationship between them are studied. Then, we investigate the effects of some column permutation strategies on P
^BB
. In addition to V-BLAST and SQRD, we also consider the permutation strategy involved in the LLL lattice reduction, to be referred to as LLL-P. On the one hand, we show that when the noise is relatively small, LLL-P always increases P
^BB
and argue why both V-BLAST and SQRD often increase P
^BB
; and on the other hand, we show that when the noise is relatively large, LLL-P always decreases P
^BB
and argue why both V-BLAST and SQRD often decrease PBB. We also derive a column permutation invariant bound on P
^BB
, which is an upper bound and a lower bound under these two opposite conditions, respectively. Numerical results demonstrate our findings. Finally, we consider a conjecture concerning x
^OB
proposed by Ma et al. We first construct an example to show that the conjecture does not hold in general, and then show that it does hold under some conditions.