Neural Network-Based Fixed-Complexity Precoder Selection for Multiple Antenna Systems

In this paper, we propose a neural network-based precoder selection method for multiple antenna systems that are equipped with maximum likelihood detectors. We train a fully connected neural network by supervised learning with novel soft labels that are derived from the error probability of maximum likelihood detection. The dimension of the input data is reduced by QR decomposition of the channel matrices, thereby reducing the number of nodes of the input layer. Furthermore, the dimension reduction improves the network accuracy. The number of connections between the layers are reduced by applying the network pruning technique, after which the surviving connections are retrained to recover the degraded accuracy due to the pruning. We also optimize the regularization method, considering not only network overfitting but also pruning and retraining. Our method achieves a near optimal bit error performance of the previous sphere decoding (SD)-based symbolic algorithm, of which complexity fluctuates depending on channel matrices. Unlike the conventional SD-based method, the complexity of the proposed method is fixed by the intrinsic characteristic of neural network, which is desirable from the perspective of hardware implementation. And the fixed complexity is lowered by pruning unimportant connections of the networks. With the aid of computer simulations, we show that the fixed complexity of the proposed method is close to the average complexity of the conventional SD-based symbolic algorithm, allowing only negligible degradation of the error performance.