Statistical Attention Layer for Neural Network Training

We examine input attention mechanism in neural networks and suggest several statistical functions to be used. We develop statistical attention functions and use them in combination with each other. The functions include the mean, maximum, minimum, standard deviation, coefficient of variation, geometric mean, range, root mean square, skewness, and kurtosis. We demonstrate the improvements obtained with the combination on a company dataset on click-through rate (CTR) prediction. When comparing the performance using four different CTR prediction models based on neural networks, the percentage improvements reach 0.14% in AUC, 5.43% in logloss, and 1.04% in F1. The statistical attention layer we developed can be included in any neural network model as a preprocessing step adapting the input and extends the reach of statistically-informed neural network architectures.