Alpha-Backpropagation Algorithm: Probability Estimation for One-Hot and Quasi-One-Hot Teachers

This study proposes theories and applications of probabilistic divergences to neural network training. This theory generalizes the cross-entropy method for backpropagation to the alpha-divergence method. This new method includes the cross-entropy method as a limited case study. An advantage is the speedup during the training phase. The target architecture is the one-hot and quasi-one-hot teacher cases. Quasi-one-hotness occurs when a teacher intentionally weakens its one-hotness moderately. In addition, we devise a reverse process for quasi-one-hotness, that is, quenching. For non-one-hot elements, this process is equivalent to annealing. The new methods automatically weigh backpropagation, reflecting the nature of one-hot and quasi-one-hot teacher data. These strategies are applicable to versatile generative artificial intelligence (AI) tools. Using generic neural networks, we investigated the degree of learning speedup. The new methods present remarkable speedups, illustrations of which require logarithmic scales. This property also allows more elaborate conditional probability estimations by partitioning a neural network. Thus, the alpha-divergence method has become meritorious for accelerating versatile edge AI developments, particularly for low-cost and low-power fine-tuning.