Integration of Stochastic Models by Minimizing α-Divergence

When there are a number of stochastic models in the form of probability distributions, one needs to integrate them. Mixtures of distributions are frequently used, but exponential mixtures also provide a good means of integration. This letter proposes a one-parameter family of integration, called
α
-integration, which includes all of these well-known integrations. These are generalizations of various averages of numbers such as arithmetic, geometric, and harmonic averages. There are psychophysical experiments that suggest that
α
-integrations are used in the brain. The
α
-divergence between two distributions is defined, which is a natural generalization of Kullback-Leibler divergence and Hellinger distance, and it is proved that
α
-integration is optimal in the sense of minimizing
α
-divergence. The theory is applied to generalize the mixture of experts and the product of experts to the
α
-mixture of experts. The
α
-predictive distribution is also stated in the Bayesian framework.