Twenty Years of Mixture of Experts

In this paper, we provide a comprehensive survey of the mixture of experts (ME). We discuss the fundamental models for regression and classification and also their training with the expectation-maximization algorithm. We follow the discussion with improvements to the ME model and focus particularly on the mixtures of Gaussian process experts. We provide a review of the literature for other training methods, such as the alternative localized ME training, and cover the variational learning of ME in detail. In addition, we describe the model selection literature which encompasses finding the optimum number of experts, as well as the depth of the tree. We present the advances in ME in the classification area and present some issues concerning the classification model. We list the statistical properties of ME, discuss how the model has been modified over the years, compare ME to some popular algorithms, and list several applications. We conclude our survey with future directions and provide a list of publicly available datasets and a list of publicly available software that implement ME. Finally, we provide examples for regression and classification. We believe that the study described in this paper will provide quick access to the relevant literature for researchers and practitioners who would like to improve or use ME, and that it will stimulate further studies in ME.

I. Introduction

Since its introduction 20 years ago, the mixture of experts (ME) model has been used in numerous regression, classification, and fusion applications in healthcare, finance, surveillance, and recognition. Although some consider ME modeling to be a solved problem, the significant number of ME studies published in the last few years suggests otherwise. These studies incorporate experts based on many different regression and classification models such as support vector machines (SVMs), Gaussian processes (GPs), and hidden Markov models (HMMs), to name just a few. Combining these models with ME has consistently yielded improved performance. The ME model is competitive for regression problems with nonstationary and piecewise continuous data, and for nonlinear classification problems with data that contain natural distinctive subsets of patterns. ME has a well-studied statistical basis, and models can be easily trained with well-known techniques such as expectation-maximization (EM), variational learning, and Markov chain Monte Carlo (MCMC) techniques including Gibbs sampling. We believe that further research can still move ME forward and, to this end, we provide a comprehensive survey of the past 20 years of the ME model. This comprehensive survey can stimulate further ME research, demonstrate the latest research in ME, and provide quick access to relevant literature for researchers and practitioners who would like to use or improve the ME model.