FastEM: an efficient EM algorithm for learning Gaussian mixture models on compute clusters

Gaussian mixture models [1] (GMMs) are commonly used to model complex probability distributions. Recent large-scale applications include clustering and image processing. The probability density function f of a GMM is a weighted sum of K multivariate (d-dimensional) Gaussian densities: $f(\mathbf{x} \mid \Theta)=\sum_{j=1}^{K} \alpha_{j} \mathcal{N}\left(\mathbf{x}; \boldsymbol{\mu}_{j}, \boldsymbol{\Sigma}_{j}\right)$, where $\boldsymbol{\mu}_{j}$ is the mean vector and $\boldsymbol{\Sigma}_{j}$ is the covariance matrix. A crucial part of most applications of GMMs is the maximum likelihood estimation of parameters $\alpha_{j}, \boldsymbol{\mu}_{j}, \boldsymbol{\Sigma}_{j}(j=1, \ldots, K)$ by the expectation-maximization (EM) algorithm given a dataset represented by $N \times d$ matrix X. An iteration of EM for GMMs [2] consists of two steps: an E-step and an M-Step. In the E-Step, given the dataset X, an $N \times K$ matrix of responsibilities $\mathbf{P}=\left[p_{i, j}\right]$ is obtained using: