Recognizing multiple persons' facial expressions using HMM based on automatic extraction of significant frames from image sequences

A method that can be used for recognizing facial expressions of multiple persons is proposed. In this method, the condition of facial muscles is assigned to a hidden state of a HMM for each expression. Then, the probability of the state is updated according to a feature vector obtained from image processing. Image processing is performed in two steps. First, a velocity vector is estimated from every two successive frames by using an optical flow algorithm. Then, a two-dimensional Fourier transform is applied to a velocity vector field at the regions around an eye and the mouth. The coefficients for lower frequencies are selected to form a feature vector. A mixture density is used for approximating the output probability of the HMM so as to represent a variation in facial expressions among persons. To cope with the case when two expressions are displayed continuously, the HMM computation is modified such that when the peak of a facial motion is detected, a new sequence of facial expressions is assumed to start from the previous frame with minimal facial motion. Experiments show that a mixture density is effective because the recognition accuracy improves as the number of mixtures increases. In addition, the method correctly recognizes a facial expression that continuously follows another one.