Face and expression recognition based on bag of words method considering holistic and local image features

This paper proposes a new framework for extracting facial features based on the bag of words method, and applies it to face and facial expression recognition. Recently, the bag of words method has been successfully used in object recognition. However, for recognition problems of facial images, the orderless collection of local patches in bag of words method cannot provide strongly distinctive information since the object category (face image) is the same. In our work, a new framework based on bag of words is presented to extract discriminative local facial features while maintaining holistic spatial information at the same time. The new method is applied to both face and facial expression recognition. Experimental results show that only using one neutral expression frame per person for training, our method can obtain the best face recognition performance ever on face images of AR database with extreme expressions, variant illuminations, and partial occlusions. For facial expression recognition, the average rate on the Cohn-Kanade database is 96.33%, which also outperforms the state of the arts.