Facial Expression Recognition Using CNN-LSTM Approach

Automated Facial Expression Recognition (FER) has remained a promising and challenging domain for Computer Vision with the appearance of Deep Learning approaches. The FER has turned up very encouraging because it mirrors our mental conditions and gives valuable facts on human behaviors. Most of this area’s research has been conducted with the posed image dataset, but posed images do not describe the human face’s genuine emotions. The non-posed images carry non-verbal knowledge among individuals in face-to-face communications. Hence, to get the human’s accurate mental state, a Posed and Non-posed Facial Expression (PNFE) dataset is built to evaluate the best performance. This paper proposed a new architecture based on Convolutional Neural Network (CNN) and Long Short Term Memory (LSTM) that classifies six fundamental expressions: happiness, anger, disgust, fear, sadness, and surprise with neutral. This combined approach is split into two phases. First, visual features are extracted by CNN by learning on the PNFE dataset, and then LSTM is applied to bound the relationship between image sequences and emotions. Besides, the result of this architecture is evaluated by confusion matrices and compared with relevant architectures and renowned FER datasets.