Pose-Robust Facial Expression Recognition Using View-Based 2D $+$ 3D AAM

This paper proposes a pose-robust face tracking and facial expression recognition method using a view-based 2D $+$ 3D active appearance model (AAM) that extends the 2D $+$ 3D AAM to the view-based approach, where one independent face model is used for a specific view and an appropriate face model is selected for the input face image. Our extension has been conducted in many aspects. First, we use principal component analysis with missing data to construct the 2D $+$ 3D AAM due to the missing data in the posed face images. Second, we develop an effective model selection method that directly uses the estimated pose angle from the 2D $+$ 3D AAM, which makes face tracking pose-robust and feature extraction for facial expression recognition accurate. Third, we propose a double-layered generalized discriminant analysis (GDA) for facial expression recognition. Experimental results show the following: 1) The face tracking by the view-based 2D $+$ 3D AAM, which uses multiple face models with one face model per each view, is more robust to pose change than that by an integrated 2D $+$ 3D AAM, which uses an integrated face model for all three views; 2) the double-layered GDA extracts good features for facial expression recognition; and 3) the view-based 2D $+$ 3D AAM outperforms other existing models at pose-varying facial expression recognition.