CLIFER: Continual Learning with Imagination for Facial Expression Recognition

Current Facial Expression Recognition (FER) approaches tend to be insensitive to individual differences in expression and interaction contexts. They are unable to adapt to the dynamics of real-world environments where data is only available incrementally, acquired by the system during interactions. In this paper, we propose a novel continual learning framework with imagination for FER (CLIFER) that (i) implements imagination to simulate expression data for particular subjects and integrates it with (ii) a complementary learning-based dual-memory (episodic and semantic) model, to augment person-specific learning. The framework is evaluated on its ability to remember previously seen classes as well as on generalising to yet unseen classes, resulting in high F1-scores for multiple FER datasets: RAVDESS (episodic: F1=0.98 ± 0.01, semantic: F1=0.75 ± 0.01), MMI (episodic: F1=0.75 ± 0.07, semantic: F1=0.46 ± 0.04) and BAUM-I (episodic: F1=0.87 ± 0.05, semantic: F1=0.51 ± 0.04).