Facial Emotion Recognition for Mobile Devices: A Practical Review

Communicating via email or various chat applications on smartphones is part of most people’s daily lives. But in written form, human communication loses a lot of valuable information, such as the facial expressions and emotions of the person you are communicating with. Thanks to techniques from the field of image processing, it is now possible to capture these non-verbal phenomena, and supplement written input with their non-verbal characteristics. In this paper, we explore the possibilities of emotion recognition from front camera images in mobile and embedded devices. A total of 63 classification and 28 regression models based on twelve different neural network architectures optimized for low performance mobile devices were trained and evaluated for success rate and latency. The training and evaluation of each neural network model is performed within the Keras API of the TensorFlow library and then converted to the TensorFlow Lite standard to reduce memory and computational requirements. Great care is taken to ensure that the entire process, from face detection to emotion classification, can operate in real time. To demonstrate and compare the performance of the evaluated models, a freely available optimized application running on Android mobile devices is created and published on Google Play, the source code of which is also available.