Emotional Talking Face Generation with a Single Image

The primary goal of our project is to generate artificial humanoid avatars, specifically talking faces from a single image and a text for enhanced human-robotic interaction. We put a specific emphasis on avatars that exhibit precise lip motion, head movement, and dynamic facial expressions. We believe that these attributes are essential components, making avatars significantly more engaging to human users. Contrary to traditional 3D modelling techniques that are commonly used in many modern state-of-the-art systems, our project aims to build avatars from machine-learned image augmentations. While numerous studies have been conducted on talking face generation systems, most have explored lip-motion in isolation from emotional facial shifts. Additionally, many methods depend heavily on audio or video inputs. In this paper, we propose an emotional talking face generation, called EmoFaceGen, which generates realistic talking face videos with emotions. Our system is unique in that it creates emotional talking face videos taking a single facial image and a text as inputs, then producing a talking face video with emotions as an output. The text input is converted to an audio source using the Text-To-Speech method for our project. Based on our findings, EmoFaceGen provides a more realistic talking face representation compared to other open-source models, highlighting a positive direction in overcoming present challenges in this area especially when considering the memory and hardware limitations associated with conventional 3D graphics methods.