A Hybrid Deep Animation Codec for Low-Bitrate Video Conferencing

Deep generative models, and particularly facial animation schemes, can be used in video conferencing applications to efficiently compress a video through a sparse set of key-points, without the need to transmit dense motion vectors. While these schemes bring significant coding gains over con-ventional video codecs at low bitrates, their performance saturates quickly when the available bandwidth increases. In this paper, we propose a layered, hybrid coding scheme to overcome this limitation. Specifically, we extend a codec based on facial animation by adding an auxiliary stream con-sisting of a very low bitrate version of the video, obtained through a conventional video codec (e.g., HEVC). The an-imated and auxiliary videos are combined through a novel fusion module. Our results show consistent average BD-Rate gains in excess of -30% on a large dataset of video confer-encing sequences, extending the operational range of bitrates of a facial animation codec alone. Our code is available at github.com/animation-based-codecs