Semantic Encoding and Decoding for Talking Head Transmission

Semantic communication focuses on transmitting only information of interest instead of raw data. It holds the potential to overcome the limitations of traditional communication methods. A critical aspect of semantic communication is semantic encoding and decoding, which deals with information compression and reconstruction. Existing key point-based talking head encoding and decoding methods encounter challenges when dealing with large changes in the pose of a human face. In this study, a semantic encoding and decoding method was proposed. Firstly, a sketch-based semantic decoding method based on generative adversarial networks was proposed. Secondly, a structure similarity loss was proposed to further enhance the performance of the sketch decoding performance. Finally, two pose adaptive semantic encoding and decoding methods were proposed to adaptively combine two modalities of semantic information. Experimental results showed that the proposed method achieved significantly better video reconstruction performance than baseline methods. The ablation study result showed that the proposed structure similarity loss can enhance the sketch decoding performance. The proposed method provides a novel approach to solving the facial distortion problem when the pose of the human face changes largely and improves the performance of semantic encoding and decoding systems.