AI-Based Saliency-Aware Video Coding

The demand for video through over-the-top (OTT) has been constantly increasing in recent years. During the COVID-19 pandemic, demand skyrocketed, hence leading to the need for better video compression. The human visual system (HVS) can quickly select visually important regions in its visual field. These regions are captured at high resolution, while other peripheral regions receive little attention. Saliency maps are a way to imitate the HVS attention mechanism. Recently, deep learning-based saliency models have achieved tremendous improvements. This article leverages state-of-the-art deep learning-based saliency models to improve video coding efficiency. First, a saliency-based rate control scheme is integrated in a high-efficiency video encoder (HEVC). Then, a saliency-guided preprocessing filtering step is introduced. Finally, the two approaches are combined. Objective and subjective evaluations show that it can lower the bitrate from 6% to almost 30% while maintaining the same visual quality.