A New Approach to Video Coding Leveraging Hybrid Coding and Video Frame Interpolation

In this work, we propose the use of a video coding method, dubbed Decoupled Interpolated Video Coding (DIVC), which blends traditional hybrid video coding with novel approaches based on Neural Networks (NNs). The DIVC approach provides a base-level representation with a reduced bit rate by dropping frames in a regular manner which can be decoded by standard hybrid video coding. Meanwhile, we also regenerate the dropped frames from the reconstructed video from the base-level representation using NN-based Video Frame Interpolation (VFI) to recover the original number of frames per second (fps). We show that the DIVC approach can improve video coding efficiency considering the perceptual quality measured with the Structural Similarity (SSIM) metric. More specifically, such approach achieved a reduction of 38.96% (AI configuration) and 5% (RA configuration) in terms of SSIM
dB
-based Bøntegaard Delta Bit Rate (BD-Rate) when compared to traditional video coding. Meanwhile, considering the Peak Signal-to-Noise Ratio (PSNR), which may not correlate as well to human perception, the discussed approach resulted in PSNR BD-Rate reduction of 16.19% in AI configuration and increase of 19.33% in RA configuration. Employing VFI to improve video coding efficiency tends to be more suited to processing video sequences with high fps or slower motion.