Spatial-Temporal Motion Compensation for Learned Video Compression

In the past few years, learned video compression has received increasing attention. However, most current learned methods rely on the bilinear warping operations in motion compensation, which is equivalent to a low-pass filtering operation and brings the distortion of reconstruction. To address this problem, we propose a spatial-temporal motion compensation for learned video compression (STMC-LVC). STMC-LVC uses a spatial-temporal motion compensation network (STMC-Net) to fully consider the spatial-temporal correlation between successive frames, performing accurate motion compensation. Specifically, STMC-Net mainly consists of the initial feature prediction module and the fusion module. The initial feature prediction module predicts reference features based on DCN to provide sufficient information for motion compensation. The fusion module obtains temporal attention information by calculating the similarity between features, performs spatial attention operations, and finally performs spatial-temporal fusion, further improving the robustness of motion compensation. In addition, STMC-LVC uses a conditional coding framework. We use a concatenation of the current feature and predicted feature as context to explore spatial-temporal correlations in feature space. Experimental results show that our method effectively improves video compression. Specifically, our model achieves an average of 33.86% and 58.70% bitrate savings than x265 (veryslow) on PSNR and MS-SSIM, respectively.