RTNN: A Neural Network-Based In-Loop Filter in VVC Using Resblock and Transformer

Recently, the neural network (NN)-based in-loop filters are actively explored and being attempted to integrate in the versatile video coding (VVC). Transformer shows better global feature processing capability than convolutional neural network (CNN), which is complimentary to CNN that has advantage of local feature extraction. Since the in-loop filter operates in the coding tree unit (CTU) with fixed and relatively small size, it is very suitable for introducing Transformer into the in-loop filter in VVC. In this paper, we propose an NN-based in-loop filter in VVC using Resblock and Transformer, named RTNN, to suppress complicated compression artifacts in VVC. The proposed RTNN is an efficient filter, which combines Resblock and Transformer to extract deep features and capture local and non-local correlation between features. Moreover, a novel attention module is designed in RTNN to better refine features by introducing auxiliary information. Besides, a multi-stage training strategy is used to consider QP distance in training and maximize the learning ability of RTNN. The proposed RTNN filter is embedded into the VVC Test Model (VTM)-11.0_Neural Network-based Video Coding (NNVC)-3.0. Experimental results show that the proposed RTNN achieves average {8.49%, 22.98%, 24.07%} and {9.23%, 22.47%, 22.31%} BD-rate reductions over VTM-11.0_NNVC-3.0 for {Y, Cb, Cr} channels in AI and RA configurations, respectively, as well as yields significant performance with acceptable complexity.