Combination Test of NNVC Tools and NN-Inter In VVC

The Joint Video Exploration Team (JVET) established an Ad Hoc Group on neural network-based video coding (NNVC), and launched the NNVC tools that integrated the low operating point (LOP) in-loop filter and NN-based intra prediction (NN-Intra) into the Versatile Video Coding (VVC). In this paper, we provide the combination test results of NNVC tools and NN-Inter in VVC to observe the complementarity and performance overlap between them. For the combination test, we use a Transformer-based reference frame synthesis (TRFS) network for VVC inter prediction, named NN-Inter. TRFS is integrated between the decoded picture buffer (DPB) and the reference picture lists (RPL) in VVC. Specifically, two adjacent decoded frames from the DPB are fed into TRFS to learn an intermediate frame, which is regarded as reference frame and inserted into the RPL for inter prediction enhancement. We set up four groups of experiments: "LOP + NN-Intra", "LOP + NN-Inter", "NN-Intra + NN-Inter", and "LOP + NN-Intra + NN-Inter". All experiments take VTM-11.0_NNVC-10.0 with NNVC tools disabled as the reference anchor. Experimental results show that the combination of LOP, NN-Intra, and NN-Inter achieves the best Bjøntegaard Delta rate (BD-rate) gains of {11.74% (Y), 19.57% (U), and 18.76% (V)} over the reference anchor. Although NN-Intra and NN-Inter demonstrate nearly additive gains due to their independent operations, significant performance overlap exists between LOP and NN-Inter due to the redundant enhancement functionality, especially for chroma channels.