Prediction of Imaging Data Based on Signal Processing Boosted by Neural Networks with Continual Learning

Modern video codecs rely on data prediction techniques to attain good compression performance. Such prediction techniques work on a block-by-block basis and rely on traditional signal processing, namely interpolation and extrapolation techniques, to predict each frame. Neural networks (NN) trained off-line have recently been shown to improve these traditional prediction techniques. However, as with any NN that requires training before operation, the prediction performance is conditioned on the training data. Furthermore, the learned parameters must be included in the compressed bitstream to reverse the prediction process when decompressing the frames, thus increasing bitrates. This work proposes a novel strategy to improve the prediction obtained by traditional signal processing by using a boosting technique that is based on fully-connected NNs (FC-NNs) that continually learn as the frames are predicted block-by-block. The proposed strategy does not require storing the learned parameters to reverse the prediction process because these parameters are refined in an on-line manner using only the data being predicted. Our evaluations show that the proposed strategy can improve the traditional block-based prediction and other related NN-based strategies with prediction accuracy gains of up to 17.064 dB PSNR. Our evaluations also show that the increase in computational complexity is negligible, with an average increase of 1.1%.

1. Introduction

The High Efficiency Video Coding (HEVC) [1] and the Versatile Video Coding (VVC) [2] standards define prediction strategies to improve the performance of video codecs by reducing the amount of data needed to represent each frame. Among these strategies, the so-called intra-prediction strategy first divides each frame into non-overlapping blocks and then predicts these block sequentially following a specific order. Such a sequential prediction relies on several prediction modes that use the pixels surrounding each block as reference. Specifically, all prediction modes are tested on each block and the one that provides the best prediction is selected. Once a block is predicted, a residual block is computed as the difference between the original block and its prediction. If predicted accurately, the residual block should contain values very close to zero, which can be compressed into a small bit-stream very efficiently by applying a linear transformation, e.g., the Discrete Cosine Transform, quantization, and entropy coding. Each compressed residual block is decompressed before continuing with the prediction of the next block because the decompressed blocks are used as reference to predict subsequent blocks (see Fig. 1). To reconstruct a frame after compression, a similar block-wise process is followed: The first compressed block is first decompressed. The recovered residual block is added the corresponding predicted block, which can be computed because the initial reference pixels are readily available and the prediction mode used is signaled into the bitstream. This first reconstructed block is then employed to reconstruct the next one, repeating the process until the last block is reconstructed.