Hierarchical video motion estimation using a neural network

In our previous papers, we have presented a neural network-based motion estimation technique that is applicable to purely translational as well as affine movements. In this paper we present a further advance, namely a hierarchical version which, while capitalizing on the merits of the previous technique, achieves some additional advantages. These are: (a) the ability to deal with very large motion action sequences; and (b) a reduction in the number of candidate macroblocks for the neural network stage and correspondingly, the number of computations needed. In this hierarchical technique, the first stage consists of one of the familiar block matching techniques, such as the LBM or FS, over very coarse pixels, to define a suitable candidate neighborhood. In the second, and final, stage a few of the neighbors of the candidate found in first stage are used in the computation of affinities via the neural network to synthesize the source macroblock. It produces 4 to 11 dB improvement in peak signal-to-noise ratio over the single-candidate methods, thus creating the potential for improved compression.