Exploiting the 3D Structures Observed in 2D Video Sequences for Motion Compensation

Video coding standards widely use 2D translational and affine motion models to compensate for the motion between frames. Noting that most 2D videos are initially captured in a 3D space may lead us to a new direction: using 3D scene geometry for 2D motion compensation. We introduce a method for rendering virtual Reference Pictures (RPs) based on the 3D information extracted from 2D video sequences. The synthesized RPs are offered to the Versatile Video Coding (VVC) encoder to serve as motion compensation references. The 3D information (camera poses and 3D scene geometry) is estimated at the encoder and transmitted to the decoder as an overhead bitstream. Multiple techniques are discussed to minimize this overhead. Simulation results show up to 5% coding gain compared to our anchor (VTM 10.0), with an acceptable increase in decoding time.