A New Approach to Decoder-Side Depth Estimation in Immersive Video Transmission

This paper presents a novel approach to decoder-side depth estimation (DSDE) in immersive video transmission. The proposal improves the state-of-the-art coding approach by changing the encoding and decoding processes so that the decoder-side depth estimator can utilize partial geometry information about the scene from several available input depth maps. Proposed changes allow for significantly faster decoding and better quality of virtual viewports presented to the viewer. The paper proposes two approaches, namely input depth map assistance (IDMA) and extended IDMA (eIDMA), both compliant with the bitstream of MPEG immersive video (MIV) standard. IDMA involves sending full depth maps for a selected subset of input views, which are then used to refine and enhance the depth maps for the remaining views. In eIDMA, the decoder additionally reprojects decoded depth maps to the remaining views and enhances them with depth patches containing difficult-to-estimate areas. Proposed methods were tested under the ISO/IEC MPEG Video Coding common test conditions for MIV. The combination of two proposals, the adaptive IDMA, was shown to outperform the current state-of-the-art DSDE approach in the rendered video quality and the computational complexity of decoder. MPEG experts have appreciated the proposed approaches, which will comply with a new DSDE profile of the incoming second edition of the MIV standard.