Hybrid Distortion-Based Rate-Distortion Optimization and Rate Control for H.265/HEVC

Mean squared error (MSE) is adopted as the distortion metric for the rate-distortion optimization (RDO) and rate control (RC) in most of the existing video coding standards, e.g., H.265/High-Efficiency Video Coding (HEVC) and H.264/Advanced Video Coding (AVC), because the pixel-wise MSE can represent the image fidelity efficiently. As videos are usually used for visual enjoyment, recently, researchers found that the MSE is not always consistent with perceptual quality, and proposed some perceptual quality-based video coding methods by exchanging the MSE with other perceptual quality metrics completely, like the structure similarity index (SSIM), just noticeable distortion (JND), etc. We believe that a sophisticated video encoder should guarantee both the subject quality and the data fidelity. By investigating the properties of the human visual system (HVS), hybrid distortion-based RDO and RC methods are proposed to improve subjective quality while avoiding fidelity degradation. Experimental results demonstrate that the proposed methods can improve subjective quality efficiently, while preserving the image fidelity of the reconstructed videos as much as possible.