New rate distortion optimization method for video coding

In this paper, we propose a new rate-distortion optimization (RDO) method for high efficient video coding (HEVC), based on texture and motion masking properties of video. Firstly, the texture perception factors of the coding tree units are defined by using the energy features of the gray level co-occurrence matrix (GLCM), and the motion regions are obtained by using robust orthogonal subspace learning (ROSL), and motion perception factor is defined. Then, we combine the texture and motion perception factors to define the perceptual scaling factor. Finally, we adjust the Lagrangian multiplier and quantization parameter adaptively according to the perceptual scaling factor. Experimental results show that the highest bit rate is reduced by 4.2% and 6.2% under the same peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) over the HEVC test model 16.0 (HM 16.0) for low-delay P configurations, respectively.