CNN oriented fast HEVC intra CU mode decision

The real-time requirements of hardwired HEVC encoder demand that, at the grain of coding tree unit (CTU), the maximum computation should be reduced by a fast CU mode decision algorithm. In addition, to realize the parallel rate-distortion optimization (RDO) of different CU modes, the current CU mode decision should not use the auxiliary information from other CU modes. Considering the above constraints, we applied convolutional neural network (CNN) to analyze the textures of source picture blocks, and then reduce the maximum number of CU modes, which will undergo the exhaustive RDO. In the CNN architecture design, we introduced the quantization parameter by considering the effect of quantization to the coding costs. We further optimized the CNN training strategy to improve the prediction accuracy. Experimental results demonstrated that, the proposed algorithm can save 63% Intra encoding time at the cost of the averaged 2.66% BDBR increase.