Fast HEVC Intra Mode Decision Based on RDO Cost Prediction

High efficiency video coding (HEVC) increases the number of intra coding modes to 35 to provide higher coding efficiency than previous video coding standards. This results in an increased encoder complexity, since there are more modes to be processed by the high resource-demanding rate-distortion optimization (RDO). In this paper, we propose a novel method to reduce the HEVC intra mode decision computational complexity and encoding time. This method is based on the prediction of the RDO cost of intra modes from a low-complexity sum of absolute transformed differences-based cost. By predicting the RDO cost, we are able to exclude non-promising modes from further processing and thereby save substantial computations. Also, a gradient-based method, using the Prewitt operator, is proposed to eliminate the non-relevant directional modes from the list of candidates. For even more complexity reduction, a mode classification is proposed to adaptively reduce chroma intra modes based on block texture. Experimental results show that we achieve a 47.3% encoding time reduction on average with a negligible quality loss of 0.062 dB for the Bjøntegaard delta peak signal-to-noise ratio when we compare our method to the HEVC test model 15.0.