RDO Transform Selection for a Multi-Transform Video Encoder

A transform selection for a multi-transform video encoding scheme (AVC possible extension) is presented. This transform selection uses rate distortion optimization (RDO) technique. This method can also be adapted to be used for coding mode-selection and quantization step selection. RDO technique is applied to a selection of transform for each residual sub-block (4×4 PELs) from an available transform collection. The mode selection methodology from [8] is improved and used for the transform selection. The [8] mode selection process uses suboptimal RDO process with one dimensional space instead of two-dimensional space, which leads to a PSNR degradation of 1-2dB compared to two-dimensional space RDO. The suggested RDO method uses two stage minimization processes: Initially, Lagrange minimization is applied to each transform. Then, second Lagrange minimization used to select transform/quantization pair. The complexity is reduced by assuming that the RD function of the scalar quantization is a monotonic function, and by use in developed bit-rate predictor. The bit-rate prediction improves the methods that are presented in [1] and [5] and achieves bit-rate degradation of 0.31% - 6.8% compared to non prediction scheme vs. a bit-rate degradation of 3.2% - 13.2% by use bitrate predictor which presented in [1] and [5]. Finally video encoding process which presented achieve up to 30% of improvement. The video encoding uses RDO selection and the bitrate predictor which developed here. The RDO process selects out of three alternative transform which given by [3].