Fast inter-mode decision and selective quarter-pel refinement in H.264 video coding

In H.264 video coding standard, there exist several inter - prediction modes that use macroblock partitions with variable block sizes. Choosing a rate-distortion optimal coding mode for each macroblock is essential for the best possible coding performance, but also prohibitive due to the heavy computational complexity associated with the required rate-distortion calculations. Likewise, sub-pel motion refinement improves the coding efficiency, but becomes a major computational bottleneck when integer-pel search is executed fast. In this paper, we present a simple strategy to reduce the complexity of quarter-pel refinement and inter-mode decision with minimum loss of coding efficiency. Based on the results of the half-pel motion estimation step, our method evaluates the likelihood of each inter-coding mode being optimal. Then, quarter-pel refinement and actual rate and distortion are computed for only those coding modes with sufficient chance of being optimal. We claim that this method minimizes optimal mode estimation error at a given level of refinement and mode decision complexity. Simulation results show that the algorithm speeds up quarter-pel search and inter-mode selection modules by a factor of about 6 with less than 0.12 dB PSNR loss.