Content-adaptive encoder optimization of the H.264/AVC deblocking filter for visual quality improvement

This paper presents a new self-adapting and content-sensitive optimization technique for the H.264/AVC in-loop deblocking filter [1], focussing on the visual enhancement of the perceived reconstruction quality. Performed frame-wisely at the encoder side, the proposed algorithm first identifies visually important image regions in the currently decoded and blocky frame, including natural and artificial edge areas, and then optimizes the filter inherent threshold decisions, finding the best trade-off between preserving natural and suppressing artificial edges. As fast optimization criterion, the low complex Edge-PSNR [2] is employed, which has proved a very good congruence to the human visual quality sensation, much better than standard PSNR. In this work, the optimization behavior of all thresholds is analyzed, showing their mostly good-natured curve shape for convex optimization, but high content dependency of the optimal values. The in-loop coding results for AVC evidence the approach's high capability for visual improvement, whose general design is easily portable to other AVC deblocking based architectures like HEVC.