Skip to Main Content
In this paper, we present an analysis of the dyadic approximation error introduced by the integerization of transform coding in H.264/AVC-like codecs. We derive the analytical formulations for dyadic approximation error and nonorthogonality error. We further classify the dyadic approximation error into a “system error” and a “nonflat error,” and proposed two models for them. We found that the “nonflat error” has a substantial impact on video quality if the number of shifting bits at decoder side (DQ_BITS) is small. We also give a theoretical justification on why scaling factors at encoder side are better to be adapted to the rescaling factors at decoder side in H.264/AVC-like codecs.