Modeling, analysis, and optimum design of quantized M-band filter banks

This paper provides a rigorous modeling and analysis of quantization effects in M-band subband codecs, followed by optimal filter bank design and compensation. The codec is represented by a polyphase decomposition of the analysis/synthesis filter banks and an embedded nonlinear gain-plus-additive noise model for the pdf-optimized scalar quantizers. We construct an equivalent time-invariant but nonlinear structure operating at the slow clock rate that allows us to compute the exact expression for the mean square quantization error in the reconstructed output. This error is shown to consist of two components: a distortion component and a dominant random noise component uncorrelated with the input signal. We determine the optimal paraunitary and biorthogonal FIR filter coefficients, compensators, and integer bit allocation to minimize this MSE subject to the constraints of filter length, average bit rate, and perfect reconstruction (PR) in the absence of quantizers. The biorthogonal filter bank results in a smaller MSE but the filter coefficients are very sensitive to signal statistics and to average bit constraints. By comparison, the paraunitary structure is much more robust. We also show that the null-compensated design that eliminates the distortion component is more robust than the optimally-compensated case that minimizes the total MSE, but only at nominal conditions. Both modeling and optimal design are validated by simulation in the two-channel case.