Sensitivity analysis of a cascade RLS-LMS algorithm for different resolution audio signals

We analyze the effect of both correlated and uncorrelated input signals on the performance of a cascade RLS-LMS based adaptive linear prediction algorithm for lossless compression of different resolution audio signals. We show that the sensitivity of the cascade RLS-LMS predictor to the input signal depends mainly on that of the RLS pre-whitener. Theoretical analysis shows that the mean deviation from the optimum error power is the same for both correlated and uncorrelated input samples and increases with the signal variance, which may lead to a degraded prediction gain of the RLS algorithm for high resolution audio signals. For real audio signals quantized at 16, 20 and 24 bits, simulation results confirm our theoretical analysis.