On competitive prediction and its relation to rate-distortion theory

Consider the normalized cumulative loss of a predictor F on the sequence x/sup n/=(x/sub 1/,...,x/sub n/), denoted L/sub F/(x/sup n/). For a set of predictors G, let L(G,x/sup n/)=min/sub F/spl isin/G/L/sub F/(x/sup n/) denote the loss of the best predictor in the class on x/sup n/. Given the stochastic process X=X/sub 1/,X/sub 2/,..., we look at EL(G,X/sup n/), termed the competitive predictability of G on X/sup n/. Our interest is in the optimal predictor set of size M, i.e., the predictor set achieving min/sub |G|/spl les/M/EL(G,X/sup n/). When M is subexponential in n, simple arguments show that min/sub |G|/spl les/M/EL(G,X/sup n/) coincides, for large n, with the Bayesian envelope min/sub F/EL/sub F/(X/sup n/). We investigate the behavior, for large n, of min/sub |G|/spl les/e//sup nR/EL(G,X/sup n/), which we term the competitive predictability of X at rate R. We show that whenever X has an autoregressive representation via a predictor with an associated independent and identically distributed (i.i.d.) innovation process, its competitive predictability is given by the distortion-rate function of that innovation process. Indeed, it will be argued that by viewing G as a rate-distortion codebook and the predictors in it as codewords allowed to base the reconstruction of each symbol on the past unquantized symbols, the result can be considered as the source-coding analog of Shannon's classical result that feedback does not increase the capacity of a memoryless channel. For a general process X, we show that the competitive predictability is lower-bounded by the Shannon lower bound (SLB) on the distortion-rate function of X and upper-bounded by the distortion-rate function of any (not necessarily memoryless) innovation process through which the process X has an autoregressive representation. Thus, the competitive predictability is also precisely characterized whenever X can be autoregressively represented via an innovation process for which the SLB is tight. The error exponent, i.e., the exponential behavior of min/sub |G|/spl les/exp(nR)/Pr(L(G,X/sup n/)>d), is also characterized for processes that can be autoregressively represented with an i.i.d. innovation process.