By Topic

The empirical distribution of rate-constrained source codes

Sign In

Cookies must be enabled to login.After enabling cookies , please use refresh or reload or ctrl+f5 on the browser for the login options.

Formats Non-Member Member
$31 $13
Learn how you can qualify for the best price for this item!
Become an IEEE Member or Subscribe to
IEEE Xplore for exclusive pricing!
close button

puzzle piece

IEEE membership options for an individual and IEEE Xplore subscriptions for an organization offer the most affordable access to essential journal articles, conference papers, standards, eBooks, and eLearning courses.

Learn more about:

IEEE membership

IEEE Xplore subscriptions

2 Author(s)
Weissman, T. ; Electr. Eng. Dept., Stanford Univ., CA, USA ; Ordentlich, E.

Let X = (X1,...) be a stationary ergodic finite-alphabet source, Xn denote its first n symbols, and Yn be the codeword assigned to Xn by a lossy source code. The empirical kth-order joint distribution Qˆk[Xn,Yn(xk,yk) is defined as the frequency of appearances of pairs of k-strings (xk,yk) along the pair (Xn,Yn). Our main interest is in the sample behavior of this (random) distribution. Letting I(Qk) denote the mutual information I(Xk;Yk) when (Xk,Yk)∼Qk we show that for any (sequence of) lossy source code(s) of rate ≤R lim supn→∞(1/k)I(Qˆk[Xn,Yn) ≤R+(1/k)H (X1k)-H~(X) a.s. where H~(X) denotes the entropy rate of X. This is shown to imply, for a large class of sources including all independent and identically distributed (i.i.d.). sources and all sources satisfying the Shannon lower bound with equality, that for any sequence of codes which is good in the sense of asymptotically attaining a point on the rate distortion curve Qˆk[Xn,YndP(Xk,Y~k) a.s. whenever P(Xk,Yk) is the unique distribution attaining the minimum in the definition of the kth-order rate distortion function. Consequences of these results include a new proof of Kieffer's sample converse to lossy source coding, as well as performance bounds for compression-based denoisers.

Published in:

Information Theory, IEEE Transactions on  (Volume:51 ,  Issue: 11 )