Near optimal lossy source coding and compression-based denoising via Markov chain Monte Carlo

We propose an implementable new universal lossy source coding algorithm. The new algorithm utilizes two well- known tools from statistical physics and computer science: Gibbs sampling and simulated annealing. In order to code a source sequence x
ⁿ
, the encoder initializes the reconstruction block as x
ⁿ
= x
ⁿ
, and then at each iteration uniformly at random chooses one of the symbols of x
ⁿ
, and updates it. This updating is based on some conditional probability distribution which depends on a parameter beta representing inverse temperature, an integer parameter k = o(logn) representing context length, and the original source sequence. At the end of this process, the encoder outputs the Lempel-Ziv description of x
ⁿ
, which the decoder deciphers perfectly, and sets as its reconstruction. The complexity of the proposed algorithm in each iteration is linear in k and independent of n. We prove that, for any stationary ergodic source, the algorithm achieves the optimal rate-distortion performance asymptotically in the limits of large number of iterations, beta, and n. We also show how our approach carries over to such problems as universal Wyner-Ziv coding and compression-based denoising.