Information Theory and Distortion

This chapter gives the basic theory for rate distortion and shows how this influence the bounds on source coding and channel coding. For source coding, distortion means lossy compression, which is used in, e.g., audio and video coding. To measure the introduced mismatch between the source symbol and the reconstructed symbol, a distortion measure is required. There are several such measures, but the two most well known are the Hamming distortion and the squared distance. The first one is typically used for discrete sources, especially for the binary case, whereas the second is mostly used for continuous sources. The chapter shows that the rate‐distortion function has the same interpretation for lossy source coding as the entropy has for lossless source coding. Finally, transform coding is described, which is the basis of, e.g., JPEG and MPEG compression.