Word-Valued Sources: An Ergodic Theorem, an AEP, and the Conservation of Entropy

A word-valued source Y = Y
₁
,Y
₂
,
...
is discrete random process that is formed by sequentially encoding the symbols of a random process X = X
₁
,X
₂
,...
with codewords from a codebook
C
. These processes appear frequently in information theory (in particular, in the analysis of source-coding algorithms), so it is of interest to give conditions on
X
and
C
for which
Y
will satisfy an ergodic theorem and possess an asymptotic equipartition property (AEP). In this paper, we prove the following: 1) if
X
is asymptotically mean stationary (AMS), then
Y
will satisfy a pointwise ergodic theorem and possess an AEP; and 2) if the codebook
C
is prefix-free, then the entropy rate of
Y
is equal to the entropy rate of
X
normalized by the average codeword length.