Probability Theory

One of the most important insights when setting up a measure for information is that the observed quantity must have multiple choices to contain information. These multiple choices are best described by means of probability theory. This chapter recapitulates the parts of probability theory. An important concept in probability theory is how two events are related. The geometric distribution and the Gaussian distribution are studied with examples. The basics of the law of large numbers begins with two famous bounds in probability theory, Markov's inequality and Chebyshev's inequality. A widely used class of the discrete stationary random processes is the class of Markov processes. The process has unit memory, i.e. the probability for a symbol depends only on the previous symbol. The behavior of a Markov chain can be visualized in a state transition graph consisting of states and edges, labeled with probabilities.