A framework for a discrete valued Helmholtz machine

The Helmholtz machine and the wake-sleep learning algorithm are interesting developments in the field of stochastic modelling due to the substitution of complex learning rules by an algorithm which uses two complementary networks which learn from each other with a simple learning rule. However, it is limited in its range of practical applications due to the binary nature of its units. This paper discusses the possibility of allowing the units to take on an arbitrary number of discrete states. Involved in this are the addition of an error term to the energy in the network and the alteration of the wake-sleep algorithm to be stochastic in nature. The difficulties associated with gradient descent in the new network are presented and some possible solutions are discussed.