Pipelined parallel contrastive divergence for continuous generative model learning

In this paper we propose a method for continuously processing and learning from data in Restricted Boltzmann Machines (RBMs). Traditionally, RBMs are trained using Contrastive Divergence (CD), which is an algorithm consisting of two phases, of which only one is driven by data. This not only prohibits training of RBMs in conjugation with continuous-time data streams, especially in event-based real-time systems, but also hinders training speed of RBMs in large-scale machine learning systems. The model we propose trades space for time and, by pipelining information propagation in the network, is capable of processing both phases of the CD learning algorithm simultaneously. Simulation results of our model on generative and discriminative tasks show convergence to the original CD algorithm. We finalize with a discussion of applying our method to other deep neural networks, resulting in continuous learning and training time reduction.