Simplified learning with binary orthogonal constraints

Deep architecture based Deep Brief Nets (DBNs) has shown its data modelling power by stacking up several Restricted Boltzmann Machines (RBMs). However, the multiple-layer structure used in DBN brings expensive computation, and furthermore leads to slow convergence. This is because the pretraining stage is usually implemented in a data-driven way, and class information attached to the training data is only used for fine-tuning. In this paper, we aim to simplify a multiple-layer DBN to a one-layer structure. We use class information as a constraint to the hidden layer during pre-training. For each training instance and its corresponding class, a binary sequence will be generated in order to adapt the output of hidden layer. We test our approaches on four data sets: basic, MNIST, basic negative MNIST, rotation MNIST and rectangle (tall vs. wide rectangles). The obtained results show that the adapted one-layer structure can compete with a three-layer, DBN.