Hyperparameters Adaptation for Restricted Boltzmann Machines Based on Free Energy

Restricted Boltzmann Machine (RBM), the building block of Deep Belief Network (DBN) and Deep Boltzmann Machine (DBM), is one of the most powerful unsupervised feature detectors. Despite its success, the challenging issue of setting its hyperparameters remains. In recent years, various types of the hyperparameters optimization (HO) algorithms have been proposed and substantially improved the performance in many supervised learning models. However, they cannot directly apply to the RBM due to its unsupervised learning strategy. Moreover, these HO algorithms typically have to train the models fully or partially for several iterations before the hyperparameters can been assess. That causes computational overhead very high, especially for the deep architectures. This paper proposes a new efficient procedure, which can online estimate the hyperparameters when training the stacked RBMs. Specifically, we optimize the three main hyperparameters (learning rate, momentum, weight-cost) simultaneously based on the free energy of the RBM by using Gaussian Process in each epoch. Extensive experiments demonstrate that the new procedure improves the performance of the RBMs significantly, and is superior to the state-of-the-art of hyperparameter optimization algorithms when training the stacked RBMs.