Training Deep Energy-Based Models Through Cyclical Stochastic Gradient Langevin Dynamics

Non-normalized log probability is assigned to data samples by energy-based models (EBMs). Applications for this functionality include image synthesis, data denoising, sample restoration, outlier identification, and Bayesian inference. Long mixing times for high-dimensional datasets are a common problem for MCMC-based sampling techniques. Some methods for scaling MCMC-based EBM training with continuous neural networks have been proposed to obtain better sampling than previous likelihood models and approaches similar to modern GAN approaches without experiencing mode collapse. This study suggests a novel approach called Cyclical Stochastic Gradient Langevin Dynamics for enhancing MCMC sampling of EBMs due to the cost of mixing times. On the MNIST dataset, we discovered that Cyclical SGLD is faster than Stochastic Gradient Langevin Dynamics (SGLD) and even more effective and reliable at producing realistic images.