Threshout Regularization for Deep Neural Networks

Regularization is key to a well-trained, robust model capable of performing well on a test set. Deep learning suffers from a huge overfitting problem due to the large number of nodes, high dimensionality of parameters, and depth. Dropout regularization is a popular method of regularizing networks with Bernoulli noise injection, unlinking connections by dropping random nodes to zero. We introduce a better version of Dropout, called Threshout, that uses Gaussian noise and soft thresholding to regularize deep neural networks. Based on wavelet subband denoising, our proposed method offers more randomness and robustness than Dropout through its flexibility, and more varied noise injection. Experimental results on four benchmark classification datasets prove that our proposed method outperforms Dropout in accuracy and in the prevention of overfitting.