Optimal Selection of Dropout Probability for Stochastic Training of Deep Residual Network

Considering the problem of low efficiency in traditional stochastic training of residual network by empirically parameters tuning for dropout probability selection, this paper proposes the relationship between the stochastic training model of deep residual network and stochastic optimal control theory, and then the expression of the optimal dropout probability is obtained by theoretical derivation. First the stochastic training model of deep residual network obtained by derivation can be approximated as a stochastic dynamic system, which means that it can provide the guidelines of stochastic learning strategies from the perspective of the stochastic dynamic system. Moreover, the stochastic training and learning process of deep residual network can be approximated as a stochastic optimal control problem. Finally, the optimal solution can be solved by dynamic programming or the minimum principle, thus the optimal selection method of the adaptive hyperparameter namely dropout probability is obtained. Experimental results are provided to illustrate the effectiveness of the optimal selection of dropout probability for stochastic training of deep residual network.