Stochastic synapses as resource for efficient deep learning machines

Synaptic unreliability was shown to be a robust and sufficient mechanism for inducing the stochasticity in biological and artificial neural network models. Previous work demonstrated multiplicative noise (also called dropconnect) as a powerful regularizer during training. Here, we show that always-on stochasticity at networks connections is a sufficient resource for deep learning machines when combined with simple threshold non-linearities. Furthermore, the resulting activity function exhibits a self-normalizing property that reflects a recently proposed “Weight Normalization” technique, itself fulfilling many of the features of batch normalization in an online fashion. Normalization of activities during training can speed up convergence by preventing so-called internal covariate shift caused by changes in the distribution of inputs as the parameters of the previous layers are trained. Collectively, our findings can improve performance of deep learning machines with fixed point representations and argue in favor of stochastic nanodevices as primitives for efficient deep learning machines with online and embedded learning capabilities.