Application of Differential Equations to the Design of Deep Residual Networks

Since the introduction of deep residual networks (ResNet) in 2015, there has been a wide range of research and applications. In recent years, due to the similarity between the network structure of ResNet and the forward Euler scheme of the numerical schemes of differential equations, researchers have started to focus on the relationship between numerical schemes of differential equations and the structures of deep learning models, and have proposed some new and improved structures and learning methods. In this paper, we focus on the application of compatibility analysis and zero stability analysis to the design of neural networks. Higher-order compatible and stable schemes are designed and applied to the cifar10 datasets with almost no change in the quantity of parameters, and higher accuracy than ResNet is obtained. Besides, we also study the schemes with different stability on small datasets, and demonstrate the superiority of stable linear multi-step schemes on small datasets. On between 20% and 40% of the cifar10 dataset, the stable scheme we propose gets significantly higher accuracy than other schemes including ResNet. We also find that as the number of categories increases, the stable scheme we propose shows great superiority to ResNet.