Leapfrog Polymorphic Neural Ordinary Differential Equation

Neural Ordinary Differential Equations (NODEs) revolutionize the way we view residual networks as solvers for initial value problems (IVPs), with layer depth serving as the time step. In this study, we propose a more efficient extension of NODEs called Leap-Frog Polymorphic Neural ODEs (LF-NODEs). LF-NODEs introduce the leap-frog updating scheme, breaking free from the specific structure of time-evolving mixtures of multiple dynamical systems. In each time step (corresponding to each residual network layer), LF-NODEs integrate the features of multiple dynamical systems (residual network layers) by employing time-variant weights from different dynamical systems. Our LF-NODEs not only overcome the limitations in the representation power of individual residual networks but also provide a more flexible and effective structure for residual networks based on multiple dynamical systems. The efficiency of the leap-frog updating scheme is theoretically derived and demonstrated. Furthermore, we expand the model by incorporating a multi-scale approach (LF-MSNODEs) to achieve even more accurate model representations. Empirical results showcase the improvements provided by our proposed methods in 2D concentric spheres task, irregularly sampled time series prediction, and classification tasks. Additionally, the results provide empirical evidence that the learned feature space enhances system efficiency with fewer parameters and function evaluations compared to the baseline methods.