Improving Transfer Time Prediction of ML Models via Auto-correcting Dynamical Systems Modeling

Machine Learning (ML) is extensively used for predicting transfer times for general purpose Wide Area Networks (WANs) or public Internet applications, but for Research and Education Networks (RENs) two major gaps exist in literature. First, RENs i.e. networks carrying large data flows have received limited attention by the networking community. RENs behave differently compared to the general purpose Internet applications and other network types. Hence, ML models from other network types cannot be used interchangeably for large data transfers. Second, the ML models are used as blackboxes to train on measured network values and then used to predict transfer times or other runtime network parameters. In this paper, we present a dynamical systems model of the large data transfers typical of RENs in the form of a system of Ordinary Differential Equations (ODEs) inspired by the Lotka-Volterra competition model. We present a transfer time prediction component called Dynamic Transfer Time Predictor (DTTP) which solves the ODEs and predicts the future transfer times. Second we formulate a loss function based on Lyapunov function called Lyapunov Drift Correction (LDC) that self-corrects the transfer time prediction errors dynamically.To design and develop our model, we studied real-world datasets consisting of over 100 million transfer records collected from platforms such as Open Science Grid (OSG), Large Hadron Collider Optical Private Network (LHCOPN), Worldwide LHC Grid (WLCG), as well as the RENs of Internet2 and ESNet. We integrate our model into well-known neural network models and regressors and present evaluation results.