Toward Digital Twin: Leveraging Pre-training Approaches for Multivariate Time Series Forecasting

Time series forecasting has been an active research area, particularly in the context of Digital Twin (DT) systems. Despite the excellent results yielded by pre-trained models in Natural Language Processing (NLP) and Computer Vision (CV), only a few studies have researched pre-training strategies for time series forecasting networks within DT systems. Recent studies demonstrate that transfer learning across multiple time series datasets does not always provide promising results, making self-supervised pre-training directly on the downstream dataset a temporarily considered optimal solution. To the best of our knowledge, the only general pre-training task in the time series field is the Masked Autoencoder. However, this approach may lead to redundant representations for downstream forecasting within DT systems. Therefore, we propose three pre-training tasks specially designed for time series forecasting within DT frameworks: Inverse Forecasting (IF), Coarser Forecasting (CF), and Anomaly Forecasting (AF). These tasks are respectively designed to capture bidirectional dependency, reduce noise, and augment data, all crucial aspects in enhancing the predictive capabilities of DT systems. By integrating the three tasks, we obtain a composite pre-training task which generally improves the forecasting results of time series models within DT systems across multiple datasets. This work contributes to the ongoing efforts to improve the accuracy and efficiency of DT systems, paving the way for more robust and reliable digital representations of physical systems.