Self-Supervised Generative Pre-Trained Model with a Learnable Mask Network for Industrial Time Series Prediction

Industrial time series prediction (ITSP) is an indispensable part of predictive control in modern industry. Recently, supervised deep learning-based methods have provided solutions with sufficient annotated data. However, there is massive unlabeled data with complex temporal features in modern industrial production, resulting in poor performance of these methods. To address this problem, a self-supervised generative pre-trained model with a learnable mask network (SSGPM-LMN) is proposed in this paper. First, the multivariate time series are made into patches channel-independently. Then, these patches are fed into a Transformer encoder with the learnable mask-reconstruction paradigm, drawing mask indices with high temporal features by calculating the cosine similarity in low-dimensional feature space to better learn general representations. Furthermore, a two-step fine-tuning strategy, including linear probing and full fine-tuning, is adopted for various downstream scenarios. Finally, extensive experimental results on case studies of ITSP and transfer learning indicate that our SSGPM-LMN achieves superior performance.