Auxiliary Knowledge-Based Fine-Tuning Mechanism for Industrial Time-Lag Parameter Prediction

Compared with the optimization technology of retraining models, fine-tuning pretrained neural networks have been widely used in industrial process monitoring because of their high precision and low training cost. However, data privacy protection in industry makes it impossible to improve the upstream pertaining model to avoid negative transfer when the target domain and the source domain are too different or the pertaining model is maliciously attacked. To solve the above problem, we proposed a fine-tuning method assisted by target domain knowledge for industrial time-lag parameter prediction. This method uses an Auto-Encoder to learn the representation of black-box and time-lag knowledge in the target domain. Then, the black-box and time-lag knowledge are used as auxiliary information to update the high-level weights of the pretrained network. At the same time, we proposed an auxiliary learning method that can dynamically update weights without introducing new parameters and provide training methods for different neural network optimizers. The experimental results on the heating furnace temperature prediction and wind condition prediction of wind farms demonstrate that the prediction performance can be effectively improved, and the defective inheritance of the pretrained model can be effectively reduced to avoid negative transfer.
Note to Practitioners
—Due to the complexity of industrial processes, the detection of key parameters is time-consuming and it is difficult to obtain labeled samples. With the widespread use of large models in computer vision and natural language processing, fine-tuning pretrained models can alleviate these challenges. However, the problem that comes with it is safety. The industrial prediction model cannot be built when the publicly available pretrained model is attacked. Moreover, the pretrained model could not be modified because the pretrained data could not be obtained. Therefore, this paper used the time-lag prior knowledge of industrial data to modify the pretrained model in the fine-tuning stage, and helps improve the pretrained accuracy and model robustness by mining the knowledge of existing data. Two groups of experiments show that the auxiliary knowledge-based fine-tuning mechanism can significantly improve the prediction accuracy and alleviate the defect inheritance of the pretrained model, which is valuable for the application of large models in the field of industrial parameter prediction.