A Multi-Scale Feature Extraction and Prediction Model Based on Dual-Layer Transformer and Pre-Normalization Strategy

Currently, efficiently mining the multi scale characteristics of ocean elements is crucial for improving the accuracy of numerical forecasting. The traditional Transformer model can only obtain single-scale features and cannot simultaneously consider both time-dimension and space-dimension features. The paper presents an end-to-end dual-layer Transformer-based model for multi-scale feature extraction and forecasting. In the first layer of the model, transformers independently extract features from the input multi-scale spatio-temporal historical data. Subsequently, the second layer fuses features by integrating the initially extracted spatio-temporal features with specific numerical attributes of the target elements, thereby simultaneously processing multi-scale spatio-temporal features. This paper has improved data processing efficiency through prenormalization techniques and utilized fused multi-scale features for numerical prediction tasks. This paper has selected eight evaluation metrics to comprehensively assess the performance of the DTransR-Net model. The experiment results demonstrate the presented model shows superior data fit and adaptability to new data compared with traditional ocean model forecasting method, classical regression models, decision tree-based models, and advanced deep learning models. The model can significantly reduce the forecasting error of oceanic elements, and obtain more precise sea surface temperature predictions.