Spatio-Temporal Transformer Network for Weather Forecasting

Spatio-temporal neural networks have been successfully applied to weather forecasting tasks recently. The key notion is to learn spatio-temporal features concurrently from spatial and temporal dependencies. Existing methods are mainly based on local smoothness assumptions where the features are learned by accumulating information in local spatio-temporal regions. However, the weather conditions in a certain spatio-temporal region are usually influenced by global meteorological changes and long-range historical weather conditions. Therefore, these methods that ignore the large-scale spatio-temporal effects can hardly learn effective features. In this paper, we propose a novel spatio-temporal Transformer network in weather forecasting to address the above challenges. The main idea is to leverage the Transformer architecture to carefully capture the multi-scale spatial and long-range temporal information in weather data. First, we propose to combine the global and local position encodings based on absolute geographic locations and relative geodesic distances and insert them into the spatial Transformer to extract the multi-scale spatial information in meteorological graphs. Then, we further capture the long-range temporal dependencies by a temporal Transformer where the attention mechanism is used to improve the representation ability and scalability of the models. Extensive experiments over real weather datasets demonstrate the effectiveness of our framework.