CMS-LSTM: Context Embedding and Multi-Scale Spatiotemporal Expression LSTM for Predictive Learning

Spatiotemporal predictive learning (ST-PL) is a hotspot with numerous applications, such as object movement and mete-orological prediction. It aims at predicting the subsequent frames via observed sequences. However, inherent uncer-tainty among consecutive frames exacerbates the difficulty in long-term prediction. To tackle the increasing ambigu-ity during forecasting, we design CMS-LSTM to focus on context correlations and multi-scale spatiotemporal flow with details on fine-grained locals, containing two elaborate de-signed blocks: Context Embedding (CE) and Spatiotemporal Expression (SE) blocks. CE is designed for abundant context interactions, while SE focuses on multi-scale spatiotemporal expression in hidden states. The newly introduced blocks also facilitate other spatiotemporal models (e.g., PredRNN, SA-ConvLSTM) to produce representative implicit features for ST-PL and improve prediction quality. Qualitative and quanti-tative experiments demonstrate the effectiveness and flexibil-ity of our proposed method. With fewer params, CMS-LSTM outperforms state-of-the-art methods in numbers of metrics on two representative benchmarks and scenarios. Code is available at https://github.com/czh-98/CMS-LSTM.