DATC-STP: Towards Accurate yet Efficient Spatiotemporal Prediction With Transformer-Style CNN

Recently, convolutional neural networks (CNNs) or vision transformers (ViTs) based Multi-In-Multi-Out (MIMO) architectures have been proposed to overcome the limitations of recurrent neural networks (RNNs) based Single-In-Single-Out (SISO) architectures. These architectures prevent the inherent limitations of RNNs, such as performance degradation and inefficient parallelization due to sequential properties. However, there are still some challenges. CNN-based MIMO architectures struggle to capture global spatiotemporal information due to the local properties of their convolution kernels. Meanwhile, ViT-based MIMO architectures are less effective at capturing local spatiotemporal information and require a large amount of computational resources due to the self-attention mechanism. To address these limitations and improve MIMO architectures, we propose a novel accurate yet efficient Dual-Attention Transformer-style CNN for Spatiotemporal Prediction (DATC-STP). DATC-STP captures both local and global spatiotemporal information through 3D patch embedding and Transformer-style CNN based attention blocks. Specifically, 3D patch embedding extracts local spatiotemporal features and reduces the size of input data including temporal, height, and width dimensions. Two Transformer-style CNN-based attention blocks treat spatiotemporal data similarly to images and capture global information with CNNs. This structure makes DATC-STP accurate yet efficient. To demonstrate the effectiveness of DATC-STP, we conduct comprehensive experiments on three widely used benchmark datasets, MovingMNIST, TaxiBJ, and KTH. We evaluated that the proposed DATC-STP achieves both competitive accuracy and efficiency. Furthermore, results of the ablation study demonstrate the effectiveness of each component of DATC-STP and highlight the potential of the proposed method.