Cloud-Type Classification Using Multimodal Integration Transformer Based on Cloud Images and Millimeter-Wave Cloud Radar Observations

The existing methods fail to simultaneously utilize the appearance information and the internal structure of clouds for cloud-type classification, resulting in incomplete cloud representation. In this article, we exploit cloud images and millimeter-wave cloud radar (MMCR) observations for cloud-type classification and propose a novel Transformer network named multimodal integration Transformer (MMITrans) to describe the complete information of clouds. To this end, we design MMITrans as three subnetworks: the vision subnetwork, the MMCR subnetwork, and the multimodal fusion network. Specifically, we extract the visual features from the cloud images through the vision subnetwork. Meanwhile, we first convert MMCR observations into several cloud-related indicators and propose the Indicator-Tokenization to effectively tokenize them and obtain the indicator features using the MMCR subnetwork. Furthermore, we propose the multimodal cross-attention in the multimodal fusion network to sufficiently fuse the visual features and the indicator features in a multiple-input way. We perform a series of experiments on cloud images and MMCR observations datasets, that is, CMD-Beijing and CMD-Gansu, and the experimental results demonstrate the superiority of the proposed MMITrans.