Correlation-Based Multimodal Fusion Method for Icing Degree Monitoring of Transmission Lines Within Internet of Things

Icing monitoring system within Internet of Things is developed to provide multimodal data for assessing the degree of icing on transmission lines. Nevertheless, existing methods relying solely on either sensor data or images demonstrate limited precision and inferior fault tolerance. This article proposes a correlation-based multimodal feature fusion approach to integrate both sensor data and imaging data, thereby enhancing the monitoring of icing severity on transmission lines and addressing the aforementioned problems. The inherent correlation characteristics present in both sensor data and images, as well as the correlation between these two modalities are thoroughly analyzed, to gain a comprehensive understanding of icing characteristics in multimodal data. Specifically, the squeeze-excitation module along with convolutional neural networks are combined to extract the temporal and spatial correlation inherent in sensor data, as well as the spatial and channel correlation inherent in images. Subsequently, the covariance matrix is used to capture the correlation between these two modalities. Moreover, with the assistance of the weight determination structure, this correlation is mapped to the fusion weights. The four-stage training strategy is introduced to guide the network training. The essentiality of the staged training approach and the necessity of correlation extraction in perceiving icing severity are validated in experiments.