MoBiM2: Multimodal Bidirectional Mamba for Two-Stage Anomaly Detection in Industrial Smelting Furnace

In the complex environments of modern process industries, timely and accurate anomaly detection is paramount for maintaining smooth, efficient, and safe production processes. Traditional anomaly detection techniques typically rely on unimodal data. However, with the advent of smart factories and advanced sensor technologies, multimodal information such as video data and time-series data has emerged as critical enablers for improving anomaly detection efficiency. To address this evolving challenge, this paper introduces a novel two-stage multimodal anomaly detection framework, termed MoBiM2, which leverages both video and time-series data. The proposed method adopts a bidirectional Mamba structure with linear complexity, effectively integrating multisource information through a shared matrix that bridges the two modalities. First, a 3D Tokenization technique is introduced to efficiently extract spatiotemporal features from video data. The initial stage of the framework, called the MBi-Mamba module, which extracts and integrates crucial information from video and time-series data via the shared matrix in the bidirectional Mamba structure. To further refine feature extraction, the second stage incorporates the EBi-Mamba module, which intensifies the interaction and fusion of multimodal information. To accommodate the diverse demands of real-world industrial applications, three model configurations of varying scales are presented. Experimental validation on a publicly available multimodal industrial smelting furnace dataset, demonstrates that the proposed method achieves accuracies of 95.71%, 94.91%, and 93.03% at different scales, respectively. Moreover, this approach surpasses existing Transformer-based and CNNbased methods, highlighting superior performance.