Development of A Multimodal Deep Feature Fusion with Ensemble Learning Architecture for Real-Time Gas Leak Detection

Gas leak detection is extremely critical, but poses many significant challenges due to the high cost and large amount of human labor and expertise necessary to complete an inspection. Our previously reported soft robot SPIRo system is designed to address these challenges. This paper proposes a multimodal deep feature fusion system with the deep forest ensemble learning model, which fuses data from gas sensors and thermal images to improve SPIRo's accuracy and redundancy. The model consists of the convolutional layers of three CNNs: the AlexNet, ResNet-50, and MobileNet, and the extracted deep features from them along with the measured gas sensor data are fused into a Deep Forest Classification (DFC) module to make the final decision on whether a gas leak is present or not. The combination of a single CNN and gas sensor data fused with a DFC is also evaluated. Two sets of data have been tested with this system. The results indicate that for the training database, the ensemble model with three CNNs achieved a detection accuracy of 98.5%, compared to 95.0% for the ensemble model with single CNN and 93.1% achieved by each individual model. For the testing database collected by SPIRo, the ensemble model with three CNNs achieved an accuracy of 88.1%, compared to 82.4% for the single CNN models with ensemble and around 73.6% for the individual models. The results from experimental data collected by SPIRo furthers the architecture's effectiveness. Not only is the proposed method more accurate, but it is also more reliable due to increased redundancy in analyzing both thermal images and gas sensor data. In addition, it is also memory efficient, making it suitable for realtime detection applications.