Semantic Segmentation of Mining Environments for Intelligent Electric Shovels Based on Multimodal Memory Enhancement

In the construction of smart mines, unmanned excavation by mining electric shovels is an essential part. In order to ensure the accuracy of electric shovel digging, it is necessary to perceive the mine environment and perform semantic segmentation. Current mainstream semantic segmentation methods primarily focus on point cloud data, effectively utilizing the three-dimensional spatial information of mining environments. However, these methods fail to leverage the rich color and texture information present in real-world environments and struggle to address scale imbalances in mining objects, resulting in suboptimal segmentation accuracy. To address this problem, we propose a memory-enhancement segmentation network (MESNet), which extracts 3D spatial features using a point cloud feature extraction branch and captures color and texture information through an image feature extraction branch. The memory enhancement module (MEM) is applied to the fused point cloud and image features, focusing on small-scale and low-frequency objects within the fused features. Experiments conducted on a large-scale mining point cloud dataset demonstrate an overall accuracy of 93.6%, validating the effectiveness of the proposed method. It also provides an effective method for autonomous environment sensing of unmanned electric shovels.