Vision-Based Human Activity Recognition Utilizing DCNN and Stacked Multi-stage Multimodal Fusion Strategy

Human Activity Recognition (HAR) possesses significant potential in pattern recognition, machine learning, wearable computing, computer vision, and notably in rehabilitation assistance. With the ability to automatically track patients' recovery progress, it addresses the issue of insufficient human resources. Presently, improving recognition accuracy hinges on the fusion of multiple information sources like physiological signals, motion sensors, and visual sensors. In this study, a multi-stage multimodal fusion strategy based on stacked integration with Deep Convolutional Neural Network (DCNN) is proposed. Firstly, the inertial sensor and depth sensor data are converted into image data, and the data enhancement method is used to extract edge information using Prewitt, and features are extracted and spliced by the DCNN to realize multimodality at the feature layer. At the decision layer, the two unimodal decisions are fused using the stacking fusion strategy to obtain the final classification decision. The experimental results show that this strategy exhibits significant advantages in human motion recognition, with accuracy rates of 99.85% and 99.51% on Kinect V2 and UTD-MHAD datasets, respectively, exceeding the state-of-the-art deep learning methods on the same dataset.