Gait Activity Classification Using Multi-Modality Sensor Fusion: A Deep Learning Approach

Floor Sensors (FS) are used to capture information from the force induced on the contact surface by feet during gait. On the other hand, the Ambulatory Inertial Sensors (AIS) are used to capture the velocity, acceleration and orientation of the body during different activities. In this paper, fusion of the stated modalities is performed to overcome the challenge of gait classification from wearable sensors on the lower portion of human body not in contact with ground as in FS. Deep learning models are utilized for the automatic feature extraction of the ground reaction force obtained from a set of 116 FS and body movements from AIS attached at 3 different locations of lower body, which is novel. Spatio-temporal information of disproportionate inputs obtained from the two modalities is balanced and fused within deep learning network layers whilst reserving the categorical content for each gait activity. Our approach of fusion compensates the degradation in spatio-temporal accuracies in individual modalities and makes the overall classification outcomes more accurate. Further assessment of multi-modality based results show significant improvements in f-scores using different deep learning models i.e., LSTM (99.90%), 2D-CNN (88.73%), 1D-CNN (94.97%) and ANN (89.33%) respectively.