Sequential Image Classification of Human-Robot Walking Environments Using Temporal Neural Networks

Robotic prosthetic legs and exoskeletons require real-time and accurate predictions of the walking environment for smooth transitions between different locomotion mode controllers. However, previous studies have mainly been limited to static image classification, therein ignoring the temporal dynamics of human-robot locomotion. Motivated by these limitations, here we developed and tested a number of state-of-the-art temporal neural networks to compare the performance between using static vs. sequential images for environment classification (i.e., level-ground terrain, incline stairs, and transitions to and from stairs). Using our large-scale image dataset, we trained several 2D encoder networks such as MobileNetV2 and MobileViT, each coupled with a temporal long short-term memory (LSTM) backbone. We also trained MoViNet, a new 3D video classification model, to further compare the performance between 2D and 3D temporal neural networks. The 3D network outperformed the 2D encoder networks with LSTM backbones and a 2D CNN baseline model in terms of image classification accuracy, suggesting that the network architecture can play an important role. However, although the 3D neural network achieved the highest image classification accuracy (98.3%), it had disproportionally higher computational and memory storage requirements, which has practical implications for real-time embedded computing for control of robotic leg prostheses and exoskeletons.