A Gesture-based Multimodal Interface for Human-Robot Interaction

Surface electromyography (sEMG) has been proposed as one of the possible input modalities for gesture or proportional control-based human-robot interaction to relieve the operator from hand-held controllers. However, when it comes to mobile robotics, the applications have been limited, often providing only direct control over the velocity of the robot. In this work, we propose a multimodal interface for controlling mobile robots in collaborative settings. The robot navigates using a combination of its internal sensors, a LiDAR, and spatial input from a person-detecting neural network. The operator of the robot is identified from the view of the robot’s camera as the person wearing an sEMG armband, although the main use of the armband is for detecting hand gestures for commanding the robot. While the navigation is dependent on the line of sight between the robot and the operator, using the armband for gesture detection allows the robot to be commanded regardless of occlusions or changes in lighting conditions. Gesture-detection neural networks are first trained and tested offline with multiple subjects. Then, the complete interface is evaluated as a proof of concept with an expert user performing a sequence of tasks in cooperation with a quadruped mobile robot (Boston Dynamics). We demonstrate the usability of the interface in a realistic environment and show great long-term online performance of the gesture detection model with an average F-score of 0.94.