A Multimodal Asynchronous Human-Machine Interaction Method Based on Electrooculography and Speech Recognition for Wheelchair Control

In the 2020s, human-machine interaction (HMI) is everywhere. However, most HMI systems perform interaction based on physical control, which does not always work well for people with motor impairments, especially people suffering from quadriplegia or amputation. In this article, we propose a multimodal asynchronous HMI method based on electrooculography (EOG) and speech recognition for wheelchair control in a virtual reality (VR) environment. Specifically, we recruited ten healthy participants to control a wheelchair in a virtual park by performing eye blinks and speaking out voice commands right after a highlighted button. The HMI system simultaneously acquired the participants’ EOG and voice signals and then made output decisions for the wheelchair movement. The participants achieved a satisfactory average accuracy of 99.1% ± 1.3% and an average response time (RT) of 1.45 ± 0.18 s during the experiment using the proposed method. After workload evaluation with NASA task load index (NASA-TLX), the overall workload of the participants using the proposed method during the experiment was at a low level. These results showed that the participants completed the wheelchair control experiment with satisfying performance using the proposed method and offered new directions for assistive HMI methods for people with motor impairments.