Extended Control With Hybrid Gaze-BCI for Multi-Robot System Under Hands-Occupied Dual-Tasking

Currently there still remains a critical need of human involvements for multi-robot system (MRS) to successfully perform their missions in real-world applications, and the hand-controller has been commonly used for the operator to input MRS control commands. However, in more challenging scenarios involving concurrent MRS control and system monitoring tasks, where the operator’s both hands are busy, the hand-controller alone is inadequate for effective human-MRS interaction. To this end, our study takes a first step toward a multimodal interface by extending the hand-controller with a hands-free input based on gaze and brain-computer interface (BCI), i.e., a hybrid gaze-BCI. Specifically, the velocity control function is still designated to the hand-controller that excels at inputting continuous velocity commands for MRS, while the formation control function is realized with a more intuitive hybrid gaze-BCI, rather than with the hand-controller via a less natural mapping. In a dual-task experimental paradigm that simulated the hands-occupied manipulation condition in real-world applications, operators achieved improved performance for controlling simulated MRS (average formation inputting accuracy increases 3%, average finishing time decreases 5 s), reduced cognitive load (average reaction time for secondary task decreases 0.32 s) and perceived workload (average rating score decreases 15.84) with the hand-controller extended by the hybrid gaze-BCI, over those with the hand-controller alone. These findings reveal the potential of the hands-free hybrid gaze-BCI to extend the traditional manual MRS input devices for creating a more operator-friendly interface, in challenging hands-occupied dual-tasking scenarios.