Shared 3D Robotic Arm Control Based on Intracortical Brain-Computer Interface and Computer Vision

Brain-computer interfaces (BCIs) convert neuronal activity into commands for controlling external devices. For example, a paralyzed patient can use BCI to control a robotic arm to drink water. However, employing intracortical brain-computer interfaces (iBCIs) to control robotic arm movements and various daily grasping tasks in a three-dimensional (3D) environment faces significant challenges. The degree of freedom (DoF) decoded from iBCI signals often falls short of the requirements for effective control of robotic arm & hand in daily activities. This study proposes a shared robotic arm control system for iBCIs. The system employs spike signals to control the robotic arm move into left or right direction. The integrated computer vision (CV) identifies the position, size, and category of the objects, facilitating more precise grasping. At last, movement direction decoding based on eight neural units achieved a classification accuracy of 93.1%, with a five-fold cross-validation error rate of 19.8%. The iBCI and CV sharedly control the robotic arm to grasp eight objects of different kinds and shapes, positioned at different locations. Furthermore, the shared control system has been validated through an online grasping experiment on a macaque. Results demonstrated the feasibility of generating practical multidimensional control of a robotic arm by merging iBCI and CV-based recognition.