LLM-Enabled Incremental Learning Framework for Hand Exoskeleton Control

It remains a formidable challenge to accurately recognize motion intentions of patients thus to control hand exoskeletons according to their volition. Current methods primarily focus on recognition of limited patient’s motion intentions, with the purpose of controlling preconfigured gestures of a hand exoskeleton for grasping objects. These methods exhibit a marked shortfall when encountering scenarios that are unexpected or not designed in advance, such as non-preprogrammed hand movements and object manipulation tasks. To tackle this issue, large language model (LLM) and speech recognition technology are employed in this study to allow the patient to control a hand exoskeleton at will. In particular, two LLMs are tailored to formulate codes of either generating non-preprogrammed gestures or dealing with unencountered objects. Additionally, an incremental learning framework is proposed to enable patients to perform both predefined and non-predefined operation tasks by integrating a natural language parser with the two LLM-based learners. The natural language parser can directly control the hand exoskeleton to perform predefined operations tasks from prestored command set, while the LLM-based learners can incrementally expand the control command set so as to enhance adaptability of the hand exoskeleton to complex activities over daily use. This study is a pioneering work in the field of hand exoskeletons, which will revolutionize the way to control hand exoskeletons. Furthermore, the proposed framework can be easily generalized to any other robots by modifying the prompt of customized LLMs, which provides a new idea to achieve autonomous learning in robotics. Note to Practitioners—The motivation of this article is to tackle the challenge of intention recognition for performing activities of daily living (ADLs) by stroke patients using a multi-degree of freedom hand exoskeleton. Existing methods for intention recognition so far can only be used for several tasks that are predefined in advance, thus none of them allow patients to control the hand exoskeleton completely at will. To surpass this limitation, an LLM-enabled incremental learning framework that integrates a hand exoskeleton controller with Large Language Model (LLM) is proposed and validated in this study. The framework offers patients an intuitive interface via voice interaction and enables patients to perform not only predefined operation tasks by the hand exoskeleton controller but also non-predefined ones that can be learned from the LLM. As a result, the hand exoskeleton controller continues to learn from the LLM, therefore is gradually able to perform all tasks in daily life. This pioneering study paves a new way in building patient-controlled hand exoskeletons with autonomous intelligence that can deal with non-predefined operation tasks in unstructured environments.