A Novel Human-in-the-Loop Multimodal Intention Fusion Method for Human–Robot Interaction

Understanding human intention plays a crucial role in the research of human-robot interaction (HRI) for service robots. Although multimodal sensing methods have shown promising results in certain scenarios, the fusion strategy cannot flexibly adapt to user preference and dynamic environment. To address this limitation, this paper proposed a human-in-the-loop multimodal intention fusion (HIL-MIF) algorithm that introduces weight factors to assess the importance of each modality. By dynamically adjusting these weight factors based on user feedback, we achieved high accuracy in intention understanding, enhancing the personalization and reliability of the interaction. In addition, a multimodal service robot system was proposed that supports three interaction modalities including gaze, voice and gestures, which can be flexibly configured into different combinations. Validation experiment was performed on target object grasping tasks and collected user subjective evaluations. The experimental results demonstrate that the HIL-MIF proposed in this paper is superior to other commonly used multimodal fusion methods in terms of both accuracy and reliability. This paper paves the way for the future tri-co intelligent robot deployment in the service industry. Note to Practitioners—In the field of service robot, accurately understanding user intentions is a critical issue, as it directly affects the success of interactions. Among the various tasks of intention understanding, we face numerous challenges. For instance, unimodal approaches are susceptible to environmental factors and have very limited applicability. Currently, multimodal intention understanding methods face challenges such as the difficulty of achieving a unified representation of heterogeneous data from multiple sources and relatively low accuracy. Additionally, users have personal pReferences regarding interaction modalities, and existing multimodal methods lack research on these preferences, making it difficult to provide highly reliable and personalized interaction experiences. To address these issues, this paper proposes a human-in-the-loop multimodal intention fusion (HIL-MIF), which integrates multimodal information from gaze, gestures, and speech. It introduces weight factors to accommodate user preferences, employs combinatorial rules to fuse multimodal data, and determines the final intent from the fused information. Furthermore, a human-in-the-loop strategy is adopted to dynamically update the weight factors based on user feedback regarding the fusion results. The primary research objective of This study is to tackle the challenges of low accuracy and poor personalization in multimodal intention understanding methods, while also adapting to interactive scenarios such as meetings and low-light environments. By utilizing the multimodal intent fusion approach to control a robotic arm for object grasping, we achieved an intention understanding accuracy of 94%. This method can be easily applied to other human-robot interaction systems. In the future, we plan to apply this method to more complex human-robot interaction tasks to accomplish more sophisticated intention understanding challenges.