Enhancing Human-Robot Interaction: Integrating ASL Recognition and LLM-Driven Co-Speech Gestures in Pepper Robot with a Compact Neural Network

This research investigates the use of compact deep neural network designs to teach the humanoid robot Pepper to comprehend American Sign Language (ASL), enhancing non-verbal interactions between humans and robots. Initially, we developed a streamlined and powerful model specifically for ASL interpretation that was optimized for embedded systems. This model ensures swift sign language recognition while minimizing computational demands. Furthermore, we incorporate large language models (LLMs) to enhance the robot's interactive abilities. Pepper can produce Co-Speech Gesture responses by carefully crafting prompts, fostering more fluid and realistic human-robot conversations. We also introduce a comprehensive software framework that encapsulates these advancements in a socially conscious AI interaction model. Utilizing Pepper's capabilities, we showcase the practicality and impact of our method in actual scenarios. Our findings underscore the significant possibilities for improving human-robot interactions through non-verbal means, bridging communication barriers, and making technology more accessible and understandable.