A multimodal emotional communication based humans-robots interaction system

A multimodal emotional communication based humans-robots interaction (MEC-HRI) system is proposed, which includes multiple emotional communication modes such as speech, facial expression, and gesture. The robots in MEC-HRI system could recognize humans' emotions and respond to humans in accordance with their emotions. Firstly, MEC-HRI system acquires the multimodal emotional cues from sensors, e.g., microphone, Kinect, eye tracker, EEG. Secondly, emotion recognition based on single modal/multimodal cues is carried out. Thirdly, robots respond to humans by facial expresion, speech, gesture, etc. Four scenarios, i.e., tour guiding, game entertainment, family services and scenairo simulation, are performed in the expriment, by which smooth communication between humans and robots is realized. In prospect, the MEC-HRI system would be applied in serving robots for smart house, education, hospital, restaurant, and so on.