Implementation of human-robot VQA interaction system with dynamic memory networks

One of the major functions of intelligent robots such as social or home service robots is to interact with users in natural language. Moving on from simple conversation or retrieval of data stored in computer memory, we present a new Human-Robot Interaction (HRI) system which can understand and reason over environment around the user and provide information about it in a natural language. For its intelligent interaction, we integrated Dynamic Memory Networks (DMN), a deep learning network for Visual Question Answering (VQA). For its hardware, we built a robotic head platform with a tablet PC and a 3 DOF neck. Through an experiment where the user and the robot had question answering interaction in our customized environment and in real time, the feasibility our proposed system was validated, and the effectiveness of deep learning application in real world as well as a new insight on human robot interaction was demonstrated.