Hello Robot, Can You Come Here? Using ROS4iOS to Provide Remote Perceptual Capabilities for Visual Location, Speech and Speaker Recognition

Mobile devices such as smartphones and tablets can provide additional sensing and interacting capabilities to a mobile robot, even extending its senses to remote locations. To do so, we developed ROS4iOS
¹
, a native port of ROS [6] allowing to seamlessly use data from mobile iOS devices to be processed on a robot. To demonstrate this capability, this video presentation illustrates how ROS4iOS has been used to implement an assistance scenario: a person in a remote location asks our IRL-1 [1] robot for assistance, and IRL-1 must recognize the person's voice, identify the remote location using images taken from the mobile device, navigate to the identified location and interact vocally with the person through the mobile device. When communication with the robot is established by launching a specific iOS application, audio from the mobile device is published on a single topic and directed toward two ROS nodes on IRL-1. PocketSphinx [4], a speech recognition toolkit, is used to obtain the person vocal commands. WISS [3], a speaker identification system, is used for speaker identification. From a vocal request made through the mobile device, the robot can identify the person and request to get images of the remote location using the rear-facing camera of the mobile device. RTABMap [5], a loopclosure detection system for visual location recognition, is used to locate the person. A map was previously built with the robot's laser range finder using the gmapping SLAM [2] algorithm, thus permitting reuse of the ROS navigation stack to plan a path and follow it safely from its current location to the requested one. Depending on its current state, the robot can also display the most suited view (i.e., phone, camera, navigation) on the iOS application. ROS4iOS can also be used to teleoperate IRL-1 through the mobile device or to discuss remotely with the robot, using the same ROS modules.