A Multi-Sensor Fusion System for Improving Indoor Mobility of the Visually Impaired

Independent movement in an unknown indoor environment is a challenging task for the visually impaired. By considering the connectivity of the corridor (room doors and stairs are all connected by the corridor), we propose an assistive navigation system to help visually impaired users navigate in corridor environments in this paper. Based on semantic simultaneous localization and mapping (SLAM), the corridor area is determined and mapped to a semantic map. Semantic path planning is then performed according to the lowest energy cost principle by taking the safety into consideration. The YOLO neural network is employed to detect and identify common indoor landmarks such as Toilet, EXIT, Staircase et at., and the system is able to give voice feedback about objects along its line of sight during the movement. This interaction helps to enhance perception about objects and places to improve travel decisions. A TurtleBot2 robot with a laptop, a RPLIDAR A2, and a Microsoft Kinect V1 are utilized to validate the localization, mapping and navigation module, while the perception module uses a ZED stereo camera to capture the objects and landmarks along its line of sight. The software modules of this system are implemented in Robot Operating System (ROS) and tested in our lab building.