Toward Assisting Blind Individuals in Exploring Unfamiliar Indoor Environments Using Multimodal LLM and Smartphone LiDAR

Blind individuals often face challenges accessing information about their surroundings when navigating unfamiliar indoor spaces. Recent advancements in artificial intelligence and smartphone-based sensing technology have opened up new possibilities for creating practical and cost-effective assistive solutions to enhance their environmental awareness and independence. In this paper, we investigate the integration of a cutting-edge large language model capable of processing multimodal inputs and outputs with smartphone-integrated LiDAR to assist blind travelers in scene understanding and obstacle avoidance in indoor spaces. We implement a smartphone application prototype that provides voice descriptions of user-captured photos and generates audio or vibrotactile feedback for nearby objects located in front of a smartphone camera. We present our initial findings from a preliminary study conducted with a blind expert in assistive technology, demonstrating our prototype's potential for enriching the indoor exploration experience of blind travelers. We also discuss opportunities and challenges regarding further improvements to our prototype.