Foresight: Empowering Visually Impaired Individuals through Multimodal Generative AI

This research paper presents an accessibility accelerator named Foresight, a system designed to reduce hallucination and enhance accuracy while assisting visually impaired individuals through a mobile application by leveraging advanced multimodal generative AI models. Existing solutions often struggle to provide accurate and contextually relevant information with seamless interaction capabilities, particularly in dynamic real-world scenarios. Foresight bridges this gap by integrating advanced AI models, including ShareGPT4V for visual analysis and Detectron2 for object grounding. The application combines their outputs through the Gemma model, which mitigates hallucinations and enhances response accuracy. Users can capture images of their surroundings, initiate detailed inquiries, and engage in iterative dialogues to refine their understanding, all within a single conversational thread per image capture. The object grounding functionality of the system highlights detected objects with distinct visual markers, improving interpretability. Additionally, it incorporates speech-to-text and text-to-speech capabilities for a fully multimodal interaction. Notably, Foresight identifies 12% more information within images compared to its base model, ShareGPT4V, demonstrating its enhanced capability in visual content analysis. Through its intuitive interface and responsive functionalities, Foresight addresses visual, cognitive, and physical challenges, offering personalized and context-aware experiences for users.