SoundEYE: AI-Driven Sensory Augmentation System for Visually Impaired Individuals through Natural Sound

Sensory augmentation is required for the visually impaired person (VIP) to compensate for the loss of vision and enhance their perception of the environment. Existing solutions for navigation aids have limited flexibility to respond to real-time changes in actions happening in a visual scene, limiting the blind individual to perceive the surroundings comprehensively and spontaneously. In this paper, we introduce an AI-driven sensory augmentation system (SoundEYE) for VIPs that will produce sound from visuals utilizing the advantages of employing natural auditory guidance which is readily comprehensible to the user. The proposed SoundEYE system incorporates a novel sound generation AI model combining visual action recognition and adversarial audio synthesis networks that can automatically generate representable sound for critical situations. This leads to a significant impact on VIPs to comprehend and deal with real-world events. Unlike the existing sound aid solutions, SoundEYE is capable of analyzing a complex scene with the presence of multiple sound sources and generating representable synchronous sound through capturing audio-visual synchronization with moving objects. In addition, we propose a comprehensive evaluation methodology to assess the performance of the system including an empirical study plan to analyze the impact of natural sound over vocal auditory cues for the VIP.