Development of an Algorithm for Audiovisual Sensory Substitution with YOLO Network Preprocessing to Reduce the User’s Sensory Overload

Research in the field of sensory substitution provides an opportunity for people with visual impairment to experience the world around them. A lot of algorithms are created for this purpose. However, even after a short period of use, the user experiences sensory overload. This study aimed to create an algorithm for visual-auditory sensory substitution with the neural network preprocessing to minimize sensory overload for user. For object recognition the YOLOv3 model was chosen. This model provides fast image processing and fine accuracy in object recognition. For test we limited recognized classes to “person” and “car” as most common objects on the street. The main advantage of the algorithm is using different waveforms for different objects. The developed algorithm was compared to popular visual-auditory sensory substitution algorithm The vOICe. The vOICe algorithm turned out to be almost twice as fast. However, the developed algorithm converted only meaningful information into sound and coped with the sensory overload problem.