Audiovisual integration system based on synesthesia for people with impaired hearing

Rather than being completely isolated, the various human sensory systems can influence each other, sometimes strongly. For example, the cognitive phenomenon known as “colored hearing” involves seeing colors in response to auditory stimuli. This type of special cognitive condition, in which one sensory stimulus evokes a different perception from that of the trigger, is known as “synesthesia.” Previous research has suggested that everyone likely has synesthesia-like recognition to some extent. Regarding colored hearing, some previous studies have shown that it obeys certain rules. The aim of this study was to develop a system that uses those rules to convert auditory information into visual information. Such a system could help people with impaired hearing to understand sound intuitively, and could have further applications, particularly in the media arts. This report presents the method for audiovisual conversion and a prototype of the system.