ISEE: A Wearable Image-Sound Translation System for Partially Sighted People

As there is a growing number of partially sighted people, many partially sighted aid systems, which use the way of converting image to sound or vibration have been designed. Phonetic prompts and vibrations are, however, very limited in information expression. Therefore, a system called “image-sound eye-substitution equipment” (ISEE) is designed in this article. This system consists of a binocular camera, Raspberry Pi, and stereo headsets. It can convert image to sound by mapping each pixel into an independent sound source based on sensory substitution technology (SST) and enables the partially sighted to browse pictures, appreciate scenery, and avoid obstacles for its high-density information expression. Its core data processing design is the SST-based image-to-sound transform (SST-IST) algorithm. The SST-IST mainly consists of three steps, i.e., the image reshape, image-to-sound conversion matrix generation, and image-to-sound transmission. First, a single-frequency pitch of pixel is generated, followed by wave interception and localization to obtain independent sound clips to express pixel information. Second, the sound chips based on head-related transfer functions (HRTFs) are stereo processed. Third, a series of single-frequency sound chips are accumulated with weights to generate a complete sound. Extensive experiments show that our SST-IST can restore image information well, and the sound restore rate can achieve 73.0%. Also, it is universally adaptive and robust to the changes in ambient light and noises within a certain range. The auditory discrimination experiments show that, with 4–5 h training, the users can recognize basic graphics by using the ISEE system.