HARK based real-time single pane 3D auditory scene visualizer empowered by speech arrow

Robot audition, which aims at multiple simultaneous human-robot communications in noisy environments, has undergone rapid development since last decade. For a common scenario, robot audition systems usually generate a huge amount of chronological data. These data fall into several categories: sound source locations and orientations, separated sound sources and recognized speeches. In order to efficiently assist people in understanding and analyzing the scenario, it is necessary to organize the data in a delicate way. To achieve this goal, we suggest Speech Arrow which conveys recognized speeches in animated arrow shaped frames. Speech Arrow is highly integrated, intuitive and vivid. In this paper, a single pane auditory scene visualizer is implemented to support Speech Arrow. The visualizer is 3D real-time virtual reality system based on the open source robot audition system HARK. It includes two versions, one of which restores original auditory scenes with offline raw data. The other analyzes and displays online instant data without any delay to help hearing impaired people. We demonstrate that our visualizer improves auditory awareness considerably with experiments.