Eyes and Ears: Automated Annotation of Audio Data Using Computer Vision

Audio annotation is a crucial, yet time-consuming process, mostly due to the effort it takes human listeners to label sound data. We present an automated system for audio annotation that leverages a camera, a microphone and an off-the-shelf computer vision model to detect objects on a video track and label the corresponding audio track. The labeled audio data is then used to train a deep learning model, which enables a microphone based system to detect objects correctly without the need for a camera. To prove the concept, we developed a simple traffic monitoring system, taking advantage of the YOLOv8 model to annotate the audio data automatically and developing a ResNet-12 model to match road sounds to vehicle types, e.g., cars and motorcycles. Our automated solution achieves a 26x improvement in labeling speed compared to the manual approach. The resulting audio model for vehicle sounds achieves 0.91 precision and 0.9 recall. The proposed system is also able to count vehicles with less than 5% error. The generic methodology for automated audio annotation using off-the-shelf computer vision models extends naturally to a broad range of use cases for example in surveillance, wildlife protection or industrial monitoring.