Automatic instrument and environmental sound recognition for media annotation of TV content

Due to the lack of annotation of their large video archives, multimedia content provider companies and television channels do not use the data in their archives to their full extent. In order to contribute with a solution to this problem, we have developed a tool that combines audio and visual information to annotate video. In particular, this tool has been used by a video production company that has given us positive feedback. The main innovation of this tool is the use of environmental sound recognition to annotate video. Here we focus on the tool's audio information extraction method, which consists of a sound recognizer that learns a small set of spectral features from the data using non-negative matrix factorization. The recognizer can be used for different purposes such as to classify musical instruments, to identify the notes that are played and to distinguish environmental sounds like water, traffic, trains and people.