Audio-visual Broadcast Transcription System Using Artificial Neural Networks

In this paper, a new system for audio and visual TV broadcast News transcription is described. In the last few years, our system for audio-only broadcast transcription has been modified with the possibility of obtaining additional visual information, especially from TV video recordings. New extension modules and algorithms mainly for visual information extraction are described in this contribution. Combined Deep Neural Networks with Hidden Markov Models (DNN-HMM) are used for audio speech signal recognition. A classification of a relevant visual signal was based on Convolutional Neural Networks (CNN). There are the additional modules for detection and identification of human faces, TV logos, and company logos in the newly developed transcription system. Another module was designed for Optical Character Recognition (OCR) of text, which occurs mainly in video recordings of TV News very often. The whole audio-visual system for broadcast transcription was tested on a relatively big database (817 hours) which has been completely transcribed. The system also includes the possibility of intelligent search in transcribed data from audio and/or visual signals.