SDRV: Real-Time On-Device Subtitles Detection, Recognition and Voicing

The paper presents a computationally efficient approach to real-time subtitle detection, recognition, and voicing (SDRV). Video text spotting and recognition have seen rapid development, with the emergence of deep learning (DL), but the problem of video subtitles detection and recognition on the fly, taking into account the limitations implied by on-device realization, has not been sufficiently addressed in the existing literature. The proposed system consists of a novel subtitle tracking module, based on the processing of three consecutive frames by a convolutional neural network (CNN)-based classifier, a subtitle text line localization module based on single shot multibox detector (SSD) architecture, and the proposed adaptation of convolutional recurrent neural network (CRNN) architecture for on-device implementation text recognition using a new chunking procedure for the CNN input and merging this data in the RNN part. The subtitles recognition results are voiced by a text-to-speech (TTS) engine, making it possible to hear the translation of the audio track in a foreign language for visually impaired people who are unable to read the subtitles in their language. On-device end-to-end evaluation of the proposed approach demonstrates high accuracy with a Word Recognition Rate (WRR) of 92.1% and low latency of 250 ms.