Synchronization and combination techniques for audio-video based handwritten mathematical content recognition in classroom videos

Recognizing handwritten mathematical content is a challenging problem, and more so when such content appears in classroom videos. However, given the fact that in such videos the handwritten text and the accompanying audio refer to the same content, a combination of a video and an audio based recognizer has the potential to significantly improve the content recognition accuracy. In this paper, using a combination of video and audio based recognizers, we focus on improving the character recognition accuracy in such videos and propose: (1) synchronization techniques for establishing a correspondence between the handwritten and the spoken content, and (2) combination techniques for combining the outputs of the video and audio based recognizers. The current implementation of the system makes use of a modified open source text recognizer and a commercially available phonetic word-spotter. For evaluation purposes, we use videos recorded in a classroom-like environment and our experiments demonstrate the significant improvements (≈ 24% relative increase as compared to the baseline video based recognizer) in character recognition accuracy that can be achieved using our techniques.