Grammar-assisted audio-video equation recognition

In this paper, we consider the problem of recognizing handwritten mathematical content from classroom videos. Since the handwritten text and the accompanying audio refer to the same mathematical characters and symbols, a combination of video and audio based recognizers has the potential to significantly increase the recognition accuracy compared to that of the individual recognizers. In this paper, we propose a novel multi-step technique for combining the output of the video and the audio based recognizers. Initial recognition results from a video based recognizer and a speech recognizer, operating independently on the handwritten and the spoken content from a classroom video, are combined with a base mathematical speech grammar to arrive at a constrained speech grammar that is specific to the content being recognized. The constrained speech grammar is then used by the speech recognizer to generate the final character recognition results. A subsequent layout analysis step, which makes used of audio cues and X-Y cuts based method, is used to arrive at the final recognized content. Experiments conducted using videos recorded in a classroom like environment are used to demonstrate the significant improvement in recognition accuracy that can be achieved using our technique.