Bimodal combination of speech and handwriting for improved word recognition

This paper presents a multimodal interface combining the use of speech and handwriting for isolated word recognition. Automatic Speech Recognition accuracy decreases as the perplexity of the task increases with the vocabulary size and the level of noise. The combination of different input modalities can improve the recognition performance. Handwriting is a modality that is natural to use, and can replace a keyboard on small portable devices, like Tablet PC's or PDA's. However this input method can be quite slow by itself. The proposed method in this paper combines both modalities by using handwriting to input only the first letters of a word, and speech to complete the word. The platform used to test this combination was a Tablet PC, using the Windows XP Tablet PC integrated handwriting recognition engine. Experiments were done based on a vocabulary of 35000 words. Relative word recognition improvements as high as 53% were obtained.