Speech Translation Statistical System Using Multimodal Sources of Knowledge

The synergic combination of different sources of knowledge is a key aspect in the development of modern statistical translators. The effect and implications of adding additional other-than-voice information in a voice translation system is described in this work. The additional information serves as the bases for the log-linear combination of several statistical models. A prototype that implements a real-time speech translation system from Spanish to English that is adapted to specific teaching-related environments is presented. In the scenario of analysis a teacher as speaker giving an educational class could use a real time translation system with foreign students. The teacher could add slides or class notes as additional reference to the voice translation system. Should notes be already translated into the destination language the system could have even more accuracy. We present the theoretical framework of the problem, summarize the overall architecture of the system, show how the system is enhanced with capabilities related to capturing the additional information; and finally present the initial performance results.