Automated language acquisition in multimodal environment

The article describes a real-time laboratory prototype system that adaptively acquires multimodal language and uses it for natural style human-machine communication. Methods for building information theoretic networks and dialogue control are presented. Example interactions are presented and results are analyzed for two modalities: speech and mouse (pen) pointer. Future work is outlined, with special emphasis on the application of multimodal language learning in information transformation.