State-machine based approach for improving robustness in multimodal control

Multimodal systems are aimed to overcome the disadvantages of unimodal systems, especially the lack of robustness. In this paper, a new approach to modeling multimodal systems, by dividing the system to independent state machines, is proffered. The state machine approach is supposed to be more intuitive to the user and to increase redundancy, especially in noisy environments. Intuitiveness is achieved by adapting each state machine to the user's expectations. Redundancy is achieved by allowing the user to use more than one modality to express the same utterance. Its task-oriented grammar allows the reuse of identical commands thus reducing the overall size of command set. The architecture also allows increased recognition accuracy by decomposing the large command vocabulary into smaller subsets. The system has been tested on a small command vocabulary in a simulated environment. The results of the comparison between pure gesture and pure voice showed that the bimodal system was superior.