Modeling multimodal integration based on colored Petri nets and feature structures

Multimodal integration is a vital technology in human-computer interaction systems. In this paper, a multimodal system's architecture is presented based on feature structures, which represent the semantic contributions of the different modalities, and multimodal integration is driven by unification of feature structures. We propose a MJHCP-net (multimodal integration-net based on hierarchical colored Petri net), which is an extension to hierarchical colored Petri net (HCPN) integrated with unification of feature structures, to model such multimodal systems. A specific multimodal system, "multimodal battle deployment editor", is used to demonstrate the modeling power of the method. Lastly, benefits and limitations of the approach are discussed.