Semantics-based software techniques for maintainable multimodal input processing in real-time interactive systems

Maintainability, i.e. reusability, modifiability, and modularity, is a critical non-functional quality requirement, especially for software frameworks. Its fulfilment is already challenging for low-interactive application areas. It is additionally complicated by complex system designs of Real-time Interactive Systems (RISs), required for Augmented, Mixed, and Virtual Reality, as well as computer games. If such systems incorporate AI methods, as required for the implementation of multimodal interfaces or smart environments, it is even further exacerbated. Existing approaches strive to establish software technical solutions to support the close temporal and semantic coupling required for multimodal processing and at the same time preserve a general decoupling principle between involved software modules. We present two key solutions that target the semantic coupling issue: (1) a semantics-based access scheme to principal elements of the application state and (2) the specification of effects by means of semantic function descriptions for multimodal processing. Both concepts are modeled in an OWL ontology. The applicability of our concepts is showcased by a prototypical implementation and explained by an interaction example that is applied for two application areas.