Modeling Graphical and Speech User Interfaces with Widgets and Spidgets

Consumer electronic devices and in-vehicle infotainment systems are increasingly integrating advanced dialog capabilities. The user expectation rises and the HMI complexity grows, including synchronization of multiple devices and cloud capabilities. The speech dialog component becomes a virtual meta-HMI consolidating all the speakable content in coordination with haptics and graphics. In this context using a dialog modeling tool adapted to all use cases, abstracted from the architecture, and playing together with the HMI, is challenging. In this paper we introduce a unified HMI modeling concept based on widgets and spidgets (speech gadgets). We leverage the concepts of graphic modeling using views and widgets together with an UML based representation and adapt those to talks and spidget components. Spidgets are combinable dialog elements grouped into talks to generate dialog states. The common UML logic representation enables implicit support of multimodal use cases.