A visual language for non-WIMP user interfaces

Unlike current GUI or WIMP style interfaces, non-WIMP user interfaces, such as virtual environments, involve parallel, continuous interactions with the user. However, most current visual (and non-visual) languages for describing human-computer interaction are based on serial, discrete, token-based models. The paper introduces a visual language for describing and programming the fine-grained aspects of non-WIMP interaction. It is based on the notion that the essence of a non-WIMP dialogue is a set of continuous relationships, most of which are temporary. The underlying model combines a data-flow or constraint-like component for the continuous relationships with an event-based component for discrete interactions, which can enable or disable individual continuous relationships. The language thus separates non-WIMP interaction into two components, each based on existing visual language approaches, and then provides a framework for connecting the two.