Cognitive engineering of gestural primitives for multi-modal interaction in a virtual environment

An important task of cognitive engineering for human-machine performance systems is to engineer the performability of a human observer in the simulated environment. For human observers to act in an environment: 1) they must perceive affordances of surrounding objects and their potential behaviors under interaction, and 2) the machine should be able to interpret human motions as an intentional drive. In this paper the concept of generative mechanisms is introduced to represent such affordances as abstract models placed in interactive pathways. The classification of gestural primitives is proposed, and implementations are presented through case studies. The application of generative mechanisms is discussed through examples for linking gestural primitives to graphic and auditory displays.