Multi-modal output composition for human-computer dialogues

The authors discuss the development of a prototype multimodal human-computer interface system, called CUBRICON, which is a research platform for investigating how knowledge-based approaches can best be applied to enhance human-computer interfaces. Four critical features of such an interface are introduced along with CUBRICON's initial approach to achieving them. Several knowledge sources have been defined within CUBRICON. These serve as the basis for CUBRICON's multimedia output planning process. These knowledge sources include: characteristics of the information, a model of the ongoing discourse, a user/task model, and a knowledge base of task domain and related interface information. CUBRICON's output planning process includes selection of appropriate media/modalities, selection of ancillary relevant information, determination of whether display resources are available, modification of resources or information to be expressed (if necessary), modification of selected output media/modalities (if necessary), and composition of the output. Examples are given for the application task domain of military tactical air control.<
>