Engaging in a Conversation with Synthetic Agents along the Virtuality Continuum

During the last decade research groups as well as a number of commercial software developers have started to deploy embodied conversational characters in the user interface especially in those application areas where a close emulation of multimodal human-human communication is needed. Incarnations of such characters differ widely in type and amount of embodiment - starting from simplistic cartoon-style 2D representations of faces, fully embodied virtual humans in 3D virtual worlds to physically embodied androids co-habiting the user's real world. Despite of their variety, most of these characters have one thing in common: In order to enter the user's physical world, they need to be physical themselves. My talk focuses on challenges that arise when embedding synthetic conversational agents in the user's physical world. Following [4], we may classify the contact between synthetic and human agents according to a "virtuality continuum" (see Fig. 1). At one extreme, we find android agents that are completely integrated in the user's physical world and even allow for physical contact with the user. Mel, a robotic penguin developed by Sidner and colleagues [5] (see image 1 in Fig. 1), is one of the most sophisticated physical agents that engages in face-to-face communication with a human user. At the other extreme, there are purely virtual environments that are populated by human and synthetic agents. A prominent example is the pedagogical agent Steve [3] (see Image 4 in Fig. 1). Steve is aware of the user's presence in the virtual space, monitors her actions and responds to them, but has no access to the external world. That is it is only able to perceive user actions that are performed in the virtual space. In between, we find a new generation of characters that inhabit a world in which virtual and digital objects are smoothly integrated. In these applications, projections of virtual characters overlay the user's physical environment or projections of real persons are inserted into a virtual world. For instance, Cavazza and colleagues [2] propose a magic mirror paradigm which puts the user both in the role of an actor and a spectator by inserting the user'svideo image in a virtual world that is populated by synthetic agents (see Image 3 in Fig. 1). In the Virtual Augsburg project (see [1]), a synthetic character called Ritchie jointly explores with the user a table-top application that combines virtual buildings of the city center of Augsburg with a real city map being laid out on a real table. Most work so far has concentrated on the design and implementation of conversational agents at the two extremes of the Virtuality Continuum. In my talk, I will report on a new generation of synthetic characters that are no longer bound to a flat screen, but able to enter a physical world and to engage in a conversation with a human user. Users and characters do not inhabit separated spaces, but share an informational and physical reality that is augmented by digital objects. As a consequence, communication has to take into account both the physical and the digital context. New forms of deixis are enabled by the manipulation of objects and movements of characters in the physical space. Further challenges arise from the realization of so-called traversable interfaces that allow human and synthetic agents to cross the border from the digital to the real world and vice versa.