Enhancing Augmented Reality Dialogue Systems with Multi-Modal Referential Information

In this paper, we present a novel approach to advancing augmented reality (AR) dialogue systems, bridging the gap between two-dimensional spaces and immersive virtual environments. We construct the “SIMMC2-Point” dataset, which transforms the original SIMMC2 dataset from virtual reality (VR) into AR environments, highlighting the additional introduced pointing modality to support understanding user's multi-modal intentions in AR. By harnessing the power of BART and CLIP models, we design the architecture of multi-modal dialogues that effectively captures spatial and attribute information. Then, a series of ablation experiments based on our designed SIMMC2-Point dataset and dialogue models underscores the significance of the pointing modality in enhancing the performance of dialogue systems across various tasks. Our work represents a crucial step forward in AR dialogue systems, facilitating seamless interactions within immersive virtual environments.

I. Introduction

The advent of machine learning and deep learning techniques has ushered in a new era of technological advancements in various domains, including computer vision, natural language processing, and human-computer interaction. Augmented reality (AR) technology, which amalgamates computer vision, sensor technology, and multi-modal human-computer interaction, has garnered widespread popularity, enabling users to interact with three-dimensional virtual worlds, delivering highly immersive experiences. Unlike conventional interaction devices, AR allows real-time interaction with virtual objects through bodily gestures, offering a more natural and human-centric experience.