Abstract:
In this paper, we present a novel approach to advancing augmented reality (AR) dialogue systems, bridging the gap between two-dimensional spaces and immersive virtual env...Show MoreMetadata
Abstract:
In this paper, we present a novel approach to advancing augmented reality (AR) dialogue systems, bridging the gap between two-dimensional spaces and immersive virtual environments. We construct the “SIMMC2-Point” dataset, which transforms the original SIMMC2 dataset from virtual reality (VR) into AR environments, highlighting the additional introduced pointing modality to support understanding user's multi-modal intentions in AR. By harnessing the power of BART and CLIP models, we design the architecture of multi-modal dialogues that effectively captures spatial and attribute information. Then, a series of ablation experiments based on our designed SIMMC2-Point dataset and dialogue models underscores the significance of the pointing modality in enhancing the performance of dialogue systems across various tasks. Our work represents a crucial step forward in AR dialogue systems, facilitating seamless interactions within immersive virtual environments.
Published in: 2023 China Automation Congress (CAC)
Date of Conference: 17-19 November 2023
Date Added to IEEE Xplore: 19 March 2024
ISBN Information: