Loading [a11y]/accessibility-menu.js
Enhancing Augmented Reality Dialogue Systems with Multi-Modal Referential Information | IEEE Conference Publication | IEEE Xplore

Enhancing Augmented Reality Dialogue Systems with Multi-Modal Referential Information


Abstract:

In this paper, we present a novel approach to advancing augmented reality (AR) dialogue systems, bridging the gap between two-dimensional spaces and immersive virtual env...Show More

Abstract:

In this paper, we present a novel approach to advancing augmented reality (AR) dialogue systems, bridging the gap between two-dimensional spaces and immersive virtual environments. We construct the “SIMMC2-Point” dataset, which transforms the original SIMMC2 dataset from virtual reality (VR) into AR environments, highlighting the additional introduced pointing modality to support understanding user's multi-modal intentions in AR. By harnessing the power of BART and CLIP models, we design the architecture of multi-modal dialogues that effectively captures spatial and attribute information. Then, a series of ablation experiments based on our designed SIMMC2-Point dataset and dialogue models underscores the significance of the pointing modality in enhancing the performance of dialogue systems across various tasks. Our work represents a crucial step forward in AR dialogue systems, facilitating seamless interactions within immersive virtual environments.
Date of Conference: 17-19 November 2023
Date Added to IEEE Xplore: 19 March 2024
ISBN Information:

ISSN Information:

Conference Location: Chongqing, China

Funding Agency:


References

References is not available for this document.