Mixed Reality IoT Smart Environments with Large Language Model Agents

The convergence of emerging technologies in mixed reality (XR), artificial intelligence (AI), and the internet of things (IoT) has given rise to new possibilities in smart space design and interaction. Specifically, large language models (LLMs) now provide forms of both conversation and control within smart spaces and this work proposes the use of LLMs as agents for hybrid physical and virtual context-aware user interactions in smart spaces. This addresses the challenge of designing and testing the utility of these systems, and the main contributions include: i) a design for convergence of an XR-IoT (XRI) and LLM driven smart space; ii) an architectural framework that allows for XRI-based LLM agents to interact in the hybrid smart space, iii) an early proof-of-concept prototype for an XRI Smart Work Space, and iv) an evaluation of the XRI LLM agent, toward future smart environments.