Learning Personalized Agent for Real-Time Face-to-Face Interaction in VR

Interactive agents in virtual reality (VR) are anticipated to make decisions and provide feedback based on the user's inputs. Despite recent advancements in large language models (LLMs), employing LLMs in real-time face-to-face interactions decision-making, and delivering personalized feedback in VR remains challenging. To address this, our proposed system involves generating and labeling symbolic data, pre-training a real-time network, collecting personalized data, and fine-tuning the network. Utilizing inputs such as interaction distances, head orientations, and hand poses, the agents can provide personalized feedback. User experiments show significant advantages in both pragmatic and hedonic aspects over LLM-based agents, suggesting potential applications across diverse interactive domains.