Pedestrian Simulation by Learning from Online Human Demonstration with VR Headset

Pedestrian simulation is essential for verifying the safety of autonomous vehicles in simulators. The goal of pedes-trian simulation is to create realistic virtual representations of the pedestrian. However, current simulations lack empirical knowledge about real pedestrian behavior. In this paper, we propose a virtual reality (VR)-based method that enables real-time interaction between an individual and a simulated traffic environment. Our human-in-the-loop approach incorporates the empirical knowledge of a VR user into the simulator. The user can view the traffic simulation by wearing a VR device, which includes a head-mounted display and two controllers and provides sparse sensor data about their position and rotation. We combine representation learning techniques to develop a full-body motion-tracking agent that takes in sparse signals from the VR device and simulates full-body motions. This agent can estimate the full-body pose and visualize a pedestrian avatar in a traffic simulation scene. To validate our method, we constructed a VR traffic simulation environment and demonstrated that our approach can produce a pedestrian avatar whose behavior closely resembles that of the real VR user.