Fully Synthetic Pedestrian Anomaly Behavior Dataset Generation in Metaverse for Enhancing Autonomous Driving Object Detection

Pedestrian detection is fundamental in the realm of autonomous driving, relying on comprehensive datasets for accurate deep-learning methods. Detecting diverse pedestrian behaviors, including rare and near-accident cases, is necessary, however, there were limitations in the development of such specific datasets. Furthermore, involving humans in data collection is risky and raises privacy concerns since recording and storing data necessitates explicit consent from all participants. To tackle this challenge, we propose MetaPed, a synthetic pedestrian dataset generated from the Metaverse, where the avatars can represent a wide range of pedestrian behaviors without exposing real individuals to potential harm or privacy violations. Our Metaverse includes both programmed pedestrian behaviors and controllable avatar movements and interactions, ensuring a comprehensive and realistic representation of pedestrian actions. To validate and ensure the generalization capabilities of our synthetic dataset’s performance in real-world scenarios, we perform both cross-dataset and intra-dataset evaluations on comprehensive real-world datasets, including KITTI, Citypersons, and INRIA Person. Our experiments demonstrate that the network trained on our synthetic dataset exhibits robust generalization capabilities for unseen real-world situations and our dataset’s inclusion significantly enhanced the Average Precision (AP) and Average Recall (AR) of the pedestrian detection model. For cross-dataset validation, we generally observe improvements in both AP and AR with models trained on all datasets. The inclusion of our dataset also improves the intra-dataset validation of the INRIA Person dataset, where the AP and AR values increase from 0.958 to 0.988 and AR from 0.697 to 0.725, respectively. For Citypersons and KITTI datasets, the AP values increase from 0.630 to 0.639 and 0.745 to 0.752, respectively. The model training on our dataset achieved the highest AP and AR values across all test datasets and the second-highest AP observed when tested on Citypersons.