METEOR: A Dense, Heterogeneous, and Unstructured Traffic Dataset with Rare Behaviors

We present a new traffic dataset, Meteor, which captures traffic patterns and multi-agent driving behaviors in unstructured scenarios. Meteor consists of more than 1000 one-minute videos, over 2 million annotated frames with bounding boxes and GPS trajectories for 16 unique agent categories, and more than 13 million bounding boxes for traffic agents. Meteor is a dataset for rare and interesting, multi-agent driving behaviors that are grouped into traffic violations, atypical interactions, and diverse scenarios. Every video in Meteor is tagged using a diverse range of factors corresponding to weather, time of the day, road conditions, and traffic density. We use Meteor to benchmark perception methods for object detection and multi-agent behavior prediction. Our key finding is that state-of-the-art models for object detection and behavior prediction, which otherwise succeed on existing datasets such as Waymo, fail on the Meteor dataset. Meteor is a step towards developing more sophisticated perception models for dense, heterogeneous, and unstructured scenarios.