MIT-AVT Clustered Driving Scene Dataset: Evaluating Perception Systems in Real-World Naturalistic Driving Scenarios

Solving the driving scene perception problem for driver-assistance systems and autonomous vehicles requires accurate and robust performance in both regularly-occurring driving scenarios (termed “common cases”) and rare outlier scenarios (termed “edge cases”). We propose an automated method for clustering common cases and detecting edge cases based on the visual characteristics of the external scene using deep learning. We apply this approach to develop a large-scale real-world video driving scene dataset of edge cases and common cases. This dataset consists of 1,156,592 10-second video clips, including 450 clusters of common cases, and 5,601 edge cases. We assign human-interpretable metadata labels (e.g., weather, lighting conditions) to the clusters through manual annotation. We further propose two automated methods for large-scale evaluation of scene segmentation models on naturalistic driving datasets that can capture potential system failures without human inspection. Video illustrations of select clusters will be made available to help with future research.