Self-Supervised Representation Learning From Temporal Ordering of Automated Driving Sequences

Self-supervised feature learning enables perception systems to benefit from the vast raw data recorded by vehicle fleets worldwide. While video-level self-supervised approaches have shown strong generalizability on classification tasks, the potential to learn dense representations from sequential data has been relatively unexplored. In this work, we propose
TempO
, a temporal ordering pretext task for pre-training region-level feature representations for perception tasks. We embed each frame by an unordered set of proposal feature vectors, a representation that is natural for object detection or tracking systems, and formulate the sequential ordering by predicting frame transition probabilities in a transformer-based multi-frame architecture whose complexity scales less than quadratic w.r.t. sequence length. Extensive evaluation in automated driving domains on the BDD100 K and MOT17 datasets shows that our
TempO
pre-training approach outperforms existing self-supervised single-frame methods as well as supervised transfer learning initialization strategies and achieves an improvement of
$+0.7$
in mAP for object detection and
$+2.0\%$
in the HOTA score for multi-object tracking.