Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress (Extended Abstract)

Most of the time series anomaly detection papers tested on a handful of popular benchmark datasets, created by Yahoo [1], Numenta [2], NASA [3] or Pei's Lab (OMNI) [4], etc. There is a strong implicit assumption that doing well on these public datasets is a sufficient condition to declare an anomaly detection algorithm is useful. In this work, we make a surprising claim. The majority of the individual exemplars in these dataset suffers from one or more of four flaws: triviality, unrealistic anomaly density, mislabeled ground truth and run-to-failure bias. Because of these four flaws, we believe that most published comparisons of anomaly detection algorithms may be unreliable, and more importantly, much of the apparent progress in recent years may be illusionary.