Anomaly Detection: A Machine Learning and Deep Learning Perspective

Anomaly detection involves finding data points that don’t follow the usual trends followed by the rest of the dataset. There exist several computationally efficient anomaly detection methods in data science. This paper explores and compares various machine learning and deep learning techniques. Anomaly detection can be a tricky task due to a multitude of reasons including highly imbalanced datasets, low amounts of data on actual frauds, unpredictable nature of the problem. The transaction in question does not necessarily need to be an outlier, which makes the problem contextual as well. Having performed fraud detection on credit card fraud and healthcare provider fraud datasets, we have been able to draw certain inferences as to which algorithms work best under which scenarios. Random Forest Classifier, XGBoost Classifier, Isolation Forests, Artificial Neural Networks, and Autoencoders are few of the prominent algorithms tested. After testing the algorithms, the results show that some algorithms work better with imbalanced datasets, while others can handle noise very well. We have also tried using certain dimensionality reduction methods in order to remove noise and improve efficiency, and certain sampling methods to handle class imbalances and then compared their results on these datasets.