Anomaly Detection in Stock Market Transactions: A Comparison of Deep Learning Methods

This study applies unsupervised machine learning techniques and deep learning models to evaluate stock market users’ past transactions to detect anomalies in transaction data regarding volume and count. First, traditional unsupervised anomaly detection methods such as support vector machines (SVM) and local outlier factor (LOF) are utilized to identify anomalies in transaction data by evaluating only past transactions and determining whether they are abnormal. This labeling enabled deep learning methods to validate and predict these transactions. Additionally, deep learning methods such as LSTM-based autoencoders, isolation forest-based LSTM, generative adversarial networks (GAN), and variational autoencoders (VAE) are employed to analyze users’ past transactions and predict the number and magnitude of future transactions. This comprehensive approach combines deep learning models with traditional methods to provide infrastructure for retrospective evaluation of transaction data anomalies and prediction of future user actions.