Anti-Money Laundering Detection Using Traditional, Deep, and Hybrid Machine Learning: A Performance Comparison

This paper evaluates the effectiveness of traditional machine learning, deep learning, and hybrid modeling techniques for anti-money laundering (AML) detection. Due to the extreme class imbalance in money laundering datasets, we conduct two experiments using random undersampling and SMOTE-based oversampling to address the scarcity of minority-class instances. The models assessed include Random Forest, XGBoost, GRU, LSTM, BiLSTM, Transformer, and hybrid variants combining deep learning encoders with XGBoost classifiers. Evaluation is performed on a highly imbalanced test set of over $\mathbf{8 5 0, 0 0 0}$ transactions, with fewer than 1,000 labeled as laundering cases. Results show that traditional models, particularly XGBoost, perform best with undersampling, achieving a recall above 76%, ROC AUC exceeding 0.89, and F1-score over 0.95. Deep learning models benefit more from SMOTE, with the Transformer achieving the highest recall of $\mathbf{7 1. 4 5 \%}$. Hybrid models, especially LSTM + XGBoost and BiLSTM + XGBoost, deliver balanced performance under both resampling strategies, with recall above 71%, ROC AUC over 0.82, and strong F1-scores. These findings underscore the value of integrating temporal modeling with structured decision trees and highlight the importance of using evaluation metrics beyond accuracy for AML tasks in highly imbalanced settings.