Optimizing Anomaly Detection in Large-scale Logs

The ALICE detector at the CERN LHC is a large and complex system that has employed system logging to keep track of progress and detect any abnormal activity that may occur. In this study, we propose a complete log-anomaly detection framework to automatically detect anomalies, with an emphasis on examining its scalability when applied to large datasets as those typical of large high-energy physics experiments. Furthermore, we investigate different factors that may have an impact on model performance through extensive tests on real-world datasets, HDFS, and CERN Infologger. The insights gained from this study will enhance the safety, efficiency, and reliability of infrastructure operations at the ALICE facility.