Anomaly Detection from Web Log Data Using Machine Learning Model

The information in the logs produced by the servers, devices, and applications can be utilized to assess the system’s health. It’s crucial to manually review logs, for instance, during upgrades, to verify whether the update and data movement went smoothly. Manual testing is insufficiently trustworthy, and manual log examination takes much time and effort. In this paper, we propose to search log files for anomalous sequences using the machine learning methods K-Means and DBSCAN. The two data representation approaches examined in this study were feature vector representation and IDF representation. The effectiveness of the deployed machine learning algorithms was examined using evaluation measures like F1 score, recall, and precision. The study found considerable differences in the algorithms’ capacities to spot anomalies, with some algorithms being better at seeing various types of abnormal arrangements than their overall prevalence. By using the study’s findings, the user might be able to spot strange arrangements after manually sifting through the log file.