LogSI: A Benchmark for System-Incremental Log Analysis

Automated log analysis plays a vital role in software operations, with deep learning methods demonstrating effectiveness for analyzing logs from individual systems. However, existing methods face limitations in efficiency, adaptability, and knowledge preservation in system-incremental log analysis. Continual learning offers a solution by expanding the model’s ability to analyze logs from the increasing number of systems. For evaluating these methods in system-incremental log analysis, we introduce LogSI, a novel benchmark with four essential abilities for system-incremental log analysis. We perform a comprehensive evaluation of various baselines on LogSI, examining their robustness against different system permutations. Additionally, we conduct an in-depth study on the factors that influence their robustness. The datasets and source code of this paper can be found in https://github.com/nonauthor/LogSIbenchmark.