AECID-PG: A Tree-Based Log Parser Generator To Enable Log Analysis

Understanding a computer system's or network's behavior is essential for various tasks such as fault diagnosis, intrusion detection or performance analysis. A key source of information describing a system's current state is log data. However, accessing this information for further analysis is often complicated. Usually, log data is available in form of unstructured text lines and there exists no common standard for the appearance of logs. Hence, log parsers are required to pre-process log lines and structure their information for further analysis. State of the art log parsers still apply pre-defined lists of regular expressions, which are linearly processed and thus render online log analysis infeasible. Furthermore, defining log parsers manually is a cumbersome and time consuming task. Therefore, in this paper we propose AECID-PG, a novel log parser generator. AECID-PG implements a density-based approach to automatically generate a tree-like parser, which reduces the complexity of log parsing from O(n) to O(log(n)). We use real log data to evaluate AECID-PG and compare its parsing capabilities to other parser generator approaches by calculating the F-score. We prove AECID-PG's broad applicability and finally demonstrate its functionality in a real world setting.