Design and development of generic web based framework for log analysis

Large scale software systems keep on generating logs for the events carried out in the past. The information recorded in these log files is very useful in debugging operation as well as for regression testing. Now days, companies are required to review their log records on regular intervals to detect and analyze the anomalies, faults or any unwanted activity that is not normal. However, when the system is complex, these log files become huge and are almost impossible to read. Often, entries are irrelevant, so combining and correlating events in huge logs is difficult, time consuming process and requires enormous computational resources. Thus this paper aims at development of generic web based framework to analyze the log files provided by the user. The built tool will parse the log files based on user selected text phrases. The developed prototype based on the assumption that a log file generally records different events based on timestamps. And each event will have its corresponding entity and pattern pairs. An entity is the attribute name given to particular entity present in similar events. A pattern is basically a value for the attribute corresponding to each entity and it is the actual point of interest. In the proposed framework timestamp is considered as the metadata for the log file and the user is required to highlight the entity and any pattern corresponding to that entity. The entity and its corresponding value are searched in the entire log file by generating regular expression dynamically. Finally, the proposed log analysis tool in this paper visualizes the highlighted entity against time using Google charts. The proposed web enabled tool is light-weight framework supporting data streaming capabilities. It is different from the existing log analysis tools in three ways. Firstly, it supports the feature of highlighting the entity-pattern pair and provides the visualizations in terms of graphs, listings, etc for the highlighted entity-pattern pair. Secondly, the tool supports generation of Regular Expressions dynamically for the highlighted entity-pattern pair. Lastly, to print and save the visualization reports as JPeg images for latter reference.