Domain Adaptive Log Anomaly Prediction for Hadoop System

Hadoop provides a powerful platform that allows reliable, scalable, and distributed processing of massive data sets across a cluster of computers. Log data record events taken place in the Hadoop system that helps to understand system activities and diagnose problems. However, system upgrades and updates often change the syntax and patterns of logs, rendering the machine-learning models that were designed for the legacy system ineffective. Retraining the machine-learning models with new data sets from scratch might improve the accuracy of the machine-learning model. Nevertheless, annotating new data sets is often time consuming and labor intensive. In this article, we propose a domain adaptive log anomaly prediction framework called LogAT to effectively transfer learned knowledge from the existing labeled data set (source domain) to the new unlabeled data set (target domain) by adopting an unsupervised domain adaption method. Furthermore, a hierarchical anomaly knowledge graph has been constructed to represent the domain knowledge that facilitates the subsequent detection and diagnosis of system faults. Extensive experiments have been conducted on public and real-world data sets to validate the effectiveness of the proposed framework as well as each module. Our results show that LogAT achieves superior performance over the state-of-the-art methods for predicting log anomalies and acquiring considerable performance improvement in terms of AUC-ROC score on different Hadoop application data sets.