Unsupervised Anomaly Detection on Distributed Log Tracing through Deep Learning

Large modern software systems have a distributed microservice architecture. Such systems are built from loosely connected and independently deployable components - microservices. Microservices interact with each other with the help of various protocols, forming a sequence of internal calls in order to process a request that comes to the system. This interaction is recorded in the form of logs, which are produced by services at various points of execution and contain useful debugging information. Distributed tracing links individual logs into traces. Thanks to this, it is possible to analyze all logs related to one specific request. Traces are extremely convenient for analyzing incidents that occur during system operation. This work is dedicated to the automatic analysis of log traces to identify abnormal behavior in the operation of distributed systems. The solution is based on the application of deep machine learning methods for analyzing log sequences. The logs are cleaned and vectorized, and then used to train a model based on the long short-term memory autoencoder. The solution is tested on the TraceBench open dataset. As a result, high values of precision and recall metrics were obtained.