Development of Performance Regression Analysis Tool using Distributed Tracing on microservice-based Application

In applications with microservice architecture, when a significant change occurs and result in decreased performance or regression, it is difficult to perform analysis to check which part of the whole microservice is the main cause of the regression due to the distributed nature of microservice. Distributed tracing can be used to analyze and determine whether there is a regression and the cause of the regression by utilizing latency data from existing operations on microservice application. This system will perform performance regression analysis using the distributed tracing tool Zipkin. Regression is then can be detected using Kolmogorov-Smirnov statistical analysis which will compare the latency data samples that occur periodically and the sample latency baseline data that represents application performance under normal circumstances. The analysis will compare the Cumulative Distribution Function (CDF) of the two samples and see if both CDF comes from a different distribution. If it is found that both CDFs originate from different distributions, it then can be suspected that there has been a performance regression because the distribution of the periodic data has deviated from the distribution of the baseline data. If a regression is detected, the system will then perform a critical path analysis to see which operations of each services are most likely contributed to the regression. The analysis will be executed by finding the difference latency of periodic and baseline operation data and it will be seen which operation’s latency difference exceeds a predetermined limit. The performance regression analysis system has been tested on a microservice application that runs on Kubernetes. The result is that out of 10 out of 11 test cases, and for every successful case, the main suspected operation that causes regression is found. Implementing the system adds an average overhead of 0.78% CPU usage and 0.67% Memory usage.