On the Use of Automated Log Clustering to Support Effort Reduction in Continuous Engineering

Continuous engineering (CE) practices, such as continuous integration and continuous deployment, have become key to modern software development. They are characterized by short automated build and test cycles that give developers early feedback on potential issues. CE practices help to release software more frequently, and reduces risk by increasing incrementality. However, effective use of CE practices in industrial projects requires making sense of the vast amounts of data that results from the repeated build and test cycles. The goal of this paper is to investigate to what extent these data can be treated more effectively by automatically grouping logs of runs that failed for the same underlying reasons, and what effort reduction can be achieved. To this end, we replicate and extend earlier work on system log clustering to evaluate its efficacy in the CE context, and to investigate the impact of five alternative log vectorization techniques. We built a prototype tool that is used to conduct an empirical case study on continuous deployment logs provided by our industrial collaborator. Questions to be answered include: (1) Can we reduce the effort needed to discover all latent issues in a set of failing runs? (2) How to best leverage the contrast between passing and failing runs to increase accuracy? (3) What trade-offs are there between effort reduction and accuracy? We present a quantitative and qualitative analysis of the results of our study. We conclude by evaluating the trade-offs, and give recommendations for applying this approach in practice.