Critical Path Analysis through Hierarchical Distributed Virtualized Environments Using Host Kernel Tracing

The dynamic nature of applications in Virtual Machines (VMs) and the increasing demand for virtualized systems make the analysis of dynamic environments critical to achieve efficient operation of such complex distributed systems. In this article, we propose a precise host-based tracing and analysis method to retrieve execution flows, and dependency flows from virtualized environments, regardless of the level of nested virtualization. Given a host operating system level trace, the Any-Level vCPU Detection (ASD) algorithm and Guest Thread-state Analysis (GTA) algorithm detect the different states of vCPUs and threads for arbitrary nesting depths. Then, the Execution-graph Construction (HEC) algorithm extracts the waiting / wake-up dependencies chains out of the running processes across VMs, for any level of virtualization in a transparent manner. The process dependency graph, vCPU state, and VM process state are displayed in an interactive trace viewer, Trace Compass, for further inspection. Our proposed VM trace analysis algorithms have been open-sourced for further enhancements and collaborative research and development. Our new techniques were evaluated with workloads generated using several well-known server applications (e.g., Hadoop, Apache, MySQL, Linux apt-get, and IMS network). The proposed approaches are based on host hypervisor tracing, which brings a lower tracing overhead (around 1 percent), is easier to deploy, and presents fewer security issues as compared to other approaches.