Real-time Application Monitoring and Diagnosis for Service Hosting Platforms of Black Boxes

Service hosting platforms typically run a large number of third-party applications that are composed of multiple communicating components distributed on a dynamic set of hosting servers. Understanding the real-time behaviors of these applications and the intricate interactions/dependency relationships among these application components is very important to service management tasks such as load balancing, capacity planning, performance debugging and fault diagnosis. In this paper, we present the scalable real-time application monitoring and diagnosis (SRAMD) tool, for applications consisting of "black box" components: software without source code available, and usually without desired logging instrumentation. SRAMD runs at application layer and requires no modification to existing applications, middleware, or messages. For each application component collocated at its hosting server, a SRAMD monitor traces the component's packet-level traffic unobtrusively, summarizes its local resource utilization and performance (e.g. response time) online, performs interactive queries (e.g. per- request resource utilization) to locate possible bottlenecks on- demand, and discovers inter-component dependence relationships statistically. The SRAMD controller simply aggregates reports from distributed monitors to construct real-time application topologies with rich runtime information. We have developed mechanisms to decentralize the computation overhead and minimize the communication cost in the monitoring and diagnosis process, and two schemes to discover application component dependency relationships in different scenarios. The SRAMD tool offers an alternative to server logs and message-level traces for service monitoring and performance diagnosis.