Application performance assurance using end-to-end user level monitoring

A new measure of performance, which uses both application integrity and traditional network response time, is proposed. Modern networked application services rely on a stack of network protocols and a host of other services many of which cross-organizational and corporate boundaries. We point out that traditional software quality assurance techniques don't scale up for post-deployment integrity checks for such applications and services. A new methodology to do non-stop post-production monitoring of networked application services for transactional integrity and time delay measurement is proposed. Specifically we describe the Telcordia/spl trade/ Application Assurance System, which we have created for measuring real-time performance of web-based applications used in commercial settings. The system measures both post-production application integrity and time delay. The measurements are carried out by sending synthetic end-user transactions and analyzing the responses. Statistical models for analyzing the data using single monitoring site as well as multiple monitoring sites are described. Creating synthetic end-user transactions is crucial for our method. The paper presents a method for generation of 'highly efficient' end-user transactions from a graphical model of the functionality of the system. Highly efficient transactions are generated using combinatorial designs. The graphical model is incrementally created using a recorder. We give several empirical examples of efficacy of this system and uses for finding performance problems.