Post Implementation Evaluation of Coverage in Software Testing Using Monitoring Tools

Software deployed into production faces a host of issues post deployment. These come due to various reasons ranging from inadequate testing to fitting into a dynamic environment that is in constant flux due to myriad changes that keep happening every day. This paper discusses why it is important to compare production and QA environments for mismatches and identify gaps in coverage. This discovery also forces options to ensure QA tests the application to almost production like load and inter-dependencies, if not the same exact set up and configuration. An important outcome of testing in similar environmental dynamics as production is the evolving of a Release Test Environment, that is separated and isolated from the regular QA testing environment. Production issues point to a same error message for multiple root causes, and unless the root causes are identified, one cannot develop a code that throws the right error messages. This technique of identifying root causes to a specific error message to be embedded in the application code needs to be developed as well and is based on data analytics. This paper is one of a multi-part series and discusses the first of the five states of data analytics which is descriptive. Some monitoring and automation tools will be discussed in this paper to show how those tools can work in sync to help in identification of missed test coverage, and how the coverage can be expanded using automated functional and performance load tests. The end goal is to arrive at a cognitive state where the system is trained to prevent defects from occurring, and the paper discusses the next research topics in that direction to go from descriptive and diagnostic to predictive, prescriptive, and cognitive states. The key to arriving at the end goal is to ensure test coverage in the Release test environment meets the production demands, for the Release test environment is where we want to implement the neural network to train and validate the prediction algorithm. Once successful, and we are confident that the prediction system works, it can then be deployed into the production environment to do real-time prediction. When we begin the discussion on diagnostic and predictive systems, development of instrumented tests and creation of an Artificial Neural Network using “bots” will be discussed, but only as an introduction to the next research paper.