The next step in verification testing of complex Systems is Automation

Automation has been proven to increase productivity, improve quality, and reduce cost in many industries. Applied heavily in manufacturing, automation has also been used extensively to validate software. This study explores the benefits and considerations of automating the execution and analysis of verification tests of complex radar systems. This testing is performed on a complete integrated system evaluating compliance against specified requirements. In the radar system laboratory, execution automation entails changing the operational environment (e.g. target types, maneuvers, etc.), simulating operator inputs into both the radar and avionics (e.g. radar settings, mode changes, etc.), and meeting objectives through a scripted sequence of actions to be performed for a particular scenario without requiring operator intervention. Analysis automation entails collection of both radar and avionics data and analysis of the collected data. The analysis provides an in depth evaluation of radar performance, such as turn-on time, number of false tracks, probability of successful electronic countermeasure detection, also without requiring operator interaction. Due to the nature of complex systems, verification testing has been challenging since the experiments are not deterministic but rather stochastic. The tight interdependencies between the software and hardware in radar systems have made repeatable comprehensive verification testing critical during radar software development, thus making automation the logical next step in the evolution of test methodologies to expedite the software/system development. To understand the benefits of automating verification testing of complex radar systems, this paper discusses the: programmatic environment in which automation is considered; verification test construct used for comprehensive testing; instrumenting radar data used to evaluate verification test criteria; characteristics of the automation system needed for successful execution and data analysis; capabilities of automation systems to evaluate a radar system's performance, stability, and functionality; automation effectiveness against each verification test type; and benefits and drawbacks to automation implementation.