Testing decision systems with classification components

Many decision tools and complex decision systems require components that use learning technology to improve the quality of the decisions, based on observations (such as sensor data). In order to employ these tools and systems in high- or medium-risk applications, the design, implementation, and deployment process needs to follow principled verification, validation, and testing procedures that assure a reliable operation. This task is far from being trivial because of the very nature of learning - a technology that provides tools for making decisions under uncertainty. Only little research efforts have been dedicated so far to validating and testing learning-based systems. This paper describes a novel tool for the testing and the validation of learning systems and a set of statistical tests that are employed by this tool for the assessment of learned classification decisions. We also describe some aspects of the underlying theoretical and experimental framework for the validation and testing of systems that learn.