System Testing of Repository-Style Software: An Experience Report

System testing based on a black box approach is a common industrial practice in information systems. Despite its widespread use, however, little guidance is available for testing engineers facing the problem of selecting the best test strategy. In previous work, we proposed to adopt functional models and related testing patterns according to the architectural style of the application under test. In this paper, we present an industrial study that applies this technique to system testing of repository based applications. We define a set of functional models abstracting different concerns of software applications: hierarchy of functions, business processes and states/transitions of application objects. The models are used to derive the functional test cases through the definition of test patterns. We applied this approach in an industrial context for over 5 years. In this paper, we analyze a data set of 37 test projects including about 22000 test cases and 1500 failures. We relate failures to the originating defect types. The study confirms that a system test strategy that uses multiple functional models according to the architectural style of the software application generates a better cost/benefit ratio than the use of just one model. The explanation is that - despite a small overlap - each model detects specific types of software defects. The results of the study can guide testing engineers in selecting the best system test strategy and significantly improve the efficiency of their work.