Evaluating four white-box test coverage methodologies

This paper presents an illustrative study aimed at evaluating the effectiveness of four white-box test coverage techniques for software programs. In the study, an experimental design was considered which was used to evaluate the chosen testing techniques. The evaluation criteria were determined both in terms of the ability to detect faults and the number of test cases required. Faults were seeded artificially into the program and several faulty-versions of programs (mutants) were generated taking help of mutation operators. Test case execution and coverage measurements were done with the help of two testing tools, Cantata and OCT. Separate regression models relating coverage and effectiveness (fault detection ability and number of test cases required) are developed. These models can be helpful for determining test effectiveness when the coverage levels are known in a problem domain.

1. INTRODUCTION

This paper discusses the results of a study aimed at evaluating four white-box test coverage techniques, viz., path, boolean, relational, and loop coverage. In the study, an experimental design was considered, the experimental design was used to evaluate the effectiveness of the chosen testing techniques, the degree of association between each of these coverage techniques and each evaluation criteria was explored, and finally, regression models that would help in predicting the percentage faults from the knowledge of coverage level of each technique were developed. The evaluation criteria were determined both in terms of the ability to detect faults and the number of test cases required. This study is inspired by the work done by [1], [2] [4], and [5], for empirically comparing some of the techniques like branch testing, data flow testing, and mutation testing.