The cloze procedure and software comprehensibility measurement

Cloze tests (i.e. fill-in-missing-parts tests) have been a long-standing measure of prose comprehension. Through human-subject experimentation, evidence was gathered to support the practical advantages of using the cloze procedure for measuring software comprehension. Cloze tests were found to be easy to construct, administer, and score and to be capable of discriminating between programs of varying comprehensibility. However, discrepancies between multiple-choice comprehension quiz results and some cloze test results for the same software suggested that certain forms of software cloze tests may not be valid. A model of software cloze tests was developed to identify a software cloze test characteristic that may produce invalid results. The test characteristic was concerned with the relative proportion of `program-dependent' and `program-independent' cloze items within a test. The developed model was shown to be consistent with software cloze test results of another researcher and led to suggestions for improving software cloze testing.