A case study of the analysis of novice student programs

It is difficult for teachers and graders to give an in-depth evaluation of student programs to the point of checking every line of code due to the amount of time checking would take. Solutions to this difficulty many involve only checking to see if the program executes correctly (dynamic analysis), glancing over the program to see if appropriate documentation to present (static analysis), and glancing over the code for any obvious problems (static analysis). A promising solution is to automate the assessment process or parts of it, however, automated solutions are difficult to construct and may be very expensive commercially. One commercial solution, however, is Verilog Logiscope which offers a limited number of licenses free to conductors. Logiscope is a static analysis checker which collects metrics on programs and allows the coverage of code to be determined when testing during program execution. It is also difficult to determine which metrics may be of use in the automated assessment of programs. This paper addresses this problem through using Logiscope to conduct a preliminary investigation with a small set of static metrics. Another metric, the code coverage of a set of test scripts run on all of the programs is also used. The metrics data collected from programs belonging to students in the beginning programming honors course, is presented and correlated to each other to determine their dependence on each order. Consideration on how to use the metrics to help students learn to program better is given.