Automatically grading Java programming assignments via reflection, inheritance, and regular expressions

A suite of programs that performs the automatic grading of Java programming assignments is described. This suite has been used in the Introduction to Computer Science course at Rutgers University since the Spring 2000 semester where it grades some 400-600 weekly assignments. The time to grade the submissions is a matter of minutes on a 500 MHZ PC. The development process for a typical assignment that includes the development of the required assignment-specific evaluation software is in the order of 8 hours. The programming system derives its ability to grade in the presence of errors from the use of the Java reflection classes, the Java inheritance mechanism, and Perl regular expressions. The reflection classes allow a student program's methods to be individually found and executed independently of how they are actually named by the student. The Java inheritance mechanism allows a flawed student method to be overridden with a known good method so the evaluation may proceed. The regular expressions allow the student's program output to be gleaned for information even if the output does not adhere to specified format. The regular expressions are also used to scan the student's source code for desirable or undesirable coding patterns. In addition, the systems programming aspects of Perl are used to extract the student programs from the environment where they are submitted and to construct the environment in which they are compiled, executed, and evaluated. To date about 32000 student programs have been graded by this system.