Development of Automatic Source Code Evaluation Tests Using Grey-Box Methods: A Programming Education Case Study

Increasing the effectiveness of programming education has emerged as an important goal in teaching programming languages in the last decade. Automatic evaluation of the correctness of the student’s source code saves teachers time and effort and allows a more comprehensive focus on the preparation of assignments with integrated feedback. The study aims to present an approach that will enable effective testing of students’ source codes within object-oriented programming courses while minimising the demands on teachers when preparing the assignment. This approach also supports variability in testing and preventing student cheating. Based on the principles of different types of testing (black-box, white-box, grey-box), an integrated solution for source code verification was designed and verified. The basic idea is to use a reference class, which is assumed to be part of every assignment, as the correct solution. This reference class is compared to the student solution using the grey-box method. Due to their identical interface (defined by assignment), comparing instance states and method outputs is a matter of basic programming language mechanisms. A significant advantage is that a random generation of test cases can be used in such a case, while the rules for their generation can be determined using simple formulas. The proposed procedure was implemented and gradually improved over 4 years on groups of bachelor students of applied informatics with a high level of acceptance.