A Parsing Technique for Enhancing Compiler Syntax Error Messages for Student Programmers

Contribution: This full research paper presents an innovative parsing technique that aims to improve syntax error messages for undergraduate students. The quality of syntax error messages generated by the new parsing technique was evaluated and compared with the messages produced by mainstream compilers. Background: Unfortunately, compiler error messages are often unhelpful. The study explains some intrinsic challenges faced in generating good syntax error messages and presents a global, local, and expression-level (GLE) parsing technique to overcome some of these challenges. GLE is a 3-phase parsing that prioritizes the parsing of the large code components over diving into all the details. The first phase parses the functional structures and ignores errors in the syntax of the smaller constructions. The second phase parses the control structures and ignores errors in the expressions and other statements. The third phase parses the expressions and statements excluded from phase two. Research Question: Can GLE parsing techniques help generate better syntax error messages? Methodology: The study evaluated the quality of syntax error messages generated by the proposed GLE parsing technique. The evaluation was done in a controlled experiment and within-group design where participants found and fixed errors in erroneous programs using accompanying error messages from different compilers. The independent variable is the compiler type. The dependent variable is the quality of syntax error messages. The quality of syntax error messages is measured by three factors: the success rate of finding errors in erroneous programs, the success rate of fixing syntax errors in erroneous programs, and mean-time-to-find and -fix erroneous programs. Three questions were used to evaluate the “helping in finding errors” quality of the error message: 1) what is the error in the program? 2) in which line is the error? 3) what is the cause of the error? One question was used to evaluate the quality of “helping in fixing errors”: “how to fix the error?” The time that participants used to find and fix a program was calculated. The participants were 51 undergraduate students in the Computer Science and Engineering department at the New Mexico Institute of Mining and Technology. Findings: The results show there is a significant statistical difference in finding errors and fixing erroneous programs using messages generated by the proposed GLE parsing technique and two mainstream compilers: GNU GCC and Microsoft Visual C++. No significant difference exists in the time-to-find and -fix. The result indicates that the proposed GLE parsing technique can help generate better error messages for undergraduate students.