Code Writing vs Code Completion Puzzles: Analyzing Questions in an E-exam

This Research Full Paper investigates the relationship between code writing tasks and other tasks like program comprehension and completion. In the context of exams, it is interesting to know whether auto-gradable comprehension and completion tasks can to some extent replace harder-to-grade code writing tasks and still test the same learning outcomes. This paper reports on the post mortem analysis of the 2019 e-exam in a university level programming course. Questions were analyzed for difficulty level, correlation, and discrimination. Program puzzles were found to correlate well with code writing, which is in line with previous findings by others. On average, code writing questions had higher difficulty, discrimination, and correlation with the total score than the other question formats - but not uniformly so. A more fine granular analysis of performance on individual interaction items sheds light on reasons why some questions worked better than others, with implications for future question design.