Measuring Programming Knowledge in a Research Context

There is a continued need for basic research on how beginners learn programming across different contexts. In research aiming at measuring effects of learning, ways of measuring programming knowledge are needed. The aim of this paper is to describe a process of establishing validity of a context-specific assessment tool for novice programming. Our case is an ongoing research project that investigates the effects of and mechanisms behind learning computing programming from hands-on experiences. For that context, we have constructed and validated an assessment tool i.e., a programming knowledge test with a connected scoring rubric based on an introductory three-hour teaching session. We relied on three ways of arguing for how our assessment of knowledge is considered valid in this process: previous research, an empirical sample, and intersubjective knowledge judgments. The main result of our study is the test with the scoring rubric and our arguments for its validity. We also present some tentative findings from the research project on the role of hands-on in learning programming. Both the area of hands-on, as well as the context-specific process, seem to be fruitful topics of future research, particularly combined with a pragmatic action and communication theory.