Towards competency based testing and feedback: Competency definition and measurement in the field of algorithms & data structures

At least since the first PISA surveys, the intentions of education are increasingly expressed in terms of competencies instead of learning goals. In consequence, learning outcomes should be measured in terms of these intended competencies as well. This holds for large-scale investigations like PISA as well as for the very small scale that is represented by the examinations that a single teaching person performs with his/her students. Yet, to assure validity, competency measurement requires proper models for competency structure and levels, which are based on empirical research. The goal of this project is to identify a set of relevant competencies in the domain of object-oriented programming (OOP), to develop a suitable competency model to represent their structure and to construct proper corresponding test instruments. This paper describes a study on the outcomes of several assignments that represents one step of this process with focus on the subdomain of data structures. The data were collected during an introductory lecture on programming in the winter term 2015/16. We analyzed the student solutions of three mini-projects that deal with the implementation of abstract data, applying a specific e-assessment system to score the students' solutions. To find suitable item sets that could be used for the definition and measurement of certain competencies, different methodologies were applied that are based on classical test theory as well as on contemporary Item Response Analysis. Up to now, several suitable item sets for the competencies were identified, which are required to implement and operate 1- and 2-dim arrays, linked lists, and binary trees.