Appropriate Number of Raters for IRT Based Peer Assessment Evaluation of Programming Skills

In order to examine the reliability of peer assessment settings for the evaluation of programming skills using peer assessment and the item response theory (IRT) technique in a small classes, the optimal conditions such as the numbers of peers and the number of tasks are investigated using parameters extracted from the surveyed data. The survey data consisted of 31 students whose partial participation consisted of joining three peer assessments out of the 5 sessions during which these took place. Peer rating conditions such as the number of peer raters or tasks are examined using mean expected standard errors. Also, the relationship between instructor's ratings and estimated ability is examined using the IRT model to look at variations in the number of peer raters and tasks. The results provide evidence that a set of guidelines could better organise peer assessment of the evaluation of programmina skills in actual course settings,

I. Introduction

The assessment of programming skills is not simple, as programming techniques and concepts of code writing need to be evaluated correctly. As a solution to this, peer assessment has been introduced into programming practice classes [1]–[3], and reports of its effectiveness regarding the stimulation of proactive learning have been positive [4], [5]. Therefore, peer assessment can be applied to a large scale classes [6].