Reasoning Skills Assessment in Information Technology National Entrance Examination Reform; a Design Science Approach

Context: Programming is a relatively difficult skill to acquire, the mean worldwide pass rate of CS1 courses being 67.7%. Programming aptitude is also difficult to predict. Solid and justified selection criteria are still disputable, which is disconcerting for both faculties and applicants. The problem becomes even more unsettling as IT industry is craving for more IT professionals and IT faculties are suffering from numerous dropouts.Goal: Programming requires a lot of reasoning, but reasoning has seldom been examined separately from mathematics in the student selections of computing disciplines in higher education. The aim of this study is to design and initially validate a novel assessment to measure reasoning skills in the entrance examination of bachelor level degree programmes in computing. Approach: Design science addresses research through creation and evaluation of new artifacts based on scientific enquiry. The design and validation of the assessment adhere to the Standards for Educational and Psychological Testing. The evaluation of the assessment included constructs and methods from both classical test theory and item response theory.Outcomes: The novel reasoning skills assessment was designed, reviewed, piloted and evaluated. The assessment measures applicants’ reasoning skills from various angles, with questions set in real world contexts. Preliminary analyses indicate acceptable validity and reliability. Reasoning skills assessment predicted performance in programming better than mathematics or verbal skills.Conclusions: The results suggest that reasoning skills are a viable predictor of performance in programming studies. More evidence for the validation of the assessment will be gathered in subsequent runs of the national entrance examination and questions refined accordingly.