Using Machine Learning to Understand Assessment Practices of Capstone Projects in Engineering

Capstone projects are used as a final assessment in undergraduate engineering degrees to ensure alignment with international standards and profession-specific competencies. Accurate and consistent assessment of these projects is therefore crucial. Variability between assessors can also have direct consequences for a student’s future career path. In light of previous work that has already indicated that variability is often high, we ask whether recent advancements in artificial intelligence and machine learning (ML) could help us gain a better understanding of what is going on in current assessment practices of capstone projects within engineering. To do this, we collect a dataset of previous capstone project reports with the marks awarded from a single department. We then have a new set of examiners re-examine the reports. We also train ML models to predict a mark given a report as input. Quantitatively, our analysis reveals a large discrepancy of roughly 12% between examiners. Qualitatively, the ML models show that marks are most affected by report length and the research area of the project. This supports previous work showing high inter-assessor variability, and showcases how ML can be used to start to uncover reasons for the differences between examiners. We also shared these findings within the department, leading to initial discussions to improve assessment practices and consistency.