Establishing the trustworthiness of scenario assignments as assessment tools for undergraduate engineering education

How to establish the trustworthiness of qualitative tools to assess students' knowledge of engineering practices, teamwork and problem solving is a question that remains unanswered for many faculty. This paper describes a process that was conducted to ensure the dependability of scenario assignments. Scenarios are a qualitative performance assessment tool designed to assess students' knowledge of engineering practices, teamwork, and problem solving. Two scoring rubrics, a holistic and an analytic, were developed to assess students' knowledge with respect to the learning outcomes associated with the scenario tool. Careful training of the scorers, analysis of the scores, and comparison of test scores determined dependability of the tool. Students were given a scenario that describes a "day in the life" problem faced by engineers, then they are asked to explain the process they would use to solve the problem. Initial findings suggest that faculty who score the scenarios with the analytic rubric can quickly identify, students' strengths and weaknesses in these areas and adapt their course to address the areas where students need attention. At the departmental or college level, scoring the scenarios with the holistic rubric is useful to assess changes in students' learning and development over time. Careful testing of the rubric revealed that the scores can reliably differentiate among student abilities.