Student-Facing Learning Analytics Dashboard for Remote Lab Practical Work

The designs of many student-facing learning analytics (SFLA) dashboards are insufficiently informed by educational research and lack rigorous evaluation in authentic learning contexts, including during remote laboratory practical work. In this article, we present and evaluate an SFLA dashboard designed using the principles of formative assessment to provide feedback to students during remote lab activities. Feedback is based upon graphical visualizations of student actions performed during lab tasks and comparison to expected procedures using TaskCompare—our custom, asymmetric graph dissimilarity measure that distinguishes students who miss expected actions from those who perform additional actions, a capability missing in existing graph distance (symmetrical dissimilarity) measures. Using a total of $N = 235$ student graphs collected during authentic learning in two different engineering courses, we describe the validation of TaskCompare and evaluate the impact of the SFLA dashboard on task completion during remote lab activities. In addition, we use components of the motivated strategies for learning questionnaire as covariates for propensity score matching to account for potential bias in self-selection of use of the dashboard. We find that those students who used the SFLA dashboard achieved significantly better task completion rate (nearly double) than those who did not, with a significant difference in TaskCompare score between the two groups (Mann–Whitney $U = 453.5$, $p < 0.01$ and Cliff's $\delta = 0.43$, large effect size). This difference remains after accounting for self-selection. We also report that students' positive rating of the usefulness of the SFLA dashboard for completing lab work is significantly above a neutral response ($S = 21.0$ and $p < 0.01$). These findings provide evidence that our SFLA dashboard is an effective means of providing formative assessment during remote laboratory activities.

I. Introduction

Student-facing learning analytics (SFLA) dashboards do not currently provide students with effective feedback for self-regulation of learning because their designs are not sufficiently underpinned by educational research and there is a lack of rigorous evaluation in authentic learning environments [1]. LA and SFLA dashboards claim to focus on the measurement, analysis, and reporting of learning [2], [3], so as to understand and optimize learning. However, LA research is rarely focused on learning [4]; designs of LA systems are seldom based on the educational theory [1]; and there is a lack of methodology for designing feedback automation [5]. For pragmatic reasons, national educational policy is driving the adoption of asynchronous, digitally mediated experiences [6], [7]. For effective learning during these experiences, a degree of challenge is required [8]. That challenge can cause disengagement if no feedback is provided to students when they need it [9], [10]. Such feedback could be provided by SFLA dashboards, yet the research focus has been on staff-facing LA [2] and only 18% of articles in a review of SFLA [11] actually report the effect of the system on student achievement, behavior, or skills.