Scaling Student Feedback in Software Engineering Massive Open Online Courses

This article presents a bot aimed at helping massive open online course participants complete programming assignments by providing them with timely formative feedback. Students’ perceptions toward the bot were positive and statistics indicate that many relied on it to complete assignments.


FOCUS: DEVELOPING YOUR SOFTWARE ENGINEERING CAREER
As a consequence, there is a pressing need to upscale the training of the future software professionals to meet the needs of the job market.Moreover, the rapidly evolving technologies in the software engineering sector require existing professionals to recycle their knowledge and stay up-to-date with their skill set.
A possible strategy to face such high demand for training new and in-service software professionals is through the use of massive open online courses (MOOCs). 2 In contrast to traditional face-to-face education, MOOCs enable flexible, open learning styles where an unlimited number of students can participate online, transcending the physical barriers of time and space.Although such flexibility allows reaching huge numbers of learners, it also brings several new challenges.For instance, students may not be able to get tutoring or help from the instructors easily, compared to traditional courses due to the overwhelming student-instructor ratio. 3A related challenge is to provide individual feedback to all students, which is crucial for learners to understand the topic under study and improve their learning outcomes. 4In fact, prior research has proven that automated feedback on programming assignments is effective for enhancing  Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
programming skills and increasing student motivation. 5,6Moreover, the quality of feedback has been acknowledged as a factor that influences student dropout, 7,8 which is more often than not very high in MOOCs.MOOCs have largely relied on peer assessment to overcome the challenge of providing student feedback.However, the quality of the feedback provided by students is often low, and it is mainly summative rather than formative assessment 9 ; i.e., students often do not get feedback designed for improving their learning while they are working on their assignments but rather feedback after submission providing a grade, but very little help to improve their learning.
The use of bots could be a solution for providing individual evaluation and formative feedback to students throughout the whole learning process.Bots are automated tools that handle repetitive and often tedious tasks.They are increasingly used in industrial and open source software projects for, e.g., ensuring license agreement signing, reporting continuous integration failures, reviewing code and pull requests, and assigning reviewers. 10In the case of MOOCs for learning software development, bots could help students assess their code's compliance with the assignments' requirements and obtain useful and timely formative feedback.In this article, we present a bot aimed at providing formative automated assessment in software engineering MOOCs.We further report on the evaluation of this bot in four MOOCs belonging to a software development specialization through a questionnaire to collect students' opinions and usage statistics.We analyze how MOOCauto helped different profiles of students improve their assignments using growth mixture models.To make use of MOOCauto for their programming assignments, instructors must create a directory with all the files needed for the assignment, including a test suite that will check students' code compliance with the assignment's requirements.Within the test suite, they must provide the feedback that they want students to obtain for each feature tested.For example, one assignment might be developing a quiz game web application in which the user scores are stored in a database.The instructor would write a test for each of the quiz features (answering a question, storing the score, etc.).When testing, for instance, the feature of saving the score, if the test does not pass, the feedback could be as follows: "When a user finishes playing, the score is not saved on the database."That way, the students will know what part of their code is not working, without being too specific, that would prevent them from debugging themselves.All the assignments, including the test suites, are made available by the teachers on an online version control system so students can easily download the latest version and instructors can update them in case any mistake needs to be fixed.

MOOCauto
After downloading the assignment from the online version control system and following the instructions to complete it, students may use MOOCauto to obtain feedback on their progress.To do this, they just need to run the command "moocauto" on their computer's terminal in the assignment directory.The bot then runs the test suite defined by the teacher, which checks their code's compliance with the assignment's requirements.On the basis of the test results, the bot provides students with textual feedback on their progress in solving the assignment, pointing to those parts that are wrong or missing, as per the feedback predefined by the teacher.Students can make use of MOOCauto any time during the development process without limitations: i.e., students can have a formative assessment that helps them continue improving their work.The traces of students' use of MOOCauto are sent to a web server that stores the level of completion achieved in each execution of the test suites (percentage of the requirements met).Instructors can follow the progress of students' completion of the assignments through an online dashboard.

Context: A MOOC Specialization in Software Development
MOOCauto has been tested in a MOOC specialization on full-stack software development consisting of four MOOCs: "Front-end development with HTML , CSS and JS," "Managing software projects with Git and GitHub," "Back-end development with Node.js,Express, and databases," and "Development of a full-stack project with JavaScript."The four MOOCs were hosted in the FOCUS: DEVELOPING YOUR SOFTWARE ENGINEERING CAREER MiriadaX platform, the lead MOOC provider in Spanish.The specialization was also offered by Universidad Politécnica de Madrid in Spain as a university-specific degree, accounting for nine European credit transfer system credits (226 h).The courses had a duration of 9-11 weeks each and were taught in Spanish.The content was structured in modules and included video lessons, slides, multiple-choice quizzes, and practical assignments in which students could make use of the MOOCauto bot to get formative feedback.The assignments provided students with an unfinished project that they needed to complete by adding additional features to the provided codebase. 11The feedback provided through MOOCauto (configured by the instructors beforehand) allowed the students to continuously evaluate their work and improve it until they met all of the assignment requirements.Once they were satisfied with their work, they could submit it for peer assessment.In this regard, it is worth indicating that students had the option of using MOOCauto to help them assess the assignments of their peers.Thus, apart from the formative assessment provided by MOOCauto, students also obtained personalized summative feedback from their peers.

Students' Perceptions
After each of the courses ended, an anonymous questionnaire was conducted to gather students' opinions of MOOCauto.The questionnaire was validated by three faculty members and had a high internal consistency (Cronbach's a = 0.97).
The sample consisted of 213 students (82.2% men, 17.8% women) who participated in at least one of the four MOOCs.The mean age of the respondents was 43.5 years [standard deviation (SD) = 11.6].Some 85% of the students had participated in a MOOC before.
The results of the questionnaire administered to collect students' perceptions of the MOOCauto bot are shown in Figure 1.According to an analysis of variance (ANOVA) test, no significant differences exist among the results obtained for each MOOC.Overall, students had a very positive opinion of MOOCauto, and thought it was easy to install and to use.They agreed that the feedback provided by the bot was useful, easy to understand, and that it helped them improve their assignments.They stated that MOOCauto increased their motivation to work on the assignments to some extent and that they preferred the feedback from the bot over that of their peers.Moreover, they stated that they would like other MOOCs to have bots similar to MOOCauto.The features of MOOCauto that the students scored the highest were the possibility to evaluate their assignments an unlimited number of times and the fact that it runs locally on their computers, without the inconvenience of having to upload their assignment elsewhere to get feedback.The respondents also valued the provision of instantaneous feedback and the availability of documentation very highly.

Usage Statistics
Anonymized trace log data were collected for each assignment evaluated using MOOCauto.The students made use of the bot throughout the complete duration of the four MOOCs (Figure 2).
Table 1 shows that the total number of users in each MOOC ranged from 255 to 158, while the number of times MOOCauto has been used ranged from 546 to 4,326.The mean number of uses of MOOCauto per user and assignment ranged from 2.1 to 6.3 although, as it commonly happens in MOOCs, many students dropped out and therefore did not continue to use MOOCauto.Students' mean time spent on the assignments ranged from 13.0 to 77.6 h, indicating that students mostly spread their work across several days.The differences in the number of uses and time spent on the assignments among the MOOCs are expected due to the different number, nature, workload, and complexity of the assignments across the courses.Moreover, the high standard deviation is an indicator of the marked differences among students participating in the same MOOC.In turn, the In contrast to traditional face-to-face education, MOOCs enable flexible, open learning styles where an unlimited number of students can participate online, transcending the physical barriers of time and space.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.mean completion level (percentage of the requirements met) reached by the students was rather uniform across MOOCs, ranging from 71.38% to 85.08%, indicating that they ended up meeting most of the assignment requirements, although the standard deviation within the courses is again rather high.
Since MOOCs are open online courses that anyone can attend, there is a considerable diversity in the profiles of learners, especially when it comes to their prior knowledge and their motivation and intentions regarding the course.For instance, some learners might be already knowledgeable about the course materials and just seeking the certification, some might be just auditing the course, and others might be interested in learning a specific topic.In the MOOCs examined in this study, such differences in students' approaches to learning are also reflected on the way participants make use of the MOOCauto tool to improve their assignments.To investigate whether MOOCauto is a useful tool for learning while accounting for the potential heterogeneity (i.e., variation within students' profiles), we used a growth mixture model.Specifically, we relied on the R package LCMM. 12e fitted a growth model to predict students' scores from the number of assessment attempts.In other words, we aimed to analyze whether students improved their compliance with the assignment requirements the more they made use of MOOCauto.The model's total explanatory power was substantial (conditional R 2 = 0.54).The model's intercept, corresponding to the mean initial score, was at 4.43 (95% confidence interval (CI) [4.25, 4.60], t(5,414) = 50.06,P < 0.001.Within this model, the effect of the attempt number was statistically  The 95% CIs and P-values were computed using a Wald t-distribution approximation.To account for students' heterogeneity, three latent classes (i.e., students' trajectories) were identified using fit indices (Bayesian information criterion, Akaike information criterion, size of smallest and largest cluster) and interpretability.Each latent class was characterized by a specific trajectory modeled by a classspecific growth mixed model (depicted in Figure 3).The first class (cluster 1) had an intercept of 1.57 (95% CI [1.28, 1.87], P < 0.0001), meaning that students started on average with a low score.The slope was 0.087 (95% CI [0.05, 0.12], P < 0.0001, which represents a small, though statistically significant increase in score with each attempt.This cluster represents students that have low initial knowledge or who rely on MOOCauto from the beginning of the assignment.These students do not seem to make use of the feedback from MOOCauto but rather make several attempts, with little improvement in their score, in a seemingly trial-and-error behavior. The second class (cluster 2) had an intercept of 3.56 (95% CI [3.25,  3.87], P < 0.0001), which represents a slightly higher initial score compared to cluster 1.The slope in this class was 0.23 (95% CI [0.19, 0.26], P < 0.0001), representing a considerable growth from one attempt to the next.This cluster represents students who use the feedback received to progressively improve their submissions and fulfill the assignment requirements.The students in this cluster (the largest in number) are the most likely to have learned from using MOOCauto feedback.
The third class (cluster 3) had an intercept of 8.17 (95% CI [7.94,  8.39], P < 0.0001), which represents a very high initial score.In this case, the slope was statistically insignificant, with a value of −0.004 (95% CI [−0.03, 0.03], P = 0.808).This cluster represents students with initial high scores, and therefore there was little room for improvement or need for feedback.Some also use it to experiment how changes in the code influence the score, as they keep using MOOCauto after they have obtained the highest scores.

Discussion
Overall, the results of this article suggest that bots can constitute a suitable solution for helping students complete their programming assignments in software engineering MOOCs, as indicated by the positive opinions of the students and the bot usage statistics.The results obtained are consistent with those of previous research in automated assessment tools for programming assignments in software engineering blended courses. 6he results of the growth mixture model point out to the existence of a large group of students who rely on MOOCauto's feedback to improve their assignments.However, two other groups exist who do not take advantage of the tool: a group that does not use the feedback received to improve and present trial-and-error behavior, and a high-achiever group that completes their assignment without the need for MOOCauto.
T he results of this article sug- gest that some of the challenges intrinsic to MOOCs, namely providing feedback to students, can be overcome through the use of bots, making this type of training suitable for software engineering.Artificial intelligence applications such as bots are increasingly gaining ground in the education field, 13 ranging from conditional automation (e.g., MOOCauto) in which teachers set the "rules," to completely autonomous technology (e.g., ChatGPT).Further research should investigate how making use of more advanced forms of artificial intelligence could enhance automated formative assessment by providing more flexibility in the evaluation of students' work and adapting the feedback to the students' characteristics.In this regard, the analysis performed constitutes a first step in this direction, as it helped us identify different profiles of usage of MOOCauto, which might benefit from different levels of intervention and support.Another interesting line of research would be to increase students' motivation to complete their assignments by adding gamification elements to MOOCauto, a technique that has proven successful in engaging MOOC participants before. 14astly, since combining different data sources can help better understand students' behavior, 15 further inspection of students' usage patterns of MOOCauto in combination with the other learning resources in the MOOC would be valuable to gain deeper insights into students' learning processes.
Sonsoles López-Pernas , University of Eastern Finland and Universidad Politécnica de Madrid Enrique Barra , Aldo Gordillo , Álvaro Alonso , and Juan Quemada , Universidad Politécnica de Madrid // This article presents a bot aimed at helping massive open online course participants complete programming assignments by providing them with timely formative feedback.Students' perceptions toward the bot were positive and statistics indicate that many relied on it to complete assignments.//Scaling Student Feedback in Software Engineering Massive Open Online Courses IT IS NO secret that software engineers are highly demanded professionals in today's job market.Almost 200,000 yearly job openings for software developers, quality assurance analysts, and testers are projected for the present decade in the United States alone.

FIGURE 3 .
FIGURE 3. Growth mixture model with three classes.

FIGURE 1 .
Results of the student questionnaire.

Table 1 . Summary statistics of MOOCauto usage
*Accounting, for each assignment, only for those students who used MOOCauto.M: mean; SD: standard deviation.FIGURE 2. Usage data of MOOCauto.Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.