Evaluating an Educational Escape Room Conducted Remotely for Teaching Software Engineering

With the rise of distance learning, new challenges have emerged for educators. Among these challenges, developing effective and motivating group activities for students in the remote classroom is one of the top priorities to be addressed. According to existing literature, educational escape rooms have proven to be engaging and effective learning activities when conducted face-to-face. However, no prior research has analyzed the instructional effectiveness of these activities when they are conducted remotely. Furthermore, none of the educational escape rooms reported in the literature has been designed for teaching software modeling. This article analyzes an educational escape room conducted remotely in a software engineering fundamentals course for teaching software modeling. A total of three evaluation instruments were used: a pre-test and a post-test to measure students' learning gains, a questionnaire to collect students’ perceptions, and a web platform for automatically gathering data on students’ interactions. The contribution of this article is two-fold. On the one hand, it provides, for the first time, evidence that remote educational escape rooms can be effective learning activities. On the other hand, it provides, also for the first time, proof that educational escape rooms are effective and engaging activities for teaching software modeling. INDEX TERMS Computer science education, distance learning, educational escape rooms, educational technology, software engineering education.


I. INTRODUCTION
Escape rooms are live-action team-based games where players discover clues, solve puzzles, and accomplish tasks in a closed place in order to accomplish a specific goal (usually escaping from a room) in a limited amount of time [1]. Nowadays, these games have become one of the leading leisure activities worldwide [2]. Proof of this fact is that, as of October 2020, the World of Escapes directory [3], one of the biggest escape room public databases, lists more than 10,000 different escape rooms spread over dozens of countries throughout the world.
In recent years, escape rooms have drawn the attention of educators and organizations in the cultural sector, which has led to these activities being used for educational purposes, not only in museums [4]- [7] and libraries [8], [9], but also in formal learning scenarios as part of the curriculum. In addition to using ludic escape rooms in educational settings, teachers have started to create actual educational escape rooms: escape rooms that require students to master field-specific knowledge and skills in order to solve the puzzles and succeed in the activity. Educational escape rooms combine some of the key concepts of game design with sound educational approaches such as active learning and collaborative learning in order to foster students' soft skills such as problem-solving, teamwork and leadership in a motivating way. Active learning is a form of learning that actively or experientially engages students in the learning process, requiring them to perform meaningful learning activities and think about what they are doing [10]. In turn, collaborative learning can refer to any instructional method in which students work together in groups toward a common goal [10]. Therefore, educational escape rooms can be considered a rather favorable atmosphere for active and collaborative learning, as they usually present a common objective on which students of the same team need to actively work together, requiring them to make adequate use of their time, resources, knowledge and skills in order to succeed.
Existing literature provides strong evidence that the use of educational escape rooms can have a very positive impact on student engagement [11]- [29] and learning [27]- [31]. The capacity of educational escape rooms to boost students' motivation and enhance their learning has made them an increasingly popular teaching activity in a wide range of areas. Prior research has examined the use of educational escape rooms in several fields, including nursing [11]- [13], medicine [14]- [17], [32], physiotherapy [18], pharmacy [23]- [28], [33], science [30], physics [34], chemistry [20], [21], [35]- [38], biology [31], [39], cryptography [19], programming [22], [29], and computer networks [40]. Nevertheless, it should be pointed out that there is a dearth of research addressing the use of educational escape rooms for teaching software engineering fundamentals and, specifically, for teaching software modeling, a topic of crucial importance that every software engineering fundamentals course must cover [41], [42]. Software modeling is the first step for solving software engineering problems and allows to produce a specification of requirements and a design representation for the software to be built [43]. Thus, it is essential for software engineering students to have a solid foundation on this topic in order to be capable of successfully accomplishing software projects. In general, software engineering has been a neglected area by most of the literature about educational escape rooms. No educational escape room experience has been reported so far addressing topics related to software engineering, with the exception of two experiences in which the activity was designed to enhance students' web programming skills [22], [29]. In this regard, it should be noted that these activities were mainly focused on practical programming skills and did not address theoretical topics typically covered by software engineering fundamentals courses such as software requirements specification and software design.
Most educational escape rooms reported in the literature were conducted as face-to-face activities in a classroom, a laboratory, an auditorium, or in another closed place. Conversely, only three works [35]- [37], to the knowledge of the authors, reported on the use of educational escape rooms conducted remotely. In these activities, termed remote educational escape rooms, the place where the escape room takes place is virtual, the students are separated in space, and teammates participate simultaneously using their own devices. All the remote educational escape rooms reported in the literature were employed to teach chemistry-related topics. Furthermore, neither of the existing works that reported on the use of a remote educational escape room analyzed the activity in terms of instructional effectiveness. Therefore, although in-class educational escape rooms have proven to be able to bring multiple benefits when it comes to student engagement and learning in a wide range of knowledge areas, there is no evidence that these benefits are preserved when these activities are conducted remotely. Therefore, further research is needed to fill this research gap. This article reports on the experience of conducting an educational escape room remotely in a software engineering fundamentals course for teaching software modeling. The contribution of this article is two-fold. On the one hand, it provides, for the first time, evidence that remote educational escape rooms can be effective learning activities. On the other hand, it provides, also for the first time, proof that educational escape rooms are effective and engaging activities for teaching software modeling in software engineering fundamentals courses.
The article is structured as follows. Next section reviews existing literature on educational escape rooms, with a focus on remote experiences, as well as previous works on gamification activities and serious games for learning software engineering fundamentals. Section III includes a comprehensive explanation of the remote educational escape room conducted. Section IV explains how the educational escape room experience was evaluated. Section V shows and discusses the results obtained from this evaluation. Lastly, Section VI finishes with the conclusions of the article and an outlook on future work.

II. RELATED WORK
As evidenced by the existing literature [11]- [34], [38]- [40], educational escape rooms are usually conducted as face-to-face activities in which students work in teams to find hidden clues, solve puzzles and ultimately achieve the final goal of the escape room before the time runs out. In these activities, solving the puzzles require the participating students to apply field-specific knowledge and skills related to the topic covered by the educational escape room, as well as performing actions typical of ludic escape rooms such as substituting symbols, opening locks, assembling physical objects or solving riddles [1]. Overall, the recent scientific literature strongly indicates that face-to-face educational escape rooms are beneficial for students' engagement [11]- [29] and learning [27]- [31].
A few educators have transcended the physical barrier and have developed purely virtual educational escape rooms [30], [31], [35]- [37]. However, two of these educational escape rooms [30], [31] were conducted exclusively face-to-face. In the case of [37], the activity was hosted in person but students were also given the possibility of playing remotely and individually as a homework assignment. Therefore, there is a total of three works [35]- [37] that have reported on the use of educational escape rooms conducted remotely.
The authors of [35] conducted a face-to-face escape room for learning chemistry in higher education, but due to the rise of the COVID-19 pandemic, they were forced to also develop a fully-digital and remote version. Students could complete the digital escape room individually or as a group through video conferencing platforms. The results of the survey administered to the students after participating in either the face-to-face or remote educational escape rooms showed that both educational escape rooms obtained very positive results in terms of motivation. Similarly, [36] reported on the creation of an online virtual escape room for an undergraduate chemistry class. The Zoom video conferencing service was used to make the activity a collaborative learning experience by separating students' teams in different rooms. Although student engagement and learning effectiveness were not measured in said study, the participating students were asked for comments about the escape room in the video conference call after completing the activity and stated that they enjoyed it and that it was a good experience in cooperative learning. Lastly, [37] reported another remote educational escape room for learning chemistry. In this experience, most participants stated that the activity was engaging and beneficial for their learning. Furthermore, participants seemed to like the use of a digital medium to conduct the educational escape room.
It should be highlighted that none of the studies reporting on remote educational escape room experiences empirically measured learning effectiveness, and hence the effectiveness of remotely conducted educational escape rooms remains as an open issue. Another gap in the scientific literature is that, although escape rooms have been used in a wide variety of fields, as shown before, no study has reported on an educational escape room for teaching software engineering fundamentals. This article fills these two gaps by analyzing the learning effectiveness and the impact on students' perceptions of a remote educational escape room for teaching software modeling.
Although no educational escape room has been conducted addressing software engineering fundamentals, there are works that have evaluated the learning effectiveness of gamified activities and serious games for teaching topics related to software engineering fundamentals. For instance, in [44], a gamification approach was undertaken in a software modeling course, incorporating elements such as points and other types of rewards (coins, items, and badges), a leaderboard, content locking, and trading. The results of the study showed that students' motivation and grades increased as a result of applying gamification to their learning process. In another study [45], a gamified collaborative tool aimed at improving participation in requirement elicitation processes was presented. Students agreed that using the tool as a way to elicit requirements was fun, interesting, and potentially more motivating than traditional approaches. Kurkovsky, Ludi and Clark [46] used LEGO Serious Play to teach software requirements elicitation in an undergraduate computer science course. The evaluation of the experience indicated a number of tangible benefits helping students achieve their learning goals. In [47], a serious computer game to teach software process modeling to undergraduate students was reported. The study showed that students who played the game increased their learning to a greater extent than students who were taught using a project-based instructional method instead. Lastly, [48] developed a serious computer game aimed at teaching stakeholder requirements definition and requirements analysis. Students who played the game considered that the game sessions were stimulating and that they helped to strengthen their understanding of the theoretical concepts learned in the classroom.
Overall, the studies presented suggest that gamification and game-based learning can be effective approaches for teaching software engineering fundamentals. Taking this into account and considering that educational escape rooms have proven to be an effective learning activity in other disciplines, it seems reasonable to presume that educational escape rooms may constitute a suitable way to teach software engineering fundamentals as well. However, this specific application of educational escape rooms poses additional challenges for instructors since this is a very theoretical subject, which can be complex and hard to integrate into these immersive and live-action experiences.

A. CONTEXT
The educational escape room analyzed in this work was conducted remotely in a software engineering fundamentals course, which is a second-year core course for all the Bachelor's Degrees offered at the Faculty of Computer Systems Engineering of UPM (Universidad Politécnica de Madrid), including the Software Engineering Bachelor's Degree. This course on software engineering fundamentals accounts for 9 ECTS (European Credit Transfer System) credits, equivalent to 225-270 hours of student work. In addition to an introduction to software engineering, the course covers software development processes and methodologies, software modeling, software design and architecture, and software testing. The aim of the conducted educational escape room was to reinforce the most important concepts related to software modeling covered throughout the course in an effective and motivating way. This educational escape room was offered to all the students enrolled in the course as a two-hour activity, whose attendance was rewarded with 0.2 additional points on the course final grade (which could be between 0 and 10). All points were awarded just for attending the activity, so students' performance during the escape room (e.g., number of puzzles solved) was not taken into account for this matter. The educational escape room was conducted approximately one week prior to a midterm exam covering the same topics as the escape room.

B. DESIGN
This section summarizes the main characteristics of the educational escape room designed for this study.

1) REMOTE
The educational escape room was conducted remotely and online through a web platform called Escapp [49], which has been specifically designed for conducting both face-to-face and remote educational escape rooms. Students were allowed to enroll in the activity in a scheduled shift, but there was also a self-paced shift which allowed them to participate in the remote educational escape room at the time of their convenience. Naturally, the students of each team participated in the activity simultaneously. All students participated in the educational escape room from their homes using their personal computers and headsets. The only software that students needed to participate was an HTML5 compliant web browser such as Firefox, Google Chrome or Safari. A videoconference room was available for students during the activity in order to allow them to notify technical problems and ask instructors for help. In this regard, it should be clarified that this videoconference room was only available for the scheduled shifts and not for the self-paced shift. The communication among team members during the activity was conducted through applications chosen by each team (e.g., Skype, Google Hangouts, Zoom, Microsoft Teams, Discord, etc.). These applications not only enabled students to communicate in real-time with their teammates, but also to share their screen at certain moments facilitating, this way, team collaboration.
The Escapp web platform was used throughout the whole process of conducting the remote educational escape room. It was used for student enrollment, team formation, management of content and multimedia resources during the activity execution, puzzle solution verification, team progress monitoring, hint management, use of gamification elements (e.g., leaderboard), and student grading and attendance.
In scheduled shifts, teachers waited for all the teams enrolled to be present and manually initiated the escape room using Escapp, allowing this way students of such teams to access the activity. On the other hand, in self-paced shifts it was the participating students themselves who initiated the escape room from the Escapp web portal. Whenever the escape room started for a team, all team members accessed an interface provided by Escapp termed "team interface", the countdown began, and an introductory video was automatically played. The team interface also provided participating students with an initial clue: a computer forensics report. In addition to providing the initial content of the activity, the team interface allowed students to request hints to solve the puzzles, to consult all the hints previously obtained as well as important clues gathered throughout the session, to access the videoconference room available for contacting the teachers, to check which puzzles they had solved, and to see the countdown (i.e., the time they have left to complete the activity) and the leaderboard in real-time. The team interface was available for students during the whole educational escape room, allowing them to perform the aforementioned actions at any time.
In order to solve the puzzles of the educational escape room, students had to access and interact with different web applications. These applications communicated with the Escapp platform through its API to verify whether students solved the puzzles. In this regard, it should be clarified that sometimes students explicitly introduced the solution for a puzzle, for instance, by typing a four-digit access code, whereas in other cases, puzzles required students to interact with some object in a specific way, for example, by selecting a specific set of elements in a periodic table in the correct order. In the remote educational escape room described in this work, the escape room state was synchronized among all the web applications involved and with the team interface provided by Escapp. Thereby, all students of the same team could be at the same point at all times, easing real-time collaboration. Furthermore, real-time notifications were shown to the students via these web applications and the team interface during the activity, including warnings of the time remaining, messages on leapfrogging in the leaderboard, and notifications each time a new hint was obtained or a new puzzle was solved by any teammate.

2) VIRTUAL
The starting point of the remote educational escape room (i.e., the place from which students begin the activity) was the team interface provided by the Escapp web platform. Furthermore, all the content, assets, and applications that students needed to consume or interact with during the activity were digital. In this regard, it should be indicated that students only needed a web browser for solving the puzzles and accessing and interacting with all the resources involved in the experience. Taking all this into account, it becomes clear that the educational escape room was completely virtual.
A great advantage of the educational escape room being remote and virtual was the possibility of performing the activity online allowing the participation of a huge number of students at the same time at a ridiculous cost. If the same educational escape room had been carried out face-to-face using physical objects, the number of participants per shift would have been limited by the site capacity, and it would have been necessary to purchase physical materials, whose cost could have been significantly high, especially if hundreds of students were to participate. For instance, a digital jigsaw puzzle can be provided to an unlimited number of students practically at no cost, but this is not an option when providing a physical jigsaw.

3) TEAM-BASED
During the activity, students worked collaboratively in teams in order to examine the virtual environment, discover and analyze files (such as text documents, software diagrams, videos, audio recordings, …), try to solve the different puzzles, and obtain hints when they got stuck. Thereby, the remote educational escape room also aimed at promoting soft skills such as teamwork and leadership. More than a week prior to the start of the activity, students enrolled, formed their own teams, and picked a shift. Students were free to select their teammates, the only restriction being that teams could have a maximum of 6 participants. Most students participated in teams of 4-6 members, although some of them participated in small teams of 3 members, in pairs, or even alone.

4) PUZZLES
The puzzles of the educational escape room had a linear structure in such a way that solving each puzzle unlocked the next one. This structure implies that participants were required to solve the puzzles in a specific sequence and could not face several puzzles at the same time. For instance, students could not try to solve a puzzle requiring them to interact with a periodic table until they found that periodic table by solving a previous puzzle. Notwithstanding, the clues needed to solve the different puzzles were provided to students throughout the whole activity. Moreover, useless clues that merely decorate the experience or acted as distractors were also used.
The puzzles of the educational escape room combined learning mechanics with game mechanics often employed by ludic escape rooms. Thereby, solving the puzzles required, on the one hand, to apply knowledge and skills specific to software modeling, and, on the other hand, to perform actions typical of ludic escape rooms such as finding hidden symbols, identifying patterns, listening to audio recordings or completing a jigsaw. All the puzzles were completely digital, so, actions involving physical objects (e.g., feel or smell an object) were not considered. A detailed description of the puzzles used in the educational escape room are provided in section III-D.
Regarding the difficulty of the escape room puzzles, it should be pointed out that these puzzles were designed with the intention of maintaining the students in a mental state in which they were neither overchallenged nor underchallenged during the whole activity, as suggested by Csikszentmihalyi's flow theory [50]. Moreover, the puzzles were designed to be of slightly increasing difficulty.

5) HINT STRATEGY
During an educational escape room, it is common for a team of students to get stuck while trying to solve a puzzle. This can occur because they lack the necessary field-specific knowledge or skills to solve the puzzle, or due to other reasons such as difficulties in understanding the game mechanics or the need for an object or clue that they have not yet found or that they have overlooked. If students remain stuck at one puzzle for too long, there is a risk of them getting bored, frustrated or even angry. This may be also counterproductive from a pedagogical point of view because it will prevent students from facing all the puzzles. Thus, taking all these into account, it becomes clear that providing students with timely help during an educational escape room is essential for its success. Of special importance is that this need should not be ignored in self-paced shifts in which an educational escape room is conducted without teacher supervision.
In the remote educational escape room analyzed in this study, a quiz-based hint approach was used in order to help participating students when they got stuck while solving a puzzle. According to this hint approach, students were allowed to request hints at any time through the Escapp web application. Each time a team requested a hint by means of this application, a quiz with five questions about software modeling (i.e., the topic addressed by the activity) randomly chosen from a pool was presented. In order to obtain the hint, the team had to answer at least four out of the five questions correctly. Teams were allowed to request as many hints as they wanted. However, a minimum time interval of five minutes between hints was established. Thereby, whenever a team received a hint, they had to wait for five minutes before they could ask for more.
Each time a student requested a hint and passed the five-question quiz, his/her team received a hint helpful for solving the puzzle they were facing. This was possible because the escape room puzzles were arranged in a sequence and the Escapp platform keeps track of teams' progress during the activity, so it knows, for each team, which puzzles they have solved, which puzzle they are currently facing and how many puzzles they have left to complete the educational escape room. Furthermore, students had the possibility of obtaining more than one hint per puzzle. A different number of hints were available for each puzzle, but all puzzles had, at least, three hints available. The hints for each puzzle were arranged in a specific order, in such a way that each hint was handed out to a team once that team had obtained all the previous hints. The first hint for each puzzle just indicated which clues students needed to gather before attempting to solve the puzzle and provided some vague help. The second hint of each puzzle provided more useful information to solve the puzzle. The last hint of each puzzle, which was usually the third one, gave away the solution of the puzzle. In the case of puzzles with more than three hints available, each of the hints between the second and the last one provided each time information than was a bit more useful than in the previous hint. Thereby, the hint approach employed prevented students from becoming frustrated and abandoning the escape room due to getting stuck, even in self-paced shifts in which no instructor was available to help the students.
Moreover, for some puzzles, not only several hints were available, but also several hint categories, each of which could have one or several hints also arranged in a sequence. In these cases, students were allowed to choose for which category they wanted the hint. The categories were used for helping students in specific parts or tasks of a puzzle. For instance, in a puzzle that required students to find different symbols, a category for each symbol was available, allowing them to request help regarding a specific symbol. In addition to specific categories, all puzzles had a general category that allowed students to ask for help toward the completion of the final goal of the escape room puzzle. VOLUME XX, 2017 6

6) DURATION
The remote educational escape room was conceived to be completed in two hours. The decision to set the duration at this time is mainly due to two reasons. Firstly, we strongly believe, based on previous escape room experiences [22], [29], that two hours is a convenient duration because it is enough time for students to learn by struggling with meaningful challenges that require the application of knowledge of software modeling, and it is not enough time for students to get tired, bored, or to lose interest. Secondly, two hours is also the typical duration of a theoretical lecture or a practical computer lab session. Therefore, this duration allows the course teachers to easily schedule the educational escape room, since it can be conducted in the timeframe reserved for any session of the course.
In the escape room experience reported in this work, each team was given two hours for solving all the puzzles and ultimately achieve the final aim. Nonetheless, once the time ran out, students were allowed to continue solving the puzzles until completing the activity. Thus, the teams that did not manage to solve all the puzzles in two hours saw how the countdown reached zero and received a notification warning that the time had run out, but they had the option to continue with the educational escape room.

7) CONSISTENT NARRATIVE
In this context, the narrative refers to the specific story of the educational escape room, including the role of the participants, their allies, the conflict, and the final goal. In order to create a highly engaging and immersive experience for students, the educational escape room was designed to be an experience in which the narrative, the puzzles and all the resources were perfectly integrated. As suggested by the "Ask Why" approach [51], each puzzle, task, item, page or resource existing in the escape room had a reason consistent with the narrative for being there. Special attention was given to ensure consistency between the narrative, the different puzzles, and the overall theme of the educational escape room.

C. NARRATIVE
The educational escape room experience started with a video recording with a confidential statement of the President of Spain (in reality, an actor playing the president), recorded in his presidential office, in which he alerts that an extremely deathly virus threatening the survival of humanity has been discovered. In this video, the president explains that a researcher who was investigating a vaccine for the virus has disappeared, but that the computer crimes unit of the national police has achieved to retrieve a software application developed by this researcher, which they are convinced that can be used to generate a vaccine for the lethal virus. However, although several software modeling diagrams and documentation created by the researcher himself could be retrieved and examined by the agents, neither of them succeeded in using the application for this purpose. For this reason, the president resorts to the students, as the world's greatest experts in software engineering, in order to discover how to use the software application designed by the researcher to produce the required vaccine and thereby save the nation and the world. Lastly, before saying goodbye and wishing the students the best of luck on their mission, the president warns them that it is crucial for the vaccine to be generated before two hours, because otherwise the antigens currently available to produce it will become tainted.
After watching the video, a classified computer forensics report elaborated by the computer crimes unit of the national police was provided to the students. This report contained more details about the case of the disappearance of the researcher who was investigating the virus, information on the software application developed by him including the URL of a web server on which the application was deployed, and a photography of his work desk taken as evidence for the investigation. From this point forward, students had to access the web application developed by the missing researcher using the URL included in the computer forensics report, and then they had to start solving the different puzzles of the educational escape room with the ultimate goal of generating the vaccine for the deadly virus before the time ran out. The next section describes in detail all these puzzles. Fig. 1 provides an overview of all the puzzles that the educational escape room comprised, indicating the order in which these puzzles had to be solved and the actions that students had to perform to solve them. Furthermore, Table I shows, for each puzzle, the learning objectives covered and the game mechanics involved. The next subsections describe each of the puzzles in detail.

1) PUZZLE 1: LOG IN TO THE RESEARCHER'S WEB APPLICATION
As mentioned before, the students started the educational escape room by watching a video recording of the President of Spain and reading a computer forensics report after that. At this point, the students were fully aware of the narrative of the activity: the story and background of the escape room, their own role in the story, and the final goal they needed to achieve. By reading the computer forensics report, the students could obtain the URL to access the web application developed by the missing researcher.
The first surprise that awaited the students was that this web application required to introduce the researcher's credentials to log in. The username was already filled in, but the students had to figure out the password. To do that, they had to view the photography of the researcher's work desk attached to the computer forensics report and notice a photography of a dog taped to the wall, on top of which there was a name hand written with a marker. If the students introduced the dog's name as the password, they could log in to the application and see the main page (shown in Fig. 2), completing this way the first puzzle.

2) PUZZLE 2: GAIN ACCESS TO THE RESEARCHER'S EMAIL INBOX
Once the students logged in to the researcher's web application, they could start exploring it. However, they were quickly able to see that they could not access all parts of this application. Concretely, the vaccine generator button was not enabled, the access to a specific folder of the file repository was blocked, and new credentials were requested when trying to access the researcher's email inbox.
In order to solve this second puzzle, the students first needed to enter the file repository and interpret the system requirements specification stored in it, which included the descriptions of the use cases and their corresponding UML use case diagrams. Thereby, the students could figure out that accessing the researcher's email inbox requires to enter a four-digit password, and that each of these four digits is related to a specific sequence diagram, but they did not yet know how.
The four UML sequence diagrams referred to by the use cases documentation could be found in a different folder of the file repository (see Fig. 3). In the same repository, there was also an interactive map that displayed numerous cities of the world and that allowed to paint on top of it. Students needed to figure out that each of the UML sequence diagrams represented a journey through different cities, which appeared in the interactive map. If the path of the journey represented by any of these UML sequence diagrams is drawn on the map, a number appears. Thereby, if these diagrams are correctly interpreted, it is possible to obtain the four digits that make up the researcher's email password. In order to obtain the password, the students just needed to sort these digits according to the information provided by the specification of use cases. In order to access the researcher's email inbox, the students also needed to provide its email address, which could be found in three different ways: by looking at a notebook that appeared in the photography attached to the computer forensics report, by reading a draft paper located in the file repository, or by examining the 'About' section of the web application.
This puzzle addressed the first learning objectives of the educational escape room: understand and interpret specifications of use cases, UML use case diagrams, and UML sequence diagrams. Moreover, this puzzle incorporated different game mechanics such as identification of patterns, using something in an unusual way, and research using information sources, since determining the path represented by each UML sequence diagram did not require just to apply knowledge on software modeling, but also to search for information about countries, tourist monuments, places and foreign currencies.

3) PUZZLE 3: UNLOCK THE RESTRICTED ACCESS DIRECTORY OF THE FILE REPOSITORY
At this point of the experience, the students could read all the emails available in the email inbox of the missing researcher. This inbox included useless advertising and promotional emails, but also emails sent by his colleagues and relatives, as well as his responses to these emails. The emails exchanged with his colleagues provided more details about the story: it seems that the missing researcher was part of a work team dedicated to investigate the virus, and that he was just about to complete a vaccine for it. At this time, the students also realized that the journeys represented by the UML sequence diagrams previously analyzed are the journeys undertaken by the other members of this work team for different aims such as collecting virus samples from different regions of the world or purchasing the necessary supplies. In one of the emails, the missing researcher informs one of his colleagues that he has added additional security to the web application restricting the access to a specific directory of the file repository. This email also contained a link to an interactive web-based periodic table and had attached a file of a UML activity diagram.
The attached UML activity diagram contained references to different chemical elements present in the periodic table.
The students had to understand and properly interpret this UML activity diagram in order to obtain a specific sequence of four chemical elements. Then, if they selected these four chemical elements in the periodic table in the right order, a passcode was automatically unveiled. Finally, unlocking the directory of the file repository whose access was restricted only required the students to introduce the unveiled passcode in a panel that popped up when accessing this directory.
While the previous puzzle addressed understanding and interpreting UML use case and sequence diagrams, this puzzle tackled UML activity diagrams, requiring students to apply knowledge on this specific matter. Regarding the game mechanics, different actions had to be carried out by the students during the resolution of the puzzle, including searching for objects (an email, in this case), establishing logical relationships among different elements, and using something (the periodic table) in an unusual way.

4) PUZZLE 4: ACCESS THE VACCINE GENERATOR
When the students accessed the directory of the file repository initially blocked, they gained access to several new files, including a set of voice recordings and a web-based jigsaw puzzle. By listening to the audio files, the students could quickly realize that these files were voice notes that were recorded by the missing researcher as a logbook while he was designing the web application so as not to forget any details on the software design. In these audio recordings, the students could hear the researcher talking in an informal and at times irreverent way, not only about the design of the software application, but also about their workmates (often criticizing and mocking them), and about mundane things, such as the number of hours he had been working non-stop, national festivities, or a rock song. Although the investigator might give the impression of not being completely in his right mind, his information on the design of the web application was reliable and very accurate.
Another important resource that became available to the students when they unlocked the restricted access directory, was a web-based jigsaw puzzle whose pieces were parts of different UML class diagrams, as shown in Fig. 4. In order to complete the jigsaw puzzle, the students needed to compose the UML class diagram of the web application by using the available pieces and hearing the voice recordings. Each piece of the jigsaw puzzle had two images, one on each side, and each image consisted of a fragment of a UML class diagram. While interacting with the digital jigsaw puzzle, the students were able to interchange pieces, as well as to flip them. The solution of the jigsaw was composed by 6 images, but the students had to pick these images from 12 two-sided pieces (i.e., 24 images). For each of the 6 fragments of the UML class diagram corresponding to the solution there were 4 different options, which means than there were more than 4000 possible UML class diagrams for completing the jigsaw puzzle, which clearly made practically impossible for students to solve it by chance in a reasonable time. In this regard, it should be pointed out that the hints provided for this puzzle allowed teams to obtain not only general help, but also help for specific pieces of the jigsaw when they got stuck. When the students solved the jigsaw puzzle, which only occurred when they managed to place every image in their correct position and clicked on a button to verify the solution, a new access code was revealed to them. At this point, they could use this code to access the only area of the web application that remained disabled: the vaccine generator.
This puzzle of the educational escape room addressed one of the most important learning objectives of the activity: understand, interpret and create UML class diagrams. The fake pieces and images of the jigsaw were designed in such a way that students needed a solid foundation on the main types of relationships (associations, shared aggregations, composite aggregations, and generalizations) and elements (classes, attributes, operations, multiplicities, association classes, etc.) of a UML class diagram. The playful aspect of this escape room puzzle relied mainly on two game mechanics: hearing (i.e., the provision of game elements through sound sources), and assembly of an object (in this case, a digital jigsaw puzzle).

5) PUZZLE 5: GENERATE THE VACCINE FOR THE VIRUS
In order to solve the last puzzle of the educational escape room and thereby succeed in the activity, the students had to generate the vaccine for the life-threatening virus by controlling a DNA synthesizer through a web interface. However, they ran into a tough final hurdle when they entered in the vaccine generator area of the web application: there were many options in this interface and the process of generating the specific vaccine was complex because it required to perform multiple steps with the DNA synthesizer, as well as to manipulate different elements. Indeed, the likelihood of successfully completing this process by chance, without following any instructions, is less than 1 out of 10 5 million. As a curiosity, it can be mentioned that this process was based on a real process for generating a vaccine, keeping this way the experience faithful to the "Ask Why" approach [51].
The first thing the students needed to do to accomplish this puzzle was to access the DNA synthesizer user manual, whose link could be found in a text available at the vaccine generator area of the web application. By examining this manual, the students could find a UML state diagram illustrating how to produce a vaccine using the DNA synthesizer software. However, although the diagram was complete, it contained six different symbols that it used to refer to different values and elements. Therefore, the students had to find the equivalence of all these symbols by carefully examining all the sections of the web application, as well as all the resources they could access.
The equivalence of one of the symbols could be found by carefully examining the 'About' section of the web application. Another symbol appeared together with a word in the photography of the work desk attached to the computer forensics report provided at the beginning of the activity. The value of another symbol could be obtained by drawing on the map the journey's path represented by a new UML sequence diagram in the same way as in the second puzzle. The equivalence of another symbol was printed in the last page of a welcome manual of a biology center available as a PDF file in the file repository. Finally, the equivalence of the remaining symbol could be found in the signature of one of the emails sent to the missing researcher. Basically, the students had to explore the whole environment to find all the necessary symbol equivalences. In this regard, it is worth indicating that specific hints for each of the symbols were available for the students to request while struggling with this puzzle.
One step of the process of generating the vaccine required students to select the virus for which the vaccine should be generated. Therefore, in addition to discover the element or value represented by each symbol included the UML state diagram, they had to figure out the official name of this virus. The only way in which they could discover this name was by reading a draft paper authored by the missing researcher and his colleagues, which was available in the file repository from the beginning.
Finally, in order to generate the vaccine, the students had to control the DNA synthesizer through the web interface according to the process indicated by the UML state diagram. To be successful in executing this process, the students needed to suitably interpret the UML state diagram, as well as to have the correct equivalences of all symbols. If some step was not correctly executed, for instance, if they did not choose the correct action, elements or synthesizer settings, an error message was prompted warning the participants about this fact and recommending them to consult the DNA synthesizer user manual. For the sake of realism, different errors messages coherent with the performed actions were included. Whenever a student managed to generate the desired vaccine successfully, completing this way the educational escape room, the screen of all the members of his/her team got filled with confetti, and a final message was displayed notifying them their success at the escape room.
This puzzle addressed the last type of UML diagram that the educational escape room intended to cover: the UML state diagram. Besides requiring students to understand and interpret UML state diagrams, this puzzle dealt with UML sequence diagrams again, although in this occasion with a sequence diagram that was a bit more intricate than the previous ones. The reason for this latter fact was that the escape room puzzles were designed to be of slightly increasing difficulty, and thus the last puzzle should be the most difficult one. This last puzzle of the educational escape room resorted to several game mechanics frequently used in ludic escape rooms, including searching for hidden objects, examining images, noticing something obvious in the environment, and symbol substitution.

IV. EVALUATION METHODOLOGY
This article aims to analyze the learning effectiveness of the remote educational escape room for teaching software modeling described in the earlier section, the students' perceptions toward this escape room, and the students' performance during this activity, including the relationships among this performance and the previous factors (i.e., learning effectiveness and students' perceptions). Three evaluation instruments were used in order to achieve this aim: (1) a pre-test and a post-test to measure students' learning gains, (2) a questionnaire to collect students' opinions, and (3) the Escapp web platform previously described, which enabled to obtain information on the students' performance and behavior in the activity by automatically gathering data on students' interactions during the remote educational escape room. The next subsection describes the sample of the study, the following three subsections describe in detail each of the aforementioned evaluation instruments, and the last subsection describes the methods used for data analysis.

A. SAMPLE
The sample of this study consisted of 162 students who were enrolled in the software engineering fundamentals course described in section III-A. All of these students participated in the remote educational escape room previously detailed, completed both the pre-test and the post-test, and fulfilled the questionnaire. Since 246 students took the midterm exam carried out approximately one week after the educational escape room, this sample represented around 66% of all course learners.
Of the 162 students who participated in this study, 132 (81.5%) were men, 29 (17.9%) were women, and 1 (0.6%) did not indicate his/her gender. Participants were aged between 19 and 36 years, being 20.7 the mean age, 20.0 the median age, and 2.2 the standard deviation. Less than half of the participating students (40.7%) had participated in a leisure or ludic escape room before, and only four (2.5%) had previously participated in an educational escape room. When asked if they like to play any kind of game, a vast majority of students (91%) answered positively, whereas 7% neither agreed nor disagreed, and only 2% answered negatively. When asked about the difficulty of the software engineering fundamentals subject, around a quarter of the students (27%) said the subject was difficult for them, nearly another quarter expressed the opposite (22%), and practically half of the surveyed students (51%) believed that the subject was neither difficult nor easy. Regarding teams, students formed a total of 41 teams with sizes between 1 and 6 members to participate in the remote educational escape room: 4 teams (9.8%) had 6 students, 12 teams (29.3%) had 5 students, 13 teams (31.7%) had 4 students, 5 teams (12.2%) had 3 students, 4 teams (9.8%) consisted of pairs, and 3 (7.3%) teams were formed by one single student who participated alone. The mean and median team size was 4.0, and the standard deviation was 1.4. According to these data, nearly 84% of the students participated in teams formed by 4-6 members, around 14% participated in small teams of 3 members or in pairs, and less than 2% conducted the educational escape room alone. A total of 34 teams (82.9%) involving 141 students (87.0%) participated in scheduled shifts where a videoconference room was available to contact instructors, whereas the remaining 7 teams (17.1%), which were formed by 21 students (13.0%), were enrolled in self-paced shifts and performed the educational escape room when they wished.

B. EVALUATION OF LEARNING EFFECTIVENESS
The pre-test was administered just before the start of the remote educational escape room to assess the students' prior knowledge on software modeling, and the post-test was conducted right afterward to assess the learning effectiveness of the escape room experience among students (i.e., the learning gains). The pre-test and the post-test had the same characteristics. Both of them gave students 10 minutes for its completion and contained the exact same ten multiple-choice questions on software modeling. These questions specifically evaluated the students' knowledge on specification of use cases, UML use case diagrams, UML class diagrams, UML sequence diagrams, and UML state diagrams, as well as the students' skills in understanding, interpreting and creating this type of diagrams. Therefore, the learning objectives assessed by the pre-and post-test and the remote educational escape room were the same. In this regard, it should be clarified that answering the questions of the pre-and post-test right not only required students to remember information and understand software engineering fundamentals, but also to apply their knowledge to analyze UML diagrams and solve specific software modeling problems. In order to prevent students from memorizing the answers of the pre-test, no feedback was provided in this first test. Thus, the correct answers were not provided to students until after they completed the post-test. Regarding this matter, it is worth indicating that, until students took the post-test, they did not know that it had the same questions as the pre-test. Another fact that should be taken into account is that, although the questions of the pre-and post-test and those questions students had to answer to obtain hints during the remote educational escape room were related to the same learning objectives, these sets of questions were completely independent, so no question of the pre-and post-test was presented during the activity. Furthermore, the grading of the educational escape room was exclusively based on attendance, so neither the pre-test score nor the post-test score counted toward the students' final grade. Thereby, students had no reason to cheat and, in addition, this approach prevented unwanted behaviors such as the one reported by [23], in which students put more effort in one of the tests because it accounted for a much more significant percentage of the grade. Regarding the characteristics of the pre-and post-test conducted, it should also be indicated that, with the aim of discouraging students from completing the tests randomly, wrong answers subtracted points from the score, although students were allowed to leave answers blank with no penalty. Given that each question was worth one point, the maximum score that a student could achieve in the pre-or post-test was 10, while the minimum score was 0.

C. EVALUATION OF STUDENTS' PERCEPTIONS
In order to gather students' perceptions toward the remote educational escape room conducted, an ad-hoc questionnaire was administered right after the post-test. This questionnaire included some initial demographic questions, a set of closed-ended questions addressing students' acceptance of the remote educational escape room, a list of statements with which participating students needed to agree or disagree using a five-point Likert scale, and an open-ended question asking for suggestions, complaints and other additional comments about the experience. The questionnaire items were aimed at assessing students' attitudes toward the use of the remote educational escape room as a learning activity, student self-reported learning effectiveness, and students' insights on different aspects of the escape room (complexity, hint usefulness and delivery, type of puzzles, immersivity, organization, duration, teamwork, etc.). Furthermore, students were also asked if they would have liked the escape room to be conducted face-to-face or to incorporate puzzles requiring interaction with physical objects, as well as whether they preferred the escape room over a traditional practical computer lab session. The items of the questionnaire are presented in the next section along with the results.
The content validity of the ad-hoc questionnaire designed for this study to examine the students' perceptions toward the remote educational escape room conducted was checked by subject area experts. Moreover, the reliability and consistency of the questionnaire were also checked using the Cronbach's α, which allows to estimate the reliability of measurement instruments, and the Kaiser-Meyer-Olkin coefficient, which provides a measure of the sample adequacy. The α of Cronbach was 0.83 and the Kaiser-Meyer-Olkin coefficient was 0.80, which are very positive results since both coefficients are equal or above 0.8 [52], [53].

D. EVALUATION OF STUDENTS' PERFORMANCE IN THE ESCAPE ROOM
Finally, the Escapp web platform was employed to automatically record data on relevant student interactions during their participation in the remote educational escape room. Specifically, Escapp was used to gather the following data: the students who made up each team, the number of puzzles solved by each team and the time required by each team to solve each puzzle, the hints obtained and requested by each team, and the teams that achieved to successfully complete the remote educational escape room, as well as the total time they required to do so. These data allowed to obtain relevant insights on the performance and behavior of the students during the remote educational escape room. Furthermore, these data also enabled to examine the relationships between students' performance in the activity and their learning gains, as well as their perceptions.

E. DATA ANALYSIS
The Shapiro-Wilk test of normality was conducted to determine whether the obtained data is normally distributed. The results of this test show that the pre-test scores, the post-test scores, and the questionnaire scores were not normally distributed. Therefore, since the assumption of normality was not met, nonparametric statistical methods were employed. Pre-test and post-test scores were compared by means of a Wilcoxon Signed-Ranks Test for paired samples. The results of the student questionnaire were analyzed by using descriptive statistics. All comparisons between different groups of students (e.g., between students who participated in scheduled shifts and students who participated in self-paced shifts) were performed using the Mann-Whitney U-test. Relationships among different variables were examined through Spearman's correlation analysis. In all cases, the correlation coefficient (r) was used as the effect size measure. According to Cohen's guidelines [54], 0.1 ≤ r < 0.3 represents a small effect size, 0.3 ≤ r < 0.5 represents a medium effect size, and r ≥ 0.5 represents a large effect size.

A. LEARNING EFFECTIVENESS
The results of the pre-test and the post-test administered to measure students' learning gains are shown in Table II, including, for each test, the mean (M), median (MED), and standard deviation (SD) of the scores attained by the students. The average score of the pre-test was 6.8 out of 10 (MED = 6.9, SD = 1.8), whereas the average score of the post-test was 8.5 (MED = 8.8, SD = 1.5). The learning gains, calculated as the difference between post-test scores and pre-test scores, reached an average value of 1.8 (MED = 1.7, SD = 1.7). A Wilcoxon Signed-Ranks Test for paired samples was undertaken to compare pre-test and post-test scores. The difference between these scores was found to be statistically significant with a large effect size. These results  [37], but these studies did not have analyzed the activities in terms of instructional effectiveness. Thus, no previous research has been conducted so far to investigate the learning effectiveness of remote educational escape rooms. Hence, this article makes a novel contribution by providing, for the first time, evidence on the learning effectiveness of an educational escape room conducted remotely. Moreover, neither of the few studies that evaluated learning effectiveness of in-person educational escape rooms previously cited conducted the activity for teaching software engineering fundamentals. Therefore, the contribution of this article is also novel because it provides, for the first time, proof that educational escape rooms are effective activities for teaching software modeling, an essential topic on which every software engineering student should have a solid foundation [41], [42].

Pre-Test Post-Test Learning Gains Wilcoxon Signed-Ranks
No statistically significant differences were found when comparing scores achieved by men and women in the pre-and post-test by means of a Mann-Whitney U-test, which was an expected result since previous research on educational escape rooms [29] did not find differences by gender in learning performance either. Likewise, no statistically significant correlation was found between the students' learning gains and their penchant for playing games or their opinion regarding the difficulty of the software engineering fundamentals course. In summary, the learning effectiveness of the remote educational escape room was not affected by gender, gaming liking or self-reported course difficulty. However, the first and second of these findings should be treated with caution due to the small number of female students and students disliking games in the sample.
The learning gains experienced by the students who participated in scheduled shifts (M = 1.8, MED = 1.7, SD = 1.8) were slightly higher than those experienced by the peers who participated in the self-paced shifts and that did not have the opportunity to contact the teachers during the experience (M = 1.5, MED = 1.6, SD = 1.4). Nevertheless, this difference had a negligible effect size (r = 0.01) and was not found to be statistically significant at an alpha level of 0.05. In this regard, it should also be pointed out that the difference between post-test scores and pre-test scores for the students who participated in self-paced shifts had a statistically significant large effect size too. Based on these results, it can be suggested that virtual educational escape rooms conducted remotely without teacher supervision can be effective self-paced learning activities. This fact opens up new possibilities for using educational escape rooms, such as their incorporation in self-paced learning environments like MOOCs (Massive Open Online Courses) and other kind of self-paced online courses, which have become especially relevant during the COVID-19 pandemic. Nevertheless, it should be taken into account that for an educational escape room to succeed in these environments, it is necessary to accurately track students' progress toward completing the activity and provide them with timely help. Thereby, it is possible to prevent participants from getting stuck at a puzzle for too long, which could put them at risk of frustration, learning losses, and dropping out the activity. Table III shows the results of the questionnaire administered to collect students' perceptions toward the remote educational escape room. For each item rated on a Likert scale from 1 to 5, the mean (M), median (MED), and standard deviation (SD) are shown in the table.

B. STUDENTS' PERCEPTIONS
Overall, the results of the questionnaire indicate highly positive students' perceptions toward the use of the remote educational escape room as a learning activity. The students had an excellent overall opinion of the activity (M = 4.6, MED = 5.0, SD = 0.6). Indeed, two out of every three students (66.7%) rated this item with a five (the maximum value), around a quarter of them (26.5%) rated this item with a four, just a few (6.2%) assigned a rating of three, and only one student out of all who participated (0.6%) indicated a negative opinion. In this regard, it is worth indicating that students' prior interest in games was not found to be a harbinger of their general opinion on the educational escape room since no correlation was found between these two variables. Another evidence of the high student acceptance of the remote educational escape room was that almost all students stated that they would recommend other students to participate in the activity (98.8%), and that they would like other courses to incorporate similar learning activities (96.9%).
Regarding the learning gains perceived by students, results show that they believed that participating in the remote educational escape room was beneficial for improving their knowledge on software modeling (M = 3.9, MED = 4.0, SD = 0.9), which is in line with the actual  In fact, a vast majority of students (83%) agreed on these three opinions. Of special interest is the fact that the students found the educational escape room to be immersive despite being conducted remotely. In face-to-face experiences, escape room designers have more resources at their disposal to immerse participants in the story such as props to create a given look or atmosphere, background music, or physical objects that provide stimuli for the participants (e.g., smells, textures, and sparkles). However, achieving immersion in an educational escape room conducted remotely is much more challenging.

Question M MED SD
The results obtained in this study are very encouraging because they prove that it is possible, for a remote educational escape room, to successfully immerse their participants by elaborating an attractive and consistent narrative. One of the keys of this success has been following the guidelines suggested by the "Ask Why" approach [51], so that all the elements that the students encountered during the experience had a reason consistent with the narrative for being there. Regarding gamification elements, it is worth indicating that students found the leaderboard motivating (M = 4.3, MED = 5.0, SD = 0.9). This is quite an interesting finding because it suggests that gamification elements like leaderboards constitute a suitable option to foster competition and a playful atmosphere in remote educational escape rooms. Although educational escape rooms are designed to be motivating and enjoyable, they are live-action team-based activities in which participants can have a feeling of rush due to the time limit and the real-time countdown, which can generate stress, especially in remote settings in which communication and collaboration among team members is more complicated and participants could be more worried about possible technical problems. In this study, around 15% of the participating students reported the educational escape room experience to be somewhat stressful and 7% to be very stressful. Nonetheless, most students felt the opposite way (M = 2.5, MED = 2.0, SD = 1.2).
Regarding the level of difficulty of the activity, results indicate that this level was appropriate, since, in general, the students did not think the educational escape room was too difficult (M = 2.7, MED = 3.0, SD = 1.0) or that it was too easy. Thus, it can be stated that it succeeded at balancing the difficulty, which is a major aspect of the experience according to the Csikszentmihalyi's flow theory [50].
Delving into the issue of difficulty, a medium and statistically significant correlation (Spearman's ρ = 0.3) was found between students' perceptions on the difficulty of the educational escape room and their perceptions on the difficulty of the course, revealing that students who thought the course was difficult thought so about the escape room too. The students believed not only that the difficulty was properly balanced, but also that they were prepared enough to succeed at the learning activity (M = 4. Although almost all students agreed that both factors affect the difficulty of the puzzles, 31% of them indicated that both factors affect equally, whereas 19% thought that the difficulty of the puzzles relied on mastering the course materials to a greater extent than in the game mechanics, and the remaining 50% thought the opposite. These results confirm that the difficulty of the educational escape room relied both on the application of knowledge and skills specific to software modeling, and the understanding of game mechanics and the execution of actions typical of ludic escape rooms. According to the results, it seems clear that some students found it more difficult to solve the educational part of the puzzles than the playful one, whereas for other students the opposite occurred. Therefore, hints delivered during educational escape rooms should provide help considering these two different factors. With respect to the organization of the remote educational escape room, the students agreed that overall it was very well organized (M = 4.3, MED = 5.0, SD = 0.9), that the initial guidance provided was enough (M = 4.1, MED = 4.0, SD = 0.9), that its supervision was adequate (M = 4.4, MED = 5.0, SD = 0.7), and that it had an appropriate duration (M = 4.3, MED = 5.0, SD = 1.0). Moreover, they rated the Escapp web platform used for managing and conducting the activity very positively (M = 4.2, MED = 4.0, SD = 0.9). Educational escape rooms are live-action team-based game-like activities whose organization can be complex, especially when they are conducted remotely. This complexity is due, in part, to the lack of tools for managing this kind of activities, as well as the lack of reported experiences, whose designs and lessons learned could be of great value for instructors interested in conducing educational escape rooms. The results of this work show that it is possible to successfully organize and conduct educational escape rooms remotely by using appropriate tools and escape room designs. In this regard, it should be highlighted that the Escapp web platform has proven to be very useful for this purpose.
Another interesting finding of this study is that less than 7% of students disagreed with the fact that the remote educational escape room should be a graded activity, whereas nearly 80% of them agreed (M = 4.2, MED = 5.0, SD = 1.1). In view of this finding and the evidence suggesting that educational escape rooms are beneficial for the students' learning, teachers of higher education institutions and of supervised or self-paced online courses could consider incorporating remote educational escape rooms as graded learning activities.
In the remote educational escape room analyzed in this study, hints were handed out to students by using the quiz-based approach previously explained in section III-B-5 with the aim of helping them in case of getting stuck at solving any of the puzzles. The results of the questionnaire show that the students had positive perceptions of this approach (M = 4.1, MED = 4.0, SD = 1.0), and that they considered the hints obtained useful to progress in the educational escape room (M = 3.9, MED = 4.0, SD = 1.2). Another fact that confirms the suitability of the hint approach is that the students considered the help provided enough because they stated that they would have not wanted any more help during the activity (M = 2.2, MED = 2.0, SD = 1.2). A medium and statistically significant correlation (Spearman's ρ = 0.3) was found between this item and students' perceptions on the difficulty of the course, indicating than those students who were more favorable to receive additional help during the escape room were the ones that thought the course was more difficult.
Almost all students (94.4%) who participated in the remote educational escape room as part of a team liked participating in this way (M = 4.8, MED = 5.0, SD = 0.6), and most of them (84.3%) disliked the idea of participating alone (M = 1.6, MED = 1.0, SD = 1.0). Moreover, students were not positive about participating in larger teams (M = 2.0, MED = 2.0, SD = 1.1). Surprisingly, this result was not affected by team size since no significant differences were found when compared the responses of the students grouped according to the size of their teams.
Regarding teamwork, most students stated that all their teammates were equally involved in the activity (M = 4.2, MED = 4.0, SD = 0.8). When asked about the communication or collaboration problems caused by conducting the educational escape room remotely, most students disagreed that there had been significant problems for this reason (M = 1.9, MED = 2.0, SD = 1.1), although it should be pointed out that around 10% of participants agreed with this statement. These are very hopeful findings because they indicate that popular real-time communication applications such as Skype, Zoom, Google Hangouts, Microsoft Teams or Discord can be effectively used for enabling team participation and collaboration in remote educational escape rooms. However, it must be taken into account that in this experience the escape room state was synchronized among all team members and hence, significantly different results could be obtained regarding teamwork and coordination in other scenarios that lack synchronization mechanisms. Furthermore, the fact that some students experienced communication or collaboration problems that would not have occurred if the activity had been carried out face-to-face, indicate that special attention should be paid by instructors to supervise and design remote educational escape rooms. Overall, the results of this study suggest that the educational escape room provided a highly favorable atmosphere for collaborative learning and that its cooperative aspect was very worthwhile from the pedagogical point of view because it effectively addressed soft skills such as teamwork and leadership, which are very important in software engineering education, as well as in engineering education in general [55]. Indeed, these skills are usually addressed in computer science degrees, as suggested by the Euro-Inf framework [42].
When asked if they would have preferred the escape room to be conducted face-to-face instead of remotely, nearly half of the students (47.5%) answered affirmatively, while one of every three (33.3%) neither agreed nor disagreed, and the remaining students (19.1%) stated that they preferred the educational escape room to be conducted remotely instead of in-class. The reason why a significant percentage of students preferred the activity to be conducted in-person instead of remotely seems unrelated to the fact that the escape room was virtual, because practically all students liked the fact that the puzzles were digital (M = 4.6, MED = 5.0, SD = 0.6) and they were not very enthusiastic about the idea of incorporating physical objects into the experience (M = 3.0, MED = 3.0, SD = 1.4). Therefore, it seems that the students who preferred the activity to be conducted face-to-face stated so because they wanted to communicate and interact with their teammates in person rather than via software applications, and not due to any other reason. The results evidencing that students liked the digital nature of the puzzles reported in this work are consistent with those of [37], in which participants of a remote educational escape room reported to like the use of a digital medium. These findings encourage educational escape room designers to incorporate digital puzzles into their experiences, especially in those cases in which the activity has to be conducted for many students and/or when reducing its cost is critical.
When the perceptions of those students who participated in self-paced shifts were compared to those of their peers who participated in scheduled shifts, only four statistically significant differences were found, all of them with small effect sizes. Firstly, for students enrolled in self-paced shifts the supervision of the activity was less adequate (self-paced: M = 4.0, MED = 4.0, SD = 0.8; others: M = 4.5, MED = 5.0, SD = 0.7; r = 0.22). This was an expected finding since in self-paced shifts there was no instructor available in case of technical difficulties or need for additional help. Nonetheless, an unexpected and very positive finding is that, despite this fact, students who performed the remote educational escape room in self-paced shifts considered, in general, that the guidance and help automatically provided was enough for successfully completing the activity and hence no additional supervision (human or automatic) was necessary. Secondly, the students of self-paced shifts thought the educational escape room was less well organized (self-paced: M = 3.9, MED = 4.0, SD = 1.1; others: M = 4.4, MED = 5.0, SD = 0.9; r = 0.13). A possible reason for this difference is that, in scheduled shifts, instructors waited for all enrolled students to be present, reminded them the instructions of the activity, and solved doubts when necessary. This could have had a positive impact on the perceptions of these students with respect to the organization of the activity. Thirdly, students in self-paced shifts perceived the activity as even less stressful (self-paced: M = 2.0, MED = 2.0, SD = 1.3; others: M = 2.5, MED = 2.0, SD = 1.2; r = 0.14). This result makes sense because these students started the activity at a time of their convenience. In scheduled shifts, it is more likely that students were worried about being punctual or finishing on time to attend or do other activities. Moreover, teams in self-paced shifts did not usually perform the activity simultaneously with other teams and therefore the feeling of competition was much less intense, which could have contributed to the students feeling stressed to an even lesser extent. Lastly, the preference of the educational escape room over a computer lab session was less pronounced for those students enrolled in self-paced shifts (self-paced: M = 3.7, MED = 4.0, SD = 1.2; others: M = 4.3, MED = 5.0, SD = 1.9; r = 0.15). This outcome could indicate that these students had a slightly less positive perception on the usefulness of the activity, although this hypothesis is not very likely because the difference on the overall opinion item was small and non-statistically significant and the difference in learning gains was non-statistically significant with a negligible effect size.
The questionnaire administered to students included a final open-ended question in which students could provide suggestions, complaints or any other comment they wished. A total of 58 out of the 162 students who participated in the remote educational escape room left a comment. Of these 52 comments, 51 (87.9%) were positive toward the experience, whereas 3 (5.2%) were negative, and the remaining 4 (6.9%) merely included suggestions to improve specific aspects. One of the negative comments was made by a student because his team could not enroll in any scheduled shift and had to participate in the self-paced shift. The other two negative comments were complaints, one about a certain hint not being very useful and the other one about technical issues with an unpopular web browser. In the positive comments, the students emphasized how enjoyable the remote educational escape room was for them (39.2% of the positive comments referred to this issue), how useful this activity was for helping them learn the topics covered (33.3%), and how fun it was (27.5%). Examples of comments emphasizing these aspects of the experience were: "It has been great, very enjoyable and enriching to learn concepts of the subject", "Very good method to motivate and teach students!", "It seems to me that looking for activities at a midpoint between playful and educational is interesting and effective", and "I really liked the feeling of immersion and the attention to detail, as well as the degree of difficulty. The most well-crafted and fun educational activity I have ever done. Congratulations and thank you very much for the effort!". The students also left many comments thanking the teaching staff of the course for taking the time and effort to organize the remote educational escape room (31.4%), as the one cited before. There were also comments referring to the innovative aspect of the activity (17.6%), asking for similar activities to be conducted in other courses (9.8%), praising the quality of the narrative (9.8%), and showing satisfaction for having participated in a team (9.8%). Regarding the remote aspect of the activity, two students indicated that collaborating online with their teammates was more difficult than collaborating in person would have been, and one of them, in spite of considering the activity fun, expressed that it would have been better to conduct the escape room face-to-face instead of remotely. Lastly, in the comments with suggestions, students proposed increasing the grade awarded for attending the escape room, minor enhancements to the Escapp web platform, and ideas for alternative narratives.
Overall, the comments are consistent with the quantitative results of the questionnaire and confirm that the students had very positive perceptions toward the use of the remote educational escape room for learning purposes, since they perceived the activity as highly enjoyable, motivating and beneficial for their learning. The comments also confirm that collaboration problems were experienced by some students due to the educational escape room being conducted remotely. However, these students represented a small percentage (around 10%) of the total and the difficulties reported were not severe in any case. Fig. 6 show, respectively, the percentage of teams that solved each puzzle of the escape room and the average time each team spent to solve each of these puzzles. In Fig. 6, error bars have been added to represent standard deviation.

Fig. 5 and
As shown in Fig. 5, almost all teams (95%) solved all the puzzles, although only slightly less than half of them (46%) did so before the time ran out (i.e., in two hours). Therefore, the success rate of the remote educational escape room (defined as the percentage of teams that successfully completed it on time) was 46%. In this regard, it should be indicated that nearly 90% of the teams managed to solve all the puzzles in 150 minutes (30 minutes more than the deadline time). Not many works in the literature have reported success rates for educational escape rooms, and the few that did so, reported very different values for this rate. Concretely, works can be found reporting a success rate of 0% (i.e., not a single team was capable of escaping) [24], 8% [22], 50% [19], [29], 67% [11], 75% [27] or 100% (i.e., all teams succeed) [32]. In the experience reported in this article, students could earn points for attending the remote educational escape room, but continuing with it after the time limit was not necessary at all. Taking this into account, the fact that almost all teams continued trying to solve escape room puzzles even after the time ran out, is a very positive finding because it suggests that the students were enjoying the activity and that they perceived it as beneficial for their learning. Naturally, this finding is consistent with those obtained from the student questionnaire.
As shown in Fig. 6, teams only needed, on average, around five minutes for solving the first puzzle, which was an expected finding because this puzzle was designed to be of very low difficulty. On average, the second puzzle required teams to spent nearly 38 minutes, the third puzzle 18 minutes, the fourth one 27 minutes, and the last one 34 minutes. Since the puzzles were designed to be of increasing difficulty, it was expected that the time required to solve each one of them was also linearly increasing. However, the time required by the teams to find the solution to the second puzzle was the longest, indicating perhaps an excessive level of difficulty for this puzzle. Another reason that could explain this lengthy time is that many teams waited for too long to request a hint for the puzzle. Apart from that, the obtained results match all that was expected. On average, the teams that managed to solve all the puzzles did so in 121 minutes (M = 121.1, MED = 121.7, SD = 26.7). Therefore, given that the activity was designed to be conducted in two hours, this figure suggests that the difficulty of the whole activity was appropriately balanced. A negative, small and statistically significant correlation (Spearman's ρ = -0.26) was found between time spent on completing the educational escape room and post-test scores, indicating that the students who performed better in the activity also performed better in the post-test. Furthermore, a positive, medium and also statistically significant correlation (Spearman's ρ = 0.31) was found between time spent and self-reported escape room difficulty, which shows that those students who performed worse in the activity were the ones who thought it was more difficult.
Another interesting outcome is the high values of the standard deviations obtained for the times spent by each team to solve each puzzle, which indicate that these times varied notably across teams. For instance, while the fastest team managed to solve the last puzzle in just 18 minutes, the slowest team needed nearly 55 minutes to do so. Our guess is that these large time differences observed are mainly due to the game mechanics involved in the puzzles. As discussed before, the difficulty of the educational escape room puzzles depends on both field-specific knowledge and game mechanics, as well as how they intertwine. However, whereas puzzles were designed in such a way that succeeding in applying field-specific knowledge mainly depends on students' knowledge of the topic covered, the understanding of their game mechanics is more dependent on intuition, different knowledge and skills such as creativity and divergent thinking, or just luck. Table IV shows the average number of hints required by each team for solving each puzzle of the remote educational escape room, the average total number of hints obtained by each team during the whole experience, and the percentage of teams that required hints. On average, each team obtained 2.8 hints throughout the activity, although there were high differences between teams. Around three out of every four teams required at least one hint to solve a puzzle during the educational escape room, whereas approximately a quarter of them managed to solve all the puzzles without any help. There was only one team that failed and refused to request hints during the whole activity. These results indicate that, overall, the students resorted to hints when they got stuck. However, the lengthy time teams waited in some occasions before requesting a hint shows a slight reluctance on the part of some students to be helped. A possible reason for this could be that these students wanted to solve the puzzles by themselves and that, in some occasions, they could not ask for help out of pride. A medium to large and statistically significant correlation (Spearman's ρ = 0.46) was found between the total number of hints required and the time spent on completing the educational escape room, showing that students who needed more help were the ones who needed more time to finish, despite having hints that provided them with useful advantages. This is a reasonable finding taking into account that, in this experience, obtaining hints required students to invest time to answer quizzes. Regarding the hints required to solve each puzzle, just two teams (4.9%) requested hints for the first puzzle. For the other puzzles, the percentage of teams that required hints ranged from 58.5% in puzzle 2 to 12.5% in puzzle 4. An unexpected outcome was that the number of hints obtained for each puzzle was not proportional to the time spent for solving them. For instance, although students needed to dedicate more time to solve puzzle 4 than to solve puzzle 3, they requested more than double the number of hints for the latter. It seems that, for some puzzles, the students gave up much earlier than for others.
As explained in section III-B-5, in order to obtain a hint during the educational escape room, the students were required to pass a quiz with five random questions about software modeling. Table V shows the average number of quiz attempts made by each team to obtain hints for each puzzle and those made throughout the whole experience, indicating also the percentage of these attempts that were successful. On average, each team made 4.1 attempts to obtain hints during the activity, 68% of which were successful, so that each team obtained, on average, 2.8 hints. As expected, the quiz success rate remained the same throughout the entire educational escape room, indicating that the pool of questions employed was appropriate in terms of size and complexity.

VI. CONCLUSIONS AND FUTURE WORK
This article analyzes the learning effectiveness of an educational escape room for teaching software modeling that was conducted remotely, as well as the students' perceptions toward this activity. Furthermore, this article also examines how the students performed in the remote educational escape room. The results of the article provide strong and unquestionable evidence that the remote educational escape room conducted was a highly effective and very engaging activity for learning software modelling. On the one hand, the difference between post-test and pre-test scores was found to be statistically significant with a large effect size, showing that the activity succeeded in producing strong students' learning gains. On the other hand, the results of the questionnaire administered to the students show that they had very positive attitudes toward the use of the remote educational escape room as a learning activity. Overall, the students stated that the activity was fun, immersive, motivating, and beneficial for their learning. Furthermore, students' responses indicate that the activity was properly organized and supervised, that its difficulty was well balanced, and that, in general terms, the escape room was well designed. As for the fact that the educational escape room was virtual and remote, most students expressed that they liked that the puzzles were digital, that the web platform employed for managing and conducting the activity was useful and effective for this purpose, and that they did not experience communication or collaboration problems as a result of the escape room being conducted remotely. In summary, the results reported by this article suggest that well-designed remote educational escape rooms are effective and engaging learning activities. Furthermore, these results help to obtain a better understanding of the benefits of using educational escape rooms and the opportunities of this novel kind of learning activities. In this regard, it is worth mentioning that the results suggest that these activities provide a very favorable atmosphere for fostering students' soft skills such as problem-solving, teamwork and leadership through active and collaborative learning, even when they are conducted remotely. This article makes two major groundbreaking contributions. On the one hand, it provides, for the first time, evidence that remote educational escape rooms can be effective learning activities. Although prior works reported on the use of this type of activities [35]- [37], neither of them assessed a remote educational escape room in terms of instructional effectiveness. On the other hand, this article provides, also for the first time, proof that educational escape rooms are effective and engaging activities for teaching software modeling, an essential topic for courses on software engineering fundamentals [41], [42]. Previous works examined the use of educational escape rooms in a wide range of fields [11]- [40], however, neither of them reported on the use of an educational escape room for teaching software engineering fundamentals. Taking all these into account, it can be concluded that this article significantly contributes to the current body of knowledge.
With the rise of online distance learning, not only as a result of the COVID-19 pandemic, but from long before through MOOCs and other distance online learning environments, it could be very interesting and captivating for instructors to find new ways of providing highly engaging group activities such as educational escape rooms in online settings. The results presented in this work are therefore useful and encouraging for instructors that wish to carry out these novel educational activities in online distance learning scenarios. In this regard, it should be remarked that this work also provides evidence that remote educational escape rooms are effective and engaging even when students participate at their own pace and without human supervision, which opens up and encourage the possibility of using this type of activities in MOOCs, as well as in any other kind of self-paced online courses.
Another interesting finding of this work related to the suitability of educational escape rooms for online environments, is that the students had very positive attitudes toward using, not only computer-based educational escape rooms, but also specifically toward using digital puzzles. This is a very positive finding because the use of digital puzzles allows to offer educational escape rooms to a massive number of students at a low cost. Furthermore, this work has proven that by using an attractive and consistent narrative and the right game mechanics, it is possible to make students feel immersed in an educational escape room even if the activity is conducted remotely. Indeed, the feeling of immersion reported in this remote experience has nothing to envy to the one reported by a previous work in which the escape room was conducted face-to-face [29].
Although the findings of this study are very encouraging and provide a strong rationale for the adoption and use of remote educational escape rooms in online settings, several aspects of the experience should be taken into account. Firstly, conducting the remote educational escape room the way it was done would have been not possible without the Escapp web platform. This software tool played an essential and irreplaceable role in the successful performance of the activity, because it was used, among other things, to manage student enrollment, to check whether students solved the puzzles, to track the progress of the students toward solving the escape room puzzles, to deliver useful hints to students when they got stuck in a timely manner, and to synchronize the escape room state between team members, facilitating this way teamwork and cooperation. Therefore, it is reasonable to think that the outcomes of conducting educational escape rooms remotely without the support of an appropriate software could be much less positive, or even not been positive at all. Based on the results obtained and the lessons learned from this experience, we strongly encourage escape room designers interested in conducting this kind of activities in a remote way to rely on software tools specifically designed for this purpose, like the Escapp platform, which has proven to be very useful for conducting remote educational escape rooms. As of today, we are not aware of the existence of any work reporting on the use of other software system for this purpose, ergo the use of Escapp constitutes another novel contribution of this work.
Secondly, it should be taken into account that, in order to conduct the educational escape room presented in this work, besides using the Escapp platform, several ad-hoc web applications had to be developed from scratch by the course teachers. These web applications were highly interactive and communicated with Escapp in order to verify puzzle solutions, synchronize its state among teammates, and show real-time notifications. Therefore, remote educational escape rooms relying on less sophisticated resources could yield different results.
Thirdly, in this experience most students participated in the educational escape room in a group. Thus, the sample of students who participated alone is too small to reach meaningful conclusions on the effectiveness of remote educational escape rooms as individual learning activities. Although escape rooms are conceived as team-based and collaborative activities for which teamwork is part of their essence, we reckon that well-designed educational escape rooms can produce positive impacts on students even when they face the puzzles individually. Taking this into account, a valuable direction for future research would be not just to evaluate the effectiveness of educational escape rooms as individual learning activities, but also to design and evaluate activities of this kind specifically designed for individual participation.
Nowadays, there is a lack of software tools specifically designed for easing the creation of resources for educational escape rooms, which is not surprising given the novelty of these activities. This lack was the reason why the web applications involved in the remote educational escape room reported in this work had to be developed from scratch, which not only required significant time and effort, but also knowledge of web technologies and programming skills, which teachers generally do not have. In view of this dearth of resources, an interesting line of future work would be to develop and evaluate authoring tools for educational escape rooms, which would make possible, not only to reduce the time spent in creating these activities, but also to allow any teacher with basic computer skills to do so. We strongly believe that this line of future work is very promising, since the success of authoring tools has already been evident in other technology-enhanced learning areas like learning objects [56]. For this reason, we plan to develop this kind of tools and use them in future educational escape rooms.
The field of educational escape rooms is a recent and growing research field in which there is plenty of research opportunities and open challenges. It is likely that in the near future there will be a need, not just for software systems specifically designed to author and conduct both face-to-face and remote computer-based educational escape rooms, but also systems for other purposes such as recommending educational escape rooms, generating learning analytics for these activities, and interacting with escape room participants using artificial intelligence techniques. Finally, it is worth pointing out that, although this work has provided evidence that educational escape rooms can be effective when conducted remotely, it has not assessed whether the benefits of conducting an educational escape room are the same regardless of it being conducted face-to-face or remotely. Therefore, an interesting future work opened by this research is to compare an educational escape room conducted face-to-face with the same escape room conducted remotely.