Analyzing Learning Effectiveness and Students’ Perceptions of an Educational Escape Room in a Programming Course in Higher Education

In recent years, educational escape rooms have started to gain momentum in the academic community. Prior research has reported on the use of educational escape rooms in several fields. However, earlier works have failed to assess the impact of this sort of activities for teaching programming in terms of learning effectiveness. This work fills the existing gap in the literature by examining an educational escape room for teaching programming in a higher education setting by means of three different instruments: (1) a pre-test and a post-test for measuring learning gains, (2) a survey for assessing students’ perceptions, and (3) a web platform for recording student interaction data during the activity. The results of this work provide, for the first time, empirical evidence that educational escape rooms are an effective and engaging way of teaching programming.


I. INTRODUCTION
Gamification is commonly known as ''the use of game design elements in non-game contexts'' [1].Over the last few years, it has gained momentum in several areas, including education.Specifically, a trend that has started to draw the attention of educators is the use of escape rooms in educational contexts.Escape rooms can be defined as ''live-action teambased games where players discover clues, solve puzzles, and accomplish tasks in one or more rooms in order to accomplish a specific goal (usually escaping from the room) in a limited amount of time'' [2].The use of escape rooms in educational contexts has proven capable of bringing many benefits for students, leveraging skills such as teamwork, leadership, creative thinking, and communication [3]- [8].
Moreover, teachers can create educational escape rooms (i.e.escape rooms specifically designed for learning purposes), which have the potential to bring additional valuable benefits for students.Since escape rooms are inherently a game concept, it has been suggested that the term ''gameful The associate editor coordinating the review of this manuscript and approving it for publication was Martin Reisslein .training'' applies even better for educational escape rooms than ''gamification'' does [9].Gameful training combines some of the key concepts of game design with sound teaching principles in order to foster students' problem-solving skills in a motivating way.One of the key elements of educational escape rooms is active learning, which is a form of learning that actively or experientally engages students in the learning process, requiring them to perform meaningful learning activities and think about what they are doing [10].By solving educational escape room puzzles, students put their knowledge of the course materials to the test and develop their skills in an active way.Active learning stems from the theory of constructivism, which claims that students construct knowledge rather than merely receive and store knowledge transmitted by the teacher [11].Specifically, social constructivism [12] advocates that knowledge is co-constructed and that individuals learn from one another.In this regard, since escape rooms usually involve participating in teams, they are a rather favorable stage for collaborative learning.Collaborative learning can refer to any instructional method in which students work together in groups toward a common goal [10].As mentioned, escape rooms usually present an objective on which students need to work together, requiring them to make adequate use of their time and resources in order to succeed.
Recent evidence suggests that the use of educational escape rooms can have a great positive impact on student engagement and learning in several fields [13]- [33].However, prior research fails to assess the impact of this sort of activities for teaching programming in terms of learning effectiveness.In a previous work [34], the authors provided empirical evidence that educational escape rooms for teaching programming are an effective way of fostering student engagement in a higher education setting.However, in said work, the learning effectiveness of the educational escape room for teaching programming was assessed solely by asking students their self-perceived increase in knowledge during the experience.Thus, the aforementioned research gap remains open to date.
This work provides, for the first time, evidence on the learning effectiveness of an educational escape room for teaching programming in a higher education setting, filling the gap identified in the literature.Additionally, further evidence on the effect of conducting an educational escape room on students' perceptions is also provided, along with an analysis of the relationships among students' learning effectiveness, perceptions and performance during the activity.
The article is structured as follows.Existing literature on educational escape rooms as well as on gamification activities and serious games for teaching programming is reviewed in the next section.Section III includes a comprehensive explanation of the conducted educational escape room.Section IV explains how the educational escape room experience was evaluated.Section V shows and discusses the results obtained from this evaluation.Lastly, Section VI finishes with the conclusions of the article and an outlook on future work.

II. RELATED WORK
Educational escape rooms constitute a fairly recent topic in academia.Despite their novelty, several experiences in a wide range of fields of knowledge have been reported in the literature.For instance, a number of researchers have reported on the use of educational escape rooms in nursing education, in which nurse students were required to prevent infections [13], perform cardiopulmonary resuscitation [16], collect blood samples [16], and report and evaluate lab results [13], [17].In the case of medical escape rooms, students have been put to the test in different areas, including surgery [19], dermatology [20], physiotherapy [21], toxicology [22], radiology [25], and pathophysiology [32].Similarly, educational escape rooms in the area of pharmacy have been reported in the literature, covering topics such as pharmacy management [14], dose calculation [18], [27], identification of medication errors [27], non-sterile compounding [29], as well as knowledge of diabetes [28] and infectious diseases [30].In the area of chemistry, prior research reported on the use of educational escape rooms for teaching chemical reactions [15] and analytical chemistry [24].Moreover, there are also works on the use of educational escape rooms for teaching cryptography [23], ICT competence [26], computer security [31], and computer networks [33].Lastly, in a previous study, the authors reported on an educational escape room for teaching programming [34].
The majority of the experiences reported in the aforementioned studies start by dividing students into teams and instructing them on the rules of the activity.In some experiences, the whole class participates at the same time, whereas in others, the escape room is conducted for only one team at a time.There is often a backstory that is revealed to students at the beginning of the escape room, usually by means of a video, which also discloses the goal that students need to accomplish in a given time limit.The goal may consist in escaping a room or achieving another specific objective.Thereafter, the countdown officially starts.Students may then begin to solve the different puzzles that make up the escape room, which lead them to accomplish the final objective.The puzzles may be of very different natures, and are usually very dependent on the field of knowledge.They usually intertwine leisure escape room puzzle mechanics (solving codes, opening locks, finding hidden elements, etc.) with domainspecific tasks (performing calculations, operating laboratory equipment, consulting information, interpreting results, etc.).Students are usually shown a countdown so they are always aware of the time they have left.Moreover, if students get stuck, teachers may give them a hint (on demand or when they consider it necessary), either for free or in exchange for something (a hint card, time, score points, etc.).All the teams that manage to accomplish the escape room goal before the time is over are considered to be successful in the activity.
Most of the studies previously reviewed in this section collected students' feedback by means of a survey after conducting educational escape rooms and agreed that these activities are perceived as fun and engaging by students [13]- [30], [34].In most works, self-perceived learning effectiveness was assessed by asking students whether they thought the activity helped them learn, to which most students replied affirmatively [16]- [31], [34].However, only four studies [27]- [30], all in the area of pharmacy, performed an objective empirical measurement of learning effectiveness, in order to determine if an actual increase in knowledge took place during the educational escape room.With this aim, in the works reported by [27]- [29], students were required to complete a pre-test prior to the activity and a post-test afterward, in order to analyze if there had been an increase in knowledge.In the educational escape rooms reported by [28], [29], results show that students obtained statistically significantly higher scores in the post-test than in the pre-test, suggesting a potential instructional benefit of these activities.Conversely, in the experience reported by [27], the results show that students scored better in the pre-test than in the post-test.The authors propose several reasons for the decrease in scores; one being that the tests were focused on specific clinical knowledge that was not adequately reinforced with the activity.Another reason identified by the authors was that the pre-test score accounted for a more significant percentage of the grade than the post-test did, resulting in students putting more effort into the former.In spite of these results, students reported that they perceived their skills had improved during the activity.Following a different approach, in the work presented by [30], students were divided into a control group and an experimental group.The former attended a case-based lecture on infectious diseases, whereas the latter participated in an educational escape room on the same topic.Results show that the control group obtained lower scores in the post-test than in the pre-test, while the scores of the experimental group remained the same.
Whereas the existing evidence on the learning effectiveness of educational escape rooms is scarce and inconsistent, evidence on the learning effectiveness of educational escape rooms for teaching computer programming specifically, and engineering and technology in general, is directly nonexistent to the knowledge of the authors.Notwithstanding, prior works have reported on the learning effectiveness of other gamified experiences for teaching this topic.For instance, previous research described the use of gamified Android applications for learning programming in C language [35] and for the web [36].Both studies found that using the applications successfully improved students' learning outcomes.Similarly, [37] showed that after adopting a gamified platform on a C programming course, students obtained significantly higher grades than before and the dropout rate decreased by more than half.Another work reported on the significant positive impact on students' performance of the use of leaderboards in short-length programming assignments [38].Furthermore, in the study conducted by [39], a control group of students learned logic and algorithms through traditional methods, and an experimental group learned the same content through gamification and affective recognition.Results show that students' learning outcomes were statistically significantly higher when taking into account the emotions of students and when they were motivated by means of gamification techniques.
Moreover, several studies have reported on the use of serious games (games designed for a primary purpose other than pure entertainment) for teaching programming.For instance, [40] used ''CMX'', a Massive Multiplayer Online Role Playing Game (MMORPG), for teaching C programming language concepts and found that students that played the game had a greater increase in learning that those who used a traditional Integrated Development Environment (IDE) instead.Another study [41] presented ''EleMental'', a serious game for learning about recursion, which proved to significantly increase students' knowledge and engagement.Lastly, [42] presented a multimedia adventure game for learning web programming, also obtaining very positive results in terms of learning effectiveness when comparing the increase in knowledge of students who played the game with those who attended a traditional lecture.
Overall, the studies presented suggest that gamification and serious games can be effective techniques for learning programming.Taking into account that educational escape rooms have proven to be an effective learning activity in other disciplines, it seems reasonable to presume that educational escape rooms may constitute a suitable way to learn programming as well.However, this specific application of educational escape rooms poses additional challenges for instructors since learning programming is hard for many students [43] and usually involves using digital resources, which can be complex to integrate into these immersive experiences.

III. DESCRIPTION OF THE ESCAPE ROOM EXPERIENCE
This section explains in detail the educational escape room analyzed in this work.

A. CONTEXT
The educational escape room was conducted in a programming course that is part of the Bachelor's Degree in Telecommunications Engineering from UPM (Universidad Politécnica de Madrid).This course is a third-year course that accounts for 6 ECTS (European Credit Transfer System) credits, equivalent to 150-180 hours of student work.In this programming course, students learn the basics of web development, including HTML, CSS, JavaScript, and more advanced technologies, such as node.js,express, and SQL.The aim of the conducted educational escape room was to reinforce the most important concepts covered throughout the programming course in a motivating and engaging way.
The educational escape room was offered to all the students in the course as an extra-credit activity that lasted two hours.Students who attended the escape room earned 0.15 points just for participating in it.The remaining 0.15 points were assigned in proportion to the degree of completion of the escape room puzzles achieved by the students.The maximum they could achieve was 0.3 points to be added to their final grade out of 10.The educational escape room was conducted in May 2019, a few days prior to the final exam.A total of 28 students attended the educational escape room.

B. DESIGN
The educational escape room was designed with the same approach reported by the authors in [34].This prior work describes in thorough detail the design principles followed and provides a comprehensive step-by-step explanation of the creation process.The main characteristics of the educational escape room analyzed in the present work can be summarized as follows: • Linear sequence: The puzzles of the escape room were arranged in a sequence in such a way that each puzzle unlocked the next one (section III-D describes each of the puzzles used in detail).Thus, students were required to solve the puzzles in a specific order.
• Pair-programming: Students were arranged in pairs with the aim of allowing them to take advantage of working as a team and enjoy the benefits of pair programming [44], [45].It should be noted that students were allowed to choose their own partner.In this regard, [46] found that, in pair-programming assignments, students performed equally whether the pairs were instructorassigned or self-selected, in spite of the fact that selfselection aided in student satisfaction.
• Hybrid puzzles: The designed educational escape room combined computer-based and physical puzzles with the objective of creating a highly engaging activity for teaching programming.
• Quiz-based hint approach: Educational escape rooms generally give hints on demand when students get stuck or when instructors consider it appropriate, sometimes applying time penalties.Instead of doing this, the hint approach adopted in the educational escape room analyzed in this work required students to earn the right to receive help by passing a five-question quiz on the course material delivered through a web application.
In order to receive a hint from the instructors, students had to get at least four out of five answers right in the quiz.They were allowed to attempt to get hints as many times as they wished, but the questions presented by the quiz application were different in each attempt.
• Large-enrollment: The educational escape room was designed in such a way that it enabled the participation of a large number of students at the same time.In order to accomplish this, the activity was hosted at a large computer laboratory and all the puzzles were designed to be replicated in an easy and inexpensive way.
• Convenient duration: The educational escape room was designed to last two hours, the typical duration of a computer lab session, giving students sufficient time to perform meaningful programming challenges, and allowing the course staff to easily replace a lab session with the educational escape room activity in the course instructional design.The educational escape room presented in this article was aimed at improving students' knowledge of the programming concepts taught in the course in a motivating and engaging way.There are four different ways in which students can learn by participating in the educational escape room: (1) solving the puzzles for which they need to understand and apply programming concepts, thus increasing and reinforcing their knowledge and skills; (2) answering the questions in the hint application which allow them to review some of the theoretical content of the course; (3) consulting the course materials that are available to them throughout the activity, and (4) working in pairs, allowing them to take advantage of pair-programming and learn from one another.

C. NARRATIVE
The overall theme of the educational escape room was science, a very popular theme used in 12% of escape rooms worldwide [2].Specifically, the overarching concept was helping create something (in this case, a cure), a topic used in 2% of escape rooms [2].The activity started by presenting the students with an introductory video (see Fig. 1) screened on the computer lab in which a person appears in a biology laboratory that has been closed due to biohazard.In the video, this person presents himself as a government agent and explains that an accident has taken place at the Severo Ochoa Molecular Biology Centre (CBMSO, in its Spanish acronym), releasing a deathly virus, and killing Dr. Gabriel Rivas, the researcher in charge of developing a vaccine against it.Another researcher has also become infected and is in a very serious condition.In order to help him recover, and before anyone else becomes infected, it is mandatory to prepare a vaccination shot, but the only one who knew how to retrieve the genetic code of the vaccine was the deceased.Luckily, he had developed a web application that he used for managing clinical trials, including the trial in which he found the cure for the deathly virus.However, although the database containing the trials survived the accident, the application did not.A preliminary, unfinished application was rescued from an old backup of Dr. Rivas's hard drive, along with some of his personal files.The application was developed using HTML, CSS, JavaScript, node.js,express, and SQL.The government agent resorts to the students, as the world's greatest experts in said technologies, in order to help him rebuild the application and gain access to the genetic code of the vaccine.They have two hours (the duration of the escape room activity) to perform this task and retrieve the genetic code of the vaccine before it is too late for the infected researcher.

D. PUZZLES
Table 1 provides an overview of all the puzzles that the educational escape room activity comprises indicating, for each one of them, the learning objectives addressed and the puzzle mechanics used.The terminology employed for naming these puzzle mechanics has been primarily extracted from [2], although new terms have been introduced.Furthermore, a flow chart of the escape room puzzles is provided in Fig. 2. The next subsections describe all the puzzles in detail.

1) PUZZLE 1: ACCESSING THE CLINICAL TRIAL APPLICATION
Each team participating in the educational escape room was assigned a laboratory desk with a computer.On top of each  desk was an envelope containing a summary of the case, which provides a link to a GitHub repository to which Dr. Rivas's hard drive files were uploaded.Knowing how to download code from this platform to their own computer constitutes the first challenge of the escape room for the students and, from the educational point of view, the first learning objective.
Once students have downloaded the content of Dr. Rivas's hard drive, they should find a folder containing the clinical trial application, and identify that it is indeed a web application built with node.js and express, such as the ones with which they have worked in class throughout the course.Their next goal is to properly install all the dependencies and to launch the application by running the pertinent commands.When they do so correctly, they may open the browser and enter the URL provided after running the start command, landing on the clinical trial application login page.In this login page, the username is already typed in, but students need to find out the password.In the case summary provided to them, it says that the password is Dr. Rivas's wife's name.If they look through the personal files in his hard drive they will find a folder containing old photos.The title of one of them, an old picture of a girl, reads: ''My wife Julia''.By inserting this name as the password in the login form, they gain access to the second screen of the application.Searching for hidden things is one of the most widely used resources in leisure escape rooms, accounting for 78% of escape rooms worldwide [2].So is noticing something ''obvious'' in the room, such as the text revealing the password hint, which is used in 49% of escape rooms [2].

2) PUZZLE 2: FINDING THE DEATHLY VIRUS
The second screen evinces the fact that the application is incomplete.It shows a ''TO-DO'' note left by Dr. Rivas which says that the screen needs to show the complete list of viruses with all their details.To perform this task, students need to find out which controller in the web server is in charge of handling the virus list page.Once they find it, they discover that the query to the database has already been written and that an array containing the complete list of viruses is passed to a template file.They need to find and modify this file in order to create a new list item for each virus.Students need to know EJS syntax in order to generate this list dynamically.A sample item is provided so they know which information they need to show.
Once students get the complete list of viruses, they need to find out which one of all the viruses is the one that infected Dr. Rivas.For the sake of authenticity, the viruses stored in the database of the clinical trial application (and therefore the ones shown to the students) were extracted from a real virus database.In order to solve this puzzle, students need to inspect Dr. Rivas's hard drive again and find a research article that he and his colleagues were writing about the vaccine of the deathly virus.Once they discover the name of the virus by reading the article, they can find it on the virus list and access its details through the clinical trial web application.

3) PUZZLE 3: GAINING ACCESS TO THE TRIALS
The detail page of the deathly virus provided by the clinical trial application shows all the basic information of the virus, as illustrated in Fig. 3. Since the virus is extremely dangerous, in order to access the trial information, it is necessary to introduce a passcode.In the observations section of the page,  there is a note written by Dr. Rivas saying that he sent the passcode to his colleagues via email.At the navigation bar of the web application, there is a button that gives access to the email inbox.The password is revealed in the observations section, but students need to search the username in the hard drive in order to log in.They can find it written down on Dr. Rivas' agenda.Once they gain access to the email inbox, they need to check the message history and see that Dr. Rivas wrote an email to his colleagues saying that he put all the necessary information to figure out the code inside locker 140 of the Telecommunications Engineering School at UPM (students' school).Students need to leave the computer lab and find locker 140, inside of which there is an envelope for each team.After fetching the envelope, they must return to the lab and see what is inside.Once they open it, they see two pieces of paper that they need to use in no specific order to find out the passcode.
One piece of paper contains a periodic table like the one shown in Fig. 4. At the bottom of the page, there is a clue in which students are suggested that they need to determine the correct order of a set of five symbols (a flask, a pill, a DNA chain, a molecule, and an atom).In one of the cells of the periodic table, they can find the flask symbol instead of the Gold element, suggesting that each symbol corresponds to a different chemical element.Thus, students need to find the missing symbol-element equivalences and figure out the correct order so as to form an ordered sequence of five chemical elements.If they were paying attention while they were exploring Dr. Rivas's hard drive in the previous puzzles, they would have noticed the equivalence of three other symbols hidden in his personal photos and documents: the molecule stands for Gallium, the pill stands for Ytterbium, and the DNA chain stands for Oxygen.Symbol substitution and searching for objects in images are very popular puzzle mechanics used in 47% and 43% of escape rooms worldwide respectively [2].
If students turn the page over, they will find several pieces of QR codes.One of these code pieces is distorted (it is not a valid QR piece) and contains the remaining symbol: the atom.If students put the paper through backlighting, they will see that the atom symbol is placed exactly on the reverse side of the Sodium element on the other side of the paper, finally unveiling the missing symbol-element equivalence.Using light is a widely adopted puzzle technique used in 54% of escape rooms worldwide [2].
The remaining piece of paper is an excerpt from a JavaScript program that deals with promises that output the aforementioned symbols.The difficulty of promises lies in the fact that they are asynchronous, so the order of execution of the program sentences is not sequential.Students need to find out the order in which the symbols are outputted by manually executing the program, putting to the test their knowledge of promises.
If students go back to the paper containing the QR pieces, they will realize that they can be combined in different ways in order to form two different complete QR codes, of which only one is valid.Students can obtain the valid QR code by folding the paper in an appropriate way.Shape manipulation is a resource used in 11% of escape rooms [2].If they scan the valid QR code, they will be taken to an online interactive application of a periodic table.By clicking on the elements (that correspond to the different symbols) in the interactive application following the correct sequence (obtained from the JavaScript program), the application will unveil a 5-digit passcode that they can introduce in the virus page in order to access the trials.

4) PUZZLE 4: FINDING THE CURE
The trial page of the virus contains hundreds of different trials.By clicking on each one of them, students can see the data and output of the selected trial, indicating whether it was successful or not.There is a button that reads ''Show successful trials only''.If students click on it, they can see that the trial list remains the same, but another ''TO-DO'' note appears saying that the database query that retrieves the trials needs to be fixed in order to show only the successful ones.Students need to find the controller in charge of performing said query and add a SQL ''WHERE'' clause in order to filter out the unsuccessful trials.If they refresh the page, they will see that only one trial was successful.By clicking on it, they can see its corresponding report.
The page of the successful trial shows a form for downloading its output, which contains the genetic code of the vaccine for the virus.At this point, students think they are already done with their task but, when they hit the download button, nothing happens.Students need to debug the code to see that the error lies in the fact that the form input does not have a name, so it cannot be correctly read on the server side.By providing the input with the same name that the controller is expecting on the server, students are able to correctly submit the form and download a file with the genetic code of the vaccine.Thanks to this code, a vaccine can be made in time to save the infected researcher and many other lives.

IV. EVALUATION METHODOLOGY
The aim of the present work is to analyze the learning effectiveness of an educational escape room for teaching programming, students' perceptions towards the activity, and the relationships among these factors and students' performance during the escape room.Three instruments were used: (1) a pre-test and a post-test to measure students' increase in knowledge, (2) a survey to collect their opinions, and (3) a web platform for automatically recording data on students' interactions during the activity in order to obtain information on the students' performance in the educational escape room.
In order to accurately measure the learning effectiveness of the educational escape room carried out, a pre-test was conducted just before the start of the educational escape room, and a post-test was conducted right afterward.Both tests contained the exact same ten multiple-choice questions on the course materials.The correct answers were not revealed to students until after completing the post-test (i.e.no feedback was provided during the pre-test preventing students from memorizing the answers).In this regard, it is worth mentioning that students did not know they were taking the posttest until the end of the activity.The knowledge required to solve the questions on both tests was akin to the knowledge that students needed to have in order to solve the puzzles of the educational escape room.The ten multiple-choice questions in the pre-and post-test assessed the entirety of the learning objectives addressed in the educational escape room.The time limit for solving each test was 10 minutes.Answering the 10 questions correctly required students not only to remember information, but also to clearly understand the main concepts covered in the activity, to analyze programming code fragments, and to know how to apply the acquired knowledge to solve specific programming problems.Therefore, the pre-and post-test allowed measuring not only knowledge acquisition but also knowledge understanding and application.Although the questions of the pre-and post-test and the ones presented by the hint application were related to the same set of learning objectives, no question from the pre-and post-test was included in the hint application (i.e. the questions of the pre-and post-test were different from the ones presented by the hint application).In the pre-and post-test, students were awarded 1 point for each question they answered right and were subtracted 1/(N-1) points for each question they answered wrong, N being the number of options in each multiple-choice question.They were allowed to leave answers blank with no penalty.The maximum score achievable by students was 10 and the minimum score was 0. This approach was aimed at preventing students from completing the questionnaires randomly.The pre-test and posttest results did not count towards students' final grades in order to avoid cheating and unexpected behaviors such as the one reported by [27], in which the pre-test accounted for a much more significant percentage of the grade than the post-test did, resulting in students putting more effort into the former.
Additionally, students' opinions on the performed educational escape room were collected through a survey that was conducted immediately after the termination of the post-test.This survey included some initial demographic questions, a set of closed-ended questions addressing students' general opinion and acceptance of the activity, and a list of statements with which they needed to agree or disagree using a 5-point Likert scale.These questions were aimed at assessing students' perceived learning effectiveness and attitudes towards the use of the educational escape room as a learning activity, as well as students' thoughts on the design (difficulty, type of puzzles, duration, hint approach employed) and organization of the escape room, the team dynamics, the immersivity of the experience, and whether students preferred the escape room over a regular computer lab session.At the end of the survey, there was a space in which students could leave suggestions, complaints, and other comments.
Lastly, a web platform developed by the teaching staff was used in order to automatically record data on relevant student interactions during the course of the educational escape room.Specifically, the following data were collected: the puzzles solved by each team, the hints requested and received by each team, and the time required by each team to solve each puzzle and to complete the escape room (in case of success).These data allowed to retrieve fine-grained information on students' performance during the educational escape room, rather than just a boolean outcome (whether they successfully completed the activity or not).Then, based on these data, relationships among this performance and students' learning effectiveness as well as students' attitudes were analyzed through Spearman's correlation analysis.
By means of the evaluation instruments utilized in this study, the authors expected to confirm the hypothesis that educational escape rooms for teaching computer programming can lead to positive impacts in terms of students' learning outcomes, as well as to provide further proof that these learning activities can be a great way to foster student engagement.

V. RESULTS AND DISCUSSION
A total of 28 students participated in the escape room.All of them were grouped in pairs, so there was a total of 14 teams.All participating students completed the pre-test, the post-test, and the survey.Of the 28 students who participated in the activity, 24 were men and 4 were women.All students were between 20 and 24 years old, being 21.1 the mean age, with a standard deviation of 1.2.In order to further characterize the sample, students had to agree or disagree with two statements using a 5-point Likert scale (1 Strongly disagree -5 Strongly agree).First, when inquired about the difficulty of the course, they neither agreed nor disagreed with the statement that it was easy (M = 2.9, MED = 3.0, SD = 0.8).Moreover, most students expressed that they like to play games (M = 4.3, MED = 4.0, SD = 0.7).

A. LEARNING EFFECTIVENESS
Table 2 shows the results of the pre-test and the post-test, including, for each test, the mean (M), median (MED), and standard deviation (SD).The average score for the pre-test was 3.3 (MED = 2.8, SD = 1.6) on a scale of 1 to 10, whereas the average score for the post-test was 5.4 (MED = 5.8, SD = 2.3).The average increase in scores was 2.1 (MED = 1.3, SD = 2.6), which implies an average learning gain of 21%.The Shapiro-Wilk test of normality distribution was used to verify that the data from the pre-and post-test scores follow the normal distribution.The difference in scores was found to be statistically significant (p-value < 0.001) at an alpha level of 0.05 when a paired-samples t-test was performed, showing that students experienced a statistically significant increase in knowledge as a result of the activity.Moreover, in order to determine the magnitude of the difference between the scores achieved by the students in the post-test and the pre-test, the Cohen's d effect size [47] was calculated.When using Cohen's d, a value of 0.2 indicates a small effect size; a value of 0.5, a medium one, and a value over 0.8, a large one.The value obtained for Cohen's d was 0.73, representing a medium to large effect size.
It should be remarked that the standard deviation of the post-test was found to be notably higher than that of the pretest, indicating bigger differences in the scores obtained by students in the former.One of the reasons for this is that students who did not complete all the puzzles in the escape room only improved their knowledge in the areas that were covered by the puzzles they did solve, thus leading to an increased heterogeneity in the post-test results.
A foreseen result was that no significant difference was found between the increase in knowledge achieved by men and women.Likewise, no correlation was found between students' increase in knowledge and their opinion on the difficulty of the course or their prior interest in games.This latter fact indicates that educational escape rooms can be effective learning activities for all students, regardless of their liking for games.
It could be argued that the results drawn from the preand post-test would not be valid if students had talked about the questions of the pre-test during the educational escape room.In this regard, it should be mentioned that, based on teacher observation, during the activity students were completely focused on solving the puzzles and trying to succeed in the escape room, and they did not talk about the pre-test at all.This was an expected behavior because time is an extremely valuable resource in an escape room and students could not obtain any benefit (e.g. a higher grade) from dedicating time to talking about the pre-test.Therefore, it can be stated that the results obtained from the pre-and post-test allow drawing trustworthy conclusions on the learning effectiveness of the educational escape room conducted.Nevertheless, the authors recognize that conducting a more extensive pre-and post-test might have allowed measuring learning effectiveness more accurately.However, the authors reckon that longer tests (the time limit for solving the tests in this study was 10 minutes) would have been to the detriment of student motivation.
The findings reported in this section are consistent with those of [29], who performed an educational escape room in the area of pharmacy and observed a significant improvement in the scores achieved by the students in the post-test (median of 83/100) compared to those of the pre-test (median of 50/100).The difference in medians between the post-and pre-test in said study was 33/100, whereas in the present study this difference was 3/10, which is surprisingly similar.Our results are also consistent with the ones reported by [28], who conducted an educational escape room in the field of pharmacy too and found that students' mean score for the post-test (81/100) was statistically significantly higher than students' mean score for the pre-test (56/100).The difference in means in [28] was 25/100 which is very close to the difference in means found in the present study (2.1/10).However, these authors identified that students' mean score for the pretest explained a statistically significant proportion of variance in the post-test score (Pearson's r = 0.21), whereas in the present work no statistically significant correlation was found between both scores.
Moreover, the findings observed in the present study match those observed in earlier studies that have examined different gamification approaches and serious games for teaching programming [35]- [42], obtaining similar results in terms of learning effectiveness.Therefore, based on these findings, it can be stated that educational escape rooms are yet another gamified activity that is capable of yielding meaningful instructional benefits.

B. STUDENTS' PERCEPTIONS
Table 3 shows the results of the student survey conducted after the educational escape room.
The results of the survey conducted in this study show that students had a very positive overall opinion on the educational escape room (M = 4.6, MED = 5.0, SD = 0.6) and thought it was a fun experience (M = 4.4, MED = 4.5, SD = 0.6).A medium correlation was found between students' prior interest in games and their opinion on the escape room (Spearman's ρ =0.3, p-value = 0.1).However, this correlation was not found to be statistically significant at an alpha level of 0.05.Students also thought the escape room was a very immersive experience (M = 4.3, MED = 4.0, SD = 0.7), and not too stressful (M = 2.0, MED = 2.0, SD = 1.1).Moreover, all of the students stated that they would like other courses to include activities like this and that they would recommend other students to participate in the escape room.The satisfactory outcomes obtained for student engagement provide further proof that educational escape rooms are an excellent way to foster motivation in programming courses, as shown in the authors' previous work [34], and as anticipated by many prior studies in other disciplines [13]- [30].
In general, students were fairly confident that the escape room allowed them to improve their knowledge (M = 4.1, MED = 4.0, SD = 0.8), which is a finding consistent with the results obtained in the learning effectiveness evaluation reported in the previous section, which was conducted by means of a pre-test and post-test.In this regard, it is worth indicating that there is a positive correlation (Spearman's ρ = 0.3, p-value = 0.14) between the measured learning effectiveness (i.e. the difference between post-and pre-test scores) and the learning effectiveness self-reported by students, indicating that many students who thought they learned actually did, and vice versa, although this correlation was not statistically significant at an alpha level of 0.05.Results show that the 18 students who stated the educational escape room helped them learn proved so by obtaining a greater score in the post-test than in the pre-test.Conversely, three students reported they had not learned and they indeed did not show any increase in knowledge.Lastly, five students reported that they learned but the test results did not back up this statement, whereas the remaining two students did not think they learned despite the fact that the test results showed they did.These results are summarized in Table 4. Regarding these results, it should be clarified that students were considered to feel they had learned when they rated the corresponding item in the survey with a score greater than three and, in the case of measured learning effectiveness, students that were considered to have learned were those who obtained a difference in scores greater than zero between the post-test and the pre-test.
When compared with the computer lab sessions performed in the course, students notably agreed that they learned more in the escape room than in a lab session (M = 4.0, MED = 4.0, SD = 0.8), and they strongly agreed that they liked the former better (M = 4.7, MED = 5.0, SD = 0.7).These results are consistent with those of [14], [30], [34], although this specific comparison was only addressed in [34].In light of these results, higher education teachers of programming courses could consider replacing some lab sessions and other similar hands-on activities with effective educational escape rooms.Another finding of this study is that students who stated they prefer the escape room to a lab session showed a greater increase in knowledge, since a medium statistically significant correlation was found between these two variables (Spearman's ρ = 0.4, p-value = 0.04).
Students slightly disagreed with the statement that the escape room was too hard (M = 2.6, MED = 3.0, SD = 1.1).A medium statistically significant correlation (Spearman's ρ = 0.4, p-value = 0.04) was found between students' thoughts on the difficulty of the escape room and their opinion on the difficulty of the course, indicating that students who found the course difficult also thought so about the activity.However, no correlation was found between students' opinion on the difficulty of the escape room and learning effectiveness (neither the one measured by means of pre-and post-tests nor the one self-reported by students).Furthermore, most students agreed that the difficulty of the escape room lied in mastering the course materials (M = 3.8, MED = 4.0, SD = 0.9), and not so much in the puzzle mechanics.They somewhat agreed that they were sufficiently prepared to succeed in the escape room (M = 3.4,MED = 3.5, SD = 1.3).Nevertheless, no correlation was found between students' self-perceived preparedness and learning effectiveness.
Regarding the organization of the educational escape room, students agreed that it was well organized (M = 4.5, MED = 5.0, SD = 0.6) and that it had an appropriate duration (M = 4.1, MED = 4.0, SD = 1.1).They also agreed that the activity was sufficiently supervised (M = 4.5, MED = 5.0, SD = 0.6), and that they received sufficient initial guidance (M = 4.0, MED = 4.0, SD = 0.9).They slightly disagreed with the statement that they wished they had received more help during the escape room (M = 2.5, MED = 2.0, SD = 1.3).A medium negative correlation was found between this variable and students' overall opinion of the activity (Spearman's ρ = −0.3,p-value = 0.13), meaning that students who thought they received enough help were the ones with a better opinion on the educational escape room.However, this correlation was not found to be statistically significant.
With regard to the design of the escape room, the results indicate that students enjoyed the combination of physical and digital puzzles (M = 4.8, MED = 5.0, SD = 0.4), and showed no preference for either one (M = 3.0, MED = 3.0, SD = 1.1).This finding indicates that the creators of the escape room managed to strike a sound balance between computer-based puzzles and puzzles that involved tangible resources.Moreover, students liked participating in the escape room in pairs rather than alone (M = 4.6, MED = 5.0, SD = 0.7) and they stated they would not have preferred to form larger teams (M = 2.1, MED = 1.5, SD = 1.2).Students agreed that all the members of the team were equally involved in solving the different puzzles (M = 4.0, MED = 4.0, SD = 1.1), thus fulfilling the objective of using a pairprogramming approach in the activity.Based on these results, it becomes clear that having students work in self-selected pairs is a suitable approach for educational escape rooms aimed at teaching programming.Lastly, students agreed with the statement that the hint approach employed was adequate (M = 3.7, MED = 4.0, SD = 1.3) meaning that, although using a quiz application as the hint mechanism can seem cumbersome at first, students learned how to make the most of it.
At the end of the survey, students were asked to pose suggestions, comments and/or complaints.Overall, most students left very positive comments that gave reason to believe that they found the activity to be entertaining, interesting and didactic.Moreover, several students thanked the staff for taking the time and effort to conduct such an innovative experience.A couple of students complained about the difficulty of one of the puzzles, stating that they would have never guessed it if it were not for the hints received.Lastly, one student suggested that the hint application should reveal the right answers after completing each quiz.However, the teaching staff purposely followed the approach mentioned earlier in order to prevent students from memorizing the answers instead of giving them some thought.

C. STUDENTS' PERFORMANCE IN THE ESCAPE ROOM
As shown in Fig. 5, all 14 teams were capable of solving the first two puzzles; 11 solved the third one too, and only 7 (50%) solved all the puzzles, thus succeeding in the escape room.Albeit very little was found in the literature on the escape rate of educational escape rooms, the range of values reported is very wide.For instance, in the educational escape room reported by [18], not a single team was capable of escaping, whereas the one in [32], stated that all of the teams did.There are also intermediate figures such as the ones reported by [29] (75% of teams), and [16] (67%), and [23] (50%).The mean time to complete the escape room was 1 hour and 53 minutes (SD = 5 minutes), and all the teams that managed to complete the activity did so in the last 20 minutes.Overall, these data suggest that the escape room was well designed in terms of difficulty and duration.
The most noteworthy result to emerge from these data is that a large statistically significant correlation (Spearman's TABLE 5. Summary of team attempts to obtain hints (N = 14).ρ = 0.6, p-value < 0.001) was found between measured learning effectiveness and the percentage of completion of the escape room puzzles achieved by the students.This implies that the students who managed to solve more puzzles were naturally the ones who learned the most during the escape room.This finding suggests that, during educational escape rooms, the more puzzles students work on, the more they learn.However, among the students who solved all the puzzles, no correlation was found between the escape time and the increase in knowledge.
Additionally, students who solved more puzzles were the ones that professed greater self-perceived learning effectiveness in the survey (Spearman's ρ = 0.5, p-value = 0.02), meaning that they also felt they learned to a greater extent than their peers.Moreover, students who solved more puzzles also had a better overall opinion on the activity (Spearman's ρ = 0.3, p-value = 0.07), although this relationship was not found to be statistically significant, and definitely preferred the escape room over a laboratory session (Spearman's ρ = 0.6, p-value < 0.001).However, this is in contrast with earlier findings [29], in which students' increase in knowledge and positive perception were independent of their teams' escape success.
Unsurprisingly, students who solved fewer puzzles stated they did not feel they obtained enough help during the activity (Spearman's ρ = −0.6,p-value = 0.02).This fact evinces that monitoring or predicting students performance during educational escape rooms could be very useful for enhancing the learning experience by driving interventions such as delivering free hints to certain teams.
Regarding hints, Table 5 shows the average number of quiz attempts to obtain hints performed by each of the 14 teams, along with the average number of those attempts that were successful (students passed the quiz and obtained a hint) or failed (students had to keep trying).Overall, there were 108 attempts to obtain hints, of which 25% were successful and 75% failed.
Regarding the impact of student interactions in the hint application on learning effectiveness, no statistically significant correlation was found between measured or selfperceived learning effectiveness and the total number of quiz attempts, the total number of failed attempts, or the hints obtained.One plausible reason for this is that after completing a quiz in the hint application, students were not shown the correct answers.If feedback had been provided for each question, students would have probably learned more when taking the quizzes, but this option was discarded because it would have allowed students to easily memorize the answers, allowing them to complete future quizzes from memory, hence obtaining unlimited hints.Despite the fact that no statistically significant relationship was found between learning effectiveness and the number of attempts performed to obtain a hint, it seems reasonable to consider that students learned when struggling to answer the questions.However, based on the results obtained, it cannot be asserted to which extent the hint approach used in the educational escape room reported in this study had any impact on students' learning.Nevertheless, we reckon that if the educational escape room had used a traditional hint approach (e.g.giving free hints to teams when they got stuck), the learning effectiveness would have been worse.
Since the quiz-based hint approach used in the educational escape room allowed students to perform unlimited attempts to get hints, a reasonable concern related to this approach was the possibility that students were continuously asking for help instead of facing the puzzles.Nevertheless, the data on students' interactions recorded during the escape room proves that having to answer a quiz prevented students from continuously requesting hints.On the one hand, this is evinced by the fact that the average number of attempts per team was 7.7 (on average, each team attempted to obtain a hint once every 15 minutes), meaning that students did not spend all of their time trying to get a hint.On the other hand, each team obtained 1.9 hints on average, which seems like a reasonable number.Previously reported escape room experiences granted a similar number of hints to students [13], [18], although it should be taken into account that the duration of these experiences were 60 and 30 minutes respectively.
As expected, students who failed quiz attempts for obtaining a hint more often were the ones who agreed to a greater extent with the fact that the escape room was too hard (Spearman's ρ = 0.6, p-value = 0.002) and that they wished they had received more help during it (Spearman's ρ = 0.7, p-value < 0.001).This was an expected finding because, on the one hand, the fewer hints a team obtained, the more difficult it became for it to succeed in the escape room, and, on the other hand, students who had trouble correctly answering questions related to course materials did also have trouble solving the escape room puzzles, whose difficulty mainly relies in mastering these materials.Furthermore, we expected that students who had experienced a higher failure rate when attempting quizzes in order to obtain hints were the ones that scored worse on the pre-test, which would mean that students who struggled the most with quizzes were the ones who had a lesser knowledge of the course materials.However, only a small non-significant correlation was found between these two variables (Spearman's ρ = −0.1,p-value = 0.8).
When students got stuck in a puzzle during the educational escape room, they had to decide whether to spend their time attempting a quiz in pursuit of a hint or to continue trying to solve the challenge.Therefore, each team had to find a balance between time spent trying to get hints and time spent solving puzzles.On the one hand, no correlation was found between the number of attempts to obtain hints performed by each team and the number of puzzles solved.This means that teams who dedicated more time to attempting to get hints were not necessarily the ones who performed better in the activity.There are two main reasons that explain this result.Firstly, the more time a team dedicates to attempting to obtain hints, the less time it has left to solve puzzles, which is the main aim of the educational escape room.Secondly, a team can perform a significant number of attempts and obtain few or zero hints, taking little advantage or wasting very valuable time.In this regard, it should be remarked that a medium to large correlation was found between success rate in the quiz application and the number of puzzles solved (Spearman's ρ = 0.6, p-value = 0.001), revealing that those teams that performed better when answering the quizzes outperformed their peers in the educational escape room.On the other hand, the data recorded indicate that hints played an essential role in the outcome of the activity: all the teams that succeeded at the escape room obtained at least one hint, whereas none of the teams that did not manage to get a hint succeeded at the escape room.Thus, no team was capable of solving all the puzzles without obtaining at least one hint, confirming that dedicating some time to getting hints was essential for teams to succeed.In spite of this fact, the existing correlation between the total number of hints obtained by each team and the number of puzzles solved was small and not statistically significant (Spearman's ρ = 0.2, p-value = 0.44).
To the knowledge of the authors, no previous study has investigated the impact of hints on students' performance in educational escape rooms.Therefore, the findings reported in this section are another unique and highly valuable contribution of this work.

VI. CONCLUSION
This paper reports the results of a case study in which an educational escape room was conducted in a programming course at a higher education institution.The main purpose of this study was to analyze the learning effectiveness, students' perceptions and students' performance on the educational escape room by means of three different instruments: a pretest and a post-test, a student survey, and a web platform that automatically records student interaction data during the activity with the aim of studying the relationships among these data and learning effectiveness as well as students' perceptions.
The difference in scores between the post-test and the pretest was found to be statistically significant with a medium to large effect size, showing that students experienced an increase in knowledge as a result of the educational escape room.This finding confirms the initial hypothesis of the present study that educational escape rooms are an effective way to teach programming in higher education, as foreshadowed by the results shown in prior works that examined the use of educational escape rooms in other disciplines [28]- [30], and those of other works that examined the use of serious games and other gamification approaches for teaching programming [35]- [39].
Additionally, the results obtained from the student survey conducted after the activity show that students had very positive attitudes towards the educational escape room and thought it was beneficial for their learning.These results indicate that educational escape rooms are highly engaging learning activities that can be used to increase student motivation in programming courses.These results, which are in accordance with those obtained in prior works that examined students' attitudes towards educational escape rooms in other disciplines [13]- [30], corroborate the results reported by the authors in a previous work [34] and provide further evidence that well-designed educational escape rooms for teaching programming are perceived by students as effective and motivational.The present study also suggests that students preferred the conducted educational escape room over traditional computer lab sessions and that they perceived the learning outcomes somewhat higher in the former.Previous works have provided similar findings [14], [30], [34], although this specific comparison was only addressed by [34].
Lastly, several noteworthy findings were revealed in this work by analyzing students' interactions during the educational escape room.The most obvious finding to emerge from the analysis is that students who learned the most in the escape room were those who managed to solved more puzzles.However, no correlation was found between completion time and increase in knowledge.Moreover, another relevant finding was that the number of hints obtained by each team did not have a statistically significant impact on learning effectiveness.Neither was a significant correlation found between the number of hints obtained and student performance in the escape room, albeit no team was capable of solving all the puzzles without receiving at least one hint.
In summary, the results of this work provide, for the first time, empirical evidence that educational escape rooms for teaching programming can have positive impacts on student learning outcomes and engagement in higher education settings.Notwithstanding, the authors acknowledge the reduced number of participants in this study as a noteworthy limitation.
An interesting future work would be to evaluate the learning effectiveness of educational escape rooms for teaching programming through a randomized control trial using a pre-test/post-test control group design, in which the experimental group participated in an educational escape room, and the control group in a lab session covering the same content.In this regard, it would also be interesting to compare the learning effectiveness and students' behavior of an educational escape room conducted using different hint approaches.Lastly, given the valuable findings emerged from the data recorded by the web platform used in this study, the authors plan to further develop it in order to produce a system capable of allowing teachers to fully manage and monitor educational escape rooms.

FIGURE 1 .
FIGURE 1. Screenshot of the introductory video screened to students at the beginning of the escape room.

FIGURE 2 .
FIGURE 2. Flow chart of the escape room puzzles.

FIGURE 3 .
FIGURE 3. Screenshot of the virus detail page in the clinical trial application.

FIGURE 4 .
FIGURE 4. Periodic table that students receive inside the envelope in Puzzle 3.

FIGURE 5 .
FIGURE 5. Number of teams that solved each puzzle in the escape room.

TABLE 1 .
Summary of the escape room puzzles.

TABLE 2 .
Results of the pre-test and the post-test (N = 28).

TABLE 3 .
Results of the student survey (N = 28).

TABLE 4 .
Contingency table for measured and self-perceived learning effectiveness.