An Innovative Multi-layer Gamification Framework for Improved STEM Learning Experience

Lately, gamification (i.e., employing game-design elements and game principles in non-game contexts) has gained massive popularity and widespread usage in various areas, including education. However, gamification deployment in education in general and remote education in particular still faces many challenges that mainly influence user quality of experience. These challenges include lack of dedicated communication-based systems, potential additional load on teachers, absence of customization and personalization for users, and no support for advanced technology-enhanced learning (TEL). This paper investigates the use of gamification for networked delivery of science, technology, engineering and mathematics (STEM) subjects. It proposes an innovative gamification framework, the NEWTON-enhanced gamification model (N-EGM), which was designed as part of the European Horizon 2020 project NEWTON. The paper also describes the proposed N-EGM model deployment in the gamification engine of a real learning management system and its associated communication and networking solution. The communication support provides easy-to-use gamification configuration functionality and efficient data collection and processing in a heterogeneous technology context. Finally, the paper evaluates the proposed N-EGM model as part of a NEWTON project pilot deployed in a Romanian school. The results demonstrate the effectiveness of the proposed gamification solution in improving both students’ learning experience and their engagement, while also increasing student knowledge gain.


I. INTRODUCTION
Educators are very much interested in new ways of increasing student motivation and stimulating learner engagement. A better-engaged individual learns more and faster and retains better the information learnt. Alongside proliferation of learning technologies into classrooms, we have seen an increase in game-based learning as a mechanism to bring together an aspect that students love (e.g. playing video games) with achievement of pedagogical objectives. The understanding of game-based learning has evolved signifi-cantly in recent years and the use of video games within the classroom and beyond is now commonplace. Beyond this, we are beginning to see how gamification, a novel avenue of research, is employed in education. Gamification is distinct from game-based learning and involves application of game rules, game mechanics and game principles in the learning process. When considering gamification in a learning context, the interest is in exploring how ideas of accumulating points, setting and achieving targets, working in competitive teams and receiving rewards can be utilised as VOLUME 9, 2021 a mechanism to stimulate student engagement and drive up the achievement of learning objectives.
In the current technology-enhanced learning (TEL) context, and especially in these pandemic times, increasing number of learners are involved in some form of online learning, whether part of formal classroom activities, or of massive open online courses (MOOC), in in-formal or nonformal learning settings. Many of these learning experiences are managed through some form of a learning management system (LMS).
Although gamification has been experimented in diverse pedagogical contexts by some researchers, gamification was implemented either by plugins to existing LMSs or manually. This is highly inefficient and results in lack of support for various types of TEL materials and situations. It is in the axis between the learner experience with a LMS and the application of gamification that the EU Horizon 2020 project NEWTON 1 (Networked Labs for Training in Sciences and Technologies for Information and Communication) sits. One of NEWTON project's goal was to "gamify" the entire learning platform so that every learning experience (interactions with traditional learning materials; interactions with various TEL materials; quizzes; assignment submissions, etc.) can be included in a gamified framework.
In this context, this paper makes the following four contributions. First, it introduces the novel NEWTON-Enhanced Gamification Model (N-EGM) which employs personalisation/adaptation, game-based learning and socialisation in conjunction with gamification in order to generate a unique social-personalized-gamified learner experience. This model has been developed as part of the NEWTON project and has been integrated into the NEWTELP Platform 2 (NEW Technology Enhanced Learning Platform), the project's newly designed LMS. N-EGM brings gamification to the context of STEM teaching and learning and enables integration of gamification in a LMS. Crucially, the paper discusses the outcomes expected from this integration process.
Second, this article describes a new gamification engine, which provides easy-to-use gamification elements, mechanisms and rules and enables efficient monitoring and application of gamification rules during students' learning process.
Third, the communication mechanism designed as a bridge between gamification and various technology-enhanced learning content in the NEWTELP Platform, including 2D/3D serious games, Virtual Reality (VR) applications and adaptive multimedia/multisensory media content is described. This mechanism provides an unified approach to collect, process and distribute efficiently data to/from heterogeneous technologies across the NEWTELP Platform.
Last, the results from one of NEWTON project's pilots which investigated the effects of using the proposed gamification framework in real life STEM education are presented.

II. GAMIFICATION IN STEM EDUCATION A. GAMIFICATION FRAMEWORKS
The term "gamification" refers to applying game elements and mechanisms in a non-game context. In many business sectors, the user experience designers have already adopted gamification concepts to maximize the engagement of users during their personal experience. In this context, gamification has also found a great application in the educational domain, by creating a new paradigm of pedagogy where the use of entertaining elements in the content and courses definition is key to motivating the learners. Some known frameworks for properly designing gamified experiences are: (i) Mechanics, Dynamics, Aesthetics (MDA) [1], (ii) Design Six (D6) [2] and (iii) Octalysis [3].
The MDA framework considers in conjunction the particular elements of a game (i.e. game rules), run-time behavior, inputs and outputs (e.g. complete levels) and desirable emotional responses sequentially. Two different MDA implementation perspectives are identified: design and player, inducing or starting from player emotions, respectively.
The D6 framework introduces a six-step design process which includes: defining business objectives, delineating target behaviors, describing the players, devising activities, including fun and deploying all required mechanics and game components. D6 enables player actions based on his/her own motivation and system feedback, while allowing user progress in the game towards the final goal.
Octalysis uses eight core drivers when organizing the gamification components in the system. They are: epic meaning and calling, development and accomplishment, creativity and feedback, ownership and possession, social influence and relatedness, scarcity and impatience, unpredictability and curiosity and loss and avoidance.
These frameworks helped advance gamification process design, but have limited contributions to gamification models and employment of gamification-related aspects such as personalisation and socialisation in the gamification process. In this space, the proposed N-EGM model and its NEWTON project platform deployment help advance the state of the art.

B. GAMIFICATION IN STEM EDUCATION
Empirical results of applying gamification in education in general, and specifically in STEM education, have been reported in many works. It is noted that the adoption of gamification mainly happens in Computer Science (CS)related courses at third educational level, such as introductory software courses [4], software engineering courses [5], [6], machine learning [7], and multimedia courses [8]. Few cases have involved STEM subjects at other education levels, for example, applied mathematics in a secondary school (level 4) [9]. Gamification helps change the user behavior, converting a mostly passive viewer from the past, as reported in [10] into an active player. This new behaviour makes the player happy to interact with the system and engage with other users, including indirectly using leaderboards and other gamification mechanisms. Among these various gamification mechanisms, the most popular ones are badges, points, levels and leaderboards [11], [12], [13]. In particular, [14] demonstrated empirically that employing a leaderboard could be very beneficial to students, if configured properly. In fact, many research works confirm that gamification has positive effects on teaching of STEM subjects if the mechanisms are designed well and educational process was conducted properly [11], [13]. Unfortunately, most existing gamification implementations were built as plugins to already-designed LMSs like the Blackboard [4], [5], Coursera [7], and Moodle [8], facing diverse limitations.
There is important evidence that gamification effectively boosts students' interest and increases their engagement and participation in courses [4], [5], [6], [7], [8], [9]. It was observed that more students were attracted to, attended and got actively involved with such gamification-enhanced courses and study activities. Additionally, compared with previous non-gamified courses, several works achieved improved performance/scores [9] and also have attracted increased participation in practical assignments, labs and research tasks [6]. Moreover, satisfaction surveys and feedback results indicated that gamification improves students' course satisfaction [5], [6], [9]. Higher satisfaction could potentially help attract more students to STEM subjects, so gamification is a positive instrument in this regard.
Along with some promising starting works and inspiring positive outcomes, there were also some not-so-satisfying results. These results in the attempts of applying gamification in STEM subject education call for future improvements from various aspects [13], [15].
Second, as pointed out in [15], target users may have very different motivation levels towards gamification and interest in various gamification components. For example, some students were not interested in games to begin with and therefore did not accept gamified systems well [6]. Such results invoke the need of a gamification solution that can accommodate students' preferences and motivations so that they could receive treatment tailored to their specific situations.
Third, there is a lack of powerful yet easy-to-use support solutions for gamification in existing LMSs. As a result, high levels of computer skills and significant time overhead are imposed on trainers during the set-up, maintaining and monitoring gamification, as well as gathering and analyzing results. This is one of the major reasons why the majority of research works performed so far happened in CS-related subjects [11], [13], [15]. This problem not only bottlenecks the adoption of gamification in a wide range of subjects and at various education levels, but also becomes a critical obstacle for running large scale pilots to achieve more significant results.
Moreover, the absence of system graphic user interface (GUI) aesthetic appeal caused by poor design or developers' lack of expertise in such an aspect or simply tight schedule could also directly lead to failure of the application of gamification [6]. These aspects have been considered when designing the NEWTELP platform and its gamification component, as part of the NEWTON project.

C. NEWTON PROJECT
The NEWTON project is a large EU Horizon 2020 Innovation Action project which designs, develops and deploys innovative solutions for Technology-Enhanced Learning (TEL) involving delivery of state-of-the-art STEM content to diverse learner audiences. The NEWTON innovative technologies include solutions for adaptive and personalised multimedia and multiple sensorial media (mulsemedia) delivery [16], Augmented and Virtual Reality (AR/VR)-enhanced learning [17], Virtual Teaching and Learning Labs (Virtual Labs), Fabrication Labs (Fab Labs) [18] and Gamification. These technologies are used in conjunction with different pedagogical approaches including self-directed, game-based and problem-based learning methods. The NEWTON proof of concept outputs have been tested in many pilots in 20 primary, secondary and third level institutions, including in schools with students with special educational needs, across six different EU countries. For example, [19] presented the results of a NEWTON pilot that show how by utilizing Fab Labs student learning experience is improved. The results of a NEWTON pilot investigating the application of VR and Virtual Labs technology among primary school students was also presented. Finally, [20] presents in details the assessment results in terms of different aspects, including learner experience, knowledge acquisition and usability.

III. GAMIFICATION AND THE NEWTELP PLATFORM
Existing LMSs such as Moodle 3 , Canvas and Open edX provide a rich set of features including multimedia integration, mobile support and assessments, etc. However, they are unable to offer adequate support for most advanced innovative technologies meant to enhance learning experience, for example, augmented reality (AR), virtual reality (VR), mix reality (MR) [21], multi-sensory media, fabrication and virtual laboratories (Fab Labs/Virtual Labs), artificial intelligencebased visualisation [22]. Moreover, although some of them do have gamification features, their gamification functionalities are rather basic and offer no support for heterogeneous technologies.
The NEWTELP Platform is a networked learning platform specifically designed to provide integration, dissemination and support for deployment and use of state of the art innovative technology-enhanced learning methods and tools. These technologies include AR/VR, Virtual Labs, Fab Labs and, last, but not least important, gamification. NEWTON is being deployed across Europe for formal, in-formal and nonformal education. Course Management System is at the core of the NEWTELP Platform and orchestrates all the other components. Content Repository stores different learning content such as lessons, tests, etc, created inside the platform. Learner Model maintains various user information such as basic ùser profiles and their operational and pedagogical characteristics to enable personalised learning experiences. In particular, pedagogical information such as students' motivation states, knowledge levels/gaps, average study times, competency and learning styles could be utilized by teachers when defining gamification rules and gamification paths. Content Upload Tool allows teachers to upload learning objects, lessons, tests, etc., created outside the platform and make them available to the platform users. The content upload can be performed from local computers or from other external networked sources and is stored in the External Content database. In their interaction with the platform, the teachers can also create interactive content such as quizzes, surveys and puzzles. The learning objects, and other content pieces along with their associated permissions/accessibility are stored in the Content Repository database.
The Gamification module has at its core a software Gamification Engine which employs gamification elements and rules in order to enable rewarding mechanisms and apply these rewards to learner actions/activities. Teachers or administrators can directly configure the gamification elements and rules, whereas learners are just affected by them, without being allowed to edit them. A set of specific APIs has been built in order to communicate and interact with the Gamification Engine and its rules and to associate learner actions/activities with these rules.
In order to enable efficient gathering of learning experience-related data during students' learning process, a Learning Record Store (LRS) along with Tin Can API (xAPI) 4 are adopted in the NEWTELP Platform. The Tin Can API enables communication and exchange of a wide range of online and offline gathered data through a set of simple vocabulary terms. These terms describe user learning experience aspects when using different technologies and employing various devices, such as PCs, mobile phones, and tablets. Depending on the nature of the learning content, data concerning user activities such as progress, scores, completion time is sent to the LRS through secure statements in JSON format in form of "actor, verb, object" with optional details such as result, timestamp, authority. The main role of the LRS database in the NEWTELP architecture is storing learning records (i.e. Tin Can statements) sent by external contents and the platform and retrieving relevant data when requested by the LMS.

IV. THE NEWTON-ENHANCED GAMIFICATION MODEL (N-EGM): A NEW GAMIFIED VISION A. N-EGM OVERVIEW
In the context of technology-enhanced learning, an enhanced gamification model N-EGM is proposed as a part of the NEWTON project aiming at providing an easy-to-operate yet powerful gamification solution for education. The basic idea behind N-EGM is that it defines a new vision for gamification where the student engagement is enabled not only by using game-elements and mechanics, but also through the integration of different concepts such as personalisation/adaptation, game-based learning and socialisation. These concepts are employed in conjunction with gamification in order to generate a social-personalized-gamified learning experience, easily integrated into LMS platforms. In this context, the benefits of game/gamification mechanisms are provided according to the profile of learners (i.e. interests, skills, learning style, etc.) by leveraging on their most popular social motivations.
Many scientific studies, e.g. [23], discussed the application of gamification in an education context using different approaches and models, with particular focus on improving motivation and engagement of students and providing teachers a powerful methodology to improve learning outcomes. Although our proposed framework is aligned with these scientific studies, we offer something more: an interconnected multi-layer model that includes the definition of the learning experience as interactive (due to serious games), personalized (due to profiling mechanisms) and shared (social network interaction), driving up student satisfaction/engagement and increasing effectiveness across learning modes.
There are very few existing gamification platforms which provide management of gamification dynamics and characteristics (i.e. definition of badges, levels, challenges), as stand-alone services. These include Bunchball Nitro 5 , Badgeville 6 and Gametize 7 . However, these platforms lack comprehensive and easy-to-use integration mechanisms with other systems, including LMSs.
Furthermore, these solutions mainly focus on business, targeting at improving employee or customer satisfaction. Bunchball Nitro was the only solution deployed in a pedagogical case study involving free K-12 digital learning 8 . It was designed to personalize instruction and drive student ownership of learning via play-lists of educational content customized to meet student needs. Apart from Bunchball Nitro, none of the other solutions has been designed for or has been applied in an educational context.
In order to address the above-mentioned problems, we aim at strengthening the deployment of gamification in an educational context. This is achieved by defining a unique logical model that takes advantage of the main features of gamification, technology-enhanced learning materials and LMS and offering an overall solution that could become a core-teaching tool, as it ensures increased interest and participation of students.
By combining different elements like interactive narrative, personalization, immediate feedback, fun elements, "scaffolding learning" with challenges that increase mastery in form of leveling up, progress indicators in form of points/badges/leaderboards, social connections and player control, it is possible to create "gamified" classrooms. This complex combination aims at creating sustained engagement because it not only considers the unique needs of the learners 5   and use points and levels in order to motivate students, but also really captures their interests.

B. N-EGM MODEL DETAILS
The NEWTON-Enhanced Gamification Model (N-EGM) has been designed by extending the main featuring aspects emerging from the different frameworks known in the literature, i.e. MDA [1], D6 [2] and Octalysis [3] and combining them with socialisation in order to generate our specific social-personalized-gamification. N-EGM is built as a multilayer integrated framework, illustrated in Fig 2. N-EGM consists of 4 layers and each of them (L1, L2, L3 and L4) implements one of the following physically disconnected, but logically inter-operable and reusable concepts.
L1 -Gamification -this layer is in charge with the configuration of game mechanics and rewarding rules to be associated with the learning content in order to create the basis for defining gamified learning experiences.
L2 -Game-based Learning -this layer defines the set of serious games and/or game-based content consuming the mechanics defined in L1. In particular, it implements the classical game dynamics and aesthetics, enabling both static (e.g. textual quiz) and graphical/animated (e.g. as a videogame for a full immersive involvement) elements-based interaction.
L3 -Profiling and Recommendation -this layer recommends a personalized and adapted game-based learning experience taking into account the learners' profile, i.e. their interests, preferences, progress, requirements and run-time behaviour. Learner data from L2 is collected by this layer (L2-L3 bi-directional communication) to enrich user profiles in order to better recommend next gamified learning content.
L4 -Socialization -this final layer enables classical social elements to be used by each of the three previous layers. L4 allows students to (i) suggest new ideas for the teaching approach, (ii) help peers in the learning stages, (iii) share own skills to solve complex problems within teams or groups, (iv) perform basic social actions e.g. make comments, "like", etc., (v) directly communicate with colleagues and teachers. Through this layer, the teachers can (i) create their private social context to discuss about specific questions such as the progress of students, (ii) join specific learners' social groups, (iii) reward students for their social activities regarding spe-  By applying N-EGM in education contexts related to STEM subjects, the aim is to improve the learning experience, leveraging on the stimulation of intrinsic and extrinsic motivations, respectively, using game-based content and gamification. Learners are expected to be motivated by the chance to acquire awards and benefits from their progress, can have personalized gamified learning pathways and satisfy their actual attitudes through interacting with their classmates in a social environment. Moreover, the teachers' workload could be mitigated since N-EGM provides a powerful tool for monitoring learners progress (for instance via leaderboards), elaborating new learning strategies based on defining appropriate rewarding rules, creating interdisciplinary personalized lessons linked through the same gamification container (single type of points, single leaderboard, etc.) and having a common social context with the entire learning ecosystem. Finally, each layer can also be used separately, implementing specific learning use cases leveraging one or all available concepts of the model.
The key strengths of the N-EGM model are: • Ease of configuration. Gamification mechanics (i.e. points, badges, leaderboards, etc.) can be easily config-ured/managed by an intuitive graphic interface. • Integration. The model is developed using techniques that simplify integration with other modules by exploiting service calls through APIs that allow access from the outside to the contents and functions that the system leveraging on N-EGM model offers. • Event-driven. The implementation is event sensitive.
For each particular event that occurs within the system, a gamification action arises (e.g., setting a specific point, updating the leaderboard). Each action is thrown and managed using ad-hoc APIs. • Modularity. Each layer is composed of several modules that interact in a multi-level architecture, while each module has a specific task and works independently: the database module takes care of storing the necessary data; the front-end module deals with interaction with users, i.e., it is responsible for acquiring the input data and processing them in compliance to predefined and invariant specifications; the back-end module allows effective operation of these interactions; the engine module allows users to configure gamification elements.

V. NEWTELP N-EGM GAMIFICATION ENGINE
The N-EGM Gamification Engine deploys the proposed N-EGM model in the NEWTELP and instantiates it such as it serves both teachers and students. From a teacher perspec-tive, the engine provides easy-to-use yet comprehensive tools to customize gamification journeys for their students. From a student perspective, the engine supports the whole learning gamification journey in the back-end and offers easy access to gamification achievements.

A. TEACHER SIDE -GAMIFICATION RULE CONFIGURATION MECHANISM
In order to assist teachers to configure the gamification mechanisms, the Gamification Engine is implemented to allow dynamic configuration of gamification elements and rules. The interaction between teachers and Gamification Engine has three stages: 1) Automatic generation of gamification rules: when the teacher creates a new course on the NEWTELP platform, a gamification container is automatically built by the engine. The container includes default gamification elements, rules and a leaderboard for this course. Whenever a new material is added to the course in the NEWTELP platform, a set of default gamification rules associated with this material is automatically generated and stored by the engine. There are different pre-set default gamification rules defined for different types of materials. For example, a default rule for a PDF teaching material is to give a student 10 gamification points when they open and view the file. This stage provides a basic gamification realization and requires minimal efforts from teachers. 2) Customization of gamification elements/rules: in order to cater for teachers' more sophisticated design of a gamification journey and accommodate the need for more flexible gamification realization associated with TEL materials, including serious games, AR/VR applications, the Gamification Engine also allows further flexibility. It enables teachers to modify existing default gamification rules introduced in Stage 1 or add and configure new gamification elements and rules via a specific gamification interface integrated in the NEWTELP platform. Fig. 3a illustrates the gamification user interface which allows teachers to view or edit existing gamification elements, mechanisms and rules associated with a course. They can also add new gamification elements and mechanisms, and configure new gamification rules by setting conditions and rewards. As shown in the figure, the gamification elements and rules supported by the NEWTELP N-EGM-enhanced Gamification Engine includes points, levels, badges, leaderboards and rewards. 3) Leaderboard view: during the period of a course session, the Gamification Engine is responsible for applying gamification rules according to student actions. This will be explained in details in the following subsection. While each student can have a detailed view of their own gamification achievements of different gamification elements, the teacher has access to the leaderboard of the whole class. The leaderboard view is also obtained through the NEWTELP gamification interface. Fig. 4 illustrates this three-stage gamification-related teacher-system interaction.

B. STUDENT SIDE: N-EGM COMMUNICATION MECHANISM AND RECORDS MAINTENANCE
To ensure effective collection of users' learning experience information and correct storage and update of their gamification status according to the gamification elements and rules set up by the teachers, a reliable communication mechanism is established. Fig. 5 illustrates this mechanisms which enables communication between for the external entities, LRS, LMS and Gamification Engine.   First, a user enrolled in a course performs authentication with LMS via a browser. After successful authentication, the user accesses course content and downloads a content application, which is stored in the External Content database. Via the downloaded application, the system uses user credentials again in order to authenticate the application with LMS. Upon success, an authorization is given to LMS, and the following data is sent back to the application: 1) endpoint URL to LRS; 2) user information in JSON format; and 3) authorization token. The user can then begin to engage with the application. During application usage process, the application generates Tin Can statements that contain information related to user's progress and achievements, which are sent to LRS through the URL obtained in the previous step. Upon receiving such statements, LRS stores them.
LMS periodically queries LRS to retrieve the Tin Can statements collected during the last period. The statements are first pre-processed by LMS for collecting user information and filtering. Then, LMS forwards them to the Gamification Engine. Upon receiving these statements, Gamification Engine checks the statements against the gamification rules stored in its database. If a certain rule is triggered, the engine updates the gamification status of the user, which is also stored in its own database.
When a student wants to view their gamification status, they can access their own gamification dashboard through the NEWLTELP platform (and the browser). Upon receiving the request to retrieve the gamification status, LMS further asks for relevant data from the Gamification Engine, which when received is then presented to the student through the platform interface, as illustrated in Fig. 3b.
In summary, the proposed Gamification Engine realizes the gamification functionality and deploys the mechanism that keeps track of students' learning activities in various applications. A comprehensive communication mechanism is also introduced to enable support for gamification. Minimal effort is required from the students, as most operations are conducted behind the scene. Students are able to view their realtime gamification status whenever they want, following one click of a button.

VI. N-EGM PILOT AND RESULT ANALYSIS
This section presents the evaluation of N-EGM, the proposed gamification model and its supporting communication mechanism. The evaluation was performed in a real educational setting, as part of the NEWTON project's GAM-LAB pilot. The pilot took place in Sf. Maria School, Bucharest-Romania. The study included 40 secondary school students with different degrees of hearing impairments. During the pilot, students participated in several STEM courses in line with their secondary school curriculum, i.e., the Atomic Structure, Ceramics, and History Museum, each including a Virtual Reality (VR) or a Virtual Lab (VL) application.
The courses were presented and gamified using the NEWTELP platform. The gamification elements employed include: • Points: The students could earn different points when they started and finished the applications as well as when they finished knowledge tests. The points were allocated according to their test results. • Leaderboard: The students were ranked on a leaderboard based on their accumulated points. • Badges: Badges were given to students when they started the first course as well as during the Atomic Structure course. The Atomic Structure course application also utilized profiling and recommendations. Inside the multi-stage application, students were exposed to learning content at different difficulty levels according to their answers to questions on personal management, desire for learning, self-control and self-efficacy at the beginning of the application and their quiz results in previous course stages. Fig. 6 is a screenshot from the teacher's account which shows the list of some gamification rules configured during this pilot. Fig. 3a illustrates the configuration of one rule that assigns 15 points when students achieved a high score (≥ 60 out of 100) in the test during the History Museum course. Students could view their gamification achievements, such as points, badges and positions on the leaderboard under their profiles at any time. Fig. 3b is a screenshot from a student account which shows this student's gamification achievements.

A. LEARNING EXPERIENCE EVALUATION
In order to evaluate the impact of the pilot, a rigorous approach to pedagogical and motivational assessment that was developed as part of the wider NEWTON project was adopted and employed [24]. In particular, the degree to which gamification features act as a motivational driver for students were primarily looked at through a gamification post-experience survey given to students at the end of the pilot. The NEWTELP platform was employed.
Major results of the gamification post-experience survey are illustrated in Fig. 7. Overall, the gamification deployed by the NEWTELP platform received very positive feedback from students. The majority of students agreed that gamification is very interesting (over 88% agreed gamification was really interesting, while barely any students claimed gamification was boring). Around 80% students welcomed learning with gamification while only around 5% were against learning with gamification. Vast majority of students were excited when earned points (48% agree and 45% strongly agree) and cared about their positions in the leaderboard (58% agree and 30% strongly agree). Also, gamification was able to encourage engagement in learning for 78% of all students.

B. KNOWLEDGE EVALUATION
Very important for the NEWTON GAM-LAB pilot was also to assess the student knowledge gain. This is as the significant improvement in terms of learner experience should not be achieved at the expense of a decrease in the knowledge accumulated by students. Pilot-related knowledge assessment tests were conducted through the NEWTELP platform before and after the pilot, respectively.
The post-test scores (mean = 1.26, standard deviation = 1.21) showed an improvement compared to the pre-test scores (mean = 4.67, standard deviation = 2.76). The results of a paired sample t-test confirmed the statistical significance of the knowledge gain improvement between the two tests with a confidence of 95% (α = 0.05, t = −7.76, p = 0.001). This demonstrates the benefit of using the N-EGM gamification model in terms of knowledge gain in comparison with a non-gamified approach.

VII. CONCLUSIONS
This paper has introduced the N-EGM model, an innovative gamification framework for future LMSs. N-EGM incorporates core game mechanics alongside socialisation elements within an approach that can be readily adopted to existing as well as new TEL solutions. The paper highlights the challenges faced when accommodating the deployment of N-EGM as part of a real life platform NEWTELP, including the design of a communication mechanism that enables seamless and efficient data exchange between various applications, LMS and gamification engine.
The gamification-related assessment results from a NEW-TON pilot which involved real life educational sessions in a secondary school were presented, demonstrating the benefits of the proposed N-EGM-enhanced gamification. The post-experience survey results presented indicate how the proposed framework is effective in improving students' engagement with the courses they attended and facilitates an increase in students' appreciation of their learning experience. Noteworthy is that the increase in learner experience was achieved while also obtaining a statistical significant improvement of the knowledge level accumulated by learners during the gamified lessons. JIM PLAYFOOT is a researcher, consultant, Co-Founder and Managing Director of White Loop London UK. Jim has particular interest in how education can be a driver of social and economic development and has worked extensively in Africa and Latin America.
CARMINE DE NICOLA received PhD degree in Mathematics at the University of Salerno. He is the CTO of Concept Solutions SRL. Previously, he was the Head of R&D projects at Beyond SRL and researcher with University of Salerno, Italy. His interests include gamification, recommendation systems, internet of things and mathematical modelling. He authored papers published by diverse international scientific conferences and journals.
GIUSEPPE GUARINO is Adjunct Professor with the Dept. of Industrial Engineering at University of Salerno, Italy and experienced ICT Researcher with over 10 years experience of working in EU and Italian R&D projects. He is an academic member of ATINER (Athens Institute for Education and Research) and author of many scientific papers in the e-learning, TEL, storytelling, cultural heritage and knowledge representation areas.
MARILENA BRATU is an Associate Professor in the Special Education Department at the Faculty of Psychology and Educational Sciences, University of Bucharest, Romania. She holds a PhD in Psychological Sciences. She possesses a rich didactic and practical experience in fields related to education, recovery and social integration of children with special needs.
FABIO DI SALVADORE has a Degree in Business Administration from University of Genoa, Italy and strong further education and training experience acquired as he practiced various business positions for management consulting firms, private SMEs and big companies. He coordinated the management of special projects concerning innovation, industrial research and experimental development co-funded under regional, national and European Programmes and mentored different innovative start-ups.