PhyRe Up! A system based on mixed reality and gamification to provide home rehabilitation for stroke patients

Stroke represents a global concern that currently affects a significant part of the world’s population. Physical rehabilitation plays a fundamental role for stroke patients to recover mobility and improve quality of life. This process is costly, considering that patients must attend face-to-face rehabilitation sessions in hospitals or rehabilitation centers. Plus, there is a lack of specialized medical staff, who are usually insufficient to properly address the growing number of stroke patients that need physical rehabilitation. This situation has been exacerbated by the COVID-19 pandemic, as some of the human resources have been devoted to fight against the pandemic, and the physical presence of rehabilitation patients in hospitals has been severely limited. This paper proposes PhyRe Up!, a novel remote rehabilitation system that uses mixed reality and gamification techniques. PhyRe Up! has been devised for stroke patients to perform therapeutic exercises at home, with great precision, and with the potential supervision of clinicians. The system aims to increase the patient’s motivation as well as maintaining the quality of performance for the exercises, similar to the obtained levels when attending face-to-face sessions with therapists. The underlying architecture combines declarative, procedural, and conditional knowledge to manage the rehabilitation process, which offers flexibility and scalability to enhance the capabilities of the proposed system. Experimental results highlight how the combination of mixed reality and gamification significantly influences the accuracy of rehabilitation exercises previously defined by therapists. Particularly, the conducted experiments in the first validation phase of PhyRe Up! shows that our proposal drastically reduces the intermediate steps required to complete an exercise thanks to the provided visual feedback. The accuracy with which the patient performs the assessed exercise for the first time is greater than when using traditional rehabilitation techniques.


I. INTRODUCTION
Physical rehabilitation is essential in the recovery process of several diseases, such as neurological diseases, physical injuries or the recovery after a surgery [1]. In the context of neurological diseases, stroke rehabilitation represents a global challenge [2]. Stroke normally occurs when a part of the brain is suddenly deprived of blood supply. Patients affected by stroke are left with disabling effects, such as loss of strength, mobility or sensitivity in some parts of their body [1]. Therefore, long-term physical rehabilitation is necessary to improve their quality of life and regain mo-bility [3]. Also, time is a crucial factor, since the sooner they are treated, the greater the possibility that patients will recover some degree of mobility [4]. Stroke is a leading cause of mortality and disability in the world, not to mention the economic costs of treatment and post-stroke recovery [5] it involves. Only in the European Union, the number of people coping with stroke is estimated to rise by 27% between 2017 and 2047 [2]. Moreover, this issue is even expected to be exacerbated because of the increase of age in elderly people which will cause a negative impact in the coming years [6].
In economic terms, however, there is a great cost regarding VOLUME XX, 2021 neurological diseases. According to the 2018 NHS Annual Report, and taking the United Kingdom as a reference, curative care and rehabilitation services assumed more than half of public healthcare spending in 2016, reaching 58.01 billion euros (57.0% of total healthcare expenditure). In Spain, as a particular example, the average cost of a patient affected by stroke was estimated at 27,711e per year [7]. More than two thirds are due to social costs, mainly informal care. Moreover, on the report of the European Brain Council (https://www.braincouncil.eu/), more than 179 million people in Europe live with neurological disorders [8]. Indeed, it is estimated that 1 in 3 will suffer from some neurological or psychiatric disorder in their lifetime. Unfortunately, the socio-economic impact is even much more accentuated in low and middle-income countries.
Generally, stroke patients do not perform rehabilitation alone [1]. Instead, they are supported by therapists in faceto-face sessions on a regular basis. This poses a problem for patients whose health condition prevent them from attending sessions in the rehabilitation center, situation which is greatly affected by the current context of the SARS-CoV II pandemic to minimize exposure [9]. On the other hand, rehabilitation sessions provided in hospitals or rehabilitation centers are sometimes not sufficient for patients, as they are limited in duration. Furthermore, there is a barrier related to motivation and engagement. Traditional physical exercises consists in performing repetitive executions of correct movements which tend to be monotonous and boring. It can cause that patients loose motivation and, in consequence, their commitment to the therapy [10]. Unfortunately, this may affect the quality of the therapy. For these reasons, telerehabilitation arises. This branch of telemedicine allows for treatments of the acute phase of disease by replacing traditional face-to-face sessions with a rehabilitation at home [11]. There is a significant number of applications under this approach, but they are generally applied to physiotherapy where virtual reality (VR) techniques are used so that the patient mimic the movements of a virtual avatar. In essence, it is a relevant approach for home rehabilitation. However, it should be noted that when the responsibility of performing these exercises is delegated to the patient, at home and with no direct supervision by the therapist, the precision of the exercises that are executed may be low, potentially affecting the quality of the rehabilitation. In this sense, they may not be well performed at home, which can lead to undesirable situations in which the effects of rehabilitation may be negative [12].
One of the extensively used technologies for rehabilitation is immersive reality, which aims to recreate artificially rehabilitation environments. Among the different alternatives, mixed reality (MR) technology is highlighted, since it allows to merge both physical and virtual spaces in an environment where real objects coexist with virtual ones [13]. In essence, MR presents a great opportunity to make possible to modify easily the needs of a patient regarding his or her condition thanks to the amazing visualizations and advanced interaction it provides. Recently, some studies have explored the potential of MR within the context of medical applications [14]. However, its potential in supporting people for remote physical rehabilitation have not yet been exploited.
Thus, this article introduces PhyRe Up!, a telerehabilitation system based on MR and gamification techniques to assist in the physical rehabilitation of stroke patients at home. PhyRe Up! makes use of telerehabilitation as a proper framework to enable physical rehabilitation at home, along with MR to provide real-time, visual feedback, and guidance for patients when performing rehabilitation exercises. This system is intended to reduce the burden of stroke patients by enabling an alternative that would reduce the cost of stroke to health systems [7]. Special attention has been paid to the module that is responsible for recommending these exercises. The precision, with which the system assists patients, allows to improve the quality of the execution as well as enabling their autonomy. PhyRe Up! relies on two cutting-edge technology devices: Microsoft HoloLens 2 TM and Azure Kinect DK TM . The first provides MR capabilities and advanced interaction, while the second significantly facilitates the patient's body tracking. Thanks to these latter artifacts, a mixed reality scenario has been created, using the real-world context from the motion tracking device (i.e. real-time body tracking) so as to interact with the synthetic information (virtual elements) added to the physical space with the visualization device. Thus, both virtual and physical world are merged to fully immerse the patient in a real rehabilitation session.
The system aims at increasing the quality time therapists can spend on their patients, facilitating how rehabilitation exercises are defined, assigned and assessed. The architecture that underpins PhyRe Up! has been designed around a module that integrates the domain knowledge, which can be augmented to address how rehabilitation exercises are defined, assessed, and recommended.
The results obtained in this work after experimentation highlight that immersive techniques based on MR allow patients at home to faithfully recreate the therapies defined by experts without their presence, and without affecting the quality and success of recovery. The quality of execution of artificially guided exercises is similar by face-to-face therapy.
The rest of the article is structured as follows. Section II positions this research within the context of other relevant research works. Then, Section III discusses in depth the architecture of PhyRe Up! Subsequently, Section IV describes the conducted experiment and Section V explains the obtained results. Then, Section VI addressed the limitations of the system from the perspective of the study and the proposal. Finally, Section VII draws the conclusions of this research article and proposes future lines of research.

II. RELATED WORK AND BACKGROUND
Virtual rehabilitation has been a promising research field during the last few years, commonly associated to elderly care and rehabilitation of patients affected by neurological diseases [15]. In fact, virtual rehabilitation has been addressed  from different points of view, and multiple disciplines have been involved. One of them is VR [16], where programs are defined as the use of computer-simulated environments that imitates physical real worlds. A systematic review and metaanalysis is presented by Dominguez et al. [17] with a focus on game-based virtual reality interventions for stroke patients. From a general point of view, a significant part of the research proposals have relied on Microsoft Kinect [18]. Initially conceived as a game device, the original Kinect has demonstrated to be a very flexible piece of hardware, clinically validated in a significant number of physical therapy and rehabilitation contexts [19]. Kinect was recently discontinued by Microsoft. The new version, named Microsoft Azure Kinect DK TM , maintains stronger integration with the cloud and the use of artificial intelligence techniques.
Augmented Reality (AR), based on superimposing virtual objects over the real world, which are affected by the user's physical interaction [20], has been more recently used to assist in the physical rehabilitation process. A review presented by Gorman et al. [21] explores relevant works in which AR technology has been adopted when performing rehabilitation after stroke. The review concludes that further investigation is required in this field. Also, a recent study is also discussed by Viglialoro et al. [22], with a focus on analyzing to what extent AR-based systems have been used in upper-limb rehabilitation and on investigating the effectiveness of AR compared to other approaches.
MR, on the other hand, merges both physical and virtual spaces in an environment where real objects coexist with virtual objects [13]. Although MR is of increasing interest within the context of virtual rehabilitation, this approach is not as mature as VR or AR. Nevertheless, this situation is changing at a fast pace, and the application of MR technology to medical applications can be considered as an emerging research field [14].
Recently, a number of contributions have addressed physical rehabilitation of patients affected by neurological diseases, particularly stroke. For example, a portable low-cost MR-based tabletop prototype is presented by Colomer et al. [23] for upper-limb rehabilitation. This research include visual feedback and gamification techniques, and a clinical evaluation is conducted to assess effectiveness and acceptance.
Similarly, the research work discussed by Aung et al. [24] proposes RehaBio, a shoulder rehabilitation system that makes use of AR and provides basic visual feedback to retrain the plasticity of the brain for patients affected by traumatic brain injury, spinal cord injury or cerebrovascular accident. In terms of interaction, the system is able to capture the video scenes related to the user's movements, render the virtual object that the user must reach, track the marker, and detect the collision between the visual objects and the real marker. Range of motion and the effectiveness were assessed, while the system was proved to motivate users. Another research work in which the use of AR stands out was recently proposed by Liang et al. [25]. In this case, tests are carried out to evaluate the impact of use AR in teaching practitioners in a stroke assessment simulation designed for clinical training. A MR device is employed to project a human face that mimics facial drooping, an actual symptom of stroke, onto a simulated virtual avatar.
A related research work is presented by Sekhavat et al. [26]. In this context, a system that relies on projectionbased augmented reality to improve the understanding level between the body perception and the movement kinematics is presented. Experiments were conducted for both unimpaired and impaired users to modify gait depending on the visual feedback provided by the system. The results show that projection-based AR outperform monitor-based systems when it comes to synchronize foot-eye coordination. On the other hand, the research work discussed by Aparecida et al. [27] addresses the clinical feasibility of an AR system for upper-limb post-stroke motor rehabilitation. Two case studies are described: i) an evaluation of upper-limb motor function and ii) another one related to the gain of motion range of shoulder flexion and abduction. The authors concluded that enhancements are generated in the patients' shoulder range of motion and speed.
The work presented by Da Gama et al. [28] introduced a rehabilitation system aimed at recognizing and classifying biomedical movements, particularly the shoulder abduction exercise. The device Microsoft Kinect was used to track the VOLUME XX, 2021 patient's joints, while basic gamification mechanisms were integrated to engage the final users. The ultimate goal was to assess the efficacy of employing a clinically-related gesture recognition tool. The authors concluded that the achieved results indicated that the use of bio-mechanical standards to recognize movements may be valuable in guiding patients during the rehabilitation process.
Another contribution is presented by Hocine et al. [29]. In this research, the authors introduce PRehab, a serious game for upper-limb rehabilitation aimed at improving the training outcomes of patients affected by stroke. The proposal revolves around the idea of generating customized game levels, so that the game difficulty is dynamically adjusted depending on the patient's skill and performance. A graphics tablet was the chosen device for the patients to play PRehab. Patients and clinicians were involved in the conducted experiments to evaluate the immersion level and the motivation.
The work described by Vamsikrishna et al. [30] introduces a methodology to facilitate palm and finger rehabilitation, using computer-vision techniques. The leap motion controller is employed to track the patient's palm and fingers, while the system is able to analyze the steps involved in the rehabilitation process without the direct supervision of a physician. Experimental validation with healthy volunteers proved that linear discriminant analysis and support vector machines, both techniques used to classify the user's movements, perform in a similar way when recognizing isolated gestures.
PhyRe Up!, the system proposed in this work, has been designed so that therapists can remotely supervise stroke patients' evolution when they complete the rehabilitation routines at home. Therapists can assign the exercises that best fit the patients' progression depending on their skills and degree of the injury. Plus, motivation plays a key role to avoid stroke patients quitting rehabilitation. In order to face this challenge, the system uses gamification techniques to turn the exercises into playful "games". In this way, patients make rehabilitation while they play and enjoy [31], maintaining their resolve and engagement level.

A. GENERAL OVERVIEW
PhyRe Up! is a telerehabilitation system based on MR and gamification techniques for stroke patients. The system can be used by both therapists and patients. In the first case, therapists can define a set of exercises and routines, which can be used as a reference to supervise how patients do rehabilitation from home. The system displays virtual 3D information in real time on a real environment, which serves as a guide for the patient to perform the exercises previously defined by the therapist. At the same time, the system automatically tracks the patient's joints and checks the correct execution of the exercises at all times. A continuous feedback exists between two of the major software modules that compose PhyRe Up!: the tracking module and the MR module. This is essential to establish proper synchronization with the execution of exercises by the patient. Figure 1 shows a graphical overview of how the data flow through the different entities involved in PhyRe Up! and their interactions. The device Azure Kinect DK TM is used to collect raw tracking data, which are then processed and sent to the MR device Microsoft HoloLens 2 TM . Patients wear the goggles on their head, which allows the system to augment the real-world information with holograms. Tracking data are used as a reference to render the virtual information (please see the upper right window where the user's point of view is represented and virtual spheres are drawn to guide the patient's physical movement). On the other hand, Figure 2 shows how the architecture of PhyRe Up! has been designed and how the different hardware and software components are related to each other. The MR device can be used by the patient when performing the rehabilitation exercises, but also by the therapist in order to define, in a visual way, the reference/gold standards. In other words, the ideal execution of the rehabilitation exercises. Thus, the therapist has the capability to test, in advance, the very same exercises that the patients will perform later on by using the same MR device. The MR therapist's module provides detailed information on this matter.
The architecture is composed of 4 main modules, which are briefly introduced next: • Domain Knowledge module. This module contains and handles the knowledge of the system. It possesses declarative, procedural and conditional knowledge needed to enable the achievement of the developed system's objectives. Moreover, all the information generated during its use will be stored and managed to properly assess and monitor the patient's evolution. A backend architecture is deployed on the cloud Microsoft Azure to guarantee the extensibility of the system and facilitate its use. • MR therapist's module. This module allows the therapist to define the rehabilitation exercises and routines that will be performed by the patient throughout the rehabilitation process. This module relies on the device Microsoft HoloLens 2 TM (https://www.microsoft.com/ es-es/hololens), which facilitates the exercise definition thanks to the use of an approach based on natural user interfaces. • MR patient's module. This module integrates the software required to manage a series of rehabilitation exercises in the form of exergames. They must be done by the patient to perform the rehabilitation routine. The exercises must have been previously defined by the therapist. Again, this module relies on the device Microsoft HoloLens 2 TM . Gamification techniques are also integrated to engage the patient during a rehabilitation session. • Tracking module. This module tracks the patient's skeleton to obtain the 3D position and orientation of the joints involved in the rehabilitation exercise. The tracking data are sent to the MR modules. The current version of PhyRe Up! relies on the device Azure Kinect DK TM (https://azure.microsoft.com/es-es/ services/kinect-dk/), a cutting-edge technology in terms of body tracking.

B. SPECIFICATION OF THE ARCHITECTURE MODULES
Next, each one of the modules that compose the devised architecture is discussed in detail, including how the interactions are carried out. VOLUME XX, 2021

1) Domain Knowledge module
This module constitutes the core of the system, since it contains the specific knowledge to achieve the objective of proposed system: the rehabilitation of patients affected by stroke.
In this module the knowledge is totally structured and organized to designate the articulation of knowledge to the rest of the subsystems. This structure is clearly differentiated in three types of knowledge, which are distinguished as follows: • Declarative knowledge. It is the factual knowledge ("knowing what"). It includes organized bodies of knowledge about the problem and context. In this way, such knowledge elements are attributed to the set of articulations that exist (i.e. J = {j 1 , j 2 , ..., j 31 }; https://docs.microsoft.com/bs-latn-ba/ azure/kinect-dk/body-joints), the constraints associated with them (i.e. C(J) = {C(j 1 ), C(j 2 ), ..., C(j 31 )} ), and the variables associated to monitor the rehabilitation exercise of patients (i.e. V = {v 1 , v 2 , ..., v k }). • Procedural knowledge. It refers to the execution of procedures, strategies, techniques or methods to achieve an end, that is "knowing how". In our context, this knowledge defines the rehabilitation exercises and the game dynamics associated with them (i.e. E = {e 1 , e 2 , ..., e n }), where each rehabilitation exercise (i.e. e y ∈ E) works a specific injury or joint (i.e. j x ∈ J) with a different degree of complexity. It is even possible to divide the set E according to the injury or joint worked. Also, the algorithms needed to track a rehabilitation exercise with the aim to check how well they are being performed are included here. • Conditional knowledge. This kind of knowledge implies knowing when and why to do something. In the context of this research, this knowledge is used for recommending the next exercise the patient should take (e i ∈ E). To do that, it takes into account the progress and results obtained in the last exercises carried out by the patient That is, it approximates the function: V k × E −→ E, that contains the therapist's knowledge about the exercises that a patient should do depending on his or her condition. To model this function we use a set of rules (i.e. R = {r 1 , r 2 , ..., r m }), being each conditional rule r i ∈ R will generally have the following form: X is a subset of input variables (X ⊆ V ), DDX is a set that represents the values that must take these variables (X), and Y is the exercise to be recommended.
Moreover, this module contains a memory element, which is responsible for saving a snapshot of the performance and the evolution of a patient. That is, a history that contains the key elements, such as number of repetitions, score, time employed in finishing the exercise, and even the failure steps, among others. A record of the exercises already performed will also be kept.  The domain knowledge module has been deployed on the cloud Microsoft Azure (https://azure.microsoft.com/es-es/) to store the rehabilitation exercises, analyze the data obtained from their execution and, above all, guarantee the security of sensitive data.

2) MR therapist's module
PhyRe Up! has been developed to facilitate a therapist the definition of rehabilitation exercises and the game dynamics associated with them. It will constitute the declarative and procedural knowledge of the system. In this sense, the interface has been designed on the promise of providing therapists the utmost freedom. Therefore, communicating between the tool and therapist is carried out through natural interaction, which means that therapists can create exercises using voice and gestures.
An exercise is designed as a game, which consist of a path composed mainly of virtual elements, such as points, rings or any other element, i.e. targets representing the trajectory of the rehabilitation exercise. The patient's joint involved in the exercise needs to pass through these virtual elements to achieve the goal. Figure 3 shows an exergame example defined by a therapist, which is framed in the box (1); (2) is the starting point which is marked in yellow to be differentiated than the the others; (3) are the points which compose the exercise, i.e. the path, and (4) the control rings. The difference between (3) and (4) is that (3) represents parts of the path that the patient must follow, and (4) represents key or strategic points in the performance of the exercise. The rings guide the exercise, and the spheres or intermediate points appear progressively as the patient reaches the control points (rings). Due to the importance of the strategic points (rings), the achievement of these means a higher score for the patient. It should be noted that the scheme of the exergame has been defined by a therapist using the MR device through multi-modal interaction.
This research work makes use of the exergame concept, which implies adding virtual elements to turn a traditional rehabilitation exercise into a gamified one.
Theoretically, the exergame (i.e. rehabilitation exercise and dynamic game) e i ∈ E defined with the system proposed herein consists of eight-tuple, . Next, we define each one of these elements: D i is a descriptive information of the exergame e i . JI i is the set related to joints involved in the rehabilitation exercise e i , that is, JI i ⊆ J.
SET i is the setup of the exercise and it is defined by means of the tuple: (sc, rp, t, c), where sc is a quantitative indicator of success in the performing of a step of the rehabilitation exercise e i (sc ∈ N); rp are the repetitions required to perform the exercise e i (rp ∈ N); t is the time needed to finish the exercise e i (t ∈ R); and c is the degree of complexity of the exercise (c ∈ N).
V E i defines the virtual elements fixed in the 3D space, such as points, rings, hoops or other elements used in the exergame e i (V E i = {ve i1 , ve i2 , ..., ve ik }, with each ve ij being a virtual element). A virtual element ve ij denoted as three-tuple (ve ij (x), ve ij (y), ve ij (z)), where ve ij (k) ∈ R and it represents the position of the element ve ij (k) in the 3D space (X(ve ij (x)), Y (ve ij (y)) or Z(ve ij (z))).
T i refers to trajectories associated with the movements that a patient will do when performing the exergame e i . T i is defined as a set of virtual points that establish the movements the patient must perform (T i = {ve ij , ve ij+1 , ..., ve in }). It should be noted that a T i is a set that may have repeated elements.
RG i is a set of rules that contains the game mechanics of the exercise e i , which are based on the interaction between JI i and T i in the 3D space to achieve an objective. Formally, RG i = (rg i1 , rg i2 , ..., rg in ), where each particular rule of the game rg ij is used to define the function • JI i establishes the joints that interact with the game rule. • V E i establishes the virtual nodes that interact with the game rule. KP I i is a set of key performance indicators that are used to monitor patient's evolution according to the performance of the exercise e i (i.e. KP I i = {k i1 , k i2 , ..., k im }). These KPIs depend on what information therapist wants to obtain after the patient finishes the exercise e i . Each k ij is defined as a pair (V k ij , DDV K ij ), where: • V k ij is a set of input variables used to define a KPI concept (V k ij = {v1 ij , v2 ij , ..., vn ij }).
• DDV K ij is a set of definition domains of the KPI's variables V k ij , that is, DDV K ij = {DDV 1 ij , ..., DDV n ij }, where DDV x ij is the definition domain of the variable vx ij . As can be observed, the definition of an exergame involves defining both declarative and procedural knowledge.
The exergames will be defined by therapists thorough an interface in such a way that the information established herein is easier for them to be introduced. This is added into the Domain Knowledge module as procedural one. Moreover, the exergames can be represented by means of strings designed in a language for physical rehabilitation, called Personalized Exergame Language (PEL). The intention to use this kind of knowledge representation is to be comparably to other rehabilitation contexts. However, the underlying details of the language, such as vocabulary, syntax, and semantics, are not provided because it is beyond the scope of this paper. This information is deeply described and explained in [32].

3) MR patient's module
This is the module that will allow the patient to perform the rehabilitation exercises. It is responsible for encouraging the patient's performance of the rehabilitation exercise during a rehabilitation session as well as evaluating how well the patient performs it. Declarative, procedural and conditional knowledge available in the domain module will be used herein.
The module presents an exergame e i to be performed by patients as a step of their rehabilitation plan (e i ∈ E). This exergame may be assigned to the patient directly by the therapist or automatically by the module using the conditional knowledge.
An example will be used to show how the module works. Let's suppose the therapist assigns to the patient the exercise shown in Figure 3. We refer it as e 5 (such that e 5 ∈ E).
As it can be seen, e 5 it is composed of ten virtual nodes (8 points and 2 rings) defined as V E 5 = {ve 51 , ve 52 , . . . , ve 510 }, and a linear trajectory defined as T 5 (ve 51 , ve 52 , . . . , ve 510 ), with ve 51 being the starting point (marked in yellow in Figure 3).
It is a drag-and-drop exercise. From a straight position, the exercise consists of taking the yellow ball with the indicated hand and dragging it through each of the virtual nodes, which are part of the trajectory, following a linear movement. Essentially, the objective of the game is to rehabilitate the right shoulder (j 12 ). For this purpose, the patient has to pass the right hand (j 15 ) through each of the virtual nodes that are part of the trajectory T 5 (in this way, JI 5 = {j 12 , j 15 }).
Since the example proposed e 5 tends to rehabilitate the right shoulder (j 12 ), the patient's joints that should not ideally move to compensate the lack of movement or strength may be left upper limb (j 5 , j 6 and j 7 ) and spine (j 1 ), i.e. C 5 (JI 5 ) = {C(j 12 )}, with C(j 12 ) = {not move(j 1 , j 5 , j 6 , j 7 )}. This warns the patient that some joints are being used to complete the exergame. It should be noted that it is also useful to prevent patient from provoking an injury.

VOLUME XX, 2021
Part of the dynamic of the game is established in the exercise setup, i.e. SET 5 = (sc = 40, rp = 4, time = 180s, c = 5). It means that the patient has to perform four repetitions in less than three minutes (180s) reaching to 40 score in an exercise with complexity grade 5. In essence, this configuration allows the module to assess how successful or unsuccessful has been the performance according to the grade of complexity.
Another aspect of game dynamic is established in the rules of the game associated with the exercise, RG 5 . At this point it is important to remark that a rule may be triggered when a joint involved JI i interacts with a virtual node V E i meeting a certain objective o j . In this particular case, RG 5 = (rg 51 , rg 52 , rg 53 , rg 54 ), where: rg 51 : IF distance(position(j 15 ), ve 51 ) ≤ α AND isGrabbed(ve 51 ) THEN score(+5ptos) ELSE f eedback(ve 51 ) rg 52 : IF distance(position(j 15 ), ve 5j ) ≤ α with j ∈ {2, 3, 5, 6, 7, 9} AND isV isited(ve 5(j−1) ) AND isT ouched(ve 5j ) THEN score(+1ptos) ELSE f eedback(ve 5j ) rg 53 : IF isInRing(position(j 15 ), ve 5j ) with j ∈ {4, 8} AND isV isited(ve 5(j−1) ) AND isCrossed(ve 5j ) THEN score(+12ptos) ELSE f eedback(ve 5j ) rg 54 : IF distance(position(j 15 ), ve 510 ) ≤ α AND isV isited(ve 59 ) AND isDropped(ve 510 ) THEN score(+5ptos) ELSE f eedback(ve 510 ) position(j i ) is a function that returns the position of the joint j i in the 3d space. distance(x, y) is a function that returns the distance between two points x and y in a 3D space. isGrabbed(x) is a function that returns true if the ball has been grabbed in x. isV isited(x) is a function that returns true if x has been previously visited according to the order of the virtual points established in the trajectory T i . isT ouched(x) is a function that returns true if x has been touched. isCrossed(x) is a function that returns true if x has been crossed. isDropped(x) is a function that returns true if the ball has been dropped in x. And isInRing(x, y) is a function that returns true if x has passed through the rings y. The parameter α is used to establish the precision of the system; its value has been defined as 0.05.
The rule rg 51 is used to check whether the patient grabs the ball that must be dragged until the final position in the starting one (i.e. ve 51 ); rg 52 is a rule to check whether the patient passes the ball through the intermediate points . The set of game mechanics GM are related to increase the score achieved when an objective is reached (i.e. score(+yptos)), or providing feedback when it is not (i.e. f eedback(ve 5x )). Visual feedback provides useful information to the patient for the achievement of the objectives, in case they have not been achieved. It provides a helpful orientation and guidance in the performance of exercises.
The game rules are defined to motivate the patient during a rehabilitation session, since feedback is presented to the patient when each game rule is triggered. Figure 4 shows an example when rg 52 is fired as a consequence of being satisfied the objectives o 2 and o 3 .
The tasks for detecting these joints, checking associated constraints satisfaction C 5 (JI 5 ), tracking motion and evaluating functions (i.e. distance, position, isInRing, isGrabbed, isV isited, isT ouched, isCrossed, isDropped), will be delegated to the tracking module due to the high computational cost. The tracking module is explained in detail in the next Subsection.
Finally, the exergames may contain metrics or KPIs to measure or monitor certain aspects in a rehabilitation exercise, for example, performance, mobility or displacement, among others. Consider that the therapist has defined in the example detailed herein a metric k 51 ∈ KP I 5 to monitor the performance of the patient. Imagine that k 51 consists of two input variables V k 51 = {score(v1 51 ), labels(v2 51 )}. Their domains are DDV 1 51 = {10, 20, 30, 40} and DDV 2 51 = {bad, normal, good, perf ect}. The patient's module is responsible for understanding this information and evaluating the exercise with respect to this definition. Specifically, the module associates a label with a score. In other words, the label bad would be matched to a score lower or equal to 10 points, the label normal a score higher than 10 and lower or equal to 20, the label good a score higher than 20 and lower or equal to 30, and label perf ect a score between 31 and 40.
Once the exergame has finished, the results of the exergame are saved into the memory of the Domain Knowledge module.
As mentioned above, there is the possibility for this module to automatically assign exercises to the patient based on his or her previous results. To this end, conditional knowledge is used. Then, a brief explanation about what conditional knowledge consists in is presented.
The conditional knowledge has been codified by means of a set of fuzzy rules (i.e. R = {r 1 , r 2 , ..., r m }) that model the function V k −→ E. In the current state of the work, the variables used to make the recommendation are: V = {dif f erence_number_steps, accumulated_deviation, dif f erence_time}. Their meaning is explained below: FIGURE 4: Patient's view. The widgets on top of the image shows the points (left) and rings (right) obtained. The "+1" particle is fired when the right hand passes through a virtual element, as well as the animation of the joint pointer.
the number of steps that the patients and the therapists performed to carried out the last rehabilitation exercise (i.e. the patient fails to pass through all the virtual points that establish the trajectory associated with the exercise). • accumulated_deviation (v 2 ) is the cumulative spatial deviation between the patient and the therapist (i.e. this calculation is based on the distance accumulated in the completion of the trajectory associated to the last exercise). In addition, the last rehabilitation exercise performed by the patient, last_exg, will be used in the rules to determine the next exercise to be performed. In this respect, a function will be applied, propose_exercise, that selects from the set of available exergames E one whose lesion and joint JI i are similar to the last exercise last_exg. This is combined with another one to return the exergames based on the complexity of last_exg, that is, those whose complexity can be higher, lower or equal.
Full details of all the rules used by the module are out of the scope of this research work, but, by way of example, some of them will be shown: Particularly, the rule r 11 makes use of three previously mentioned parameters, that is, difference_number_steps, ac-cumulated_deviation and difference_time, which are relevant for recommending an exergame that might be added in the next rehabilitation session. This rule means that if i) the patient has performed a number of steps moderately different to the therapist, ii) the spatial deviation between them is considerable, iii) and the time invested varies reasonably, it would be ideal that the patient repeats the last exergame.
On the other hand, we do think that justifying the use of fuzzy logic when designing this module would be beneficial. At this stage, there is no attempt to provide an accurate assessment of how much patients have progressed, but rather to provide them with guidance that they can understand. In this respect, using linguistic variables makes it easier for them to interpret this information.
As a consequence of the work done by this module, the patients will receive detailed feedback on the work they have done, with an explanation about the areas in which they have performed well, and others which may still require more attention. In addition, on a broader level, patients will be informed whether they have truly done well the routine.
Interestingly, the approach presented herein shows the potential to define aspects related to physical rehabilitation. The underlying idea of this method is to be extensible and customizable enough to be used in other rehabilitation contexts.
As a result of the example detailed in this section, the complete execution of the exercise is shown in Figure 5, where multiple key frames have been selected. This figure is the composite of frames following the order from left to right and from top to bottom.

4) Tracking module
This module is responsible for running real-time body tracking recognition of the users' movements when they perform rehabilitation exercises. Essentially, it obtains 3D spatial coordinates of the human body joints, that is, the position given VOLUME XX, 2021 as a three-tuple (x, y and z) ∈ R 3 and the rotation given as a four-tuple (w, a, b and c) ∈ R 4 . The rotation is expressed as a normalized quaternion. Principally, the process to perform body tracking is used through Azure Kinect DK TM device. Fundamentally, this hardware device has been utilized along with Microsoft HoloLens 2 TM . In essence, the MR device cannot track the subject's body. Therefore, an additional system has been required to obtain the information of the users' joints JI i involved in the rehabilitation exercise e i , in real-time without wearing markers or sensors, among others.
Specifically, the human movement recognition, in an exergame of this approach, is mainly composed of skeleton acquisition and distance measurement. Firstly, the human skeleton is obtained and the spatial coordinates of the skeleton joints JI i are calculated (position function). Then, it calculates the distance between the particular joint involved j i and the virtual node ve ij (distance function). Also, this module determines whether or not the subject has touched or crossed the virtual node target (isT ouched and isCrossed functions) or other functions has been triggered on it (i.e. using isGrabbed, isV isited, isDropped and isInRing functions).
A coordinate system has been created to share the position and rotation of the joints. It is due to the fact that HoloLens 2 device cannot obtain, at least accurately, the information of the human joints. In this respect, the tracking device obtains the spatial coordinates of the human body joints and they are shared to be used by the MR device. This can be graphically observed in Figure 6.
Fundamentally, an AR marker is placed on the Azure Kinect device to obtain, through Vuforia computer-vision library (https://developer.vuforia.com/), the information required. By using the position of the head p h = (x h , y h , z h ) FIGURE 6: Overview of the coordinate system. and the position of the marker p k = (x k , y k , z k ), the vector between the head and the device Azure Kinect DK TM is computed: It should be noted that MR device is the origin coordinate, while the Azure Kinect DK TM device has its own one, created in the AR marker location, to have the adequate position of the tracked joint. The x and y axes of the tracking coordinate system is inverse to the ones managed by HoloLens 2 TM . Therefore, once a joint position is obtained by the tracking device, the aforementioned components need to be multiplied by −1 to be them aligned with the HoloLens coordinate system.
Particularly, the actual distance between the person and the camera is calculated using the module | − → v k |. It is mainly performed to ensure that the confidence level of the key points is optimal, since high or low distances may influence the quality of body tracking recognition; it is recommended a range between 0.5 -3.86 m distance (please, refer to https://docs.microsoft.com/es-es/ azure/kinect-dk/hardware-specification). At this point, combining the vector between the head and the tracking device − → v k , and the position of the particular joint involved j i , the point from the head is: Then, the Euclidean distance is used to get the distance between the joint and virtual node, being X(x 1 , x 2 , x 3 ) the joint and Y (y 1 , y 2 , y 3 ) the virtual node in the spatial coordinate system: (3) By using the above information, the MR device is capable of determining whether the joint involved j i has touched or crossed a virtual node ve ij . It is carried out using a value internally defined as a threshold.
However, it should be noted that this process should show low latency, since a high latency may significantly affect the feedback that patients/therapists receive on behalf of the MR module. To achieve the best performance, a combination of i) an approach based on the asynchronous processing of the frames, by means of queues, and ii) a multi-threaded architecture, is proposed. The main thread of the tracking application obtains the depth frames through Azure Kinect DK TM and queues them. Additional application threads are waiting in the queue to retrieve the raw images. These threads will then process the frames in the depth camera to obtain the depth map and, in this way, the patient's joints JI i . After obtaining the frame, and calculating the depth map and the position of the joints, the data are sent to the MR device via the UDP protocol, as it is faster than TCP. In this context of interactive graphics, speed is considered to be especially relevant.

IV. RESULTS
This section describes the experiments conducted to empirically highlight the benefits of the proposal, which mainly influence the accuracy with what the subject recreates the physical exercises defined by the therapist. The accuracy of the executed exercises, along with perseverance, is key to achieve effective recovery, reducing therapy times and maximizing mobility [4].
A representative group of 25 anonymous volunteers of different sexes (12 men and 13 women) of different ages The subjects who took part in the experiment used adjustable weights from 1 to 3 kg to limit their movements with the aim of simulating injuries in the upper extremities.
Volunteers signed an informed consent agreement which stated that the data collected would be anonymized and treated only for research purposes.
The experiments were composed of several steps. First of all, a therapist defines, with the MR device, one or several rehabilitation exercises of different levels of difficulty. The difficulty varies depending on the range of the movement, the existence of changes of the direction in the movement, and the number of involved dimensions in 3D space. Secondly, the subjects visualize a video of the exercise to be performed (without considering the use of the MR device). This is related to a traditional rehabilitation, that is, patients that perform exercises in a clinic trying to recreate the exercises with no visual reference by not using any kind of technology. Finally, the MR device Microsoft HoloLens 2 TM is used afterwards by the subjects through which they receive visual feedback about the exercise to be performed. Figure 7 graphically shows a simple example of an exercise defined by the therapist. It is defined as a semi-circular movement that must be performed with the right hand. Figure 8 depicts the movement as it is constantly appreciated by the user thanks to PhyRe Up! The rings represent control points that the user must physically reach, while the spheres represent the path to be followed. The yellow sphere represents the current point where the subject is located.
On the other hand, Figure 9 shows, by means of a blue line, the previous exercise defined by the therapist. In red, all the attempts made by different subjects without having a constant visual feedback are represented. As can be ap-  Figure 7 considering how it is rendered by PhyRe Up! preciated, the trajectories made by the subjects greatly differ from the trajectory defined by the therapist, despite being a relatively simple exercise. The greatest differences occur in the Z axis because the subjects lose the sensation of depth when watching the video in two dimensions.
Once the exercises were performed without the visual guidance provided by PhyRe Up!, the very same exercises were performed again, but this time with the assistance of PhyRe Up!, thus receiving continuous visual feedback. Figure 10 shows a comparison between the trajectory defined by the therapist (blue line) and the trajectories made by the subjects when using PhyRe Up! (green ones). In this occasion, it can be clearly seen how the subjects' joint makes a much more faithful trajectory regarding the one defined by the therapist.
In addition, Figure 11 includes both modes, i.e. paths made with and without visual feedback, along the path defined by the therapist. In this comparison the differences between one mode and the other can be seen at first sight. A simple path has been chosen in the previous graphs to demonstrate that the differences between one mode and another are already apparent from a simple exercise. In the case of complex exercises the differences are even more magnified.
Finally, the visual information included above is complemented by the data shown in Table 2, which summarizes the results obtained in the 25 conducted tests and compares the performance of exercises with, and without visual feedback with respect to the exercises defined by a therapist. The reference exercise was, again, a semi-circular movement with the right hand. The columns in the table, from left to right, represent the following items: test identifier, number of intermediate points or steps followed by the subject to perform an exercise without visual feedback, deviation on the x-axis, deviation on the y-axis, deviation on the z-axis and accumulated deviation on the three axes. These mea-FIGURE 9: Result of the rehabilitation exercise without feedback. In blue, the exercise defined by the therapist; in red, the trajectories traced by the test subjects. Each red line represents a single execution of an exercise performed by a subject without visual feedback. FIGURE 10: Paths followed by the subjects of the experiment when using PhyRe Up! In blue, the exercise defined by the therapist; in green, the trajectories followed by the test subjects. Each green line represents a single execution of an exercise performed by a subject with visual feedback. surements are computed by adding up the current position of the joint and the next control point. On the other hand, the last five columns have the same meaning, but regarding the execution of the exercise when PhyRe Up! was used. This information reflects that the visual feedback provided by our proposal reduces the number of intermediate steps to complete an exercise and decreases, to a great extent, the accumulated deviation. The latter implies that the accuracy of the performed exercises is much higher.
As a result, it can be noted that the MR-based visual  feedback mode dramatically reduces the total number of steps the subjects need to complete an exercise, as well as the total deviations from the therapist-defined points, which can be also observed in Figure 12. However, it should be highlighted that this result cannot be generalized for all population. Concretely, the last three subjects had more steps FIGURE 12: Graphic representation of the total deviation in the three axis with and without the feedback offered by PhyRe Up!. The y-axis represents the cumulative deviation measured in centimeters. The greater the deviation, the greater the difference between the movement defined by the therapist and the one made by the subject. and deviations after using the new technique compared no feedback, which are different to other subjects. The reason for this consideration may be attributed to the subjects' age, since they are found in the third age range. It seems to be logical because it is unfortunately usual that people who are older find themselves inexperienced using technology. VOLUME XX, 2021 FIGURE 13: Graphic representation of the deviation in the z-axis with and without the feedback offered by PhyRe Up!. The y-axis represents the cumulative deviation in centimeters on the z-axis between the movement defined by the therapist and the one performed by the subject. Table 3 shows the number of total intermediate steps that were needed to perform the 25 tests, the deviations accumulated individually on each of the axes and the total deviation in the 3D space. The last row shows the difference between not using PhyRe Up! and using the proposed system. Despite having taken an exercise whose path is simple, the differences are remarkable, especially in the Z-axis (see Figure 13). This could be when the subjects do not use PhyRe Up! since they watch an exercise by means of a traditional 2D video and lose the depth reference. However, the AR lenses and the 3D feedback offered by PhyRe Up! allow the subjects to have an accurate representation of the whole exercise.

V. DISCUSSION
Stroke represents a global challenge that affects significantly a part of the society, particularly elderly people. Plus, it is expected that population aging will have a negative impact on the coming years. In this context, physical rehabilitation is essential for stroke patients to recover mobility and improve quality of life. However, the face-to-face supervision of these therapies implies that the patients, and commonly their relatives, have travel related time and costs. In other words, a high cost both for patients/families and health services. Visual representation systems, supported by immersive technology, allow to recreate virtual rehabilitation environments at home. These solutions combined with accurate skeleton tracking methods have been proposed in the last few years for patients for home rehabilitation without the need for a continuous and face-to-face supervision.
The proposed system PhyRe Up! aims to help stroke patients perform rehabilitation exercises by evaluating how accurately they adjust to the physician's requests. The experiment presented herein aims to assess the evaluation of accuracy, as this is a key aspect when performing rehabilitation exercises. Validation of the system for accuracy is the first step before applying for a clinical trial with patients, who have suffered moderate or severe stroke, according to the levels measured by the National Institutes of Health Stroke Scale (NIHSS) scoring system. The need for future randomized control trials is required in order to assess feasibility and effectiveness of proposed system on the recovery of stroke patients.
Interestingly, the data obtained from the conducted experiment shows that our proposal drastically reduces the intermediate steps required to complete an exercise thanks to the visual feedback that PhyRe Up! provides (see Section IV). Furthermore, the accuracy of the performed exercise is higher compared to traditional rehabilitation techniques. The employed gamification-based approach complements the visual and auditory feedback by rewarding the patients when they achieve a goal. It is important to highlight that previous research has shown to be effective and motivating [23], [24], [27], but they do not mention an improvement on the accuracy of the exercises. Only one research [26] showed a better performance in terms of a decrease in steps to complete an exercise. However, this presents a limitation regarding the usability of the system. These aspects should highlight the benefits of our proposal which could help physical rehabilitation exercises be well performed and motivated, while being also user friendly and more portable than those shown in last research. However, new approaches, methods, and techniques need to be devised in order to maintain the quality of therapeutic exercises performance and patient motivation. The objective is to reduce the risk of patients dropping out the treatment.

VI. LIMITATIONS OF THE STUDY
The experiment conducted has been oriented towards the evaluation of accuracy and data acquisition, as this is a key aspect when proposing a system that allows autonomous but guided rehabilitation. This first step is essential before evaluating its usefulness with real patients who have suffered a moderate or severe stroke, according to the levels measured by the National Institutes of Health Stroke Scale (NIHSS) scoring system.
Once the proposal presented herein has been accepted by the scientific community, a series of clinical trials, according to the guidelines provided by physicians, will be carried out in a second phase. These are intended to be performed in rehabilitation center rooms where the system will be deployed. The technological requirements of the proposed system are not excessive (MR headset, hand-tracking system and standard laptop). In fact, it should not be considered as high-cost since it aims at reducing the burden of stroke treatment by facilitating treatment and by fighting the lack of specialized staff [7], whose cost is significantly higher. The main objective of the research, in this phase, is to show the usefulness, and even the acceptance, of the system in patient rehabilitation, studying and analyzing the influence of motivation on the patient's commitment.
Lastly, in a third phase, a clinical trial will be conducted to analyze the feasibility and effectiveness of the system for its intended use, which is to allow effective rehabilitation at home. In this phase, the system should be prescribed by the physician and approved by the administration, being left on loan for the duration of the treatment in those special cases that require it.
Apart from this, we hope that two aspects will be improved in the long term: 1) the cost of the technology applied in the proposed system herein is expected to be reduced, and 2) people are progressively acquiring digital skills. These expectations are intended to make the system available for everyone in a short period of time.

VII. CONCLUSIONS
In this work we have introduced PhyRe Up!, a non-intrusive system based on MR and gamification techniques designed to rehabilitate stroke patients at home. The adopted interaction mechanism and the knowledge from the therapist make it possible to adjust the rehabilitation routine to the patient's needs. The use of gamification components aims at maintaining motivation while the patients recover their lack of mobility. The feedback provided through our approach is also intended to provide guidance to ensure that rehabilitation exercises are correctly performed, that is, they are accurately executed, similar as when the therapist supervises them in person. Temporal and movement accuracy aspects have been especially considered when designing PhyRe Up!, since these positively affect the recovery success rate. An experiment has been conducted to validate these aspects with a group of 25 subjects performing rehabilitation exercises with traditional methods and using the approach proposed herein (rehab with MR device). The results seem to be promising because the MR-based visual feedback mode appears to improve the rehabilitation. The total number of steps that needs to be completed by a subject seem to be reduced, as well as the total deviations from the therapist-defined points. However, it should be highlighted that this conclusion cannot be generalized for all population considering the limitation of the sample size, blinding and methods used to define the experiment.
As future lines of research, and once the system has been validated in terms of its efficacy with attention to its accuracy, we want to launch two clinical trials. One to assess its efficiency and other one to analyze its influence on the patient rehabilitation. Furthermore, other aspects will be measured with the aim of exploring the efficacy of the system on patient recovery. The objective will be to evaluate the degree of improvement of a patient when the system is used continuously in a treatment.
Therefore, the data collected in these clinical trials will be of interest for the further improvement of the system, PhyRe Up!. Moreover, we are confident that these data could be used by machine learning algorithms to obtain a set of rules to guide the rehabilitation of each patient, offering personalized recommendations in a dynamic way, thus adapting their rehabilitation plan according to their level of progress. This will also be a line of work in the future.