Subjective Evaluation of the Effect of Exoskeleton Robots for Rehabilitation Training

Recently, the interest in the lower-limb exoskeleton robot is increasing within rehabilitation practice. While research on advancing the gait rehabilitation exoskeleton robot (ER) is being actively conducted from a technical point of view, studies evaluating usability and user value of ERs are insufficient. In this study, an evaluation tool was developed and verified, and an evaluation of an ER-based rehabilitation training program was conducted using the evaluation tool. A total of 102 rehabilitation patients (51 using ER and 51 under the existing training program [ETP] without robots) and 38 therapists participated in this experiment. The ER used in this study was Exowalk (HR-02, HMH Co. Ltd, South Korea). As a result of the study, the usability of the ERs consists of trust and overall satisfaction. The patient group using the ER showed statistically higher trust and overall satisfaction scores than the patient group with the ETP. On the other hand, the user value of the ER includes self-direction and security. The self-direction score of the patients using the ER was significantly higher than that of the patients with the ETP, but the security score showed the opposite tendency. This study is unique in that measures were developed to evaluate the usability and user value of the gait rehabilitation ER, especially for the Exowalk, verified by various user groups, and statistically analyzed to be used as a reference for the design of usability and user value in other gait rehabilitation ERs.


I. INTRODUCTION
Gait is the most basic method of transportation and has a significant impact on one's independence and quality of life Van Iersel et al. [1]. Gait problems can change a person's life regarding independence and also may cause significant health issues, both short-and long-term. Gait disability is quite common, as the global population is ageing Mikolajczyk et al. [2] and the stroke closely related the gait disability affects about one of six people globally Hobbs and Artemiadis [3].
The associate editor coordinating the review of this manuscript and approving it for publication was Dingguo Zhang .
According to the National Center for Health Statistics (NCHS), it is known that approximately 7.5% of the adults in the United States have difficulties in walking [4]. People with lower-limb dysfunction cannot conduct normal gait mobility, due to the blockage of their neural pathways. According to the medical research, physical rehabilitation, such as repeated exercises, is able to reactivate the neural pathways of the patients Tan et al. [5].
Repetitive gait training with human assistants is the most common method for gait rehabilitation Yoon et al. [6]. However, it is a labor-intensive process and requires large amounts of time and efforts from the therapists Low [7].
This also hinders patient access to rehabilitative gait training difficult -at present, due to the medical staff shortage and the growing number of patients, there are saturated waiting lists Pavón-Pulido et al. [8] and Benjamin et al. [9] and patient access is increasingly recognized as a serious problem.
A possible solution for lessening the patient access problem may be to employ lower-limb rehabilitation exoskeleton robots (ERs). Recently, electromechanical systems, such as exoskeleton robots, have been popular and got an interest in the rehabilitation community for their ability to automate the time consumed therapy works. Sale et al. [10] and Calabrò et al. [11]. Various lower-limb rehabilitation ERs have been developed to support the rehabilitation of different joints, gaits, and daily life activities Zhou et al. [12].
Robotic rehabilitation for gait training may provide consistent and efficient treatment reducing the workloads of the therapists. While research on advancing the gait rehabilitation ER is being actively conducted from a technical point of view, however, studies evaluating user experience (UX) of ERs are insufficient and this represents a serious research gap. In addition to utility, the experience of patients during the robotic rehabilitation treatment, including different dimensions of usability and user value, must be considered Low [7]. Usability is the degree to which a product or system can be used to achieve specified goals with effectiveness, efficiency, and satisfaction in a certain context (ISO, 9241-1:2001); while user value is the value that is satisfied when a user interacts with a product or system Park and Han [13]. Hill et al. [14] conducted the systematic literature review and analyzed studies that included a user perspective on exoskeleton technology. Only about 3 studies out of a total of 4619 articles were analyzed as actual participants directly evaluated ERs from the user perspective, and these studies also used different evaluation indicators. As there is no common tool to evaluate user experience, previous studies have used different evaluation tools. Kuzmicheva et al. [15] used evaluation factors of ''satisfaction'' and ''ease of use'' in terms of usability, and ''safety'' in terms of user value. Eicher et al. [16] evaluated usability based on two factors: usability and acceptability. Several studies have utilized the system usability scale (SUS) for usability of ERs Meyer et al. [17], Tsai et al. [18], and Ciobanu et al. [19]. SUS has the advantage of quick method, but it focuses only on user interface Bangor et al. [20] and has little to do with the usability of ERs for rehabilitation. Fundarò et al. [21] utilized Psychosocial Impact of Assistive Devices (PIADS) to measure the psychological impact of the robot assisted gait training. The PIADS is a measure to evaluate the psychological impact of using general assistive technology Day et al. [22], but it does not reflect the perspective of ER-based rehabilitation training.
The objective of the current study was to develop a usability and user value evaluation tool for ERs and ER-based rehabilitation training. An evaluation tool was developed by merging items from literature and verified through exploratory and confirmatory factor analysis. An assessment of an ER-based rehabilitation training program was conducted using the evaluation tool. The findings from the evaluation are discussed.

A. PARTICIPANTS
In this study, a total of 102 patients (64 males and 38 females) and 38 therapists participated in the evaluation of an ER-based rehabilitative training program, after excluding patients whose trials were suspended according to criteria determined by the person in charge, such as when a subject withdrew consent to participate in a clinical trial or could not follow the occurring procedure. The participants' average age was 56 years old (SD = 22.53). Of the participants, 51 patients were rehabilitated by using a particular type of ER -Exowalk R (HR-02, HMH Co. Ltd, South Korea), and the other 51 patients were rehabilitated within the existing training program. The existing rehabilitation training program (ETP) is a conventional gait therapy method in which the therapist assists the patient's gait from the side or rear. The 38 therapists consisted of those who assisted in the use of the ERs. The experiment was conducted at Dongguk University Ilsan Hospital, Chungnam National University Hospital, and Seoul National University Bundang Hospital in South Korea.

B. APPARATUS
The ER used in this study was Exowalk R (HR-02, HMH Co. Ltd, South Korea). This device is an active type ER that allows patients with severe disabilities to walk onboard. The dimensions of the robot are 1,180 mm, 980 mm and 1,350 mm in width, length, and height, respectively, and it weighs 230 kg. The footrest does not actually touch the ground but, instead, has the effect of moving forward at a constant speed through a rear motor (see Figure 1). Instead of simply walking on the treadmill (e.g., stagnant), this ER makes the patients feel like they are really walking forward because the robot moves according to their directions. The exoskleton was chosen because a preliminary result Nam et al. [23] showed its effectiveness for gait training and its positive impacts on user confidence and desire for continued use. Nam et al. [23] reported the therapeutic effects of Exowalk-based gait training (pre-vs. post-training) on walking ability of stroke patients in terms of functional ambulatory category (FAC). For the control group (patients with ETP), FAC was 2.44 ± 1.55 in the pretraining and 2.75 ± 1.53 in the post-training. On the other hand, for the experimental group (patients with ER), FAC was 3.22 ± 1.31 in the pretraining and 3.78 ± 1.44 in the post-training. The results showed that the FAC between preand post-training sessions were improved in both groups and indicated the therapeutic effectiveness of the ER in terms of rehabilitation. The difference of FAC between pre-and posttraining in the experimental group was statistically significant (p < 0.05).
Exowalk R has various advantages over the four types of lower-limb rehabilitation ERs and devices described earlier. Exowalk R allows gait training while moving forward instead VOLUME 9, 2021 of being kept at a position, and no additional tuning operations are required. It is easy to use, and there is no need for an operator's assistance after the initial setting. In addition, the patient can easily move straight forward, backward, and rotate left and right through the hand controller, and the walking speed can be adjusted, according to the degree of rehabilitation.
Exowalk R can be directly initiated by the patient. The robot supports the patient's lower body from behind. It allows the patient to have a more stable standing posture by placing both arms on the upper-limb cradles. When the patient moves in the desired direction, the electric wheel of the robot turns at a predetermined speed, making the entire robot travel. The robot can be controlled by the patient by default, but the therapist controls it separately from the side in order to ensure safety. The parts that are controlled at this time are walking speed, pattern, and training time. The difference between Exowalk R and conventional devices is the former has no harness for the upper-body of a patient because it holds the lower body at the thighs, calves, and feet.

C. MEASURES
Measures that can compare the aforementioned different situations were developed, verified, and used. Finally, usability (rehabilitation effectiveness, trust, and overall satisfaction) and user value, which both consider human value framework for quality-of-life evaluations Schwartz and Bisky [24], were measured by using a set of 10-point subjective rating scales.
First, for usability evaluation, rehabilitation effectiveness indicates whether the treatment protocol is successful. Trust deduces whether the treatment protocol is reliable enough to be received repeatedly. Overall satisfaction indicates whether the entire procedure is fulfilling.

D. EXPERIMENTAL PROCEDURE
Participants proceeded with the experiment after receiving and agreeing to a sufficient explanation of the purpose and procedure of this study. In preparation for safety accidents related to the walking rehabilitation robot, participants received training and guidance from a medical doctor and physical therapists before the training. In order to prevent a hazardous error, an automatic stop function was implemented on the ER. All of the existing rehabilitation treatments (e.g., central nervous system development treatment and strength training) were performed except for walking training in both the experimental group (the group using the ER) and the control group (the group under the ETP) participating in the experiment. The control group performed conventional gait rehabilitation, a gait therapy in which a physical therapist guides and makes the patient walk in a traditional way. In both groups, the walking practice session was administered for 30 minutes. This 30-minute walking practice session was considered to be one session, and a total of 20 sessions were conducted for each participant, five times a week within a four-week span. There was no break during each of the 30-min intervals. After the rehabilitation training was completed, questionnaire items on the usability and user value of the walking rehabilitation ER or the ETP were filled. The participants were asked to fill out the questionnaire, based on the walking practice sessions.

E. DATA ANALYSIS
The rehabilitation effectiveness, trust, and overall satisfaction questionnaire items were refined and verified through statistical analyses. In this process, exploratory and confirmatory factor analyses were employed. Based on the refined questionnaire, the analysis of variance (ANOVA) was applied for comparison between the ER and ETP groups. The data were filtered by applying Cho's sampling method for reducing insincere inputs Cho et al. [25]. As with the usability measures, the processed data were checked for differences between the groups by applying the ANOVA.

A. RELIABILITY AND VALIDITY OF MEASURES
Cronbach's alpha and exploratory factor analyses were applied to confirm the reliability and validity of the questionnaire items. As a result of the Cronbach's alpha analysis, the internal consistency of the questionnaire items was confirmed. The alpha values of rehabilitation effectiveness, trust, and overall satisfaction were 0.913, 0.823, and 0.888, respectively.
After applying the exploratory factor analyses, the questionnaire structure, as shown in Table 1, was finally developed. The initially developed questionnaire included four items for rehabilitation effectiveness, six items for trust, and six items for overall satisfaction (Appendix 2). Then, it was refined through the backward elimination process with the factor analysis. The elimination criteria consisted of two conditions: (1) factor loading should be greater than or equal to 0.6, and (2) there should be at least three items in one factor. The type of factoring method was the principal component analysis, and varimax method was selected as the factor rotation method. The suitability of the data was confirmed by two measures, the Kaiser-Mayer-Olkin measure of sampling adequacy and Bartlett's test, and the values were 0.873 and <0.05, respectively. The confirmatory factor analysis was applied to verify the refined structure of the questionnaire. As for the absolute fit indexes, the goodness-of-fit index (GFI) was 0.886 and the root-mean-square-error of approximation (RMSEA) was 0.118. In the incremental and parsimonious fit index series, the normed fit index (NFI) was 0.911; the comparative fit index (CFI) was 0.943; the ratio of chi-square to its degrees of freedom (CMIN/DF) was 2.502. Overall, the composition of the questionnaire items was accepted and adopted in this study, based on the aforementioned higher adequacies within the statistical data.

B. USABILITY MEASURES
For usability analysis, rehabilitation effectiveness, trust, and overall satisfaction scores were measured according to the questionnaire items selected by the factor analysis. The final scores were calculated by obtaining the average of the items included in each factor. Based on these scores, ANOVA was applied to test whether there were significant differences between the participant groups. In the case of rehabilitation effectiveness, there was no significant difference (p < 0.01) between the participant groups (Fig. 2). Patients who completed the ETP tended to be the most effective.
In the case of both trust and overall satisfaction, a statistically significant difference (p < 0.05) was observed among the participant groups ( Fig. 3 and 4). The patient group using the ER score was higher than the score of the patient group with the ETP.

C. USER VALUES
As a result of ANOVA, there were statistically significant differences in the two user values of self-direction and security (Fig. 5). There was no statistically significant difference in other values.

A. THE USABILITY ASPECTS OF THE ER FOR GAIT REHABILITATION
In this study, it was confirmed that the usability of the walking rehabilitation robot constitutes effectiveness, trust, and overall satisfaction. According to the International Organization for Standardization (ISO, 9241-1:2001), the usability of a software is the degree to which a product or system can be used to achieve specified goals with effectiveness, efficiency, and satisfaction in a certain context. However, unlike the definition of usability provided by ISO, the trust between the users and the ERs may be an important factor for evaluating the usability of an ER. Therefore, the trust can be considered in evaluating ERs, and this should be developed to improve the usability of the ERs by patients' perspectives.
This study evaluated the usability of the gait rehabilitation ER by patient groups: patients using the gait rehabilitation ER, and patients using the ETP. In previous research, few studies have evaluated the usability aspects of gait rehabilitation ERs by dividing user groups. In Kuzmicheva et al. [15]'s study, the gait rehabilitation ER was operated from the perspective of patients and therapists, and the patients had high usability scores, but the therapists had relatively low usability scores. However, since only three therapists participated in this study, the generalizability of the study findings seems limited. In Eicher et al. [16]'s study, usability was evaluated by dividing the young and elderly, and it was confirmed that there was no significant difference in usability evaluation scores between the two groups. As a result of this study, it was concluded that, when the gait rehabilitation ER was worn, the usability scores were significantly higher in terms of trust and overall satisfaction compared to the group that did not wear it.
Also, this study conducted a usability evaluation of the ER for gait rehabilitation, using subjective ratings. The questionnaire was systematically developed and verified through exploratory factor and confirmatory factor analyses. In the previous research, studies on evaluation tools that assess the usability of ERs were insufficient. Each research developed their own questionnaire without any verification process (Kuzmicheva et al., B). In general, the system usability scale (SUS) was utilized in many existing studies to evaluate the usability of ERs (Tsai et al. [18], Ciobanu et al. [19], and Eicher et al. [16]. The SUS evaluates 10 aspects related to system usability through the questionnaire Brooke [26] and is mainly used as a quick method for assessing software systems. However, the SUS is an inconvenient and incomprehensive evaluation method because the correlation between each item is high Lewis and Sauro [27], and mainly focuses on user interfaces (e.g., standard OS-based software interfaces, web pages and applications, cell phones, etc.) Bangor et al. [20]. However, the ER for gait rehabilitation is in the form of a robot in which an interface and hardware are combined, and since the actual user wears and uses the ER, other tools were needed to evaluate the usability of ERs. Therefore, in this study, the usability evaluation measures were identified and verified based on statistical analyses. The questionnaire, which incorporates the usability measures, is particularly useful for evaluating the interfaces of ERs efficiently and comprehensively.

B. THE USER VALUE ASPECT OF ER FOR GAIT REHABILITATION
The purpose of this study was to investigate the values that patients consider most important while using gait rehabilitation ERs. In addition, we tried to investigate whether there were any significant differences between the user values after each patient proceeded with the gait rehabilitation ER and general rehabilitation ETP. As a result of comparing the two patient groups (ER vs. ETP), only the 'self-directed' and 'security' items showed different scores among the two groups. First, the patient group using the gait rehabilitation ER evaluated the self-directed items higher than the patient group using the ETP. It is presumed that the patients using the ER were able to practice walking independently. This level of freedom fits well with the design intention, which allows a patient to control the ER's walking speed or pattern. In addition, while wearing the ER, the patient uses the electric wheel of the ER to walk when the patient wants to move in their desired direction. This functionality of the ER promotes the self-direction of gait rehabilitation by providing freedom to the patients.
Second, the participants who used the ETP felt safer than who utilized the ER. This could be caused by two reasons: (1) the users' fear was caused by first encountering a robot that performs rehabilitation training, and (2) there is human assistance in the ETP. It is thought that different results may arise if the rehabilitation training with the ER is carried out long-term. Indeed, according to Kuzmicheva et al. [15]'s study, the satisfaction, ease of use, and safety aspects all tended to increase as the patient was further exposed to the gait rehabilitation ER. However, since safety is still an important item that gait rehabilitation ERs must have Wu et al. [28], future research needs to be conducted to improve the safety of ERs.

C. LIMITATIONS
This study contains a user group of a heterogeneous nature. The study could not be carried out with various types of patients, so the experiment was conducted only on patients recovering from strokes (cerebral hemorrhage or cerebral infarction). In addition, this study used only one type of ER. Exowalk R is a commercialized gait rehabilitation ER, which is an immersive device; therefore, the results could differ from other commercialized ER types in terms of usability and user value. Lastly, the usability and user value were evaluated as a single unit, so the change in usability and user value in the longitudinal aspect could not be confirmed. In particular, considering the previous study that user value changes with time, it is necessary to study in the longitudinal aspect later Kim et al. [29].

V. CONCLUSION
In this study, the usability and user value of the gait rehabilitation ER were assessed, based on the patients' use of the ER and the ETP, to develop an evaluation tool for advancing the interactions between users and ERs. We divided user groups into patients who participated in rehabilitation programs using ERs, and patients who participated in the ETP. The usability of gait rehabilitation ERs was composed of effectiveness, trust, and overall satisfaction. Compared to patients with the ETP, patients using the ERs were found to feel significantly higher trust and overall satisfaction. In terms VOLUME 9, 2021 of user value, patients utilizing the ERs exhibited significantly higher 'self-direction' and lower 'security' values compared to patients treated within the ETPs. It is understood that the patients' feelings of safety regarding the ERs are still insufficient, and the development of the ERs necessitates enhancing security in the future. The results of this study are significant because the usability and user value of the rehabilitation ER were identified. The identified measures will be useful to efficiently and comprehensively evaluate the interactions between users and ERs to allow for advancement in human-robot interaction for effective gait rehabilitation.