Contents Introduction
Persistent motor dysfunction in the hands severely affects the quality of life of stroke survivors by limiting their activities during daily life [1]. Impaired finger extension is one of the most common symptom after stroke [2] and leads to poor upper limb recovery by diminishing hand opening function [3]. Despite physical and occupational therapy, approximately 55% of stroke survivors still suffer from long-term hand disabilities after three to six months of rehabilitation [4]. Even four years post-stroke, nearly 70% of stroke survivors still experienced non-use or disuse of the affected arm [5]. These studies indicate the need for improved hand rehabilitation techniques and strategies for stroke survivors.
Although high-intensity, task-oriented, and repetitive training has been proven to be one of the best approaches to inducing neuroplastic changes for restoring hand function [6], [7], it is difficult to achieve without a high level of engagement by stroke survivors. Engagement is defined as deliberate effort and commitment to work toward the goals of rehabilitation interventions, with active participation and emotional involvement in the therapies [8]. Previous studies have suggested that the level of engagement is closely related to the level of functional improvement [9], [10]. Also, engagement has been reported to be associated with reward-related dopaminergic systems in the brain that facilitate neural plasticity and motor learning [11], [12]. However, stroke survivors are easily distracted and lose interest due to the repetitive and monotonous nature of conventional rehabilitation training, which results in a decrease in engagement [13]. Such training even leads patients to neglect the training prescribed for recovery [14]. To address this, various approaches have attempted to find ways to induce and sustain engagement during rehabilitation training.
As promising tools to promote engagement, virtual reality (VR) games have been utilized with rehabilitation programs over the past decade [15], [16], [17]. In VR games, training conditions can be modified through content manipulation, so task-oriented, high-intensity, and repetitive movements can be effectively implemented in a state of a high level of engagement [18]. In addition, using VR games in rehabilitation has proven to be effective in recovering motor function by integrating visuoauditory feedback related to movement and cognitive processes [19]. Recently, contact-free hand tracking technology has been used as a control input for playing such VR games, but there are few studies of its application in hand rehabilitation [20], [21], [22]. Even in those few cases, the number of task repetitions is insufficient to achieve neuroplastic change and functional improvement [23]. Also, the possible gestures that can be trained are limited since the condition (range of motion, degree of spasticity, etc.) of the stroke survivor’s hand was not considered.
In this study, we present a VR-based hand rehabilitation system that incorporates a rhythm game with vibrotactile feedback, designed to facilitate task-oriented hand gestures for stroke survivors based on their individual hand conditions. The inclusion of a music-based rhythm game in our rehabilitation paradigm is supported by the potential benefits it offers to stroke survivors. Music-based interventions, particularly rhythm-based training, have been shown to improve not only motor and cognitive outcomes but also the emotional and psychological aspects of stroke survivors [24], [25], [26]. Rhythm games specifically can promote motor learning by enhancing timing and coordination in movements, as they require users to synchronize their movements with the beat or rhythm of the music [27], [28]. With their enjoyable and motivational nature, introducing rhythm games is expected to encourage high repetitions of movements and sustain a high level of user engagement by varying the music selection and difficulty level.
Our system incorporates a bimanual task based on research demonstrating the effectiveness of bimanual training in stroke recovery, specifically showing that repetitive bilateral arm training with auditory cueing leads to significant improvements in motor function compared to unilateral exercises for stroke survivors with hemiparesis [29]. In our VR game system, players need to make the same gestures at an approaching target which is customized to their hand condition. When they perform the gestures correctly with precise timing, multisensory feedback consisting of vibrotactile, visual, and auditory feedback is provided at the same time. Previous studies have shown that this combination of triple sensory modalities achieved better results in terms of attention than single or dual sensory modalities [30]. For these reasons, the proposed rehabilitation system is believed to enable intensive, task-oriented, and repetitive hand training with a high level of engagement.
To validate our hand rehabilitation system, we used functional near-infrared spectroscopy (fNIRS) to measure the brain activation of healthy subjects and a stroke survivor. With fNIRS, we can directly observe brain activity while subjects perform various motor, sensory, and/or cognitive tasks. Our primary focus is on understanding cortical activation patterns during the intervention, which enables us to examine the potential effectiveness of our proposed rehabilitation system. Specifically, fNIRS is a non-invasive method for measuring the hemodynamic response associated with activation of the cerebral cortex based on the intrinsic optical absorption of blood [31]. Until now, only a few studies have used fNIRS to measure brain activation induced by rehabilitation training using VR, and the area of interest has also been limited to motor and sensory functional areas [32]. In this study, in addition to the sensorimotor area, other areas such as the prefrontal, premotor, and somatosensory association areas that are likely to be activated while using the proposed hand rehabilitation system were set as areas of interest.
Methods and Material
A. Hand Rehabilitation System
We developed a hand rehabilitation system consisting of a gesture-controlled VR rhythm game and vibrotactile feedback, as depicted in Fig. 1. We opted for a head-mounted display (HMD) to provide an immersive VR experience, as HMD-based systems elicit greater presence and immersion than 2D screen-based systems, enhancing user engagement and motivation during rehabilitation [33]. To implement the system, we employed Meta Quest 2 (Meta Platforms, Inc., Menlo Park, CA) as the VR HMD, C2-HDLF vibrotactors (Engineering Acoustics, Inc., Casselberry, FL), and the fNIRS system to measure brain activation during gameplay for experimental validation (Fig. 1a). Compared to other VR devices, the Meta Quest 2 has the unique advantage of enabling hand tracking using only embedded cameras, without extra hardware. It can track hands at up to 60 Hz by using a multi-stage process to estimate hand pose and finger joint angles in real-time [20]. Regarding the hand-tracking accuracy of Quest, Abdlkarim et al. reported an average fingertip positional error of 1.1 cm, an average finger joint angle error of 9.6 degrees and an average temporal delay of 0.038 ms [34]. To provide vibrotactile feedback strong enough to be felt even by stroke patients, we selected the C2-HDLF, a moving magnetic actuator capable of generating strong localized tactile sensations. It delivers a high displacement output range of 0.5 mm to 1.3 mm in the 50–160 Hz range.
Hand rehabilitation system. (a) The configuration of the system. Participants wear a VR HMD and fNIRS cap on their head and vibrotactors on the volar side of both wrists. Four hand-tracking cameras built into the VR HMD recognize the participant’s hand gestures. (b) VR scenes shown to the player when playing the VR rhythm game. From the target generator, a target gesture of each hand is created and approaches the player’s hand. The player is required to match the gesture of their hand to the approaching target. (c) The process of the hand making contact with a target gesture. When the player makes a proper gesture toward the approaching target, the color of the player’s hand changes to the same color as the target, and the target explodes in the event of a collision, leaving multiple cubic debris flying which disappear in a second.
To develop the VR rhythm game for the hand rehabilitation system, the Unity 3D game engine (version 2020.3.10f1 LTS release) was chosen as the software platform since the Meta Quest 2 has a software development kit (SDK) for development in Unity 3D, and the Oculus Integration asset (version 33.0), which includes a hand and finger tracking application programming interface (API). In addition, TDK-API (version 1.0.6.0) was used to generate vibrotactile feedback according to events that occurred in the game.
Through the VR HMD, VR scenes are shown to the player when they play the VR rhythm game (Fig. 1b). Target gestures for each hand are generated by a target generator located in front of the player and approach the hand of the player to the rhythm of the music. The player is required to match the gesture of their hand to the approaching target. If the player makes the same gesture at the approaching target, the player’s hand color changes to the same color as the target and the target explodes in the event of a collision (Fig. 1c). Specifically, when the hand makes the same gesture as the target and collides with it, the system provides a 250 ms pulse of vibrotactile feedback, simulating the impact of the target exploding, along with visual and auditory feedback of the explosion event. In particular, only the hand that performed the gesture correctly is provided multisensory feedback. Two vibrotactors were attached to the volar side of the wrist since previous studies reported cerebral sensory cortex activation and restoration of sensory function when vibrotactile feedback was provided to the wrist [35], [36]. In addition to visual feedback, such as the explosive effects of the target gesture, the game score is displayed in the upper left corner of the field of view in real-time because it can support enhanced user motivation [37].
B. Hand Gesture Recognition
Our hand rehabilitation system requires the player to repeatedly perform the same gestures as the target gesture according to the rhythm of the music. As shown in Fig. 2, three gestures were selected as target gestures to be recognized: finger extension, medium wrap, and lateral pinch. The reasons for selecting the above three gestures are as follows. First, spastic clenched fist deformities can be commonly observed in stroke survivors due to spastic hypertonia of the finger flexors and weakness of the finger extensors [38], and finger extension is the motor function most likely to be impaired by clenched fist deformities [39]. Second, medium wrap and lateral pinch are the two most important types of grasp that can be used to grasp most types of objects [40].
Gesture recognition method. There are three target gestures to be recognized, and each classified using 3D distance (DIST) between the 22 hand points and the reference point. Prior to gesture recognition, each player’s unique set of 22 DISTs for the three gestures is pre-recorded as reference gestures. During gameplay, if the difference between the DIST of the current gesture and the pre-recorded reference gesture is less than a predetermined threshold value for all hand points, the player’s hand gesture is recognized as the target gesture, and the color of the player’s hand changes to match the target.
The algorithm for gesture recognition is as follows. We employed a hand model consisting of 22 hand points and a reference point (Fig. 2). Real-time 3D distance (DIST) is calculated between each of the 22 hand points (
Before playing the VR game, we pre-recorded each player’s unique set of 22 DISTs for each of the three gestures and saved them as DISTs of reference gesture. This personalized reference gesture is essential because, even when performing the same gesture, the hand posture varies depending on an individual’s motor ability, especially among stroke survivors. When setting the reference gesture for the affected side, we used the individual’s limited posture as the reference. In contrast, for the non-paretic side, a fully performed gesture served as the reference. During gameplay, if the difference between DIST(
In this study, we set the threshold value, indicating the required recognition accuracy, to 2 cm, using an empirical approach to determine a suitable level of difficulty. Target gestures are randomly generated as one of three gestures according to the beat of music and approach towards the baseline, where the player’s hands are positioned, at the preset speed. To sequentially eliminate the approaching targets, players must imitate the same gesture of the nearest target to have their gesture to be recognized as the target gesture. Then, as the player’s hand color becomes the same as the target, the player must maintain the gesture until their hand comes into contact with the nearest target, causing the target to explode and disappear. The player then needs to either keep the current gesture or switch to another gesture, depending on the subsequent target gesture coming towards the baseline.
C. Validation Experiment
1) Participants:
Eleven healthy subjects (25.4 ± 4.4 yrs) and one stroke survivor with mild left-sided hemiplegia (22 yrs) were recruited for the validation of our hand rehabilitation system, as shown in Table I. The stroke survivor had experienced a neonatal ischemic stroke in conjunction with apnea over 22 years ago, resulting in lesions on the lateral side of the right SM1. He had a Modified Barthel Index score of 75, reflecting his capacity for basic self-care tasks with some assistance needed for more complex activities. He was able to perform finger opposition and flexion well, while finger extension was less smooth but still possible. The experimental protocols were approved by the Institutional Review Board at the Korea Advanced Institute of Science and Technology (KH2022-001, approved on 01/28/2022); written informed consent was obtained from each subject before participation. The experiment was conducted in accordance with the latest Declaration of Helsinki.
2) Experimental Setup and Procedure:
To measure brain activation, NIRSport2 (fNIRS, NIRx Medical Technologies, Glen Head, NY, USA) with continuous near-infrared light of two wavelengths (760 nm & 850 nm) was used. After establishing a region of interest (ROI) in the cerebral cortex, a custom channel montage was employed to measure task-related brain activation. Based on the international 10–20 system [41], the optodes and channels are arranged as shown in Fig. 3. The distances between source and detector were kept to 3 cm and the channel was placed in the center between them. 16 detectors and 16 sources constructed 51 channels in total and the channels were divided into 4 groups to cover the ROIs as follows: The primary sensorimotor cortex (SM1), the premotor cortex & the supplementary motor area (PMC&SMA), the somatosensory association cortex (SAC), and the prefrontal cortex (PFC).
fNIRS optodes and channel configuration. (a) Thirty-two fNIRS optodes (16 detectors and 16 sources) that formed 51 channels were used to cover 5 Regions of interest (ROI): the prefrontal cortex (PFC), the premotor cortex & the supplementary motor area (PMC&SMA), the primary sensorimotor cortex (SM1), and the somatosensory association cortex (SAC). The placement of each optode was based on the anatomical landmark location of the international 10–20 systems. (b) The 3D schematic images depicting the position of the optodes in the head scalp. Columns from left to right: lateral view of the left hemisphere, superior view, lateral view of the right hemisphere. (c) Schematic representation of the protocol. The experiment consisted of 12 tasks and 13 rests based on the block design paradigm. The duration of rest and task blocks is 20 seconds. A: anterior, P: posterior, L: left, R: right.
Based on the block design paradigm [42], detailed in Fig. 3c, measurements were performed with a total of 10 repetitions in a block consisting of 20 seconds rest, 20 seconds task of VR-based hand rehabilitation, and an additional 20 second rest section. In the task Section of the block, subjects played a VR rhythm game with the music. In the rest Section of the block, the music stopped and subjects were asked not to move and rest. In this manner, a session comprises 21 blocks, taking 420 seconds. Note that the target gestures given to the player are totally randomized for each subject.
The song “Believer” by Imagine Dragons, which has a strong beat and is widely used in many commercialized rhythm games, was selected as the music for the VR rhythm game. The original song is about 200 seconds long and has a tempo of 125 bpm. However, we found that the original tempo was too fast, making it difficult to manage for those who first tried our rehabilitation system. Therefore, we halved the tempo to 63 bpm, which generated a total of 209 target gestures throughout the song, with 21 gestures in 9 blocks and 20 gestures in 1 block. The target gestures are generated alternately between the left and right side, with the order of generation starting from the left side. Thus, participants perform a total of 209 gestures, consisting of 105 left-hand gestures and 104 right-hand gestures throughout the game. For each participant, the number of missed targets was counted, and the success rate was calculated.
Details of the measurement procedure and instructions provided to the subject are as follows. As a subject enters an experiment room insulated from noise and light, he or she is asked to sit down in a chair facing the wall. Then the subject is instructed on how to wear the VR HMD, and the operator attaches vibrotactors to both wrists of the subject. After placing the optode holder cap on the subject’s head and inserting the optodes into the cap, we give a few instructions before the measurement starts, such as “close your eyes during the rest section” and “do not voluntarily move the other body parts except your hands.” These instructions essentially complete the preparation for the measurement.
Motion artifacts can be generated by body movement of the user — especially the head — which can lead to the decoupling of optodes from the scalp and subsequently produce high-frequency spikes and shifts in the baseline measurements [43], [44]. In order to prevent head movement, we designed all the contents provided in the VR rhythm game so that they could be viewed at a glance without head movement. Before the experiment, the height of the desk was adjusted to subjects, and we instructed the subjects to take a comfortable posture while placing the central or ulnar sides of the dorsal wrist on the desk to ensure that only hand movements occurred. Throughout the experiment, we closely monitored the participants’ entire body, with particular attention to the contact area between the wrist and the desk, to verify that no unintentional body movements took place. After the experiment was over, we collected verbal feedback from the subjects to verify the absence of any unintentional body movements during the session.
3) Data Analysis:
Data analyses were carried out using the open source software Homer3 (version 1.33) implemented in MATLAB R2021a (Mathworks, Natick, MA, USA) [45]. First, the raw NIRS signals were converted into optical density (OD). Afterward, motion artifacts in the optical density data such as baseline shifts and high-frequency spikes were detected and corrected using a hybrid method based on the spline interpolation method and Savitzky-Golay filtering [46]. The data were then low-pass filtered with a cut-off frequency of 0.5 Hz in order to remove high-frequency oscillations. Subsequently, the changes in the OD signal were converted into concentration changes of oxygenated hemoglobin (HbO) and deoxygenated hemoglobin (HbR) by employing the modified Beer-Lambert law with a partial pathlength factor of 6 [47].
The hemodynamic response function (HRF) of HbO (or HbR) was estimated with a general linear model (GLM) approach that uses iterative weighted least squares [48]. The HRF was modeled as a series of consecutive Gaussian temporal basis functions with a standard deviation of 1 s and their means were separated by 1 s over the time range of −2 s to 20 s [49]. After the model extraction, each HRF was normalized using the mean amplitude from −2 s to 0 s before the task onset, which is the point when the rest period ends and the task begins. Then, the activation value of HbO (or HbR) representing the task was calculated by subtracting the mean HbO (or HbR) response amplitude of the baseline period from that of the task period. The baseline periods are defined as the 2 s of fNIRS measurements preceding the onset of the task period. Considering the delay in hemodynamic responses, we used intervals between 5 and 15 s after the task onset as the time window to calculate the mean response amplitude in the task period. We selected this time window to capture the peak of the hemodynamic response, which typically occurs around 5–6 s after the stimulus onset, and its subsequent decline [50]. Additionally, a previous study demonstrated higher classification accuracy using the average HbO value within the 5–15 s window compared to the entire 20 s task duration, supporting that this window captures the most task-relevant hemodynamic changes [51]. In this way, the activation value of HbO (or HbR) was calculated for all participants and channels as an indicator for statistical analysis.
To test the assumption of normality of the activation values for each channel, the Shapiro-Wilk test was performed and found that all data followed a normal distribution for all channels at the significance level
A projection of the estimated hemodynamic response of HbO onto the brain cortex was carried out using the MATLAB-based software AtlasViewerGUI (version 2.16.1). It was used to transfer optode positions into Montreal Neurological Institute coordinates (MNI) [55] by registering the 3D location of the optodes and channels to the head surface of the Colin Atlas, and projecting them onto the cortical surface [56]. We then found the corresponding Brodmann Area (BA) based on the automatic anatomical labeling (AAL) database, and used that information to assign each channel to ROI [57]. Then, an image of the cortical activation was reconstructed by solving the inverse problem with a regularization scaling parameter = 0.01. Finally, the mean hemodynamic response of HbO between 5 and 15 s during the VR-based hand rehabilitation was projected onto a brain cortex.
4) Survey on User Experience:
To better understand how participants felt about the VR hand rehabilitation system, we carried out a survey after the experiment. The survey used a simple 5-point Likert scale to assess user engagement, presence, motivation, and the role of vibrotactile feedback in enhancing these factors. Although the survey was conducted at least six months after the experiment, all participants remembered the experiment except for one healthy subject who could not be reached, and the survey provided helpful insights into the user experience. The survey had five statements about engagement, presence, motivation, and the impact of vibrotactile feedback. Participants were asked to rate their level of agreement with each statement after using the VR system. The rating scale ranges from 1 (strongly disagree) to 5 (strongly agree).
Results
Here we present five main results regarding the success rate of the VR rhythm game, the significantly activated channels, the cortical activation map, the time course of the hemodynamic response function, and the survey results. Group average results for the healthy subjects and the individual results for the single stroke survivor are reported.
A. The Success Rate of Gesture Matching
The success rate of gesture matching was obtained by counting the number of missed and mismatched target gestures among a total of 209 target gestures presented in the VR rhythm game. Table II shows that the success rate of gesture matching among healthy subjects ranged from 68.6% to 99.3% with a mean of 90% and a standard deviation of 10.7%. The success rate of the stroke patient was 79.6%.
B. The Significant Channels
Table III shows the results of paired t-test analysis for HbO activation values in the healthy group for each channel and the corresponding ROI, comparing the VR game condition to the resting condition. This analysis revealed significant increases in HbO in 19 out of 51 channels with a significance level of 0.01 and 10 degrees of freedom. And the ROIs involving these channels were both sides of SM1 and SAC, the right sides of PFC and PMC&SMA, and the center of the SM1 and PMC&SMA.
C. The Cortical Activation Maps
Fig. 4 shows the results of the cortical activation maps for the healthy group and one stroke survivor stimulated by the VR-based hand rehabilitation system with vibrotactile feedback. An increased HbO response is highlighted in red while a decreased HbO response is highlighted in blue, respectively, when comparing the VR game condition to the resting condition. Regardless of whether the subject was a healthy subject or a stroke survivor, increased cortical activation was observed for all ROIs (i.e., both sides of SM1, SAC, PFC, and PMC&SMA). In the stroke survivor, the SM1 activation area was wider on the contralesional side than on the ipsilesional side. In addition, a widespread decrease in HbO response was found in both lateral sides of the brain of the stroke survivor, compared to healthy groups.
HbO cortical activation map of the brain surface during the VR-based hand rehabilitation task compared to the resting condition. (a) Group mean HbO map of the healthy group. (b) Individual HbO map of 1 stroke survivor. Each reconstructed image is averaged over the time course from 5 sec to 15 sec. An increased HbO response is highlighted in red while a decreased HbO response is highlighted in blue. The color bar indicates the scale of the concentration change in micromol/L. Red dots are source optodes, and blue dots are detector optodes. A: anterior, P: posterior, L: left, R: right.
D. The Time Course of Hemodynamic Response Function
Fig. 5 and Fig. 6 present the temporal change in the HRF of HbO (or HbR) related to the VR-based hand rehabilitation task and vibrotactile feedback in 4 ROIs. The averaged HRF time course with confidence bands for the healthy group is depicted in Fig. 5. It is observed that both sides of the brain have similar time course trends in all ROIs except the prefrontal cortex. In the SM1 and SAC areas, HbO showed a tendency to increase after the start of the task and then gradually decreased until the end of the task, and HbR showed the opposite tendency. Regarding the PMC&SMA area, a temporary increase was confirmed immediately after the start of the task, but a tendency to decrease rapidly was observed. While PFC showed a tendency to increase in the right brain, it was found that there was almost no change throughout the task in the left brain.
Group-level HRF time course of oxy and deoxy hemoglobin changes (red and blue respectively) of the healthy group during VR-based hand rehabilitation. Four ROIs (PFC, PMC&SMA, SM1, and SAC) are divided into two hemispheres, and the HRF time course is averaged for the channels contained in each ROI. The shaded areas show 95% confidence bands. A: anterior, P: posterior, L: left, R: right.
The HRF time course of oxy and deoxy hemoglobin changes (red and blue respectively) of the stroke survivor. Four ROIs (PFC, PMC&SMA, SM1, and SAC) are divided into two hemispheres, and the HRF time course is averaged for the channels contained in each ROI. A: anterior, P: posterior, L: left, R: right.
The individual HRF time course of a single stroke survivor is shown in Fig. 6. A similar time course was observed in both sides of SM1, right PFC, left PMC&SMA, and left SAC; HbO showed a tendency to increase after the start of the task and then gradually decreased until the end of the task, and HbR showed the opposite tendency to HbO. However, the right SAC showed a similar trend to the above-mentioned area except for the initial decrease after the start of the task. The left PFC and right PMC&SMA showed a similar trend, with both HbO and HbR gradually increasing and decreasing.
E. The Survey Results on User Experience
We received responses from all subjects except one healthy subject. As shown in Table IV, the results revealed that the all participants found the VR hand rehabilitation system engaging (63.6% agreed and 36.4% strongly agreed). In terms of presence, 72.7% of participants agreed or strongly agreed that they experienced a sense of presence, while 27.3% uncertain or disagree. They also reported being motivated to continue using the system for their training (90.9% agreed or strongly agreed). The inclusion of vibrotactile feedback was found to contribute positively to both engagement (81.8% agreed and 18.2% strongly agreed) and presence (63.6% agreed and 27.3% strongly agreed). Notably, responses from stroke survivor, marked with an asterisk in Table IV, showed a very positive perception of the system, including engagement, presence, motivation, and the beneficial impact of vibrotactile feedback on their experience.
Discussion
In this study, we proposed an immersive hand rehabilitation system based on VR rhythm games and vibrotactile feedback, which is capable of performing task-oriented, repetitive, and intensive tasks with a high level of engagement. Compared with previous studies that developed VR-based hand rehabilitation systems using contact-free hand-tracking technology, our study has the following three differences [21], [22].
First, the reference gestures used for gesture recognition can be personalized for each participant, and how precisely the gesture should be performed can be adjusted by changing the threshold value. As shown in Table II, with a recognition threshold value of 2 cm, the gesture matching success rate of the stroke survivor was 79.6 % and the average success rate of 11 healthy was 90 % (SD = 10.7 %). A similar accuracy was found in the study by Yuan and Folmer [58], in which the accuracy ranged from about 75 to 90 % when novice players played a commercial rhythm game four times. This indicates that an appropriate threshold value for gesture recognition was selected and that all participants, including the stroke survivor, performed the gestures correctly according to these criteria.
It is important to note that two healthy participants exhibited lower success rates compared to the stroke survivor, despite having better motor ability. This observation suggests that factors related to the game other than motor ability, such as prior experience, familiarity, and preference, could contribute to the observed variation in success rate [59]. Direct confirmation with both healthy participants and the stroke survivor revealed that the healthy participants had less experience with the VR game, showed less preference and familiarity with the game than the stroke survivor did. This highlights the necessity to take these factors into account when implementing game-based rehabilitation interventions and interpreting results within the context of a gaming-based rehabilitation system.
Previous studies have utilized built-in hand recognition functions which could recognize only limited hand gestures, such as poke, pinch, and finger opposition, and did not consider how accurately the hand movements were being performed. In contrast, for this study three training motions— lateral pinch, medium wrap, and finger extension, among the many possible hand motion options, were selected. These motions have proven to be the most commonly used postures for gripping and releasing objects of various shapes and sizes in daily life [40], and have also been adopted as target movements in recently developed soft and exoskeleton robots [60], [61].
Second, a rhythm game was selected as the VR game content to facilitate intensity control and repetitive training. Previously, Ogun et al. selected daily life behaviors such as handling cubes, picking up vegetables from a bowl, and kitchen experience [21]; Pereira et al. selected farm activities such as harvesting crops, milking the cow, and making cheese [22]. Those studies reported that high user engagement was successfully induced, however, the number of repetitions of tasks was insufficient to achieve neuroplastic change and functional improvement [23]. It has been reported by previous research that the number of repetitions must be at least 300 times a day to achieve functional improvement of the upper extremity [62]. Our system guided participants to perform 209 gestures throughout a single 200-second song, excluding the time for rest period required for brain activity measurement. This indicates that our system is capable of facilitating a significant number of repetitions in a daily routine for recovery. In addition, the intensity of training could be easily adjusted by manipulating the type of song and the frequency or pattern of the target appearance.
Third, vibrotactile feedback was used to induce a high sense of presence. It is now widely used in video games and VR games to the extent that it is difficult to find products without vibration tactile feedback among commercial game controllers, and it has been reported to provide a stronger sense of presence than audio-visual feedback alone [63]. It was attached to the median nerve of the wrist to provide vibration without interfering with hand tracking, and the effect of vibrotactile feedback will be discussed in detail using the brain activation results described later.
For the system validation, fNIRS was used to monitor the cortical activation and time course of the hemodynamic response function in four ROIs (PFC, PMC&SMA, SM1, and SAC) during the VR-based hand rehabilitation task compared to the resting condition. As is evident from the significantly activated channels shown in Table II and cortical activation maps in Fig. 4, the introduction of our system gave rise to significant activations in both sides of SM1 and SAC, and the right sides of PMC&SMA and PFC. It is plausible that our findings result from a synergistic effect due to the combined influence of factors integrated into our system, such as VR game, music-based rhythm, and vibrotactile feedback. This is supported by previous studies reporting brain activity associated with each of these factors; the brain regions activated by our VR system are consistent with and overlap those activated by individual factors in previous research. Specifically, VR game can activate the PFC, PMC&SMA, and SM1, as demonstrated in studies examining a finger opposition rhythm game without music [64] and participants catching an approaching ball in VR using a button press with their index finger [65]. Additionally, music has been shown to activate the PFC, SMA, and SAC, with activations influenced by the tone and pitch of the music [66], [67], as well as during the listening of popular songs [68]. Finally, activation of the somatosensory cortex and SAC has been observed when vibrations were applied to the index finger [69] or the volar side of the wrist [70].
To further understand the potential mechanisms underlying the activation of specific brain regions in our study, we consider the following explanations: First, SM1 activation might be primarily due to the hand gestures performed during VR gameplay, as SM1 plays a crucial role in motor execution and sensorimotor integration during voluntary hand movements [71], [72]. Second, the activation of PMC&SMA is thought to be due to the planning and execution of the sequential hand gestures required to play the VR rhythm game that aims to remove each approaching target in a sequential manner. Findings by Halsband et al. [73] support this explanation, as they revealed that the PMC&SMA are crucial for the generation of sequential movements, and stroke survivors with lesions in these areas had difficulty reproducing rhythmic sequences by tapping a keyboard, even when provided with rhythmic auditory stimuli before the experiment. Third, SAC activation could be stimulated by the multisensory feedback, which includes vibrotactile feedback provided when a correct gesture is made in the VR rhythm game. This interpretation is in line with previous research findings [74], [75]: Reed et al. observed SAC activation during tactile object recognition tasks, while Andersen et al. found that SAC activation occurred when movement-related multisensory feedback was provided. Finally, there are several potential reasons for the observed PFC activation. As the PFC is known for its role in connecting visual information with motor responses and exhibiting increased activation during sensorimotor tasks with high cognitive load [76], the constant decision-making required in our VR system to respond to randomly generated targets amplifies cognitive demand. Additionally, the PFC functions as a hub for processing motivation- and attention-related information [77], potentially aiding in the voluntary allocation of attention to the VR rhythm game content.
Notably, the activation area of the PMC&SMA and PFC was more extensive in the stroke survivor compared to the healthy group. This finding is in line with previous research that reported the bilateral recruitment of PMC&SMA in stroke survivors during motor tasks, suggesting that more effort is required by the brain to accomplish the same motor task as healthy subjects [78]. Moreover, previous study has demonstrated prominent PFC activation during VR-based upper limb rehabilitation in stroke survivors, indicating that the brain reallocates the workload as a compensatory strategy, thereby preferentially increasing the activation of PFC [79].
As to the Group-level HRF time course of the healthy group, an increase in HbO and a decrease in HbR, the typical patterns of response to stimuli, were observed during the VR-based hand rehabilitation for all ROIs in Fig. 5. An increase in HbO means an increase in blood inflow from the artery to the activated area, and a concomitant decrease in HbR means an increase in oxygen metabolism [80]. Our findings are in line with previous studies in which the hand rehabilitation task is associated with an increase in cerebral blood flow and brain oxygenation [81].
For the individual HRF time course of the stroke survivor, the same phenomenon was observed in the bilateral SM1, right PFC, left SAC, and left PMC&SMA, as shown in Fig. 6. In the right SAC, an initial decrease in HbO was observed shortly after the task onset, followed by a return to the baseline level and a subsequent increase. This finding could be explained by previous research suggesting that the activation of the SAC may be modulated by attentional shifts between cognitive tasks and somatosensory or auditory stimuli [82], [83]. Specifically, decreased HbO in the SAC is associated with increased perceptual demand during cognitive tasks, whereas increased HbO in the SAC corresponds to a more focused response to auditory and somatosensory stimuli. In this context, the stroke survivor may shift his attention from the cognitive task to the auditory or somatosensory sensations after the task onset in order to perform the VR game using his affected left hand.
There are two additional remarks worth mentioning. First, there was a difference in the interpretation of activation between the results of significantly activated channels and those of the cortical activation map. Specifically, there was no significantly activated channel in the left PMC&SMA and left PFC, however, the cortical activation maps showed an increase in HbO response for those areas. The reason for this discrepancy is considered to be high subject variability resulting in wider confidence intervals, as shown in Fig. 5. Second, both HbO and HbR tended to increase in the left PFC and right PMC&SMA of the stroke survivor, as shown in Fig. 6. A possible explanation for this phenomenon is an increase in oxygen supply due to increasing HbO and oxygen consumption due to increasing HbR, which is considered to have occurred at the same time [84]. As a further explanation for the increased HbO and HbR in the left PFC, our result is in line with the previous study that reported that a similar phenomenon occurred when performing a high cognitive task with sustained attention [85].
The survey results highlight the potential benefits and positive user experience of the VR hand rehabilitation system. Most participants, including the stroke survivor, reported being engaged and motivated to continue using the system, and experienced presence during gameplay. These factors are essential for effective rehabilitation, as sustained engagement and motivation with the presence of VR can lead to improved therapy adherence, which may result in better functional outcomes. Moreover, the survey results indicate that the inclusion of vibrotactile feedback positively influenced engagement and presence. It suggests that providing this additional sensory input can enhance the realism of the virtual environment, offering users a more immersive experience. It is worth mentioning that the survey took place six months after the initial experiment. While this time gap may have influenced participants’ memories of their experience, the overall positive feedback indicates that the VR hand rehabilitation system made a lasting impression.
Although it was possible to develop an immersive hand rehabilitation system and confirm its effectiveness, the present study is limited by the small number of participants, limited power, and lack of control conditions to isolate the specific effects of the system’s components. In future studies, we aim to include more control conditions to better understand the unique contributions of each component within our system, such as the VR game, music-based rhythm, and vibrotactile feedback on brain activation, user engagement, and potential enhancement in motor function. Moreover, the placement of vibrotactile feedback, which is not directly on the hand, should be addressed in future research to provide more relevant feedback to users. Furthermore, it is crucial to conduct a clinical intervention study to verify the therapeutic effect of our proposed system on the functional restoration of stroke survivors. This involves identifying the severity of hand conditions that can still benefit from our system, such as range of motion and degree of spasticity. Subsequently, subject-specific target gestures considering these limitations should be applied to stroke survivors.
Conclusion
We developed a hand rehabilitation system with a VR rhythm game that can be controlled by hand gestures and provides vibrotactile feedback. Our experimental results demonstrated that our system successfully activates brain areas associated with motor planning/execution, multisensory integration, and attention. This approach is especially suitable for subject-specific hand rehabilitation because various types of target gestures can be set, based on one’s hand condition. In addition, it allows repetitive training flexibility, with higher intensity and precision by adjusting the game difficulty, such as the frequency of target appearance, changing the rhythm according to various music, and the threshold value for hand recognition. Such adjustments can be based on the user’s current condition, and are important to increase the volitional drive of the stroke survivor to accomplish the task. The presented system is extensible to other parts of the upper limb, such as the wrist, elbow, and shoulder. In addition, the system is a potentially useful tool for teaching proficient hand skills for various purposes in other areas.