Combining FES and Exoskeletons in a Hybrid Haptic System for Enhancing VR Experience

Robotic technology and functional electrical stimulation (FES) have emerged as highly effective rehabilitative techniques for individuals with neuromuscular diseases, showcasting their ability to restore motor functions. Within the proposed study, we developed and tested a new hybrid controller combining an upper-limb exoskeleton with FES to enhance haptic feedback when performing task-oriented and bimanual movement, like pick-and-place, in a virtual environment. We investigated the performance of the proposed approach on eight unimpaired participants providing haptic feedback either only by the exoskeleton or by the hybrid system. The hybrid control presents two different modalities, assistive and resistive, to modulate the perception of the load. FES intensity is calibrated to the subjects’ biomechanical properties and it is adjusted in real-time according to the real-time motion of the upper limbs. Experimental results highlighted the ability of the hybrid control to improve kinematic performance: in both hybrid modalities subjects reduced the target matching error(values between 0.048±0.007 m and 0.06±0.006 m) without affecting the normal motion smoothness (SPARC values in the hybrid conditions range from −2.58±0.12 to −3.30±0.13). Moreover, the resistive approach resulted in greater metabolic consumption (1.04±0.03 W/kg), indicating a more realistic experience of lifting a virtual object through FES that increased the perceived weight. The innovation in our hybrid control relies on the modulation of muscular activation during manipulation tasks, which could be a promising approach in the clinical treatment of neuromuscular diseases.


Combining FES and Exoskeletons in a Hybrid
Haptic System for Enhancing VR Experience L. Buatier de Mongeot , Graduate Student Member, IEEE, E. Galofaro, F. Ramadan, E. D'Antonio, F. Missiroli , Member, IEEE, N. Lotti , M. Casadio , and L. Masia , Senior Member, IEEE Abstract-Robotic technology and functional electrical stimulation (FES) have emerged as highly effective rehabilitative techniques for individuals with neuromuscular diseases, showcasting their ability to restore motor functions.Within the proposed study, we developed and tested a new hybrid controller combining an upper-limb exoskeleton with FES to enhance haptic feedback when performing task-oriented and bimanual movement, like pick-and-place, in a virtual environment.We investigated the performance of the proposed approach on eight unimpaired participants providing haptic feedback either only by the exoskeleton or by the hybrid system.The hybrid control presents two different modalities, assistive and resistive, to modulate the perception of the load.FES intensity is calibrated to the subjects' biomechanical properties and it is adjusted in real-time according to the real-time motion of the upper limbs.Experimental results highlighted the ability of the hybrid control to improve kinematic performance: in both hybrid modalities subjects reduced the target matching error(values between 0.048±0.007m and 0.06±0.006m) without affecting the normal motion smoothness (SPARC values in the hybrid conditions range from −2.58±0.12 to −3.30±0.13).Moreover, the resistive approach resulted in greater metabolic consumption (1.04±0.03W/kg), indicating a more realistic experience of lifting a virtual object through FES that increased the perceived weight.The innovation in our hybrid control relies on the modulation of muscular activation during manipulation tasks, which could be a promising approach in the clinical treatment of neuromuscular diseases.

I. INTRODUCTION
N EUROLOGICAL diseases like Spinal Cord Injury (SCI), Multiple Sclerosis, or Stroke, affect significantly the motor and somatosensory systems, leading to physical disabilities and reduced muscle strength [1], [2], [3], [4].Strong emphasis is placed on the rehabilitation of the upper limb motor functions to boost independence in performing Activities of Daily Life (ADLs) and promote social reintegration [5].As a result, there is a considerable demand for efficient rehabilitation approaches.It has been demonstrated that intensive robotic rehabilitation is successful for inducing recovery after stroke and effective for SCI [6].
Rehabilitation robotics, such as end-effector robots and exoskeletons, offer high precision and repeatability in training sessions, potentially establishing standardized, quantitative methodologies for treating motor impairments in clinical settings [7].
Functional Electrical Stimulation (FES), instead, is a technique that induces muscle contraction using electrical stimulation delivered via either superficial or implantable electrodes, [8].It prevents muscle atrophy, preserves or increases functional range of motion, strengthens muscles and encourages cortical reorganization and neural plasticity [9].However, it is noted that FES can lead to quicker fatigue compared to voluntary muscle control, and precise control of FES-induced joint motion can be challenging due to its non-linear nature [10].
The concept of Hybrid Systems arises as a promising approach for rehabilitation.These hybrid systems combine both robotic technology and FES to aid and potentially restore upper limb functions [11].One of the primary goals of these systems is to optimize neuroplasticity by delivering multiple sensory input to the central nervous system [12].FES and robotic devices collaborate to support functional movements, enabling users to perform ADLs [13].FES stimulates muscles, while robots assist either by providing gravity compensation on the limb or by actively influencing joint motions.FES can be easily included into lightweight wearable and textile devices [14].
Early investigations into hybrid systems did not typically involve a simultaneous activation of both systems.For instance, some studies primarily focused on providing FES while robotic devices were designed to assist and steady upper limb.These robotic devices included features such as compensating gravity and immobilizing specific degrees of freedom through the use of brakes to reduce muscle fatigue, like in Ambrosini et al.'s and Wolf et al.'s studies [15], [16].
In contrast, other hybrid systems allowed for the concurrent activation of joint sets, actuating some degrees of freedom with FES while the robot handled the remaining degrees of freedom, as presented by Varoto et al. [17].In many cases, these solutions incorporated FES to stimulate hand muscles (for grasping), while the robot assisted with arm and forearm movements like in Ajiboye et al. [18].
Recent research, however, has shifted towards shared control systems that integrate FES and robotic support on the same single joint, usually the elbow, providing optimized collaborative assistance [19], [20], [21], [22].Some of these studies focused more on compensating robots power requirements by using FES to initiate or to complete motion.Different approaches have been explored to assist motion while preventing muscle fatigue.These include initiating motion either through FES and then completing it using wearable robots or viceversa, adjusting FES parameters (pulse width, intensity, or frequency) to achieve the desired torque without compromising muscle performance [20], [23], [24].
Both robotic technologies and FES can be used with Virtual Reality (VR) systems to enhance patient's motivation during therapy, while providing consistent measurements of kinematics and kinetics [11], [25].Exoskeletons and end-effector robots provide haptic feedback through force fields and torques applied to users' hands and arms.FES has been lately employed to provide haptic feedback to render, for example, virtual walls and virtual weight perception [24], [26], [27], [28].
So far, there are no multi-joints hybrid systems that combine exoskeletons and FES in a shared and simultaneous actuation.Furthermore, research on hybrid systems with VR remains limited.Our group has already developed a solution integrating an elbow exosuit with a wearable stimulator to render a load on the hand during flexion performed in a virtual environment [24].This approach can offer continuous haptic feedback and stimulation to multiple muscles, providing multisensory feedback through force fields generated by robots and through modulation of electrical stimulation.The primary obstacle for hybrid systems is establishing an effective shared control between FES and exoskeleton to enhance the efficacy of the haptic feedback.This is mainly guaranteed by a robust FES controller, which relies not only on the control approach, but also on the number of targeted muscles and on the correct positioning of the electrodes.
In the current study, we developed a multi-joint hybrid system consisting of a rigid bimanual actuated exoskeleton and a FES suit, as displayed in Fig 1 .An immersive virtual environment simulated a scenario for a functional bimanual task (i.e.pick-and-place that involves shoulder and elbow flexion-extension).This system provided haptic force feedback during object manipulation via robot and via FES.The FES controller is structured in two ways: (1) motion assistance (assistive) and (2) haptic feedback rendering (resistive).Realtime modulation of FES intensity aimed at reducing muscle fatigue in biceps, triceps, anterior and posterior deltoids [23].These modalities differ in the stimulation sequence: either agonist or antagonist muscles of elbow and shoulder joints.We hypothesized that: (i) the hybrid haptic feedback approach would result in improved kinematic performance compared to using haptic feedback produced solely by the exoskeleton; (ii) the hybrid mode would increase the level of effort required by the subject to complete the task, especially in the resistive condition.To validate these assumptions, hybrid controllers are compared with an exoskeleton-alone controller by assessing metabolic expenditure and kinematic outcomes.

II. HYBRID SYSTEM DESIGN A. Actuated Exoskeleton for the Upper Limbs
The Arm Light Exoskeleton Rehab Station ALEx-RS (Wearable Technology, Italy) is a robotic tool designed for neuromotor rehabilitation of upper limbs [29], [30].It features six degrees of freedom (DoFs) per limb, with three at the shoulder, one at the elbow, and two at the wrist, with only the shoulder and elbow actuated.The exoskeleton utilizes a tendon-driven transmission system that is low in inertia, providing back drivability and smooth motion.It covers 92% of the upper limb workspace, has gravity and friction compensation, and each arm is equipped with four brushless motors and four optical incremental encoders.
For the present study, we adopted a haptic control design that we had previously deeply tested [31].This framework is designed to produce bimanual force feedback at the device end-effectors (EEs) by using the "god-object method" through an impedance control.

B. FES Stimulator
The FES stimulator, named Teslasuit® (VR Electronics Ltd, London, UK), is a wearable full-body suit divided in jacket and trousers.The suit delivers electrical stimulation to the user through up to 80 channels, consisting in pair of shaped electrodes placed on anatomically suitable locations.The jacket component of the suit contains 62 electrodes controlled via Wi-Fi.

C. Hybrid System
The hybrid system (Fig 1 ), merging exoskeleton and FES, operates in accordance with the real-time motion, recorded by the encoders of the exoskeleton.The two technologies are interconnected by the high-level controller, which detects the manipulation of virtual objects in the virtual environment.The FES stimulator is connected to the haptic interface implemented for the exoskeleton through the Teslasuit SDKs for Unity 3D (Unity Software Inc., Copenhagen, Denmark, version 2020.2.13).

D. Virtual Scenario
The virtual scenario, rendered via Unity 3D software, was responsible for providing hybrid haptic interaction with the rendered virtual objects during a pick-and-place task.The scenario (Fig 1) was comprised of a virtual room with a table (starting position), a moving shelf (target), and a virtual object (VO).The end-effectors of the exoskeleton were represented by two blue spheres, able to deform proportionally upon the interaction with VO.

III. REAL-TIME ADAPTIVE CONTROLLER
The controller (Fig 2) consisted of two layers: the Exoskeleton controller, which was designed to provide stable force feedback via motor actuation, and the FES controller, which delivered electrical stimulation to superficial muscles of the upper limbs regulating its timing and intensity calibrated by means of a biomechanical model.A high-level controller, Virtual Haptic Unit, was responsible of detecting collision events between EEs and VO enabling both exoskeleton and FES actions during bimanual manipulation of the VO.

A. Virtual Haptic Unit -VHU
The high-level controller, named Virtual Haptic Unit (VHU), was developed via C# and Unity3D.Its purpose was to visually represent the virtual scenario and handle the computations for haptic rendering according to the user's movements and manipulations in the virtual environment.At a rate of 200 Hz, VHU detected the virtual collision between the exoskeleton EEs and the VO.The haptic response was designed to prevent the user from penetrating the VO, employing the "god-object algorithm" [32].According to this method, each hand was represented by three main spheres: • End-Effector (EE): This sphere, not visible in the virtual environment, corresponded to the real EE's position in the virtual scenario.• God Object (GO): Also invisible, the GO represented the ideal EE position constrained to the surface of the manipulated VO. • Visual Sphere (VS): A visible sphere fed back to the user either the EE position (when no interaction with VO is detected) or the GO position (during collision of EE and VO).This sphere provided visual information of the contact force, its shape deformed, squeezing onto the surface of the VO proportionally to the applied forces and, therefore, to the distance between EE and GO.These spheres were used for various computations, such as determining interaction forces through an Impedance Controller and indicating position within the virtual environment.Detailed explanations can be found in a previous study [31] and in the following section, while in

B. Exoskeleton Controller
The low-level controller, running at 1kHz, was developed on a second dedicated workstation to provide force feedback during interactions with the VO by means of an Impedance Controller.Feedback forces (F E E ) were computed considering the penetration of the EE spheres into the VO by means of a mass-spring-damp model: a virtual elastic component was proportional to the distance between the EE spheres and GO spheres, whereas a virtual viscous term was proportional to the mutual velocity of the two spheres during motion.
The computed force values F E E , one per EE, were sent back to the device at each exoskeleton arm via a shared Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.memory communication protocol.Subsequently, the Torque Computation module converted forces into joint torques (τ ) for each motor by using the transpose of the Jacobian.

C. FES Controller
The FES controller delivered electrical stimulation to superficial arm muscles for elbow and shoulder flexion-extension.Its function was enabled during bimanual manipulation of the VO and its intensity was modulated according to the upper limb kinematics and the estimated muscular activation.
We employed a rectangular biphasic waveform with frequency set at 60 Hz, pulse width limited to 350 µs, and the intensity (I 0 ) determined through a calibration procedure conducted prior to the experiment for each participant.The aim of the calibration was to detect the minimum intensity at which the user could detect the slightest signal, whereas the maximum was determined based on the user's subjective tolerance.
We assumed that injecting an appropriate level of current, that mimics normal muscle activity, makes it possible to deliver a modulated stimulation pattern.In this way it can be perceived as a realistic contraction and helps in mitigating muscle fatigue [23].This is why we performed a modulation of FES intensity by means of a biomechanical model, a similar approach as in Sierotowicz et al. [33].
The biomechanics of the upper limbs motion was analyzed by means of a bimanual MOBL musculoskeletal model compatible with OpenSim version 3.2 [34].The model was scaled on subjects to adjust the dimensions and properties of the model's components to estimate muscular activation according to the upper limb motion.
Such an estimation was performed by means of Static Optimization, an extension of the inverse dynamics provided by OpenSim [35], [36].It offers quantitative estimation of muscle activation levels, ranging from 0 to 1, for several muscle groups involved in a given task.In the context of this research, Static Optimization has been employed for two main reasons.Firstly, it has been a valuable tool to identify which of the numerous muscles groups are predominantly engaged in the bimanual pick-and-place task.Secondly, we obtained a quantitative relationship between human motion and muscle activation level.
More specifically, the Static Optimization tool determines the muscle activation that minimize the sum of squared muscle activation while satisfying the task of generating the desired joint moment τ j .
where k m is the activation level of muscle m at each time instance of the movement, f (F 0 m , l m , v m ) represents the estimated muscle force constrained respectively by its forcelength-velocity properties of muscle m, r m is the moment arm of muscle m about the j th joint axis.
The optimization involves minimizing the energy cost function J: that is a weighted sum of squared muscle activation levels k m , normalized values between 0 and 1, respectively meaning "no muscular activity" and "complete muscular activation".Through this estimation we identified the muscles groups predominantly engaged in the task: the long head of the biceps, the long head of the triceps, anterior and posterior deltoids.See Consequently, we obtained C m (q i ), the relationship between k m and the angular trajectories of the corresponding joints (biceps and triceps refer to the elbow angle q 1 while anterior and posterior deltoids refer to shoulder angle q 2 ): where coefficients a 4m , a 3m , a 2m , a 1m and a 0m are specific for each muscle.
In the current hybrid control, modulation consisted of multiplying FES intensity I 0 (t), provided by the calibration procedure, with C m (q i ): The task was structured into two phases, lift and drop: after grabbing the VO from the starting position, the user proceeded flexing elbows and shoulders joints to reach the height of the target.This action predominantly activated biceps and anterior deltoids (lift phase).Then, when the user aimed to place the VO on the target, extensions of the elbows and shoulders involved triceps and posterior deltoids activations (drop phase).
The FES controller was designed to deliver stimulation according to the motion phase.In our control approach, the motion phase was detected by analyzing the sign of the first derivative of the angular trajectory of the elbow joint q ′ 1 (t), tracked by the exoskeleton encoders.A positive value (q ′ 1 (t) > 0) indicated an elbow flexion (lift phase), whereas a negative value (q ′ 1 (t) < 0) corresponded to an elbow extension (drop phase).
We designed two modalities to provide FES during the virtual pick-and-place task: • Resistive Stimulation (RS): this mode was designed to increase weight perception.FES was targeted to antagonist muscles.• Assistive Stimulation (AS): it was intended for supporting muscular activity.FES was directed towards agonist muscles.It is worth to mention that our FES controller defines four functions (Equation 3) to modulate FES intensity for each muscle.However, they do not differ according to the modality.In RS mode, for example, the controller activates the triceps to the same extent as the biceps by employing C biceps , as well as the posterior deltoids stimulation is modulated by C ant Deltoids .

D. Hybrid Controller
The multi-joint hybrid controller integrated the Exoskeleton controller and FES controller in a parallel structure (Fig 2).The Exoskeleton controller was operated using the Unity graphic engine within the "VHU" module, which detected collisions between the EEs and the VO.This event allowed for simultaneous force feedback to the EEs and the application of electrical stimulation to the corresponding muscles.This process generated either augmented haptic rendering or human motion assistance, depending on the chosen modality.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

IV. EXPERIMENTS A. Subjects
The experiment involved a group of eight healthy and right-handed participants naive to the task (seven females and one male, 24.6±3.0 years old, 166±5.3cm height, and 57.7±6.2 kg weight).All participants had no history of musculoskeletal or neurological diseases.Before beginning the experiments, all participants signed informed consent forms.The experimental protocol was approved by Heidelberg University Institutional Review Board (S-287/2020): the study was conducted following the ethical standards of the 2013 Declaration of Helsinki.The experiments were conducted at the Aries Lab of Heidelberg University.

B. Experimental Protocol
Participants were instructed to take a seat at the exoskeleton workstation (ALEx-RS), to wear the FES stimulator (Teslasuit), and the 3D visor (Oculus Rift S).The Teslasuit must be worn at least 20 minutes before the task to obtain the proper fitting between the suit electrodes and the skin.To collect data on metabolic energy expenditure, we utilized a portable metabolic analyzer (K5-Cosmed, Rome, Italy).To establish the average metabolic cost at rest, subjects were required to engage a four-minute breathing session before starting the experiment.The net metabolic cost was computed by subtracting the metabolic data recorded at rest from the data obtained during each experimental session.As illustrated in Fig. 1, the task involved grabbing and lifting the VO from the table (starting position) and placing it on the moving shelf (target).It is important to mention that, in all conditions, the hybrid system exploited the exoskeleton controller to provide haptic feedback during the manipulation of the VO as well as the weight of the VO itself.On top, the FES changed across the different conditions, as explained as follows.
The experiment included three conditions, proposed to subjects in a random order: • Assistive Stimulation (AS E xo+F E S ): combined use of bimanual exoskeleton and FES stimulator components.FES was applied to the agonist muscles during movement to assist motion during the task; • Resistive Stimulation (R S E xo+F E S ): combined use of bimanual exoskeleton and FES stimulator.The stimulated muscles were the antagonists to resist the motion by generating opposite physiological joint torques: subjects' perception of the load was increased.• No Stimulation (N S E xo ): only exoskeleton was used to generate the haptic force feedback.Each target-set consisted of nine repetitions of the pick-andplace task at the three possible shelf heights (high, middle, low).Each condition (AS E xo+F E S , R S E xo+F E S , and N S E xo ) was performed on different days to prevent participants from experiencing muscular fatigue.In each condition one had a total of 11 target-sets, broken down as follows: • Familiarization phase: consisting of one target-set to allow participants to experience the virtual environment and to better understand the task.
• Training phase: a total of 10 target-sets were completed, where participants received haptic feedback from the hybrid system according to the aforementioned conditions.The whole experiment comprised 33 target-sets (11 for each condition).

C. Data Analysis
The study evaluated participants' performance based on their metabolic consumption, which was measured using a metabolic analyzer operating in mixing chamber mode, and on their kinematics, which were recorded by exoskeleton sensors at a frequency of 200 Hz and filtered using a Savitzky-Golay filter with a 10 Hz cutoff frequency.
The study estimated the metabolic cost of three different conditions (AS E xo+F E S , R S E xo+F E S , and N S E xo ) by analyzing the volumes of oxygen (V O 2 ) and carbon dioxide (V C O 2 ).The net metabolic cost (P) was computed according to Péronnet and Massicotte's equation [37], which was adjusted for the subjects' weight (W) and then subtracted by the average cost at rest.
Movement accuracy was evaluated by calculating the matching error (ME), which is the difference between the EEs positions (x eei , y eei , z eei ) while placing the VO on the target and the target position (x T i , y T i , z T i ): The normalized smoothness, as in [38], was also considered by calculating the arc length of the EEs speed profile's Fourier magnitude spectrum (Spectral Arc Length -SPARC).

D. Statistical Analysis
The performance metrics were first averaged across repetitions and then compared across conditions using a repeated-measures analysis of variance (rANOVA) and we considered as the within-subjects factor the experimental modality (AS E xo+F E S , R S E xo+F E S , and N S E xo ).Data normality was evaluated using the Shapiro-Wilk test, and the sphericity condition was assessed using the Mauchly test.Statistical significance was considered for p-values lower than 0.05.We performed a post-hoc analysis using paired t-tests with Bonferroni correction to evaluate pairwise differences between conditions.Reported measurements are presented as mean ± standard error (SE).

A. Metabolic Cost
Results illustrated in Fig. 4, on the left side, show the comparison between AS E xo+F E S , R S E xo+F E S and N S E xo .It is noticeable that the hybrid condition (R S E xo+F E S : 1.04±0.03W/kg) resulted in higher metabolic consumption compared to the condition with only the actuated exoskeleton (N S E xo 0.67±0.03W/kg).The statistical test indicated a significant effect of the R S E xo+F E S modality (p=0.008) with respect to the N S E xo condition, generating an average increase in metabolic cost of 35.73±7.81%.
Instead, in the case of the AS E xo+F E S condition (0.88±0.04 W/kg), the metabolic outcomes indicated no statistical difference with the N S E xo condition, even if it recorded an increase in metabolic expenditure of 20.5±11.1%.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

B. Kinematics
Kinematic analysis focuses on the performance of the pickand-place task from the grabbing at the starting position until the successful target reaching.In Fig. 3 we reported the EE trajectory on the sagittal plane for a sample subject.It shows that FES is delivered only during bimanual manipulation of the VO that is also the time when the user performed the pick-and-place task.Results of the kinematic assessments are displayed in Fig. 4 (central and right side) for the three different target heights in the sequence N S E xo , R S E xo+F E S and AS E xo+F E S .Specifically, the central part of Fig. 4 shows the average matching error, which is used to evaluate the accuracy of the movement, whereas on the right of of Fig. 4 the normalized smoothness is reported.
We found that subjects performed with best accuracy in the R S E xo+F E S condition with lower matching error values (M E high =0.048±0.007m, M E middle =0.05±0.006m; and M E low =0.06±0.006m), whereas the N S E xo condition generated the worst performances in terms of accuracy.The AS E xo+F E S modality (M E high =0.054±0.007m, M E middle =0.06±0.008m; and M E low =0.07±0.011m), improved significantly the performances, but not as much as in the R S E xo+F E S condition.
Statistical analysis revealed significant differences (p<0.05) between the hybrid conditions (AS E xo+F E S , R S E xo+F E S ) and the N S E xo condition, pointing out the immediate effect of the hybrid system on human kinematics.
Users' performances were evaluated also in terms of the smoothness of the movement.The formula provided in [38] computes the SPARC index, which is a negative dimensionless parameter where smoother movements are characterized by values close to zero.Results demonstrated that the hybrid system did not affect the smoothness of the kinematic performance, as confirmed by the statistical analysis which did not identify any significant differences across conditions (AS E xo+F E S modality: S P A RC high =−2.58±0.12, S P A RC middle =−2.863±0.14, S P A RC low =−3.27±0.14.R S E xo+F E S modality S P A RC high =−2.72±0.13,S P A RC middle =−3.03±0.12,S P A RC low =−3.30±0.13).

VI. DISCUSSION
Numerous neuromuscular disorders result in motor impairments, including conditions like Spinal Cord Injury (SCI), stroke, and multiple sclerosis.Substantial evidence supports the idea that intensive motor rehabilitation can promote the recovery of motor functions and the maintenance of muscle tone in individuals with residual motor capabilities or paralyzed limbs [6].
Over time, various technological solutions have been developed to address these challenge, ranging from robotics with the use of exoskeletons or exosuits to neuroscience with the study of neuroplasticity to aid in motor relearning [39].Additionally, the application of virtual reality in the medical field is gaining increasing interest [25].In the realm of rehabilitation, virtual reality offers the opportunity to create virtual environments and scenarios, allowing for customized functional exercises for each patient.This stimulates cognitive engagement as patients interact with the virtual world, thereby enhancing motor control for performing ADLs.To ensure the effectiveness of virtual reality usage, it is crucial to prioritize visual quality and provide additional sensory feedback to the user, such as auditory signals, tactile sensations, and improved proprioception [12].
In the context of this study, a hybrid system has been developed that combines a bimanual exoskeleton for the upper limbs with a wearable FES device.This integration harnesses the strengths of both components, enhancing their effects.The system was employed in two primary modes: the "haptic augmentation" domain, which assessed the impact of using multi-modal kinesthetic feedback generated by FES and the robot, and the assistance domain, which evaluated how FESassisted movement, combined with robotic haptic feedback, can be effective in rehabilitation treatments.
The majority of hybrid systems are typically categorized by their objectives, which include either replacing lost function, offering support for reduced function, or facilitating the restoration of function through therapeutic interventions [40].However, a notable absence in the field of hybrid systems is the integration of multimodal feedback for users.This entails combining VR to provide visual feedback, simulating various scenarios and functional tasks, along with the use of exoskeletons and FES to deliver both human motion assistance and kinesthetic feedback.
This study introduced a new hybrid system featuring a modular control for adjusting FES intensity during a bimanual manipulation task performed in a virtual environment.Results demonstrated that the hybrid control is both robust and effective in delivering haptic feedback via various proprioceptive channels which involved both force feedback as well as muscular stimulation, all synchronized according to the real-time virtual manipulation.
To the best of our knowledge, this is a novel combination of different technologies and a new paradigm in the field of haptics.This innovative approach not only allows realistic simulation of physical interaction with virtual dynamics, but also for the modulation of muscular stimulation, hence controlling the internal force generated by human muscles and mitigating muscular fatigue.
Intensity modulation was carried out under the premise that injecting a suitable amount of current resembling typical muscle activity would make it possible to induce a realistic muscle contraction and simultaneously helps in reducing muscle fatigue [23].[41].The use of electrical stimulation in conjunction with robotic rendering provides unprecedented combination of control paradigms able to enhance the haptic sensation: antagonistic muscles must be stimulated to create muscle tension and increase the effort needed to complete a specific task [24], [27], [28].Furthermore, FES can be applied to agonist muscles to facilitate proper muscle activation and give the sensation of a lighter movement and provide also training by properly activating those muscle groups which should be responsible of the movement [27].
In this research, the Exoskeleton controller was specifically developed to provide kinesthetic feedback to the user during bimanual interaction, rather than to actively assist or guide upper limb motion.Consequently, in a rehabilitation context, it is essential for patients to possess some residual mobility, which is then further enhanced through FES in the assistive mode.
The findings of this study align with existing literature, underscoring that FES is a suitable technology for delivering more realistic haptic feedback during interactions with objects in a virtual environment.Previous work by Lopes et al. [26] delved into the integration of haptics with walls and heavy objects in VR through FES.They demonstrated how introducing haptic feedback via electrodes on the user's arms could enhance the sense of presence in virtual interactive applications, however without conducting a quantitative analysis of system performance.In contrast, this study scrutinized two key physiological metrics of participants: metabolic consumption and kinematic performance [24].

Hybrid FES and Robotic rendering increase muscular activation and metabolic consumption
This result is consistent with previous works, which showed that the metabolic cost increase with the exercise intensity [24], [42].The findings revealed that the use of the hybrid system led to higher effort compared to relying solely on robotic rendering.This is attributed to non-voluntary motor unit recruitment pattern facilitated by FES [43].The metabolic cost was at its peak in the R S E xo+F E S condition when antagonistic muscles where stimulated, making users perceive the action of lifting a load compared to the N S E xo condition.Conversely, the AS E xo+F E S condition also resulted in increased metabolic consumption as additional muscle fibres were recruited, but without a statistically significant difference from the N S E xo condition.This is an interesting finding: in the AS E xo+F E S mode real-time modulated FES intensity induced functional muscular contractions preventing muscle fatigue.Moreover, FES targeted specific muscle groups for only a few seconds (1.74 ± 0.32 seconds) at a time, alternating between Biceps/Anterior Deltoids and Triceps/Posterior Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
Deltoids according to the motion phase (lifting/dropping).As suggested in the study [10], this short duration of stimulation is unlikely to produce a consistent fatigue effect.This is one of the reasons why we chose to perform each experimental session on different days: we aimed to mitigate any residual effects from previous sessions (e.g., soreness, fatigue. . .), but also to reduce excessive variability in metabolic consumption, with morning sessions being the most optimal timing.FES promotes higher motion stability and better kinematics outcomes The analysis of kinematic outcomes provided not only an understanding of the task execution performance in the hybrid condition but also the effects of muscle fatigue on upper limb kinematics [44].Both the R S E xo+F E S and AS E xo+F E S modalities revealed enhanced movement precision, as demonstrated by the reduced matching errors detected when reaching the target.In contrast, the N S E xo condition resulted in inferior kinematic performance.These observations indicated that the hybrid system takes the best of the two components resulting in improved movement execution, as discovered also in previous works [19], [21].It is worth noticing that the condition RS E xo+F E S , in which antagonist muscles were stimulated, provided the lowest matching errors when compared to the other conditions.This finding could indicate that provoking cocontraction (activation of both agonist and antagonist muscles) might provide a higher stabilization of the movement, yet requiring a larger metabolic consumption.The analysis of kinematic outcomes provided not only an understanding of the task execution performance in the hybrid condition but also the effects of muscle fatigue on upper limb kinematics [44].Both the R S E xo+F E S and AS E xo+F E S modalities revealed enhanced movement precision, as demonstrated by the reduced matching errors detected when reaching the target.In contrast, the N S E xo condition resulted in inferior kinematic performance.These observations indicated that the hybrid system takes the best of the two components resulting in improved movement execution, as discovered also in previous works [19], [21], [24].It is worth noticing that the condition RS E xo+F E S , in which antagonist muscles were stimulated, provided the lowest matching errors when compared to the other conditions.This finding could indicate that provoking cocontraction (activation of both agonist and antagonist muscles) might provide a higher stabilization of the movement, yet requiring a larger metabolic consumption.The smoothest condition was the N S E xo , which confirms that, by adding additional resistive torques by means of robotic devices it is possible to reduce movement variability [45].However, it was not significantly smoother than the hybrid conditions.The addition of FES-induced resistive torque does not result in improved smoothness yet, due to the non-linearities of FES.Nevertheless, their FES controller successfully delivered FES without significantly affecting motion smoothness, a favorable outcome compared to prior studies that involved antagonist muscle stimulation [24].Referring to another study, the suggestion was made to improve motion smoothness through FES by targeting both agonist and antagonist muscles simultaneously, as proposed in Ruppel et al. [46].This approach offers an intriguing solution worth considering for potential implementation and further investigation in future research.
Given that the kinematic analysis of the hybrid conditions showed enhanced movement accuracy and the preservation of natural smoothness, we can infer that the hybrid controller employed in this experimental protocol effectively prevented the degradation of human kinematics caused by muscle fatigue.
Limitations of the study Despite our study yielding promising outcomes, there are several limitations to our work.The developed controller was tested on a restricted number of unimpaired subjects.The availability of a single suit size precluded the inclusion of a wide range of participants in terms of anthropometric measures.Additionally, we focused exclusively on a limited set of stimulation muscles related to the upper limb region.Expanding the scope to include other muscles (e.g., forearm muscles) in future research could be beneficial for improving haptic feedback in manipulation tasks.The calibration procedure established FES intensity modulation in advance for each participant using a biomechanical model tailored to the specific user and functional task.It would be intriguing to investigate approaches that incorporate real-time estimation of muscular activity to run online and to be adapted to a wider range of functional tasks and motions.

VII. CONCLUSION
Taking into account the mentioned limitations, the study's findings revealed that the hybrid controller effectively managed haptic feedback delivered concurrently by the exoskeleton and FES, leading to precise and accurate movements.Furthermore, the heightened weight perception due to FES feedback required greater effort, as evidenced by the metabolic consumption analysis.These results are consistent with literature and indicate that the innovative hybrid control, tailored to subjects' biomechanical characteristics and aligned with real-time muscle activation, could serve as a valuable resource for promoting motor relearning and possibly inducing muscle tone when such an approach is used in neurological condition.

Fig. 1 .
Fig. 1.Hybrid FES-Exoskeleton system for upper-limbs enhanced haptic feedback and the visual scenario rendered on the 3D visor during the pick-and-place task.

Fig. 2 .
Fig. 2. Real-time control.a) The real-time control framework of the hybrid system consists of two main controllers interconnected by the VHU module: the Exoskeleton Controller and the FES Controller.The VHU is the high-level controller that detects and measures the penetration of the end-effectors (EEs) into the virtual object (VO), providing information in real-time of the manipulation of the VO.The Exoskeleton Controller is a low-level controller providing torque τ exo to the user through the impedance controller.The FES Controller modulates the electrical intensity Im to send to each muscle, based on the real-time upper-limbs movement (elbow and shoulder angle) according to a predefined biomechanical model.The Hybrid System can be either resistive (RS exo+FES ) or assistive (ASexo+FES).They differ in the stimulation sequence, involving either antagonist or agonist muscles.
Fig 2 top left we provided a visual representation of the god-object algorithm.

Fig 2 -
right side for a visual representation of k m used in FES modulation.

Fig. 3 .
Fig. 3. EEs trajectory EEs trajectory profiles in the sagittal plane over 30 s for a sample subject are shown for each experimental modality.Highlighted ranges correspond to the execution of the pick-and-place task, when haptic feedback is delivered.On the right, a representation of the k m functions that modulate FES intensity.

Fig. 4 .
Fig. 4. Average Metabolic Consumption.Comparison of metabolic cost (mean ± SE) in the three experimental conditions NS Exo , RS Exo+FES , and AS Exo+F ES .Arrows denote the incremental metabolic expenditure of the hybrid system with respect to the NS Exo .Average Matching Error.Comparisons between conditions, distinguishing High, Middle and Low targets, are provided.Movement accuracy (mean ± SE) determined by the matching errors (ME).Average Motion Smoothness.Normalized Smoothness (mean ± SE), evaluated according to the Fourier magnitude spectrum's arc length of EEs speed.Comparisons between conditions, distinguishing High, Middle and Low targets, are provided.