A Novel Sensory Feedback Approach to Facilitate Both Predictive and Corrective Control of Grasping Force in Myoelectric Prostheses

Reliable force control is especially important when using myoelectric upper-limb prostheses as the force defines whether an object will be firmly grasped, damaged, or dropped. It is known from human motor control that the grasping of non-disabled subjects is based on a combination of anticipation and feedback correction. Inspired by this insight, the present study proposes a novel approach to provide artificial sensory feedback to the user of a myoelectric prosthesis using vibrotactile stimulation to facilitate both predictive and corrective processes characteristic of grasping in non-disabled people. Specifically, the level of EMG was conveyed to the subjects while closing the prosthesis (predictive strategy), whereas the actual grasping force was transmitted when the prosthesis closed (corrective strategy). To investigate if this combined EMG and force feedback is indeed an effective method to explicitly close the control loop, 16 non-disabled and 3 transradial amputee subjects performed a set of functional tasks, inspired by the “Box and Block” test, with six target force levels, in three conditions: no feedback, only EMG feedback, and combined feedback. The highest overall performance in non-disabled subjects was obtained with combined feedback (79.6±9.9%), whereas the lowest was achieved with no feedback (53±11.5%). The combined feedback, however, increased the task completion time compared to the other two conditions. A similar trend was obtained also in three amputee subjects. The results, therefore, indicate that the feedback inspired by human motor control is indeed an effective approach to improve prosthesis grasping in realistic conditions when other sources of feedback (vision and audition) are not blocked.


I. INTRODUCTION
T HE amputation of the hand is a sudden and traumatic experience that impairs a person's autonomy and has lasting effects on their quality of life [1].It causes numerous physical, functional and psychological challenges, including difficulties with the execution of daily tasks [2] and returning to work [3], as well as emotional distress [4], phantom limb pain [5], and appearance-related concerns [6].
Importantly, some of the lost motor functions can be regained using myoelectric prostheses.In these devices, users activate muscles in their residual limb, and the recorded myoelectric signals are translated into prosthesis commands, thereby enabling an intuitive execution of basic hand movements.However, the restoration of function is only partial, as most commercial prostheses, apart from two recent systems [7], [8], do not convey somatosensory information back to the user, although such feedback is essential to establish and maintain effective motor behavior [9].The users of myoelectric prostheses often complain about the lack of sensory feedback and wish to reduce their visual attention to the hand when using it in daily life activities [1].Accordingly, there have been various attempts to restore sensory information and hence close the control loop [10], [11], [12].Invasively, feedback can be provided by subdermal electrical stimulation [13] or through direct stimulation of peripheral nerves [14].Non-invasive approaches include skin stretching mechanisms [15], [16], visual [17] or auditory [18], [19] cues, and electrotactile, [20], [21], or vibrotactile [22], [23] stimulation interfaces.However, developing a feedback interface that is effective in realistic conditions of prosthesis use, when incidental feedback sources such as motor sound and visual observation are not blocked, is still an open challenge [24], [25].
Reliable force control when grasping an object is of crucial importance; applying excessive force could potentially damage the object, whereas the object could be dropped if the grasp is not strong enough.Accordingly, the information on the generated grasping force is a commonly selected feedback variable in the literature [10], [11], [24].It is usually provided using sensory substitution, as when the force magnitude is encoded in electrotactile or vibrotactile stimuli delivered to the skin [25].Such feedback allows the subjects to carefully modulate the generated force while holding an object [26], [27], or adapt their feedforward commands (myoelectric signals) across trials in routine grasping tasks [28], [29], [30].
More recently, the concept of EMG feedback was introduced in [31], where the amplitude of the recorded myoelectric signal was conveyed to the subjects.Usually in this approach, the discrete levels of myoelectric activity are associated with different feedback patterns [32], [33].The feedback, therefore, allows the users to adjust their muscle contraction and position the myoelectric signal within the desired level while closing the prosthesis.Most prostheses are controlled proportionally; the contraction intensity defines the prosthesis closing velocity and, by extension, the grasping force generated upon contact (e.g., faster closing results in stronger grasps).The users can thereby exploit EMG feedback to predictively control the grasping force.Other approaches that can facilitate predictive strategies were also presented in the literature, for instance, joint velocity feedback [34].
Grasping in non-disabled people is a combination of anticipatory, feedforward motor planning and feedback-driven corrections [35], [36], [37].Grasping starts with the visual assessment of object properties (shape, size, and weight), and this information is then used to anticipate the force required to stably grasp and lift the object.In this phase, the subject relies on the internal models of object dynamics established through experience.If the prediction, however, does not match the reality (e.g., an object is heavier than expected), the initial grasping force will be modulated based on the received feedback.This enables fast and routine grasping that is at the same time robust to uncertainty and disturbance.
In the present manuscript, we introduce a novel method to closing the loop in prosthesis force control designed to facilitate the aforementioned biological processes that are substantially disturbed in prosthesis users (e.g., changed relation between the muscle contraction and grasping force).More specifically, the novel approach uses an array of vibration motors to transmit both EMG and force information as feedback to the subject.The EMG feedback was conveyed during prosthesis closing to provide the prediction of the force that the hand would generate upon contact.Once an object had been grasped, the feedback switched to transmitting the information on the actually generated force, thereby allowing force corrections after the hand was closed.Therefore, although still very different from the biological force control in nondisabled individuals, the novel approach aims to facilitate both predictive and corrective control processes when grasping with a myoelectric prosthesis.
We assessed the subjects' ability to employ and benefit from the proposed control scheme.While the novel method can potentially improve the control, it might be challenging for the subjects to interpret the feedback as it abruptly switched between the two very different feedback signals (EMG and force).To test the effectiveness of the combined feedback, 19 subjects (16 non-disabled and 3 transradial amputees) performed a set of functional, force-matching tasks, inspired by the "Box and Block" test, with six target force levels in three different conditions (no feedback, EMG feedback, and combined, EMG and force feedback).The main research question was if the combined feedback would improve the accuracy in generating the desired forces, while we also expected that it would likely decrease the speed of task execution due to additional corrections that can be performed in this condition.

A. Subjects
Sixteen non-disabled subjects (11 females, 5 males, 24 ± 2 years) and three subjects with transradial amputation (see Table I) performed the experiments approved by the Ethics Committee of Special Hospital for Rehabilitation and Orthopedic Prosthetics in Belgrade, Serbia.All subjects were informed about the aims of the study, and they signed a consent form before commencing the experiment.

B. Experimental Setup
1) Non-Disabled Subjects: The setup (Fig. 1, left) used in the study comprised: (1) a Michelangelo hand prosthesis with two dry electrodes for EMG recording (Otto Bock Healthcare Products GmbH, AT) [38], (2) a custom-made vibrotactile system with four coin-type vibration motors with eccentric rotating mass, (3) a laptop computer (Intel(R) i5-3230M @ 2.60 GHz, 4 GB RAM, 17" screen) with a Bluetooth communication interface, and (4) a standard equipment for the "Box and Block" test [39].All subjects performed the experiment in a standing position.To ensure almost isometric muscle contractions, the distal part of their dominant forearm and the hand were immobilized using a 3D-printed splint, which was, due to its weight, suspended using an elastic rope.The splint included a special compartment in which the electronic equipment for the Michelangelo prosthesis and the vibrotactile system was placed.
The Michelangelo hand [38] is a non-backdrivable prosthesis with mechanically coupled fingers that can open and close, and a thumb that can move into opposition.The prosthesis can therefore perform lateral and pinch grasps, and it was also equipped with an active wrist rotation unit.However, in the present study, the lateral grasp and wrist rotations were disabled, allowing the subjects to open and close the hand only in pinch mode.The prosthesis has an embedded force sensor, a proprietary design of Otto Bock, that measures the generated force between the two main digits (i.e., thumb and index finger).The maximal force the Michelangelo hand can produce when closing in pinch mode is 70 N [38].The prosthesis was attached to the splint and positioned approximately below the subject's hand.Two surface EMG electrodes were positioned on the proximal part of the subject's forearm, over the wrist flexor and extensor muscles, which were identified for each subject individually using palpation.Once the positions were determined, the electrodes were strapped using an elastic brace.To close the prosthesis, the subjects activated their wrist flexor muscles, whereas the activation of extensor muscles led to the prosthesis opening.The active electrodes integrated analog circuits for amplification, rectification, and low pass filtering of the measured EMG signals with a cut-off frequency of 3 Hz.
The vibrotactile system provided EMG and force feedback to the subjects and included four vibration motors placed equidistantly around the subject's forearm.The motors were attached to the forearm using an elastic brace positioned proximally to the electrodes.The motors were oriented vertically so that the side of the motors was in contact with the skin as such placement produced clearer sensations.The driver board of the vibrotactile system was connected to the Michelangelo hand and the computer via Bluetooth.It conveyed the prosthesis sensor data (EMG signals and grasping force, sampled at the rate of 100 Hz) to the computer, activated vibration motors according to the predefined feedback scheme, and sent commands to the prosthesis computed from the recorded myoelectric activity (see Section II-C).According to the datasheets of the components, the round-trip latency in the control loop was less than 50 ms and it was mainly due to the bidirectional Bluetooth communication between the internal prosthesis controller and the vibrotactile system.Importantly, none of the subjects complained about the system's responsiveness.Inside the system, the sampled EMG signals were additionally filtered using a digital low-pass filter with a cut-off frequency of 0.65 Hz to obtain a smooth envelope.This value was selected based on a recent study [33], which showed that a low cut-off frequency led to the overall best performance during closed-loop prosthesis control.
The equipment for the "Box and Block" test was placed on the desk which height was adjusted so that the subjects could easily grasp wooden blocks and transfer them over the barrier.A simple with a graphical user interface was developed to assist the subjects during different phases of the experimental procedure (see Section II-E.2) and was displayed on a standard laptop computer placed on the desk in front of them, right next to the "Box and Block" setup.However, when performing the main experimental task (see Section II-E.1), the screen Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
was turned toward the experimenter so that the subjects could not see it.The app generated the order of the target forces, which the experimenter then verbally indicated to the subjects, allowed the experimenter to measure the time (start and stop buttons), and recorded the results (e.g., prosthesis sensor data).
2) Amputee Subjects: The setup for the amputee subjects (Fig. 1, right) included the same components as in the non-disabled group and they also performed the tasks in a standing position.However, for each of the amputee subjects, a specially designed, customized socket was produced, which contrary to non-disabled subjects was not additionally suspended during the experiment, thereby mimicking the real-life use of the prosthesis.The Michelangelo hand was connected to the socket, whereas the electronics of the prosthesis and the vibrotactile system were placed in a special compartment, inside the distal part of the socket, between the subject's arm and the prosthesis' rotation unit.Two EMG electrodes were integrated into the socket and their location was determined by a prosthetist, who used standard procedure to identify the positioning so that each electrode produced a good control signal.The prosthetist also decided on the exact positioning of the vibro motor array within the socket, considering the socket size and EMG electrode locations.The motors were oriented vertically and placed equidistantly around the residual forearm.To reduce the spread of vibrations through the rigid material, they were integrated into the socket using a damping mechanism designed by Otto Bock.Specifically, each motor was mounted in the middle of a mechanical spring attached across a ring-shaped frame, that was inserted into the inner side of the socket.The vibration kinetic energy was, therefore, partly dissipated through the displacement of the spring.This allowed the generation of more localized sensations, but also prevented the potential movement of EMG electrodes due to vibrations, which could lead to movement artifacts in the recorded EMG.

C. Combined EMG and Force Feedback
To avoid excessively high muscle contractions that could lead to muscle fatigue, the myoelectric signal was normalized to 60% of the maximal voluntary contraction (MVC), whereas the dead zone threshold was set to 10% MVC to prevent involuntary prosthesis movements [33].The normalized myoelectric signal was divided into six intervals that were mapped to the prosthesis command input using a piecewise linear mapping (so-called nonlinear mapping [40]), as shown in Table II.Importantly, most commercial prostheses, including the Michelangelo hand, function proportionally.The normalized command signal can be regarded as a reference input for the prosthesis motor (e.g., motor input voltage).Hence, when the prosthesis is open, the command input determines the velocity of closing (1 corresponds to the maximum speed).However, once the prosthesis contacts a rigid object and the motor stalls, the command input sets the desired force (1 corresponds to the maximum force).The myoelectric signal was mapped to the command input (Table II), and hence, the higher the myoelectric signal, the faster the prosthesis closed and the stronger the grasping force after contact.Therefore, there is an approximately one-to-one correspondence between the levels of the myoelectric signal, closing velocity, and grasping force defined in Table II.For instance, if the myoelectric signal was maintained within level 2, the prosthesis would close with level 2 velocity and produce exactly the same level of grasping force.If the subject then increased the myoelectric signal after contact, the prosthesis responded with a further increase in grasping force (e.g., from level 2 to level 3 or higher).Otherwise, as the prosthesis is non-backdrivable, the force achieved upon contact (e.g., level 2) was maintained until the object was released (e.g., by generating wrist extension).
As indicated before, the vibrotactile interface provided discrete EMG and force feedback (Fig. 2a).During prosthesis closing, the feedback indicated the level of the myoelectric signal, whereas 1 s after the hand contacted the object, the feedback switched to conveying the level of grasping force (Fig. 2b).The switching time of 1 s after the touch onset was chosen in order to have a smooth transition from EMG to force feedback, as this was the time sufficient for the grasping force signal to increase and stabilize (see Section II-D and Fig. 3).
Each level of EMG/grasping force was associated with a different pattern of activation across the motors (so-called spatial encoding) (Fig. 2c).If the feedback signal (EMG or grasping force) was within the first interval, the motor on the lateral side was activated.As the signal increased, other motors were activated sequentially in a clockwise direction.More precisely, the motors on the posterior, medial, and anterior sides of the forearm were sequentially activated as the signal reached the second, third, and fourth levels, respectively.If the signal was within the fifth level, the two motors on the lateral and anterior sides were activated.Finally, all four motors vibrated when the signal was within the maximal, sixth level.The discrete encoding scheme was selected because it is easy to interpret and has been effectively used before [27], [30], [32], [33].The vibration intensity for the motors was defined for each subject individually before commencing the experiment (see Section II-E).

D. Closed-Loop Control Approach
Providing the EMG feedback during prosthesis closing allowed the subjects to predictively control the force that the hand would generate after grasping an object.Specifically, to produce the desired force level, the subjects were instructed to use online EMG feedback to reach that same level of muscle contraction, and then maintain the myoelectric signal within that interval until the prosthesis completely closed.As explained in the previous section, the level of myoelectric signal upon contact defined the level of the generated grasping The intensity of muscle contraction was translated into the prosthesis velocity (nonlinear mapping).During the prosthesis closing, vibrotactile EMG feedback indicated the level of the myoelectric signal, allowing the subject to predict the grasping force that the hand would generate.1 s after the prosthesis closed around an object, the vibrotactile feedback switched to indicating the level of actual grasping force, allowing the subject to correct the generated force.(b) A timeline of a single grasping trial that indicates the moment of the feedback switch.In both (a) and (b), the green pathway corresponds to predictive strategy, whereas pink color stands for corrective strategy.(c) Two feedback signals (EMG and grasping force) were discretized into six levels, and each level was conveyed to the subjects by activating vibration motors around the forearm (full circle indicates active motor).
force after contact (Fig. 3a).One second after the touch onset, vibrotactile stimulation started conveying the information on the grasping force, and this allowed the subjects to notice if the desired level was indeed achieved.Importantly, as shown by the feedback signal in Fig. 3, the transition from EMG to force feedback was smooth, as the feedback level remained the same after the feedback switch.The smooth transition was the result of the proportional relation between EMG and force, as explained in the previous section.If the generated force was lower than the target, they could then correct the force by carefully increasing the intensity of muscle contraction until the feedback indicated that the force increased (Fig. 3b).Once the desired force level was reached, the subjects could relax their muscles, and due to the non-backdrivability of the prosthesis, the attained force was maintained until they generated the command to open the hand (i.e., contract the wrist extensor muscles).This is a usual behavior in myoelectric prostheses, which aims to minimize muscle fatigue because the user can relax the muscles while holding an object, but it also prevents a controlled decrease in force.Therefore, in the present study, downward force corrections were not possible.

E. Experimental Protocol
The experimental assessment was divided into two days.During the first day, the subjects received training to familiarize themselves with the myoelectric control, vibrotactile feedback, and the experimental task that was inspired by the "Box and Block" test.On the following, validation day, they performed the experimental task in three conditions, i.e., without vibrotactile feedback, with EMG feedback, and with combined EMG and force feedback.In both EMG feedback and no feedback conditions, the relationship between the myoelectric signal, prosthesis command input, and grasping force remained the same as with the combined feedback (Table II), but the three conditions differed in the type of sensory information conveyed to the subjects through the vibrotactile interface (Table III).Specifically, as explained in Section II-C, in the combined approach, EMG feedback was provided during the prosthesis closing, whereas after the prosthesis closed, the vibrotactile stimulation conveyed the information on the grasping force.In the condition with only EMG feedback, the feedback was also provided throughout the whole grasping trial, i.e., during the prosthesis closing as well as after the hand closed, but it always conveyed information on the generated EMG.In the no feedback condition, the stimulation was turned off.To mimic a realistic scenario, in all three conditions, the subjects were not deprived of incidental sensory information (e.g., their vision, the sound of the prosthesis, and their natural muscle proprioception).The order of conditions was randomly selected for each subject.
Both training (day 1) and validation (day 2) sessions lasted between 2 and 2.5 hours.1) Experimental Task: Despite using the "Box and Block" setup, our task was different from the classic "Box and Block" test.While the latter focuses on the speed (number of blocks transferred during a predefined time interval), our task measured the accuracy of force control.Therefore, at the beginning of each trial, the subjects were asked to grasp a single wooden block with the prosthesis using a three-digit pinch grasp (i.e., between the prosthesis thumb, index, and Two representative trials illustrating the use of combined feedback for prosthesis control (a) without and (b) with force correction after contact.The plots show the EMG signal, grasping force, and the signal indicating discrete feedback levels conveyed to the subject, recorded in two grasping trials when the target force was level 4 (gray areas in the bottom panels).The discrete levels were conveyed to the subject using vibrotactile stimulation patterns shown in Fig. 2c.The color of the feedback signal indicates the variable conveyed by the stimulation (orange for EMG and purple for force).The two vertical lines are the touch onset and the feedback switch (from EMG to force), 1 s after the touch onset.(a) The subject used vibrotactile EMG feedback to contract the wrist flexor muscles, reach the fourth interval of the normalized EMG range, and maintain the myoelectric signal within that interval until the prosthesis completely closed.The force feedback indicated that the generated force level was indeed level 4, and the subject relaxed the muscles as the task was successfully accomplished.(b) In this trial, the subject did not manage to reach the fourth level of EMG before the prosthesis closed.Consequently, the actually generated force was one level lower than the target.The subject realized this by perceiving the vibrotactile force feedback and then used the feedback to increase the force to the next level.
middle finger), produce the given force level that was verbally indicated by the experimenter, and transfer the block from one side of the barrier to the other.During the training phase (see Section II-E.2 and Fig. 1), 10 blocks were placed in the same compartment, and then after the subject transferred all the blocks, they were replaced by the experimenter.During testing (see Section II-E.3), however, to eliminate a potential impact of the clutter on performance, only a single block was used, which was replaced by the experimenter after each trial.Each of the six force levels was used five times as the target resulting in 30 trials (6 levels × 5 presentations), which were presented in a randomized order.Therefore, not only did the subjects need to grasp a wooden block and transfer it to the other side of the barrier, as in the standard "Box and Block" test, but they also needed to produce the given grasping force to consider a trial successful.Since the Michelangelo prosthesis is non-backdrivable, once the subjects assumed that the desired force was generated, they were allowed to relax their muscles and transfer the block, knowing that the achieved force would be maintained.The maximum force level recorded during the grasping trial (i.e., from the moment the object was contacted until it was released) was regarded as the generated force (trial outcome).If the generated force was higher or lower than the target level after releasing the object, the subjects were informed verbally that they made a mistake.This corresponded to the feedback that the subjects would likely receive in the daily-life application as they could see if the object slipped from the grasp (force too low) or if it was deformed/broken (force too high).Importantly, they were not time-limited when performing the task.Before the next trial, the prosthesis was returned to the neutral position in non-disabled subjects and fully opened in the amputees.
2) Training (Day 1): First, the stimulation intensity for each vibration motor was determined.Specifically, each motor was repeatedly activated for two seconds, while the vibration intensity was increased in steps of 10% of the normalized range until the subjects reported that they felt vibrations and could recognize the stimulus location reliably, without any discomfort.After that, the subjects were familiarized with the vibrotactile feedback scheme.To this aim, a vibration pattern for each EMG/force level (Fig. 2c) was delivered three times for two seconds, while the experimenter verbally indicated the level.To test the subjects' ability to recognize the patterns, they were delivered randomly (each pattern three times in total) for two seconds and the subjects were asked to identify the level conveyed by the feedback.The correct answers were provided by the experimenter (reinforced learning).The procedure was repeated until the subjects identified all levels correctly.
In the following phase, the subjects were familiarized with the myoelectric prosthesis control in combination with vibrotactile feedback.Initially, they opened and closed the prosthesis several times, while vibrotactile feedback conveyed EMG and force levels.In addition, the generated myoelectric signal and grasping force were displayed as vertical bars, while the vibration motors were shown as circles Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
(full when activated) on the computer screen in front of the subjects.The bars changed color to indicate the levels conveyed by the vibrotactile feedback.The subjects were then asked to close the hand and produce different levels of grasping force several times by relying on the provided vibrotactile and visual feedback.Importantly, they were instructed to associate the colors they see on the EMG/force bars with the vibrotactile sensations on the forearm.Thereafter the procedure was repeated, but instead of simply closing the prosthesis, the subjects needed to grasp a wooden block with different force levels several times.Again, they were advised to utilize vibrotactile feedback and visual cues displayed on the screen while performing the task.Finally, they were asked to repeatedly close the prosthesis, and grasp a wooden block with a given force level (indicated by the experimenter) using vibrotactile feedback, but this time, they could not see the screen, as this is how they would perform the main experimental task (Section II-E.1).Each of the six force levels was given as the target five times, consecutively.The experimenter verbally disclosed the actual, generated forces.
The final training step was to perform the experimental task using the setup of the "Box and Block" test (Section II-E.1) identically as it would be performed the following day, thereby simulating the validation phase.Importantly, throughout the training session, the experimenter verbally assisted and instructed the subjects to ensure that they were well-prepared for the upcoming validation.After finishing all the aforementioned steps for one condition, the whole procedure was repeated for the remaining two conditions with a break of about ten minutes in between.
3) Validation (Day 2): As a quick reminder, the subjects repeated the first two training phases (feedback and closed-loop prosthesis control familiarization) but with fewer trials.They then performed the experimental task using the setup of the "Box and Block" test (Section II-E.1) in two blocks of 30 trials, with a short break in between, collecting 60 trials in total for each condition.After finishing the test in one condition, there was a longer break of about 10 minutes, before repeating the whole procedure for the remaining two conditions.Target and generated force levels as well as the time elapsed from the moment the target force level was verbally indicated to the subjects until they released the object were recorded in each trial of the test.

F. Data Analysis
The main outcome measure to assess the accuracy of force control was the success rate expressed as the percentage of successful trials, defined as trials in which the subjects correctly grasped and transferred the block while applying the desired force level.Put differently, the successful trials were those in which the maximum force generated during the trial was equal to the target force.The second outcome measure, which assessed the speed of task execution, was the time that the subjects needed to perform a successful trial.The success rate was computed overall and per target force level for each subject, whereas the task time was computed per level.Additionally, in the condition with the combined feedback, the force generated at the beginning of each grasp (i.e., at the moment the feedback switched from EMG to grasping force) was recorded in successful trials to investigate the impact of the predictive (EMG feedback) and corrective mechanisms (force feedback) on the overall performance.If this initial force was equal to the target, this means that the subjects accomplished the task by relying solely on the predictive approach, with no need for corrections after the prosthesis contacted the object.However, the results for the highest, sixth force level were excluded from the analysis, since this level was trivial to achieve in all conditions -the subjects would simply generate maximal muscle contraction (i.e., the myoelectric signal was in saturation) to produce the maximal force when grasping a block.
For the group of non-disabled subjects, their success rates, overall and per target force, as well as their average task completion time per target level were compared statistically across conditions (i.e., no feedback, EMG feedback, and combined feedback).The success rates were also compared across the target force levels within each feedback condition to investigate if the feedback enabled a more stable performance across levels.For each of these comparisons, the appropriate statistical tests were selected using the following approach.First, the Lilliefors test was used to assess the normality of the data.In case the data were normally distributed, a one-way repeated measures ANOVA was performed.Post hoc analysis included pairwise comparisons using t-tests with Bonferroni correction.If the assumption of normality was violated, a nonparametric, Friedman's test was used, while the Wilcoxon signed-rank test with Bonferroni correction was applied for multiple comparisons.The threshold for statistical significance was set at p < 0.05.The statistical analysis was not performed for amputee subjects due to the small sample size, and their results were shown individually.

III. RESULTS
The overall success rates for non-disabled subjects in three conditions are shown in Fig. 4. The ANOVA test revealed that the success rates (mean ± standard deviation) in both conditions with feedback were significantly higher compared to the condition with no feedback (53±11.5%).In addition, the combined feedback (79.6±9.9%)outperformed the EMG feedback only (66.7±12.1%).
Fig. 5 depicts the success rate obtained for different target force levels in the three conditions for the non-disabled subjects.The two plots show the same results but grouped to compare performance across the target forces in the same feedback condition (Fig. 5a) and across feedback conditions for the same target force level (Fig. 5b).In the condition without feedback (Fig. 5a), there was a significant drop in performance for target level 3 (40±13.7%),and 4 (47.5±16.5%)compared to level 1 (68.1±23.7%)(Friedman's test, p=0.034 and p=0.029, respectively).On the contrary, when either EMG or combined feedback was provided, there were no statistically significant differences between the target levels.
Furthermore, when analyzed per level (Fig. 5b), the success rates with combined feedback (80.6±13.9%,73.1±19.9%,78.1±14.7%, and 87.5±10% for levels 2-5, respectively) were significantly higher compared to no feedback Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

Fig. 4.
The overall success rate for the non-disabled subjects in three conditions (no feedback, EMG feedback, and combined feedback).The boxplots display the distribution of individual results, whereas the average success rate (mean ± standard deviation) is indicated by the small black-filled squares.The performance was the highest when the combined, EMG and force feedback, was provided (79.6±9.9%), and the lowest for the no-feedback condition (53±11.5%).The black horizontal lines indicate statistically significant differences (p<0.001 - * * * ).
The distribution of forces generated at the beginning of each grasp in successful trials with combined feedback (i.e., at the moment when the feedback switched from EMG to force) over all subjects is shown in Fig. 5c.In the great majority of cases, the subjects achieved the target force or one level below immediately upon grasping the object (1 s after contact).In level 1, the initial force was equal to the target in all cases, but then the number of such trials (initial force = target force) steadily decreased for higher target force levels.The percentage of initially correct forces out of all successful trials for the middle target levels (2-5) was consistently lower with combined feedback (overall light blue in Fig. 5c) than with only EMG feedback (74.4% vs 97.2%, 71.8% vs 92.5%, 53.6% vs 90.3%, and 42.9% vs 81%, for levels 2-5, respectively).
The time to successfully complete the task for different force levels is reported in Fig. 6.The combined feedback increased the time with respect to no feedback in all target levels except the first (10.6±2.4 s vs 8.7±1.2 s, p=9e-3; 9.4±1.9s vs 7.6±1.5 s, p=6e-4; 9.5±2.1 s vs 7.5±1.3s, Fig. 5.The success rate for each target force level and three conditions (no feedback, EMG feedback, and combined feedback) obtained in the of non-disabled subjects.The results are grouped to analyze subjects' performance for different force levels in the same condition (a), and to compare their performance in three conditions for each, specific target level (b).The boxplots display the distribution of individual results, whereas the average success rate (mean ± standard deviation) is indicated by the small black-filled squares.The black horizontal lines indicate statistically significant differences (p<0.05 - * , p<0.01 - * * , p<0.001 - * * * ).The panel (c) shows the success rate for each target level with combined feedback over all subjects and trials.The bars are divided into segments to indicate the force level achieved at the beginning of a grasp in successful trials (i.e., the force recorded at the moment the feedback switched from EMG to grasping force).p=2e-4; and 10.6±3 s vs 7.1±1 s, p=5e-4 for levels 2-5, respectively).When comparing combined to EMG feedback, the task time was significantly lower with EMG feedback for level 2 (9.2±1.4 s, p=0.038), level 3 (7.9±1.4 s, p=9e-5), Fig. 6.
The time the non-disabled subjects needed to perform a successful trial in the experimental task for each target force in different conditions (no feedback, EMG feedback, and combined feedback).The boxplots display the distribution of individual results, whereas the average task completion time (mean ± standard deviation) is indicated by the small black-filled squares.The black horizontal lines indicate statistically significant differences (p<0.05 - * , p<0.01 - * * , p<0.001 - * * * ). 4 (7.8±1.7 s, p=9e-5), and level 5 (7.7±2 s, p=9e-5).However, there was no statistically significant difference in the task time between the EMG feedback and no feedback for any of the target levels.The ANOVA test was used to compare the results across conditions for level 2, whereas Friedman's test was employed for levels 1, 3, 4, and 5.
Finally, the results obtained from three amputee subjects are shown in Table IV.Overall, A2 and A3 performed noticeably better with combined feedback compared to EMG feedback (70% vs 52% for A2, and 88% vs 60% for A3), whereas A1 did not benefit from the novel scheme.In fact, A1 performed slightly better with EMG feedback compared to combined feedback (76% versus 70%).The lowest overall success rate was obtained in the absence of any vibrotactile feedback; coincidentally, it was 40% for all three subjects despite their per-level performance, as shown in Fig. 7a, was always different.Furthermore, all subjects achieved a higher success rate when provided with vibrotactile feedback (EMG or combined) compared to no feedback for all target levels (Fig. 7a), except A2 for target level 2. In fact, A2 and A3 showed a consistent trend for levels 3, 4, and 5, where the success rate was the lowest with no feedback, better with EMG feedback, and highest with combined feedback.The distribution of initial forces in the successful trials performed by amputees when using combined feedback followed the same trend as in the non-disabled group.A1 reached the target force for levels 2-5 immediately after grasping the object in 55.6% of the successful trials, whereas in the rest of them, he was only 1 level lower than the target.The initial force generated by A2 was equal to the target level (2-5) in 34.5% of the successful trials, 1 level lower than the target in 62.1%, and 2 levels lower only once.Finally, A3 produced the desired force level (2)(3)(4)(5) immediately upon grasping in 41.2%, 1 level lower than the target in 50% of the successful trials, and more than 1 level lower in 3 trials (i.e., 2 levels lower twice and 3 levels lower once).Regarding the time to successfully accomplish the task,  A3 exhibited the same trend as non-disabled subjects, i.e., the time was similar with EMG and no feedback and higher for combined feedback.In the case of A1 and A2, the time increased consistently from no feedback to EMG and then combined feedback (Fig. 7b).

IV. DISCUSSION
The present study proposed a novel method to close the loop in prosthesis control by facilitating both predictive and corrective processes characteristic of biological grasping strategies [35].The performance when using the prosthesis to accomplish a functional task with the novel method (combined feedback) was compared to that achieved with EMG feedback and no feedback in non-disabled and amputee subjects.
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
The results obtained in non-disabled subjects showed that both conditions with feedback (EMG and combined feedback) led to significantly improved performance compared to the condition without vibrotactile feedback (Fig. 4).Importantly, the performance in the combined approach was significantly higher than with the EMG feedback only.Therefore, although a previous study [32] demonstrated that EMG feedback outperformed conventional force feedback in a similar forcematching task, the results obtained in the present work show that the subjects can exploit these two feedback variables when they are provided sequentially, through the same vibrotactile interface, resulting in a significantly increased overall performance.
Despite its benefits, EMG feedback does not guarantee perfect accuracy [32], [33].The subjects can fail to maintain the myoelectric signal within the desired level, or they are not fast enough to reach the target level during prosthesis closing, as illustrated in Fig. 3.After contact, the prosthesis responds differently, and the force is not anymore smoothly proportional to the command input (myoelectric signal) but changes stepwise and abruptly.In these cases, therefore, when using only EMG feedback, the subjects could actually produce an incorrect grasping force without being aware that they made a mistake.The combined feedback provided force information after contact and this allowed the subjects to identify those cases and correct the force, thereby increasing the overall success rate.This is indeed confirmed when analyzing per-level performance (Fig. 5), which shows that the combined feedback particularly improved the success rate over the EMG feedback when there was less time for the predictive phase (faster closing for levels 4 and 5).
Nevertheless, the improvement in performance with the combined feedback comes with a cost, i.e., a significantly increased time to successfully complete the task compared to the remaining two conditions (without feedback and with EMG feedback) for the middle target forces (levels 2-5) (Fig. 6).This outcome is not surprising since, in case the subjects did not initially reach the target level using the predictive EMG feedback, they needed additional time to correct the generated force after the prosthesis closed (Fig. 3b).The increase in task time when introducing feedback was also previously reported with other methods [20], [41], [42].However, the difference in time might decrease after a longer period of using the prosthesis with the combined feedback, as the subjects would likely become more skilled in executing force corrections.Another approach to characterize the performance of the combined feedback would be to assess the impact of time constraints on performance, more specifically, to assess the speed-accuracy trade-off functions afforded by this novel feedback modality, following the approach described in [43].Interestingly, the EMG feedback improved performance compared to no feedback, without increasing the time to accomplish the task (Fig. 6).This reflects the predictive nature of EMG feedback, which allows the subjects to adjust the grasping force already while closing the prosthesis.
In general, the benefits of feedback (EMG or combined) were most pronounced for the middle target levels (i.e., levels 2-5 for combined, and levels 2-4 for EMG in Fig. 5b).
In fact, when comparing across force levels, the performance without feedback exhibited a characteristic "U-shaped" profile, whereas for both feedback conditions, the performance profile was relatively flat (Fig. 5a).Therefore, providing feedback allowed the subjects to achieve robust control that was not impacted by the target force.
Interestingly, the analysis of the initial grasping forces (Fig. 5c) showed that, although the same information (level of EMG) was conveyed to the subjects until they grasped the object in both feedback conditions (EMG and combined), they had two different approaches to produce the target force.Namely, in the EMG feedback condition, they aimed to generate the desired force immediately upon contact by reaching the corresponding level of EMG within a short period of prosthesis closing, knowing that the information on the actual force would not be provided subsequently.Therefore, as expected, they relied solely on the predictive strategy.However, with the combined feedback, they switched to a more conservative approach, which was particularly evident for higher target forces (Fig. 5c).Namely, instead of predictively aiming directly for the target level, they often used EMG feedback to generate the initial force that is close to the target (e.g., one level lower) and then they increased the force to the desired level using force feedback (Fig. 3b).The subjects therefore employed and clearly benefited from both predictive and corrective control strategies.This allowed them to minimize the chance of overshooting the target, which is a "catastrophic" event as the force could not be decreased in a controlled manner, while also minimizing the number of subsequent force corrections (e.g., shift the force only 1 level up after contact).
Although the three amputee subjects also clearly benefited from the additional vibrotactile feedback (EMG or combined in Table IV), one subject (A1) did not manage to further increase the performance using the combined approach compared to only EMG feedback.This might be because the subjects were not explicitly notified about the moment when the feedback switched from EMG to grasping force, and indeed, some of them reported they lacked this indication.This could be implemented by providing a tactile cue marking the contact event, as in [44].Another challenge is that the feedback becomes very different before and after contact due to the nature of the feedback signals; it changes fast when conveying the information on EMG as the subjects can easily modulate the myoelectric signal, whereas it changes slowly and abruptly when transmitting the force.Some subjects indeed reported that it was difficult to adapt to this change and adjust the strategy within a single grasp.
EMG and force signals have been combined before but the feedback was provided visually [31], [45], which is an ideal interface for the perception and integration of information, but not really convenient for clinical application.In [31], the high information bandwidth of the visual display was used to convey the signals simultaneously.This could be implemented using vibration motors by designing different vibration patterns, for instance, continuous activation for force and pulsating profile for EMG level, but the perception and interpretation of such encoding might be challenging.
Additionally, the experimental setup in [31] was not directly clinically relevant (e.g., prosthesis on the table) and this aspect was improved in [45], but the feedback was again provided visually and it included additional information (EMG, force, contact, and aperture).
In the present study, EMG and grasping force were provided sequentially using the same vibrotactile interface.Since non-disabled individuals perceive muscle proprioception and grasping force through different sensory pathways, it would make sense to convey those variables using different interfaces (e.g., two sets of motors).However, using the same motors results in a more compact interface, and this is an important practical advantage when integrating the feedback inside the socket, as we have done in the amputee subjects.
In the present experiment, subjects were asked to grasp the same rigid object (a wooden block) using different target force levels.This is different from the natural grasping task, where the grasping forces are scaled according to the characteristics (e.g., weight, shape, etc.) of different objects.Nevertheless, our experimental approach allowed us to distribute the target levels over the whole working range of the prosthesis.To somewhat mimic the natural, ecological feedback (e.g., object broken, damaged, or slipped from the grasp), the experimenter verbally disclosed at the end of each trial if the generated force was too high or too low with respect to the target.
In principle, the prosthesis can generate forces even without grasping an object (e.g., closing the prosthesis placed on the table so that the fingers press against each other, as done by non-disabled subjects in [40]).Nevertheless, our objective in the present study was to investigate the impact of additional vibrotactile feedback in realistic conditions, where the subjects needed to apply the desired force, while also carrying the whole setup on their arm (without suspension in amputees) and taking care to position the hand to grasp the block correctly.The latter aspect was "tested" by transferring the block over the barrier, as the block could fall out if not stably grasped.
Finally, as demonstrated before [46], predictive and corrective processes have complementary roles during the prosthesis grasping force control.Specifically, if the control is reliable, the subjects may establish a reliable prediction of the forces they produce, allowing them to adjust their commands before the prosthesis closed, whereas corrective strategies can be employed to refine their judgments in case of an inaccurate prediction.Indeed, it seems that prosthesis users tend to establish such control behavior even in the absence of explicit feedback.Specifically, they form predictive models [47] while the incidental sources of sensory information inherently available in myoelectric prostheses [48], such as subjects' vision, their inner muscle proprioception, and the sound of prosthesis movements can provide a reliable estimation of the prosthesis state (e.g., prosthesis velocity) [49], [50], enabling them to implicitly close the control loop [24].With the combined feedback proposed in the present work, the predictive and corrective processes become explicit, leading to improved grasping force control.However, experience and training play an important role in establishing both predictive and corrective strategies, and this can affect the use of supplemental feedback [30], [51], [52], [53].The present study was not designed to assess the impact of learning and, therefore, the next step in this research is to investigate if and how predictive and corrective mechanisms change during prolonged use of the combined feedback.

V. CONFLICT OF INTEREST
Christian Hofer, Michael F. Russold, and Mario Koppe are employed by Otto Bock, a producer of the Michelangelo hand, which was used as an example of a myoelectric prosthesis in the present study.

Fig. 1 .
Fig.1.The experimental setup used in non-disabled subjects (left) and in amputee subjects (right).(Left) The subject placed his/her forearm in a 3D-printed splint, with a Michelangelo prosthesis attached below.Two EMG electrodes and four vibration motors were strapped around the forearm using elastic braces.(Right) A custom designed socket, with the Michelangelo prosthesis, mounted on the subject's residual limb.Two EMG electrodes and four vibration motors, positioned equidistantly around the residual limb, were integrated into the socket.A simple graphical user interface was displayed on a laptop computer during the familiarization phase, but the screen was removed from the subjects' view when performing the main task (see text).

Fig. 2 .
Fig.2.The closed-loop control scheme with combined feedback.(a) The subject closed the prosthesis by activating the wrist flexor muscles.The intensity of muscle contraction was translated into the prosthesis velocity (nonlinear mapping).During the prosthesis closing, vibrotactile EMG feedback indicated the level of the myoelectric signal, allowing the subject to predict the grasping force that the hand would generate.1 s after the prosthesis closed around an object, the vibrotactile feedback switched to indicating the level of actual grasping force, allowing the subject to correct the generated force.(b) A timeline of a single grasping trial that indicates the moment of the feedback switch.In both (a) and (b), the green pathway corresponds to predictive strategy, whereas pink color stands for corrective strategy.(c) Two feedback signals (EMG and grasping force) were discretized into six levels, and each level was conveyed to the subjects by activating vibration motors around the forearm (full circle indicates active motor).

Fig. 3 .
Fig. 3.Two representative trials illustrating the use of combined feedback for prosthesis control (a) without and (b) with force correction after contact.The plots show the EMG signal, grasping force, and the signal indicating discrete feedback levels conveyed to the subject, recorded in two grasping trials when the target force was level 4 (gray areas in the bottom panels).The discrete levels were conveyed to the subject using vibrotactile stimulation patterns shown in Fig.2c.The color of the feedback signal indicates the variable conveyed by the stimulation (orange for EMG and purple for force).The two vertical lines are the touch onset and the feedback switch (from EMG to force), 1 s after the touch onset.(a) The subject used vibrotactile EMG feedback to contract the wrist flexor muscles, reach the fourth interval of the normalized EMG range, and maintain the myoelectric signal within that interval until the prosthesis completely closed.The force feedback indicated that the generated force level was indeed level 4, and the subject relaxed the muscles as the task was successfully accomplished.(b) In this trial, the subject did not manage to reach the fourth level of EMG before the prosthesis closed.Consequently, the actually generated force was one level lower than the target.The subject realized this by perceiving the vibrotactile force feedback and then used the feedback to increase the force to the next level.

Fig. 7 .
Fig.7.The success rate (a), and the average time needed to successfully complete a single trial (b) for each target force level, obtained in three amputee subjects (A1, A2, and A3) performing the task in three conditions (no feedback, EMG feedback, and combined feedback).

TABLE I THE
DEMOGRAPHICS OF THE AMPUTEE SUBJECTS

TABLE II UPPER
BOUNDARY FOR EACH DISCRETE LEVEL OF THE MYOELECTRIC SIGNAL, PROSTHESIS COMMAND INPUT AND GRASPING FORCE ON THE NORMALIZED SCALE