Artificial Perception and Semiautonomous Control in Myoelectric Hand Prostheses Increases Performance and Decreases Effort

—Dexterous control of upper limb prostheses with mul-tiarticulated wrists/hands is still a challenge due to the limitations of myoelectric man–machine interfaces. Multiple factors limit the overall performance and usability of these interfaces, such as the need to control degrees of freedom sequentially and not concurrently, and the inaccuracies in decoding the user intent from weak or fatigued muscles. In this article, we developed a novel man– machine interface that endows a myoelectric prosthesis (MYO) with artiﬁcial perception, estimation of user intention, and intelligent control (MYO–PACE) to continuously support the user with automation while preparing the prosthesis for grasping. We compared the MYO–PACE against state-of-the-art myoelectric control (pattern recognition) in laboratory and clinical tests. For this purpose, eight able-bodied and two amputee individuals performed a standard clinical test consisting of a series of manipulation tasks (portion of the SHAP test), as well as a more complex sequence of transfer tasks in a cluttered scene. In all tests, the subjects not only completed the trials faster using the MYO–PACE but also achieved

Abstract-Dexterous control of upper limb prostheses with multiarticulated wrists/hands is still a challenge due to the limitations of myoelectric man-machine interfaces.Multiple factors limit the overall performance and usability of these interfaces, such as the need to control degrees of freedom sequentially and not concurrently, and the inaccuracies in decoding the user intent from weak or fatigued muscles.In this article, we developed a novel manmachine interface that endows a myoelectric prosthesis (MYO) with artificial perception, estimation of user intention, and intelligent control (MYO-PACE) to continuously support the user with automation while preparing the prosthesis for grasping.We compared the MYO-PACE against state-of-the-art myoelectric control (pattern recognition) in laboratory and clinical tests.For this purpose, eight able-bodied and two amputee individuals performed a standard clinical test consisting of a series of manipulation tasks (portion of the SHAP test), as well as a more complex sequence of transfer tasks in a cluttered scene.In all tests, the subjects not only completed the trials faster using the MYO-PACE but also achieved

I. INTRODUCTION
W HEN we grasp an object, we usually do not think about how we do it.The type of grasp, tuning of the wrist posture, and movement of the fingers seem to happen automatically.Years of training have led to highly dexterous brain/hand interaction that mainly happens subconsciously and includes the use of a variety of touch and proprioceptive sensors as well as stereovision through our eyes.Sudden loss of a hand dramatically changes this situation with an impact on a person's ability to perform work-related, social, and daily life activities.Current myoelectric prostheses are advanced robotic systems that aim to mechanically mimic their biological counterparts [1].Indeed, modern models have as many as 24 actuated degrees of freedom, allowing individual finger movements [2], and an almost complete range of wrist movements [3].However, in a relatively short time, the user must learn how to control these functions with interfaces that currently do not allow large information transfer rates and without feedback from touch or proprioception.
The current human-machine interfaces (HMIs) are inadequate because their control bandwidth is limited, i.e., the number and rate of user commands transmitted to the prosthesis are not high enough to fully exploit the advanced mechatronic capabilities of modern-day upper limb prostheses.As discussed by Ning et al. [4], the HMIs for prosthesis control are developing at a slower pace than the mechatronic technologies, and are therefore unable to efficiently accommodate the increasing complexity of prosthetic hands [5], [6].Myoelectric HMIs based on two control signals have been developed in the early 1960s and are still the state-of-the-art control systems in commercial active prostheses [4].These interfaces provide control over a single degree of freedom (DoF).Additional DoFs can be controlled by a switching command of the user (e.g., muscle cocontraction).This solution This work is licensed under a Creative Commons Attribution 4.0 License.For more information, see https://creativecommons.org/licenses/by/4.0/ is cognitively demanding and slow, especially when controlling more than two DoFs [7].To extend the number of controllable DoFs, the pattern classification of multichannel EMG signals has been investigated for decades [8].In this approach, the user can activate the desired function among several possible ones.However, functions are still controlled sequentially (one at a time) (e.g., wrist rotation followed by hand closing).Conversely, simultaneous control of multiple functions can be achieved by regression control algorithms, although for a limited number of DoFs [9].
A characteristic common to all myocontrol methods is that the user needs to provide the commands for all the phases of the movement.This may lead to muscle and/or cognitive fatigue because such control is tedious, unintuitive, and unnatural [7].Achieving effective myocontrol, therefore, requires a high level of physical and mental fitness.This contrasts with the natural symbiotic link between the brain and the hand, which allows the hand to move effortlessly to perform a target task.
In this study, we implement and evaluate a system based on sensor fusion that supports the user in controlling a prosthetic hand by automatizing its functions during reach and grasp interactions.This approach counterbalances the shortcomings of conventional myocontrol by sharing control with the user.Recently, multimodal sensor-fusion techniques [10], [11] have been applied in several HMIs for assistive systems to improve control capabilities while reducing the cognitive burden of myocontrol.By relying on multiple sensor modalities, sensor fusion can be employed to interpret the user's behavior as well as the state of the prosthesis and the environment, thereby providing context information.The context data can then be used by the prosthesis controller to implement a certain degree of automation.This approach can improve ease of use, robustness, and/or performance compared to the conventional HMIs, which rely only on the explicit command input generated by the user [12], [13].Sensing and interpretation of the context have been widely used for wheelchair control [14]- [17], and have recently gained popularity in prosthetics as well [18]- [20].
Prosthetic control is a challenging scenario for cooperative automation since some DoFs are under direct control of the artificial controller, while others are controlled solely by the user.For example, a transradial amputee controls the position and orientation of the prosthesis by moving the residual limb in space, while the controller adjusts the wrist and preshapes the hand [21].In previous approaches, myoelectric interfaces have been supplemented by systems for automatic reduction of users' compensatory movements during reaching using inertial data [22], prevention of object slippage by interpreting measured forces [23]- [25], or automatic grasp estimation through computer vision [21], [26]- [28].However, in all these cases, the automatic assistance was limited to specific functions and integrated with the most basic myoelectric control (two channels with switching).The user was also required to perform explicit actions to activate the assistance, such as centering the gaze at the target object [21], issuing a predefined myoelectric command [28], or performing a specific arm movement [21].
The present article describes a system that integrates a classification-based myoelectric control [MYO; implementing the linear discriminant analysis algorithm (LDA) [29]] with a novel interface for comprehensive artificial extero-and proprioception and autonomous, adaptive prosthesis control (PACE).The resulting system, which we will refer to as MYO-PACE, enables simultaneous control of four prosthesis DoFs and continuously reacts to a change in user intention or environment without the need to be explicitly triggered.This results in a level of control dexterity of prosthetic hands and wrists that has never been achieved before.The MYO-PACE interface mimics the dexterity of the human motor control by replacing the biological sensors involved in hand preshaping with their mechatronic counterparts: The human eyes, proprioceptors, and somatosensory receptors are replaced by a variety of sensors, placed externally and/or embedded in the prosthesis, including cameras, inertial units, motor encoders, and force sensors.Moreover, the system provides an intuitive augmented reality (AR) feedback to the user.Therefore, the proposed multimodal interface is the first system in which advanced myocontrol (LDA) and advanced scene perception are combined to implement continuous decoding of user's intentions and reactive control of a dexterous hand prosthesis with actuated hand and wrist (four degrees of freedom).
In this study, we also present an extensive validation of the proposed system and comparison with state-of-the-art methods for experimental tasks involving manipulation in cluttered scene (multiple simple objects) with both able-body and amputee participants and complex manipulation tasks (single complex object) with end users.The first task was selected to challenge the MYO-PACE with a complicated scene that includes multiple modeling and interaction possibilities, whereas the second task was a standard clinical test for assessing the quality of prosthesis control.We postulate three hypotheses.First hypothesis is that the MYO-PACE increases the overall performance in both tests (cluttered scene and a clinical test).Since the system is active only during the preparation of the grasp (wrist orientation and hand preshape are automatically regulated before the grasp occurs), our second hypothesis is that the performance improvement can be explained in terms of decrease in duration of this particular phase.Finally, our third hypothesis is that the autonomous support provided by the system decreases the overall myoelectric activity of those muscles that are involved in the prosthesis control.
The myoelectric interface comprises an array of eight dry active electrodes with adjustable gains.They acquire the EMG data and provide linear envelopes required to implement myoelectric prosthesis control.The linear envelopes are sampled at 100 Hz by an embedded prosthesis controller, and the data are transferred to the host PC via a Bluetooth connection.
The IMU is attached to the custom-made socket via a 3D printed connector.The IMU measures the absolute orientation of the prosthetic hand with respect to the local coordinate system (i.e., yaw, roll, and pitch angles).The information on prosthesis orientation was necessary to adjust the wrist according to the orientation of the participant's forearm.Data were streamed wirelessly to the host PC at a sampling rate of 30 Hz.
The Creative SR300 replaces the low-resolution camera embedded in the glasses, which are worn by the user.It is mounted on the glasses by using a 3D printed support.This ensures that the camera is facing the same scene that the user is looking at.It simultaneously acquires both color and depth (RGB-D) images at a resolution of 1920 × 1080 pixels and 640 × 480 pixels, respectively.The camera is a central component of the system operation.The RGB-D data are streamed through a USB port at a rate of 30 Hz.The depth data is employed for scene modeling, whereas the RGB-D data is used for ego-motion estimation (i.e., camera's motion relative to a rigid scene [30]).
The three retro-reflective markers are placed on the custommade forearm splint in positions that are unlikely to be occluded while performing the task.The relative position of the markers is captured at 30 Hz by the infrared sensor of the SR300 camera.The marker position is fed into the proprioception module to provide the position of the prosthetic hand, relative to the object.
The AR Meta Glasses, connected to the computer through an HDMI interface, superimpose digital holographic images to the real world, allowing the user to receive visual feedback directly overlaying the observed scene she/he is interacting with.The holographic images are generated by projecting 960 × 540 pixels screens on two semitransparent glasses located in front of the user's eyes.The refresh rate of the AR feedback is 30 Hz.
The Michelangelo hand prosthesis provides simultaneous opening and closing of all fingers with two grip types (palmar and lateral), as well as an actuated wrist with pronation/supination and flexion/extension.The prosthesis is mounted on a custommade ergonomic socket, which was connected to the left forearm of able-bodied subjects, positioning the prosthesis below the hand, as shown in Fig. 1.The prosthetic hand weights 600 g and is, in the case of able-bodied subjects, positioned 12 cm distal from the subject's hand.The custom adaptor weights 700 g and this weight is distributed around the forearm of the participant.The four-position encoders (thumb, fingers, wrist rotator, and wrist flexor) measure fingers aperture and hand orientation relative to the socket.They are required to reach specific positions for each DoF of the prosthesis.A single force transducer placed at the base of the thumb measures the grasping force, used to detect when the hand is holding an object.A bidirectional communication protocol, running over a Bluetooth interface at 100 Hz, allows for sensory and control data transmission.
The host PC 1) receives data from the sensors of the prosthesis, the myoelectric interface, the inertial unit, and the camera; 2) processes these data to obtain context information and to make decisions on the prosthesis control; 3) sends control commands to the prosthesis; and 4) visualizes the feedback to the user through the AR glasses.In addition, the host PC also implements a user interface for the execution of the experimental protocol (e.g., starting and stopping), system setup, and monitoring.The algorithms were implemented using C++ for scene generation, Unity 3D (Unity Technologies, San Francisco, US) for scene management, prosthesis preshaping and AR feedback, and MATLAB 2017a and Simulink (MathWorks, Natick, US-MA) for myoelectric inputs decoding and prosthesis communication (CLS Toolbox [31]).
The unique functions of the MYO-PACE system, as described below, are achieved through the integration of inputs from all the system components.As shown in Fig. 2, the PACE system comprises a pipeline of processing modules leading into the autonomous preshape control block.When the information from all components has been fused into a complete model of the scene, including the objects and the prosthesis, the PACE commands can be computed and transmitted to the prosthesis.

III. CONTROL SYSTEM
The MYO-PACE interface relies on the control commands generated by the MYO (Fig. 2(a)) and the PACE (Fig. 2(b)) control interfaces in order to control the multiple DoF prosthesis.The two interfaces operate simultaneously to generate myoelectric (volitional) and automatic control commands for the prosthesis, respectively.These two command streams are then multiplexed (Fig. 2(c)) to select the one that will be transmitted to the prosthesis.In the end, the AR feedback interface (Fig. 2(d)) provides visual feedback to the user based on the information received from the system.Fig. 2. Overview of the novel MYO-PACE system.The system integrates (a) the PACE interface, (b) the myoelectric (MYO) interface, (c) the control commands multiplexer, and (d) the augmented reality feedback interface.The novel PACE interface (a) continuously perceives the user and the environment, estimates user's intentions and reactively, autonomously and simultaneously adjusts the four DoFs of the prosthesis.The manual control unit (b) interprets the myoelectric inputs of the user and proportionally controls the corresponding DoF of the prosthesis.The control commands multiplexer (c) defines which signal (manual or automatic) will control the prosthesis.The augmented reality feedback unit (d) provides the user with intuitive information regarding the state of the system and the environmental perception, allowing him/her to supervise its operation and, if necessary, intervene.

A. PACE Interface
The PACE interface (Fig. 2(a)) comprises three modules: artificial perception, user intent estimation, and autonomous control.The artificial perception module [see Fig. 2(a-1)] tracks the prosthesis (and the user) movements and models the user's environment.The user intent estimation module [see Fig. 2(a-2)] predicts the object that the user wants to grasp.The autonomous preshape control module [see Fig. 2(a-3)] determines the optimal strategy to preshape the prosthesis for grasping and generates a continuous stream of commands to the prosthesis to configure the hand and wrist.
1) Artificial Perception Module: The artificial perception module [see Fig. 2(a-1)] has three submodules: artificial proprioception, artificial exteroception, and scene generation.They acquire proprioceptive and exteroceptive data from the sensors and combine them in order to track the user-actions (e.g., the prosthesis position and movement) and detect the changes in the environment (e.g., an object that is reallocated).
The artificial proprioception submodule acquires information about the user (i.e., hand tracking and ego-motion) and the prosthesis (joint angles configuration and grasping force).The camera is used to track the three retro-reflective markers located on the prosthesis, thereby defining the coordinate frame of the prosthesis with respect to that of the camera.The IMU sensor is then initialized based on this reference frame and used to track the prosthesis orientation.Ego-motion is estimated using the RealSense proprietary APIs [32] distributed with the camera, providing the information about the movement of the camera (user's head) and, therefore, the part of the scene that the user presently observes.The sensor data assessing the current state of the prosthesis (i.e., wrist angles, hand aperture, grip type, and grasping force) are collected through the AxonBus interface of the Michelangelo hand.Overall, the errors for the estimation of the prosthesis position and orientation in space are <2 cm and <5°, respectively, as determined in pilot tests.
The artificial exteroception submodule acquires depth images of the environment through the RGB-D camera and uses computer-vision algorithms to model the objects present in the scene (i.e., estimate shape, size, position, and orientation relative to the camera reference frame).In the first step, the algorithm filters each acquired point cloud to remove all the points which are not part of the objects (prosthesis, table, wall, and artifacts) and then fits primitive geometrical shapes to the remaining points to model the objects in the scene.Specifically, the points of the point cloud belonging to the prosthesis are removed based on the artificial proprioception information, while the table and the wall are identified by employing the random sample consensus method (RANSAC, [33]).Then, the algorithm based on RANSAC fits a horizontal plane to the point cloud; if found, this plane is considered as the table surface on which the objects are lying, and its points are filtered out.For the wall, if the RANSAC-algorithm identifies a second perpendicular plane with more than 40% of the point cloud points, these points are also removed.In the next step, the locally convex-connected patches algorithm (LCCP, [34]) is used on the filtered point cloud in order to cluster and segment the remaining points.In the last step, very small clusters (less than ten points) are directly removed, whereas the others are modeled through RANSAC, selecting the best fit between basic geometrical shapes (spheres, boxes, and cylinders).No a priori knowledge about the objects' properties (e.g., color features and size) is required.The result is a robust system that can handle unknown objects.
To obtain a reliable and consistent set of objects in the scene, newly modeled shapes are not immediately accepted and rendered in the scene.The algorithm first detects the possible collision between the new object models and the current scene model.If the collision is detected, it is assumed that the group of colliding models represents the same real object.Using the Mahalanobis distance, the center of the group is computed, and the object model closest to the center is selected to represent the real object.This object is then rendered and analyzed by the autonomous control system.The object model is removed from the virtual scene if it is not detected for more than ten consecutive frames, while it is in the camera's field of view.
The scene generating submodule is responsible for fusing the artificial proprio-and exteroception data and for generating and updating the real-time 3D representation of the scene in front of the user.When an object is grasped, it is temporarily removed from the scene, just to be immediately remodeled in its new position when the user releases it.To be able to track the objects even when they come out of the user's sight, simultaneous localization and mapping (SLAM, [35]) techniques are employed.The scene is not updated when the user moves his head faster than 0.1 m/sec or 0.17 rad/sec.The dynamic model of the scene provides the system with comprehensive information about the environment, which is the basis for the robust and predictive estimation of user intention and automatic configuration of the prosthesis.
By modeling the object using geometric primitives, the artificial exteroception module filters out sensor artifacts (i.e., spatially and temporally unstable clusters of points) coming from the vision system due to the movement of the camera, for instance.Moreover, since small and temporally unstable object parts would inevitably lead to large perturbations in preshape selection, this approach effectively stabilizes the system decisions and increases its robustness.On the other hand, this also makes the system less sensitive to objects (or object parts) having any dimension smaller than 2 cm, since these might be sometimes discarded already in the modeling phase.
2) User Intention Estimation Module: The user intention estimation module [see Fig. 2(a-2)] relies on estimated prosthesis position and orientation (from the artificial proprioception submodule) and the scene model (from the scene generating submodule) to interpret the user intention to grasp an object.For this purpose, each object in the scene is assigned a semispherical selection volume.The right part of the selection sphere is "cut" for quick deselection when the prosthesis moves from the object, preventing thereby unnatural pronation movements (i.e., trying to grasp an object from the right-hand side using a left-hand prosthesis).When the prosthesis approaches the object, a proximity value is computed according to the following formula: where D H-O is the distance between the hand (H) and the object (O), minD is the minimal distance to select an object, and hyst the hysteresis coefficient (equals to 1.16 if the object is selected, 1 otherwise).The higher the proximity value, the deeper is the prosthesis in the semispherical area.The object with the highest proximity value is then selected.The hysteresis coefficient (hyst) automatically increases from 1 to 1.16, and ensures the stability of the selection (Fig. 3).

3) Autonomous Preshape Control Module:
The autonomous preshape control module [see Fig. 2-a3] determines the Fig. 4. Illustration of the wrist preshape for the different grasps.The dark green and dark blue lines represent, respectively, the rotation and flexion axes of the wrist at rest.During a palmar side grasp (first row), the wrist rotation angle (R) is calculated as the sum of three elements: (a) the rotation to bring the palm parallel to the table, (b) a 40°degrees counter-clockwise rotation (adjusted to each subject by maximum ±10°), and (c) a small additional rotation to compensate the missing wrist abduction DoF (proportional to the angle (d) between the wrist rotation at rest (dark green) and the hand-object vector (e)).The flexion (F) is computed as the angle to bring the palm perpendicular to the object (f), slightly adjusted based on the prosthesis specific shape and user preferences.For the palmar top grasp, the wrist rotation (R) is the angle necessary to orient the palm perpendicular to the hand-object vector (g), slightly adjusted based on user preferences.The wrist flexion (F) is computed as the angle necessary to keep the prosthesis rotation axis perpendicular to the table vertical vector (h), corrected by a user-specific factor (±10°).
prosthesis preshape and orientation in order to facilitate the optimal grasp.The system decides on the grip type, amount of wrist flexion/extension, pronation/supination, and the size of the hand aperture.Three different scenarios are possible.If the object or at least the side of the object that the user wants to grasp is smaller than 3 cm, then the lateral grasp is selected.If the hand is above the object (hand height > 1.5 × object height), the palmar top grasp is selected.Otherwise, the hand will select palmar side grasp.For both lateral and palmar top grasps, the hand flexion axis is maintained horizontal while the palm is oriented toward the object by rotating the wrist, as illustrated in Fig. 4 and defined in equations ( 2) and (3): where ) are the angles between the vertical vector of the coordinate system of the object and the right and forward vectors of the forearm.The variables θ Corr top−rot and θ Corr top−flex , respectively, are the offsets for correction of rotation and flexion.These are adjusted for each subject individually and are in the range of (−10°, 10°).
In the case of the palmar side grasp, the hand is positioned to an angle of 40°from the horizontal, which is additionally adjusted proportionally to the angle between the forearm and the hand-to-object vector, in order to compensate for the lack of wrist abduction.The prosthesis flexion is continuously updated so that the palm maintains perpendicular orientation to the object, adjusted by the constant offset of 20°and an additional correction factor that is determined for each subject individually where is the angle between the forward vector of the forearm and the vector going from the hand to the object.The variables θ Corr side−rot and θ Corr side−flex are the offsets for correction of the rotation and the flexion, which are adjusted for each subject individually (−10°,10°).The proportional gains of 0.12 and 0.8 in ( 4) and ( 5) were determined heuristically.
The hand aperture is set to be 2 cm larger than the thickness of the object along the trajectory that the fingers follow while closing.The generated set of multi-DOFs adjustments is continuously converted into smooth velocity trajectories and sent to the control commands multiplexer.Since the calculations are continuously performed, the system reacts instantaneously to changes in both prosthesis position and external environment.

B. Myoelectric Interface
The myoelectric interface (Fig. 2b, MYO) acquires linear envelopes from the array of eight EMG electrodes located on the ipsilateral side of the subject, extracts amplitude features (RMS), and classifies them into proportional single-DoF velocity commands for the prosthesis using LDA.Six classes have been implemented to control the DOFs of the prosthesis: close/open hand, flex/extend, and pronate/supinate the wrist.The user can cycle between the two available grasp types (i.e., palmar or lateral) through a short, strong activation of the hand extensor muscles.This approach reduced the number of classes that the user needed to generate (from seven to six) and made the myocontrol easier and more robust.Even after extensive training, the amputee subjects were unable to reliably generate six distinguishable muscle activation patterns.For this reason, a four-class control mode has also been implemented.In this mode, the control of the wrist is separated from that of the hand.The user can switch between wrist control and hand control by generating a short contraction (<0.3 s) of the flexor muscles.

C. Control Commands Multiplexer
In the terminology of shared control [36], the PACE and MYO controllers implement traded autonomy, where the multiplexer block (Fig. 2) switches the control from fully automatic to fully volitional depending on the system state.The control commands multiplexer (Fig. 2c) sends to the prosthesis the commands produced by the myoelectric interface when an object is grasped, or the user generates myoelectric inputs.In all the other cases, the multiplexer transmits the commands generated by the PACE interface (i.e., autonomous control).When the user stops producing myoelectric activity and no force is detected by the fingers (the hand is not holding any object), the control transitions from the manual to the automatic mode after a small delay of 0.75 s.

D. Augmented Reality Feedback Interface
The augmented reality feedback interface (Fig. 2d) employs the AR glasses to provide the user with visual feedback regarding the system operation.Icons are displayed in the top corners of the glasses to indicate to the user if the scene is being updated (blue cube), if an object is selected (yellow target) and if myoelectric inputs are detected (green hand).Additionally, all the detected objects in the scene are presented to the user through holograms, overlaying the real objects, with intuitively coded colors: Detected objects are blue, whereas selected objects are yellow with green faces to indicate the estimated approach side.The selected objects become green when the user sends myoelectric inputs.Fig. 2d shows the augmented reality feedback interface and the sets of icons and object holograms that are visualized on the glasses.
Fig. 5 illustrates the operation of the MYO-PACE system during the interaction with a single object.The snapshots were recorded during an experiment and show the subject's view of the scene through the AR glasses.They illustrate the interaction with one representative object, with scene modeling and updating (a), user intention detection, and automatic control of the prosthesis (b), (c), and (d) and manual control and manipulation (e), (f), (g), and (h).

IV. EXPERIMENTS
Amputee subjects and able-body subjects performed a custom-designed relocalization task to assess the MYO-PACE in a scene with cluttered objects (cluttered scene interaction test-CSI).In addition, the two amputees were also tested in manipulation tasks mimicking activities of daily living (SHAP).
In all tests, the performance of the MYO-PACE interface was compared to state-of-the-art myocontrol (LDA classifier).Each control scheme was trained and evaluated in two separate sessions.The order of the systems was randomized across subjects.The subjects trained the pattern recognition algorithm first, and then they received instructions regarding the tasks.

A. Subjects
Two amputee subjects (59 and 61 years old) and eight male right-handed able-bodied participants (from 23 to 30 years old) volunteered for this study.All subjects signed a written consent form that was approved by the Ethical Committee of the University Medical Center Göttingen (22/04/16).Three able-bodied subjects had previous experience in myoelectric control.Both amputees regularly used transradial myoelectric prostheses.

B. Experimental Tasks: Cluttered Scene Interaction Test (CSI) and SHAP
A CSI task has been designed by taking inspiration from the Southampton Hand Assessment Procedure (SHAP) test [37].The "abstract object" part of the SHAP test requires the participants to grasp one of the six abstract objects and to place it in the corresponding place holder.Our test extended this concept by 1) placing six objects simultaneously on the table [L 200 × W 105 × H (60-90 cm, adjusted individually)] (increasing the scene modeling and intention detection complexity), including two cd boxes (130 × 20 × 130 mm), two tea boxes (130 × 65 × 80 mm), one can (117 mm height, 65 mm diameter), and one plastic peach (around 75 mm diameter) (Fig. 6), and 2) by asking the subject to manipulate and relocate each object in one continuous, uninterrupted trial (altering the prosthesis pose after each manipulation).Fig. 6a shows the setup.Unlike in the SHAP test, the participants performed the CSI test while standing, which gave them the ability to utilize a wider variety of motions and body postures.The participants were also encouraged to grasp the objects as they preferred, which required the autonomous control to react to the possibly inconsistent behavior of each participant.For able-body subjects, the prosthesis was fixed to a splint, which immobilizes the hand, resulting in isometric muscle contractions during EMG signals generation.The amputee subjects were fitted with sockets that integrated eight EMG electrodes.
In order to extend the assessment to more complex objects used in daily living, 17 out of the 24 tasks of the SHAP test were included in the assessment.The remaining seven tasks were excluded since they were not compatible with the current setup.The daily life activities "Jar Lid," "Glass Jug Pouring," "Carton Pouring," "Lifting a Heavy Object," and "Lifting a Light Object" were included in the test.The combination of abstract objects and the activities of daily living (ADLs) covers a large variety of objects, rendering the evaluation more clinically relevant.Two amputee subjects participated in this experiment.Based on the scene model and prosthesis motion, the system infers user intention to grasp the central object from the front-left side.The object assumes a yellow-green texture, the prosthesis is automatically preshaped, and the yellow icon is displayed.(c) User changes his mind and moves the prosthesis toward another object.The system responds by indicating the new selection and reshaping the prosthesis to grasp the object from above.(d) However, the user moves the prosthesis toward the table and the system detects the intention to grasp the object from the side.The object texture is updated, and the prosthesis automatically changes the wrist orientation.(e) User approaches the object and generates muscle activity for closing the prosthetic hand.The hologram of the target object becomes green and the icon for myoelectric inputs is visualized (manual control mode).The time spent by the user to prepare the prosthesis for grasp and to close the hand counts as the myoelectric control time.(f) User has grasped the object and transports it without generating myoelectric signals.The system is in the manipulation phase even if the user does not generate myoelectric commands.(g) User manipulates the object (changes its orientation) by generating myoelectric commands to control the wrist degrees of freedom.The time spent performing this action counts both the time spent to generate myoelectric control signals and the time to control the wrist.(h) User releases the object in the new position where it will be automatically remodeled by the system as soon as the user stops generating myoelectric commands.

C. System Initialization and Calibration
The system was initialized as soon as the camera was turned ON, and the first frame was acquired.The system used this frame as a reference point for the computer-vision based tracking.Three components required calibration before the system could be adequately used: the AR glasses, the IMU, and the myoelectric interface.The calibration of the AR projection was done manually based on user feedback.To calibrate the IMU, the user held the prosthesis in front of the RGB-D camera to compute the relation between the camera coordinate system and the three markers of the prosthesis.
The myoelectric interface was calibrated before each test session in two different arm positions (shoulder 20°extended and elbow 70°flexed, shoulder 75°extended, and elbow fully extended).The myoelectric control was then calibrated in three steps: 1) the isometric maximal voluntary contractions (MVC) for each LDA class were measured; 2) the data for the classifier training were collected by asking the subject to perform the movement at two contraction levels (40% and 80% of the MVC); and 3) the thresholds and gains that regulate the class detection and proportional control of the prosthesis were fine tuned.Amputee subjects were first tested on their ability to produce six different muscle patterns.If they were not satisfied with the prosthesis controllability, they tested the four-class control.

D. Procedure
Each control scheme was trained and evaluated in two separate sessions.The order of the systems was randomized across subjects.Fig. 6 illustrates 1) the setup and initial positions of the objects for the CSI test, 2) the first three object relocations, 3) the next three relocations, and 4) the objects in their destination positions.
A training session (approximately 2.5 hours) preceded each evaluation session, which was performed on a separate day.During the training session, the participants practiced and calibrated either LDA or MYO-PACE.The training session allowed the subjects to understand both control schemes, as well as to practice in grasping objects by generating appropriate muscle activations.While practicing, the subjects also received explanations about the movements generated by each of the systems (MYO-PACE and LDA).The goal of the training session was for the subjects to feel confident in their ability to skillfully interact with the prosthesis during the subsequent evaluation session, with both control systems.During each experiment, the subjects had to interact with a set of objects using either the MYO-PACE or LDA control.The interaction included grasping, lifting, reorienting, transporting, and releasing the objects.During the CSI test, the subjects interacted with every object once, except for the peach, which they relocated again after the first manipulation.The scene was designed to assess the performance of the MYO-PACE in the situation where it was critical to update the scene constantly.As the subjects replaced the grasped objects, the system needed to remodel the moved objects and cope with a cluttered scene that included multiple and composite objects, such as the peach on the tin can.
The CSI test consisted of 10 trials (per condition; MYO-PACE and LDA).The reduced SHAP test was not repeated (i.e., one trial per condition), as instructed in the SHAP manual, and the tasks were performed according to the protocol in the manual.There was no time limit for performing the trial, but the subjects were instructed to perform it as fast as possible while at the same time avoiding excessive compensatory movements.No instructions were given regarding how to grasp or manipulate the objects.During the CSI test, the subjects only needed to follow the specified order of interactions with the objects (Fig. 6b and  c).The experimenter observed the task execution and the trial were repeated in the following cases: 1) while using MYO-PACE, the subject continuously provided manual commands, preventing the automatic unit from preshaping the prosthesis; 2) the user performed excessive compensatory movements; 3) the subject dropped an object while performing the trial; 4) hardware stopped functioning (e.g., the camera froze; < 5% trials).The trial was deemed finished when all the objects had been relocated to the destination positions and oriented correctly, Fig. 6d.The scene model by the MYO-PACE interface was cleared between each trial (no previous knowledge about the scene configuration was passed on to the next trial), and the prosthesis was reset to its neutral position.

E. Data Analysis and Hypotheses
The main hypothesis was that MYO-PACE system substantially increases subjects' performance in accomplishing the tasks compared to LDA.This hypothesis was tested by comparing the time required to complete the experimental tasks (primary outcome measure in SHAP and CSI test) between the two conditions (MYO-PACE vs. LDA).
To test the further hypothesis that the MYO-PACE system decreases the overall use of myocontrol, the time during which the user generated myoelectric inputs to control the prosthesis was also logged (myocontrol time-secondary outcome measure).The total time was therefore measured from the start of the trial until its completion, whereas the myocontrol time consisted of all those intervals in which the user generated myoelectric control inputs in any phase of the trial.In addition, the myocontrol time was divided into hand (i.e., aperture) and wrist control (flexion and rotation).
The two outcome measures were further examined separately between grasp preparation (preparation time) and object manipulation (manipulation time).The preparation time accounted for the time before the grasp, which corresponded to the total time the user needed to preshape and position the prosthesis and grasp the object.The manipulation time was the time interval between the object grasp and the subsequent object release.Because the autonomous system supports the preshaping of the prosthesis before the grasp, we expected a decrease in grasp preparation time with an unchanged manipulation time when using the proposed system.
For the CSI test, the averaged values of each outcome measure have been compared between the interfaces (MYO-PACE vs. LDA).For the SHAP test, task durations passed the Kolmogorov-Smirnov normality test and have been therefore compared using a paired t-test.The rest of the variables did not pass the Kolmogorov-Smirnov normality test.Therefore, the nonparametric Wilcoxon signed-ranks test was used to evaluate statistically significant differences.The results of the CSI test are reported as median (interquartile range); those of the SHAPs tests are reported as cumulative duration of the tasks.The significance level was set at p < 0.05.

A. Experimental Task 1-Cluttered Scene Interaction Test
In the CSI test, 160 trials (2 control methods × 10 trials × 8 subjects) were performed by able-bodied and 32 (2 control methods × 8 trials × 2 subjects) by amputee subjects.To minimize the influence of training on the performance outcomes, only the last six trials were used for analysis.All subjects successfully performed the tasks in both MYO-PACE and LDA conditions.The indications above the asterisks stand respectively for a significant difference (p < 0.05) in the total amount of time (Tot), the time spent controlling the prosthesis manually (hand and wrist; Myo), and time spent controlling the wrist manually (Wrist) for each phase of the task.
1) Able-Bodied Subjects: Fig. 7 depicts the summary results for the performance.The total trial time was significantly longer for LDA [63.3(23.4)s] than for MYO-PACE [50.5 (7.2) s], indicating a better overall task performance with MYO-PACE.The decrease in overall time was mostly due to the improved preparation phase, which decreased significantly from 38 (16.8) s for LDA to 32.9 (4.2) s for the MYO-PACE system.
The subjects generated substantially more myoelectric activity with the LDA interface compared to the MYO-PACE interface [34.6 (18.3) s vs. 20.6 (6.7) s].Specifically, a significant difference was observed during the preparation phase, where subjects reduced myocontrol time from 17.9 (13.2) s with LDA to 9.1 (2.9) s with MYO-PACE.The time spent for the wrist control was significantly smaller for the MYO-PACE during both grasp preparation time and total trial time.
2) Amputee Subjects: The first amputee was significantly faster with MYO-PACE [from 66.8 (9.1) s to 57.0 (6.7) s], while no significant difference has been observed for the total trial time for the second amputee (Fig. 8).
Both amputees significantly reduced the time spent generating myoelectric activity to control the prosthesis.Specifically, they reduced the time to control the wrist during the preparation phase and the total duration of the trial.Interestingly, when using the MYO-PACE system, the second amputee significantly decreased the overall manipulation time, as well as the manipulation myocontrol time, including the wrist manipulation control time.

B. Experimental Task 2-SHAP Tests
The SHAP test was executed for a total of four repetitions (two control methods × two subjects).Both subjects successfully performed the tasks in both MYO-PACE and LDA conditions.The cumulative duration of the test and the total time spent using myocontrol are divided in preparation and manipulation phase.The indications above the asterisks stand respectively for a significant difference (p < 0.05) in the total amount of time (Tot), the time spent controlling the prosthesis manually (hand and wrist; Myo), and time spent controlling the wrist manually (Wrist) for each phase of the task.Fig. 9 shows the results of the SHAP tests.The total trial duration, the time spent using myocontrol to control the prosthesis as well as the time spent in controlling the wrist were significantly reduced when using MYO-PACE.No difference was observed in the manipulation phase between the control systems, whereas in the preparation phase, the total myoelectric control time, as well as the wrist manipulation time, were all significantly reduced when using the MYO-PACE.

VI. DISCUSSION
We presented a novel semiautonomous system based on sensor fusion (MYO-PACE) for the control of an advanced prosthesis (i.e., Michelangelo hand with wrist rotation and flexion/extension unit).By design, the automatic control is only active during the preparation phase of the grasp, whereas the control during the manipulation phase is based on the user volitional myoelectric input (LDA).Therefore, the system configures the prosthesis in anticipation of the grasp on behalf of the user, relieving him/her of the tedious manual adjustments that are characteristic of this phase.
The MYO-PACE interface and the state-of-the-art LDA myoelectric control interface have been compared by asking the subjects to interact with several simple objects in a customdesigned cluttered scene (CSI) as well as a variety of dailylife objects in the clinically validated SHAP test.The overall setup, as well as the different manipulation scenarios, aimed at assessing the usability of the system in realistic and clinical conditions.
Semiautonomous control has been used in prosthetics before but in a very limited context.The scene analysis was based mostly on simple computer vision algorithms, where the subject needed to target the object (e.g., marking it with a laser [38], aiming with the prosthesis [26], or centering it with the camera [21], [39], [40]).Moreover, in contrast to our system, which continuously updates and fuses new information with every acquired frame (SLAM), the previous solutions could assess only a simple scene by acquiring and processing a single snapshot (picture).In addition, since these systems were unable to perceive the user's location in the environment, they could not estimate his/her intention to grasp the object and they thus needed to be triggered by an external signal [28], [38]- [40].Conversely, the system described here utilizes sensor information from several sources (including depth sensing, inertial units, marker tracking, and sensors embedded in the prosthesis) to assess multiple objects, the user, and his/her surroundings, continuously and dynamically.The scene is analyzed by building a model in the background, which is populated and refreshed online as the subject looks freely around the scene.
The LCCP algorithm has been selected since it can segment the objects into parts.This can be applied to detect objects that are stacked together or to break-down larger objects (or objects with more complicated geometry) into graspable segments.This feature has been exploited in the present experiment several times.For example, in Fig. 5, the peach is modeled as a separate object from the can on top of which it sits, and in a similar fashion, the system is able to differentiate between the water container and its handle in the SHAP test (not shown in the picture).None of the previous solutions [21], [28], [38] is capable of this functionality.
By integrating proprioception, exteroception, and a real-time computation, MYO-PACE can configure the prosthesis according to the user intention in a cluttered scene without any explicit triggering by the user.The real-time tracking of the prosthesis and of the object allows computing the relative orientation of the prosthesis and therefore actuating multiple DoFs of the wrist and hand.As human movements are performed by simultaneous, synergistic motions of multiple joints, this simultaneous preshaping of the multiple DoFs of the prosthetic device generates more naturally appearing motion compared to conventional control (LDA).Furthermore, by controlling multiple actuators simultaneously instead of sequentially, a significant reduction in the duration of the preparation phase is achieved.Consequently, MYO-PACE reaches a level of interaction dexterity that substantially surpasses the state-of-the-art systems in upper limb prosthetics.

A. Semiautonomous Control Improves Performance
To validate the hypothesis that performance improves when acting on a cluttered scene, the system was tested using the CSI test, which included a sequence of tasks that constantly varied the pose of the prosthesis and therefore left very little room for mental preparation between manipulations.The results showed a significant reduction of the total time needed to perform the test, confirming that the novel man-machine interface significantly outperformed the state-of-the-art myocontrol for able-bodied subjects.One of the amputee participants also performed significantly better with MYO-PACE, which is consistent with our hypothesis, whereas the second only showed a trend in improving his performance.Importantly, the results obtained in this experimental context also demonstrate that the LCCP algorithm enabled the MYO-PACE to efficiently control the prosthesis in a cluttered environment (CSI test) or even in those cases when the objects were geometrically complex (SHAP test), which would not have been possible with other systems [28].
We also hypothesized an improvement in performance during a single manipulation of a complex object in a clinically relevant setting, which has been investigated using a SHAP test.The SHAP test consists of daily-life objects that require a series of complex manipulations.Interestingly, most of the manipulations can be successfully completed without extensive wrist usage in the grasp preparation phase, and therefore, an initial assumption was that the support provided by the MYO-PACE system could have only a limited impact.Nonetheless, the use of MYO-PACE significantly improved the overall performance of both amputee subjects also in these conditions.
We separated our analysis for the different phases of each task.The time for the preparation phase decreased significantly both in the CSI and in the SHAP tests, thus confirming our second hypothesis that the most significant effect of MYO-PACE would involve grasp preparation.The only exception to this observation was the decrease in object manipulation time, but not preparation time, in the CSI test by Amputee 2. An explanation for this counterintuitive finding is that the subjects could freely decide the strategy to accomplish each task.Using LDA control, they could perform a fast but nonoptimal grasp using fewer DoFs than with the proposed system.However, a suboptimal grasp could have slowed down the subsequent manipulation phase.
The usage of the MYO-PACE interface also reduced the time spent on myoelectric control for both able-bodied and amputee subjects in both tasks, which confirms our third hypothesis.Prolonged muscle contractions could lead to the development of muscle fatigue and worsen pattern classification performance [41].Therefore, by reducing the total muscle contraction time, the MYO-PACE shows the potential of limiting the impact of muscle fatigue on myoelectric control and increasing the overall control efficiency.
In conclusion, it has been demonstrated that the semiautonomous control system MYO-PACE improves the performance of prosthesis control in comparison with the state of the art (LDA) in both complicated sequences of object relocalization for able-bodied participants and single standard manipulation of complex objects in a clinically established protocol for the two amputees.The increase in performance is directly associated with the decrease in the duration of the preparation phase of the movement.Finally, as anticipated, the usage of the MYO-PACE system leads to a significant reduction of muscle usage.
The experimental assessment in the present study focused on directly evaluating functional performance instead of assessing the accuracy of individual system components (e.g., correct object recognition and/or grasp identification).This was a deliberate choice since functional scores are more relevant for clinical applications, as discussed in [42].

B. Semiautonomous Prosthesis Overcomes Some Limitations of Myocontrol
Pattern classification and regression represent the state of the art in prosthesis control [43], and commercial systems have appeared on the market [44], [45].However, their applications for amputees are still limited by several factors [4], [46], [47].After an amputation, only some of the muscles remain in the residual limb.This can reduce the number of discriminable muscle patterns that can be produced by an amputee, which directly limits the number of prosthesis DoFs that he/she can control.And indeed, this limitation has been encountered in the present study, since the two amputee subjects were not able to produce six different muscle patterns even after extensive training.Therefore, unlike able-bodied subjects, the amputees were not able to directly address each prosthesis DoF: The number of pattern recognition classes had to be reduced to four, and a myoelectric switch was introduced to change between hand and wrist function.An additional limitation of pattern recognition implemented using surface electrodes is that the muscle patterns associated with a specific movement can change due to several factors, such as repositioning of the limb, sweating, shift in electrode locations, and muscle fatigue, ultimately leading to misclassifications and hence unwanted movements of the prosthesis [4], [46], [47].The novel semiautonomous system (MYO-PACE) overcomes these limitations by operating all the DoFs of the prosthesis without the use of muscle activations in the preparation phase of the grasp (unless the user explicitly indicates that he/she wishes to take over the prosthesis control).And indeed, the performance during the CSI test was less variable across subjects with MYO-PACE (i.e., interquartile range for the total duration time of 7.2 s for MYO-PACE vs. 23.5 s for LDA).The difference in performance between the two amputees in the LDA condition is also reduced when using MYO-PACE because the system reduces the level of myocontrol skill required to handle the prosthesis complexity.

C. Limitation and Future Development
Although the MYO-PACE significantly outperforms the clinical state of the art myoelectric control, it is also substantially more complex in terms of required computational power and additional hardware.Moreover, some of the system components, such as the augmented reality glasses, are cumbersome and not particularly comfortable to wear.These limitations could be solved in the future iterations of the system.This technology is fast-developing and miniaturized components (e.g., depth cameras [48], augmented reality glasses [49]) with improved characteristics are being constantly developed.The computational resources that the computer vision requires can also be reduced by modifying the software to analyze only the parts of the scene of interest (e.g., around the prosthesis), or by delegating these to cloud computing as proposed in [50] and [51].The technical limitations of the SR300 camera led to occasional modeling errors for the small objects (thinner than 2 cm).Importantly, the hardware component with the biggest influence on ergonomics is the AR interface, which is not essential for the system operation.The AR feedback assists the user in recognizing the system errors, but the same can be achieved by looking at the prosthesis (e.g., noticing that the hand preshaped in a wrong grasp) once the system working principles are clear to the user.Therefore, this component could be removed if the vision sensors would be embedded in the prosthesis, as done in [52] and if the overall system robustness would not require the user to monitor the system operation.
In the present study, the MYO-PACE system controlled two DoFs of the wrist and a hand aperture in a two grasp types (palmar and lateral).Nevertheless, the approach could be easily extended to include automatic selection among several grasp types in a dexterous prosthesis based on the object properties estimated by the artificial exteroception module.Finally, it would be interesting to explore the possibility of combining the MYO-PACE system with an underactuated prosthesis [53]- [55], [56], where the control complexity during grasping is already reduced intrinsically by the mechanical design.

VII. CONCLUSION
In conclusion, the proposed control approach enables a smooth and natural prosthesis motion, as well as a level of dexterity that substantially surpasses the state-of-the-art pattern recognition myocontrol.

APPENDIX
The dataset(s) supporting the conclusions of this article is (are) available upon request.

Fig. 3 .
Fig. 3. Projections of the object selection volumes on the objects plane.The objects are selected depending on the distance between the prosthesis (P) and the objects.Additionally, the selection can only happen inside the selection volumes of the objects.The projections on the table of such selection volumes are shown in the figure (blue for the unselected objects, and yellow for the selected one).

Fig. 5 .
Fig.5.Example interaction with an object using MYO-PACE showing the view from the user perspective (AR glasses).(a) Real objects in the scene are modeled and blue holograms are superimposed over the objects.The top-left blue icon indicates that the system actively updates the scene.The time encompassed from now until the object grasp counts as the time to prepare for grasp (a-d; preparation time).(b) Based on the scene model and prosthesis motion, the system infers user intention to grasp the central object from the front-left side.The object assumes a yellow-green texture, the prosthesis is automatically preshaped, and the yellow icon is displayed.(c) User changes his mind and moves the prosthesis toward another object.The system responds by indicating the new selection and reshaping the prosthesis to grasp the object from above.(d) However, the user moves the prosthesis toward the table and the system detects the intention to grasp the object from the side.The object texture is updated, and the prosthesis automatically changes the wrist orientation.(e) User approaches the object and generates muscle activity for closing the prosthetic hand.The hologram of the target object becomes green and the icon for myoelectric inputs is visualized (manual control mode).The time spent by the user to prepare the prosthesis for grasp and to close the hand counts as the myoelectric control time.(f) User has grasped the object and transports it without generating myoelectric signals.The system is in the manipulation phase even if the user does not generate myoelectric commands.(g) User manipulates the object (changes its orientation) by generating myoelectric commands to control the wrist degrees of freedom.The time spent performing this action counts both the time spent to generate myoelectric control signals and the time to control the wrist.(h) User releases the object in the new position where it will be automatically remodeled by the system as soon as the user stops generating myoelectric commands.

Fig. 6 .
Fig. 6.Experimental task overview.The figure illustrates the experimental setup and task sequence (annotations 1-6).(a) Initial position of the user, prosthesis, and objects at the start of the trial.These positions were the same for every trial.The destination of each object was marked by a label.The elastic compensated for the weight of the prosthesis, reducing the fatigue.(b) Repositioning of the two CD boxes (1, 2) and of the objects-set (3) {tin can, peach}.(c) Rotation of the first tee box (4), translation of the peach from the top of the tin can to the top of the tee box (5), and transfer of the second tee box (6).(d) Final position of the objects at the end of the trial.

Fig. 7 .
Fig. 7. Trial time decomposition for the cluttered scene interaction (CSI) test performed by able-bodied subjects.The trial execution time and the total time spent using myocontrol are decomposed in preparation and manipulation phases.The indications above the asterisks stand respectively for a significant difference (p < 0.05) in the total amount of time (Tot), the time spent controlling the prosthesis manually (hand and wrist; Myo), and time spent controlling the wrist manually (Wrist) for each phase of the task.

Fig. 8 .
Fig. 8. Trial time decomposition for the cluttered scene interaction (CSI) test performed by amputee subjects.The trial execution time and the total time spent using myocontrol are decomposed in preparation and manipulation phases.The indications above the asterisks stand respectively for a significant difference (p < 0.05) in the total amount of time (Tot), the time spent controlling the prosthesis manually (hand and wrist; Myo), and time spent controlling the wrist manually (Wrist) for each phase of the task.

Fig. 9 .
Fig. 9. Performance of the two amputee subjects during the SHAP tests.The cumulative duration of the test and the total time spent using myocontrol are divided in preparation and manipulation phase.The indications above the asterisks stand respectively for a significant difference (p < 0.05) in the total amount of time (Tot), the time spent controlling the prosthesis manually (hand and wrist; Myo), and time spent controlling the wrist manually (Wrist) for each phase of the task.