Simultaneous sEMG Recognition of Gestures and Force Levels for Interaction With Prosthetic Hand

The natural interaction between the prosthetic hand and the upper limb amputation patient is important and directly affects the rehabilitation effect and operation ability. Most previous studies only focused on the interaction of gestures but ignored the force levels. This paper proposes a simultaneous recognition method of gestures and forces for interaction with a prosthetic hand. The multitask classification algorithm based on a convolutional neural network (CNN) is designed to improve recognition efficiency and ensure recognition accuracy. The offline experimental results show that the algorithm proposed in this study outperforms other methods in both training speed and accuracy. To prove the effectiveness of the proposed method, a myoelectric prosthetic hand integrated with tactile sensors is developed, and surface electromyography (sEMG) datasets of healthy persons and amputees are built. The online experimental results show that the amputee can control the prosthetic hand to continuously make gestures under different force levels, and the effect of hand coordination on the hand perception of amputees is explored. The results show that gesture classification operation tasks with different force levels based on sEMG signals can be accurately recognized and comfortably interact with prosthetic hands in real time. It improves the amputees’ operation ability and relieves their muscle fatigue.

Therefore, to regain their ability to grasp, it is necessary to help them wear prosthetic hands. Upper limb amputees will receive rehabilitation training and fit prosthetic hands after surgery. However, due to the high failure rate, inflexible operation, and excessive weight of the current affordable commercial prosthetic hands for ordinary families, another limitation of the current commercial surface electromyography(sEMG) hand is that the prosthetic hand will return to a neutral position from the previous gesture, usually a completely open position [1], which is not able to continuously and smoothly control the prosthetic hands. Upper limb amputees cannot achieve friendly human-computer interaction in the process of using prosthetic hands. Therefore, most upper limb amputees choose to use cosmetic prosthetic hands (only the appearance is similar to normal hands for viewing purposes). They prefer to use a healthy hand instead of a prosthetic hand to complete daily operations [2], [3]. Using a prosthetic hand to complete some daily operations could improve the physical and psychological aspects of the amputee person's quality of life and could avoid atrophy of upper extremity muscles due to prolonged inactivity. Therefore, it is a meaningful topic to allow amputees to comfortably use a prosthetic hand to maintain normal daily life and continuous rehabilitation of residual limbs. At present, there are two main ways to control the prosthetic hand: noninvasive surface electromyography (sEMG) and targeted sensory reinnervation (TSR) and targeted-muscle reinnervation (TMR) surgically invasive sensors [4], [5]. Invasive sensors obtain a less noisy and higher quality signal [6], but sEMG is more convenient and does not cause harm to upper limb amputees. Many amputees are unwilling to undergo reoperation or use invasive sensors, and invasive sensors need to be replaced after a period of time due to electrode wear or displacement. This paper develops a prosthetic hand control system that assists amputees who are not willing to use invasive sensors.
A major application scenario for sEMG signals is decoding gestures [7], [8]. Reference [9] collected the sEMG data of twelve subjects with five gestures, and the average recognition accuracy reached 98.7% by the neural network. [10] proposed the sEMG compression scheme to classify gestures based on compressed covariance sensing. The proposed method was verified by NinaPro open-source data that contained forty-nine gestures and achieved good results. Reference [11] studied the influence of convolutional neural network (CNN) algorithm parameters on gesture recognition. By studying the effect of hyperparameters on each gesture of eighteen subjects, This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ they found that no matter what the network configuration is, some actions such as handshake or hand extension are better recognized. Reference [12] proposed a new method for automatically identifying the most important sEMG channels for classifying gestures. The proposed method generates discriminative codes based on the regularization of time series sEMG data. Compared with the case of using all channels, the identified significant sEMG channels increase the classification accuracy by 11%. Reference [13] collected sEMG data of fourteen gestures and found that misjudgments of gestures often occurred in daily life. And there is a wide range of scenarios where gestures are used in daily applications. Moreover, gestures are used in a wide range of scenarios in daily applications. Reference [14] maps the classification recognition of gestures to audio playback control and controls the control operation of the audio armband through different gestures. Reference [15] proposes a new multisensor guided gesture recognition system for the remote operation of surgical robots, which uses long short-term memory (LSTM) networks to recognize gestures and uses the detected gestures to perform a set of human-robot collaboration tasks on a surgical robot platform, which shows that gesture recognition has great potential in daily applications as well as human-robot interaction scenarios.
Various of the research on the interactive control of prosthetic hands focuses on the recognition of gestures and online classification control [16], [17]. Gesture control alone is not enough to satisfy the daily life of amputees, and force control is also very important. For example, when they want to grasp fragile objects, satisfactory force control is needed. It can help them complete daily operations better and avoid high-intensity activities of the residual limb muscles all the time. The grasping types and forces of healthy subjects have been estimated by sEMG signals [18]. The absolute error of the grasping force estimated through 16 sEMG channels is 2.52%, but currently, only offline prediction is achieved, and no online test is performed. Four surface sEMG sensors are arranged in the forearm muscles to predict the force of the wrist with two degrees of freedom [19]. Reference [1] used a deep recurrent neural network (RNN) network to train nine subjects, and six gesture classifications of three different force levels were obtained, but force levels and gestures were not recognized at the same time. Furthermore, the deep long short-term memory (LSTM) network training model used is relatively large, and it is difficult to realize the online recognition and control of the prosthetic hand. Reference [20] used sEMG signals to estimate the force of the residual limb from 200 g to 1000 g and carried out an online evaluation. Reference [21] designed seven gestures for 31 healthy subjects and adopted a hierarchical model. Seven gestures are recognized first, and then two specific gestures are detected; only two force levels of the two gestures are recognized.
In summary, the online recognition of simple gestures or force levels is realized and the online synchronization of the two needs further research. Therefore, we will design a real time operating system for different force levels and different gestures to control prosthetic hands to help amputees complete their daily activities. The gesture and force level control are combined and applied to the prosthetic hands with tactile sensors. Compared with previous studies, force level control will enable people to complete daily operations more flexibly and have better human-computer interaction. Feel more relaxed operations that require lower force levels and relieve muscle fatigue. This research may improve the usage rate of prosthetic hands.
The main contributions of this paper can be summarized in the following points: (1) A novel method of simultaneous sEMG recognition of gestures and force levels is proposed. The accuracy of gesture/force level recognition for healthy subjects was 94.75%/86.5%, and that for amputee subjects was 78.3%/76.3%.
(2) The sEMG dataset of three gestures with three force levels is established. This dataset includes four healthy subjects and three amputee subjects.
(3) A series of grasping experiments of the prosthetic hand with tactile sensors are implemented.
The paper is organized as follows. Section II introduces the system design for the interaction between human and prosthetic hands. Section III introduces the recognition algorithms. Section IV describes the EMG dataset, and the experimental results are presented. This paper ends with the conclusion in Section V.

II. SYSTEM DESIGN
The proposed method of simultaneous sEMG recognition of gestures and force levels is shown in Fig.1. First, data are collected using sEMG sensors and force sensors and then fed into a neural network for offline training. The sEMG data are then read in real time and fed into the trained model for online control and prosthetic hand grasping. Next, the hardware and the detailed experimental process are introduced.

A. Hardware Equipment
An sEMG acquisition system, a force device, and a prosthetic hand are used in the experiment.
The electromyographic activity of each subject was recorded with the British Biometrics' sEMG wireless acquisition kit. It consists of a wireless receiver and 6 wireless sEMG sensors, which are shown in Fig.1 and Fig.3. The sEMG sensor samples data at 2000 Hz, the precision is 0.0001 mV, and the range is −2 mV to 2 mV.
A force device is used to evaluate force levels, as shown in Fig.1 and Fig.8. It consists of the force sensor and bracket. The sensor is fixed on the bracket. Its precision is 0.1 N, and the range is 0-200 N. While measuring the sEMG signal, we simultaneously use a force sensor. For a healthy person, although they have both healthy hands, the force sensor is necessary to remain in one stable fore level. For example, while they are wrapping, they can make the value of the force sensor approximately 5 N to stay at a low force level. After several actions, they can keep one stable fore level without a force sensor, which is also a training process. For amputees, their residual limb's feelings are weaker, so the force sensor is more important.
Furthermore, the prosthetic hand uses the blue tooth for communication. The developed prosthetic hand has 6 degrees of freedom, and each fingertip is equipped with a tactile sensor array that can help the prosthetic hand implement three force levels of grasping force. Therefore, the subjects can use different force levels, and their muscle will be less fatigued compared to maintaining the same force level. The tactile sensor adopts an array structure, which can collect the contact force information of the fingertip more comprehensively. The prosthetic hand can achieve a variety of grasping actions in daily life, and the execution time of each action is less than 2 s. Meanwhile, flexible silicone material is equipped to the surface of the fingertip to increase friction and ensure stability. The performance of the prosthetic hand is adequate for our grasping experiments.
The prosthetic hand has a total of 5 fingertip tactile sensors. The size of each tactile sensor is 15 mm × 15 mm × 0.2 mm. Each tactile sensor includes 25 contacts, and the average value of 25 contacts is taken to calculate the pressure with an accuracy of 0.01 N, an acquisition frequency of 50 Hz, a response speed of 2 ms, and a measurement range of 0-15 N.

B. Offline Recognition
Offline recognition is implemented, and 6 sEMG sensors are first used to evaluate the influence of the number of channels on the accuracy. On the advice of the doctor, we chose 6 locations to place the sensors, corresponding to 3 different muscles. As shown in Fig.3 numer1,4 represent the flexor digitorum superficialis (FDS), number2,5 represent the flexor carpi radialis (FCS), number3,6 represent the extensor carpi radialis (ECA). The 3 selected muscles are the anterior group of the forearm muscles, which are mainly responsible for flexing the fingers and palms. We divide the six sensors into 2 groups (gruop1: 123 and group2: 456). After our experiments (details are in section IV), three sensors (one group) are selected that fully ensure accuracy, wear comfort and can measure the muscle.
Then, the active segments of the collected sEMG signal are extracted, and the threshold is set. The raw sEMG data collected in the experiment, including the active segment (motion state) and the resting segment (rest state), are shown in Fig.2(a). This study uses a sliding window with a length of 6000 (3 s) and a step size of 128 to extract the active segment of the raw data. First, the data are observed, and the voltage threshold is set. For example, we set the threshold to 0.005 mV for this set of data and then use the sliding window to process the data. If the average voltage value of the data in the sliding window is more than the threshold, it is regarded as the active segment; otherwise, it is regarded as the resting segment. The sliding window processing process is shown in Fig.2(a). The processed data are shown in Fig.2(b). Then, the preprocessed data are fed into the neural network for training. Meanwhile, the gestures and force levels are used as labels, but only sEMG data are used for training data, and the force levels and gestures are identified from the sEMG data. The force levels of the subjects are trained by pressing the force sensor. Three levels, including low, medium, and high, are selected.

C. Online Recognition
In the online recognition, the real-time sEMG signals are captured, and the model trained in the offline step is used for Fig. 2. Sliding window. (a) Describes the process of performing a sliding window on the raw sEMG signal, observing the data and setting a threshold, then using the sliding window to calculate the mean value of the data within the window. If the mean value is greater than the threshold, it is classified as a motion segment, and vice versa as a resting segment. The length of each movement of the sliding window is the step size, and finally, the raw data are divided into motion and resting segments. (b) The data after extracting the active segment and completing the splicing. simultaneous recognition of gestures and forces levels. After the trained model is discriminated, the recognized gesture and force level are output, and the score value t of the output category is output. If t is more than the set threshold, the recognized instruction will be transmitted to the prosthetic hand and control the prosthetic hand to make the corresponding action; if t is less than the threshold, keep the resting state. We complete different gestures by changing the rotation and bending angle of the prosthetic hand and determining the force level by the pressure value of the fingertip, such as a low force level. The fingertip pressure threshold is set to 5N. When the pressure exceeds this threshold, the motor locks, and the prosthetic hand no longer continues to move.

A. Network Structure
The deep learning algorithm has the advantages of high training speed, without manually extracting features. Currently, some researchers have successfully applied the deep learning method to sEMG [22], [23], [24] or electroencephalogram (EEG) [25] signal classification and explored several effective network frameworks. Reference [26] and [27] transmitted raw sEMG signals as an input vector to a CNN classifier and achieved good results. sEMG signals look like time series, but in fact, they only change in a short time from the resting state to the gesture. The continuous state after the gesture is completed is no longer a time series of changes. Since the training speed of LSTM is slower than that of CNN, and it is more effective in time series problems, our study chooses to use CNN.
We choose to convert the two-dimensional sEMG signal (sequence × number of channels) into a three-dimensional image (1 × sequence × number of channels) and use CNN to train. Meanwhile, this paper adopts the CNN structure as the main body, and the network structure is shown in Fig.4. Gestures and force levels can be classified at the same time by using our CNN framework called G&F-CNN (Gestures and Force levels CNN). In G&F-CNN, we used the batch normalization (BN) layer to speed up the convergence speed and place the problem of gradient explosion and used the dropout layer to prevent the problem of overfitting.
The computer hardware used is Intel Core i9-11900H, 32G memory, 16G video memory, an eight-core processor, and an NVIDIA RTX3080 graphics card. The training framework is based on the Keras of TensorFlow. The maximum number of iterations and batch size are 3000 epochs and 256, respectively. All dropout parameters are set to 0.3. During the training process, the overall data are divided into three parts: 80% is the training set, 10% is the validation set, and 10% is the test set. In the iterative process, when the accuracy of the new model is greater than that of the previously trained model, it will be saved, and we choose the model with the best performance after the iteration. The learning rate is 0.0001. Because the BN layer and the dropout layer are used, there is no need to lower the learning rate, there is no overfitting, and the convergence speed is very fast. The optimizer chooses the Adam optimizer, the parameters of the optimizer are the default parameters, the loss function uses categorical-cross entropy and the loss function chooses categorical-cross entropy. In the neural network, we use 6 convolutional blocks as shared layers. Each block contains a 2D convolution layer, a BN layer, a ReLu activation function layer, a dropout layer (dropout=0.3), and the convolutional layer detailed parameters (kernel size, step size, etc.). In the network structure diagram, the BN layer uses default parameters; after the shared layer, it is the task block. We have two task blocks: one is the gesture task block, and the other is the force levels task block. Each task block contains a 2D convolutional layer, a flatten layer, a fully connected layer, a softmax activation function layer, and finally output k values (k is the number of categories), and the largest number of k values corresponding to the category is the output result. Reference [14] indicates that various nontarget gestures will be performed in daily life activities, resulting in many false positives and activations. Therefore, this article adds threshold activation to the softmax classification and sets the threshold to 0.85. If the classification is more than this threshold, a classification gesture will be output. When the threshold is less than this value, the prosthetic hand will remain at rest.

B. Multitask Learning Method
Reference [21] used multiple classifiers to classify gestures and force levels. In the output of the gesture classifier, if it is a specific gesture, other trained force level classifiers are called to classify the force levels. We used a multitask learning structure to replace multiclassifiers for training. Six channels of sEMG signals are used as input, force level features and gesture features are extracted from EMG signals to achieve recognition, and gesture categories and force levels are output at the same time.
Multitask learning is applied in various fields, such as medicine [28], robot operation [29], and vision [30]. Compared with multiclassifiers, multitask learning and training are faster, only the need to train one model is more convenient, and the training effect of multitask learning is better than that of multiclassifiers. Multitask learning also has the effect of reducing overfitting. In this study, the first 6 convolutional layers are shared, and each task is classified by adding a convolutional layer, flatten layer, and Softmax after the number of shared layers.

IV. EXPERIMENTAL RESULTS
Gesture and force level experiments of healthy persons and amputees are designed. The dataset is built to evaluate the performance of the proposed method. Furthermore, real time interaction experiments for prosthetic hand grasping are implemented to prove the effectiveness.

A. Dateset
We invited 4 healthy subjects and 3 disabled subjects to participate in the experiments. Four healthy subjects are males approximately 25 years old, and all right hands are used in the experiment and are called sub1-sub4. Three disabled subjects are called sub 5-7, sub5 with the right residual limb amputated for 25 years, no phantom limbs and no tactile feedback; sub6 with the right residual limb has amputated for 3 years, have phantom limbs and no tactile feedback; sub7 with the left residual limb has amputated for 3 years, have phantom limbs and have tactile feedback. They are required to make 3 gestures under 3 force levels. Three gestures are wrapped, 2 finger pinch (2FP), and 3 finger pinch (3FP), as shown in Fig.7. The three force levels are low, medium, and high.
During data collection, we used a computer screen to prompt the subjects to carry out the instructions. Six wireless sEMG sensors are used for healthy subjects, and three wireless sEMG sensors are used for amputees. Meanwhile, the force sensor is used as a supervision to ensure the right force level of the subjects.
Every subject's experiment included 9 motions (3 gestures*3 force levels), and every motion included 20 trials. At the beginning of the experiment, the screen displays the tips of the required force level and gesture pictures. Then  subjects can select different buttons to determine the status of "start execution", "rest", etc. Then each trial lasted for 5s and rests for 5s. Each motion ends for 2 minutes to rest, and every 3 motions end for 5 minutes to rest to ensure that the muscles are not in a state of fatigue. The demonstration of data collection is shown in Fig.8 and the experimental design of our dataset is shown in Fig.6.
For healthy subjects, we collected data from 6 sEMG sensors, and the sEMG data came from the hands of the individual gesturing during the experiment; for amputees, we collected data from 3 sEMG sensors, and the sEMG data came from For amputees, they can only make gestures under different force levels through imagination, and the characteristic signal obtained only by imagination is not very strong, so two experiments are designed for studying the influence of a healthy hand on the residual limb. The first set of experiments is to make only the residual limb hand do actions are called the control experimental group. The second set of experiments is to let him use the normal hand and the residual limb of the disabled to perform the same action at the same time and is called a collaborative experimental group. We want to improve the perception of residual limbs through healthy hands. The demonstration of data collection is shown in Fig.9.

B. Recognition Performance
We compared G&F-CNN with LSTM, support vector machines(SVM) and CNN+LSTM [31]. The learning rate was set to 0.0001. G&F-CNN, LSTM and CNN+LSTM were trained for 3000 epochs, and SVM stopped when SVM was trained to the optimal accuracy. The algorithm evaluation index is accuracy, the definition of accuracy is R/N, N is the total number of training samples, and R is the number of successful classification samples. The classification results of different algorithms on gestures and force levels are shown in Fig.5(a) and Fig.5(b). All the results (comparison of algorithms, comparison of different training methods, etc., through 5-fold cross-verification). Each individual dataset was divided into 10 data subsets, and the proportions of the training set, validation set, and test set were 7:1:2. Data subsets 0 and 1 were used for the first time as a test set. The remaining 8 data subsets took 1 random subset as a validation set, the remaining 7 as a training set, and a second used data subsets 2 and 3 as a test set for a total of 5 cross-validations. The recognition algorithms of multitask learning (MTL) and multiclassifier learning (MCL) are implemented for comparison. The convergence time of multitask learning is 2/3 that of multiclassifier learning and achieves higher recognition accuracy. Fig.5(c) shows the gesture recognition results in 6 sEMG sensors and 3 sEMG sensors of sEMG data on healthy subjects. Fig.5(d) shows the accuracy of force levels in two different channel numbers and two different training methods.
The results show that the convergence speed of the 3 sEMG sensors training result is increased from 2000 epochs to 3000 epochs compared with the 6 sEMG sensors training result, and the accuracy rate is slightly reduced, but it still maintains a high level (the difference is within 5%). To improve the comfort of the amputee and the convenience of wearing it, 3 sEMG sensors are used for online experiments.
We used G&F-CNN to classify and recognize gestures and force levels in different healthy individuals, and the detailed results are shown in Table I and Table II.(FP means finger pinch). Fig.5(e) and Fig.5(f) show the results on two experimental groups of amputees using G&F-CNN, 3 sEMG sensors, and a multitask learning method to train the model. In the collaborative experimental group, we found that using the healthy hand and the residual limb at the same time caused the subjects to be distracted, and the longer the amputation time and the weaker the perception ability of the residual limb, the greater the negative effect of distraction. Sub5 has been amputated for a long time, and using both hands together is very uncoordinated and requires a lot of distraction, so the experimental results of the collaborative experimental group are even worse; the positive effects of bimanual synergy canceled out, and the experimental results of the two groups were similar; the residual limbs of Sub7 had stronger perception ability and did not need to be distracted too much, so the experimental results of the collaborative experimental group were better. This discovery opens up new research ideas for the study of amputee hand coordination.

C. Grasping Experiments of Prosthetic Hand
To prove that the subjects using different force levels can make the operation more comfortable, we asked the subjects to perform exercises at different force levels until they were fatigued. Fig.11 shows the sEMG data of the subjects making fist movements at three different force levels. Fig.11a shows the high force level. The subjects began to fatigue in approximately 38 seconds. The sEMG level cannot be maintained; Fig.11b shows the mid force level, and the subject begins to fatigue at approximately 120s and cannot maintain the EMG level; Fig.11c shows the low force level.
For muscle fatigue determination, frequency domain studies are also necessary. According to studies [32], [33], the mean power spectrogram and median frequency are important factors in determining muscle fatigue. Fig.10 and Table III show the mean power spectrogram and median frequency changes before and after fatigue at different force levels. Regarding the selection of the fatigue time point, since it was considered that the low force level did not produce fatigue, the last 5s of data was selected assuming the fatigue state and compared to observe whether there was a difference to determine whether there was real fatigue, and the fatigue time of the medium force level and high force level were assumed to be 120s and 38s, respectively. In Fig.10, the mean power frequencies represented by subplots a and b do not differ significantly, indicating that muscle fatigue did not occur at low force levels, and the mean power frequencies represented by subplots c and d differ significantly, verifying the fact that muscle fatigue occurs at medium force levels. Similarly, subplots e and f verify the fact that fatigue occurs at high force levels. As shown in Table III, before and after the fatigue of medium and high force levels, there was a significant difference in median frequency, which verified the fact of fatigue, and the difference was smaller for low force levels, which did not produce fatigue. Fig.12 shows tactile sensor data of wrapping at different force levels making the results more intuitive. Low force level gestures do not fatigue the subject, causing a decrease in sEMG amplitude. The lower the force level is, the lower the fatigue level of the tested muscles, which makes the manipulation more flexible and comfortable.
The online EMG recognition results are used for interaction with a prosthetic hand. The EMG data input trained model and the classification results of gesture and force are output in real-time to control the prosthetic hand to make corresponding actions.   We asked healthy subjects and amputees to perform 20 motions on the prosthetic hand online using each of the different gestures and force levels. Table IV and Table V show the number of successful motions and online accuracy of healthy subjects and amputees in completing assigned tasks. Fig.13 shows the gestures and force levels interaction between the healthy subject and the prosthetic hand. The upper part of the figure shows the interactive effects of the three gestures. The lower part of the picture shows the effect of  three levels of force through the prosthetic hand pinching a rubber ball. Fig.14 shows the interaction between the amputee and the prosthetic hand. Different objects are grasped by different gestures. A warp gesture is recognized to grasp a water bottle. A two-finger pinch gesture is recognized to grasp ping-pong. A three-finger pinch gesture is recognized to grab a rubber ball. Furthermore, when grasping light objects, he can use a low level of force. When grasping heavy objects, a high level of force can be used. When grasping general objects, he can use medium-level force. Hence, the method of interaction with the prosthetic hand is more flexible. It improves the grasping ability and can effectively alleviate the muscle fatigue of the residual limb.

A. Analysis of the Results
This paper investigates the classification recognition and online control of prosthetic hands based on sEMG signals for simultaneous gestures and force levels. The average gesture/force recognition accuracy of healthy subjects was 94.75%/86.5%; the average recognition accuracy of gesture/force levels of amputee subjects was 78.3%/76.3%. Compared to previous studies, [14] conducted classification recognition of gestures and force levels for 31 healthy subjects with an accuracy of 98.78%, and the accuracy of force level recognition for two specific gestures was 98.80% and 96.09%, but this study did not classify gestures and force levels at the same time, and its recognition method was as follows: if a specific gesture was detected as a predetermined gesture in the process of gesture recognition, a specific trained model was called to If a specific gesture is detected in the process of gesture recognition, the trained model is called to classify the force level; compared with this study, the accuracy of gesture recognition reaches a high level, and the accuracy of force level recognition has room for improvement. The real-time nature of the sEMG control system requires that decisions be given within 300 ms of the end of the action execution [34]. The delay of this study's system is approximately 160 ms, which has reached the standard of real time operation.
The experimental results show that the accuracy rate of sub1 is significantly lower than that of the other three subjects for the following reasons: 1. Individual variability. It is normal for each individual to have differences in accuracy; 2. Distraction. Subject 1 may not pay enough attention during some motions in the experiment; 3. Body size. Subject 1 was fatter than the other subjects, and the detected sEMG signal characteristics were weaker. The experimental results show that the force level classification result is significantly lower than the gesture classification result. The reason may be that in force level data collection, subjects perform actions based on their senses only, which leads to inaccurate force levels done to some motions of subjects in data collection. For example, when conducting fist-mid force level data collection experiments, subjects may make fist-high force level actions, resulting in datasets containing a small number of errors.

B. Future Work
In this study, we propose an operating system based on sEMG signals to simultaneously recognize hand gestures and force levels and control the prosthetic hand in real time, and the outlook of future work is as follows: 1. The prosthetic hand used in this study is a rigid prosthetic hand, and soft prosthetic hands can be used in the future, such as the soft prosthetic hand used in the study [35], which is lighter in weight, more flexible in operation, and has better human-computer interaction in comparison. 2. The hand gestures designed in this study were fixed movements, and continuous decoding of hand movements can be studied in the future. in [36], the reference electrode was placed at the wrist and olecranon for healthy participants and amputees, respectively, for continuous estimation of multiple degrees of freedom wrist torques. This is one of the future research directions of this study. 3. Handedness synergy for disabled people can be used as a rehabilitation tool for long-term experiments and followup of patients; for example, amputees are asked to perform movements with both hands together as daily training and observe whether their residual limb perception is enhanced after a certain period.

VI. CONCLUSION
In this paper, a real-time electromyography prosthetic hand control system is designed that uses the G&F-CNN network to classify sEMG signals for multitasking recognition of forces and gestures. Three gestures with three force levels are recognized online to improve the interaction with a prosthetic hand. Force interaction can achieve finer control and alleviate muscle fatigue. It helps amputees use prosthetic hands more conveniently and comfortably in daily life. A series of experiments are implemented to prove the superior performance of the proposed method. This research has great potential for application and can be applied to daily life operations and human-computer interaction scenarios, such as forming a hybrid brain-computer interface operating system with electrooculogram(EOG) and sEMG [37], controlling wheelchairs through gestures [38], etc. In the end, the average gesture recognition accuracy of healthy people reached 94.75%, and the average recognition accuracy of force was 86.5%; the average recognition accuracy of the gestures of amputees was 78.3%, and the average recognition accuracy of force was 76.3%. The feasibility of the proposed system was verified through grasping experiments.