Quantitative Analysis on the Interaction Fatigue of Natural Gestures

With the popularity of natural user interface (NUI), natural gesture interaction has become the mainstream. Using improper natural gestures for a long time will cause muscle fatigue, which leads to an increase in pathological problems such as tenosynovitis. To avoid the harm caused by improper gestures, this paper selects three daily interactive gestures as the research objects including browsing information, playing games and typing, and divides them into nine independent gestures. After denoising, filtering, segmenting, and extracting the parameters of the acquired surface electromyography (sEMG) signals, the time-domain, frequency-domain and time-frequency-domain are analyzed. The characteristics of the envelope waveform and power spectrum threshold, as well as the fatigue characteristics of nine independent gestures are obtained. The long short-term memory (LSTM), one of the recurrent neural network (RNN) methods is used to train nine independent gesture models. The fatigue characteristics of the integrated gestures are predicted by the trained LSTM series model. The energy consumption characteristics of integrated gestures in smartphones and PCs are obtained. It is found that the simple behavior of browsing in the integrated behavior is suitable for natural interactive gestures, and complex behaviors such as games or typing on PC have lower energy consumption than that of smartphones. Among the independent natural gestures, the energy consumption of clicking is higher than that of dragging. Compared with a behavior for the same purpose, the energy consumption of mouse gestures for PCs is much lower than that of smartphones. This study of gesture fatigue provides a theoretical basis for the design of gestures and the development of internet user products.


I. INTRODUCTION
Steve Mann proposed the concept of ''nature user interface (NUI)''. NUI is a type of user interface designed to make users feel as natural as possible [1]. The direct interaction between the human and natural interfaces is called the natural gesture interaction. The natural gestures can directly control the NUI, which helps users to get a good immersive experience. However, improper natural gestures will cause different forms of fatigue, resulting in damage to the soft tissue of the hand, ''mobile phone hand'', tenosynovitis [2] and so on. The tendon sheath is a double-layer sleeve such as a closed synovium tube, which protects the tendon. Injury The associate editor coordinating the review of this manuscript and approving it for publication was Alberto Botter . and inflammation of the tendon and tendon sheath will be caused if the tendon is excessively rubbed for a long time due to improper gestures. Tenosynovitis will lead to swelling and deformation of fingers and inconvenience of movement [3].
To avoid muscle damage caused by improper gestures, it is necessary to study gesture fatigue. Wigdor et al. [4] proposed that the problems of gesture input in natural interactions are 'the occlusion problem' and 'the fat finger problem'. The interaction mode between the hand and interface was paid more attention, but the fatigue of hand gestures was less noticed. Some scholars tried to determine the method of gesture optimization by studying the natural interfaces. Steven [5] divided the smartphone screen into three areas, including an easy area, an ok area and a reach area to research the interface influence. With the rapid updating of VOLUME 8, 2020 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ smartphones, research on the interactive difficulty in screen hardware has limited guiding effect on the gesture design. Many scholars have used a variety of methods to study the fatigue of gesture. The Swedish psychologist Gunnar Borg first proposed the concept of ''subjective physical feeling'' in 1972 and proposed the ''rating of perceived exertion (RPE)'' test method on this basis. Because the subjective assessment of the body's physical condition comes from its material basis state, the RPE score of the Borg table is closely related to the current physiological and biochemical indicators (such as metabolic rate, heart rate, hormone levels, etc.) [6]. Perceptual engineering as a traditional research method of fatigue degree, is mainly based on subjective evaluation and lacks an accurate objective basis. Zhuang [7] used the Borg subjective evaluation method and signals in traditional signal analysis to evaluate the fatigue of somatosensory gestures. Wang et al. [8] regarded the arm as a multi rigid hinge structure, simplified the upper limb as the mechanical model with three segments and seven degrees of freedom, and established a fatigue evaluation model based on biomechanics. However, the traditional analysis of sEMG or biomechanics is only for simple and independent gesture signals, lacking studies for integrated and comprehensive gesture signals from real situations. A sEMG signal is a kind of alternating current (AC) voltage signal, which can reflect the degree of muscle fatigue. The signal of bioelectricity is generated when the electrical pulse sequence is triggered by the α-motoneurons of the spinal cord and transmitted to the superficial muscle fiber along the axon. The original sEMG signal appears under the guidance of the surface electrode. One-dimensional voltage time series signals can reflect the neuromuscular activity to a certain extent [9]. The maximum frequency of sEMG signal power spectrums is generally between 30 ∼ 300Hz [10]. There is a good linear relationship between the amplitude of the signal and the force produced by the muscle [11], which makes sEMG signals one of the important tools to evaluate ergonomic design. Chen et al. [12], Marina et al. [13], and Zhang et al. [14] used sEMG signals as the main medium to research the fatigue issues. The power spectrum of the signal is the square of the signal spectrum. According to the Parseval theorem, the energy of the real signal is equal to the average power spectrum to calculate the energy consumption. On the basis of the power spectrum of gesture sEMG signals, the energy consumption is calculated as the basis of fatigue. Aiming at the fatigue problem of integrated gestures based on smartphones and PCs, this paper extracts the sEMGbased features of time domain, frequency domain and timefrequency domain to analyze the fatigue characteristics of independent gestures, and establishes prediction models to obtain the differences of fatigue characteristics of integrated natural gestures.

A. SELECTION OF SEMG SIGNALS
The contraction of each muscle in the hand is controlled by a tendon sheath connecting to a specific position of the hand. The extension of five fingers mainly includes the extension of the thumb and the other four fingers. The active muscles of four fingers flexion are the flexor digitorum superficialis and the flexor digitorum profundus, which are located in the lower layer of the flexor carpi. The active muscles of thumb flexion mainly include the adductor pollicis, opposites pollicis, flexor pollicis brevis and flexor pollicis brevis. Muscle of thenar refers to four muscles in the lateral group of hand muscles: the abductor pollicis brevis, flexor pollicis brevis, palmaris pollicis, and adductor pollicis. Muscle of hypothenar refers to three muscles in the small finger side of palm: the abductor digit minimis, flexor digit brevis, and palmaris digit minimis. According to the characteristics of the collectability of sEMG signals of different muscles [15], the flexor digitorum superficialis, extensor digitorum of the forearm, muscle of thenar and muscle of hypothenar groups are selected to study their fatigue degree. The four muscles of sEMG signal acquisition selected in the experiment are shown in Fig. 1.

B. SEMG SIGNALS ACQUISITION OF GESTURE
There are important differences in operation between the most frequently used smartphones and computers, which include natural gesture interaction and third-party GUI touch such as mouse and keyboard, respectively. Different operation gestures have their own fatigue characteristics and analogical value. This paper mainly focuses on natural gestures, selecting three gestures commonly used in daily life as the research object, and compares them with the same operation gestures on the computer to evaluate their fatigue, aiming to improve the monotonicity of evaluation of natural gestures.
The EMG wireless signal collector and ergolab software of Beijing Jinfa Technology Co., Ltd. are used to collect the original sEMG signals. A Lenovo computer with a screen size of 23 inches are used in the PC experiments. The screen resolution is 1920 × 1080 and the screen density is 424ppi. The iPhone XS Max smartphone is used as the mobile terminal. The screen size is 6.5 inches, the screen resolution is 2688 × 1242, and the screen density is 458 ppi. There are five graduate students aged 20-35 as subjects, all of whom are right-handed. The width and thickness of their hands are in line with the proportion of height, and in line with the national standard of GB/T 26158. The signal acquisition process based on the smartphone is used as an example, as shown in Fig. 2. Firstly, the sEMG signals of one-handed and two-handed gestures are collected by sEMG sensor and signal acquisition units. Secondly, the signal receiver receives and transmits the original sEMG signal to the computer. Finally, the software matching with the sensor displays the collected signals and conducts preliminary analysis. The sampling frequency is 1024 Hz.
According to the user's habits of operation gestures on mobile phones and computers, they put their arms on the table, supported by the elbows, and are mainly tested by the movement of their hands. There are two main forms of smartphone gestures including one-handed and two-handed. The test gestures are shown in Fig. 3. In Steven Hoober's book Designing mobile interfaces, the vertical screen is divided into three areas: easy area, OK area and hard to reach area. This method of division is suitable for the 4.5-inch smart phone, but not suitable for the modern smartphone with its large screen and complex use environment. Therefore, combining the interaction design of APP keys in IOS and Android systems, this paper divides the large screen into three parts including upper, middle and lower for the split screen sEMG test of the G1 and G3 gestures, as shown in Fig. 3.
This paper focuses on the natural gestures, one-handed and two-handed operations, as shown in Fig. 3. The testing process is divided into two types: independent behavior and comprehensive behavior. Comprehensive behavior simulates the way smartphone are used in daily life. There are three kinds of behaviors with high frequency: browsing microblog or WeChat, playing mobile games and typing for smartphones and computers. The integrated gestures of comprehensive behavior can be divided into 9 independent gestures. Independent behaviors include natural gestures G1-G6 based on smartphones and G7-G9 based on computers. There are G1 one-handed click, G2 one-handed horizontal drag, G3 one-handed vertical drag, G4 two-handed click, G5 two-handed horizontal drag, G6 two-handed vertical drag, G7 one-handed mouse click, G8 one-handed middle mouse VOLUME 8, 2020  button scroll, and G9 two-handed keyboard input, as shown in Table 1. Each independent behavior is a mechanical repetitive action. The fatigue tests of sEMG signals use a large screen smartphone at 6.5 inches. The explanations are shown in Table 2 and Fig. 3. The illustrations of nature gestures G1-G6 can be seen in Fig. 3(c). The independent gesture test is within 10 seconds, and the time of the integrated gesture test is divided into 5-10 minutes.

III. METHODS
The experimental flow is shown in Fig. 4. Signal acquisition is divided into two categories: independent gesture signals and integrated gesture signals. The independent gesture signals are filtered and segmented by noise reduction to obtain time-domain and frequency-domain features. The time-domain features include absolute mean value (MAV) and variance (VAR). The wavelet envelopes of different gestures are further obtained, and the frequency-domain features are used to obtain the power spectrum of sEMG signals, which are used to analyze the fatigue characteristics of the independent gestures. The sEMG signals of the independent gestures are decomposed by wavelet to obtain wavelet coefficients as the input gate information of LSTM.
Nine independent gesture models are trained to predict the fatigue degree of the integrated gestures and to analyze the fatigue characteristics of the integrated gestures.
A. DATA PROCESSING 1) SIGNAL DENOISING During the sEMG signals acquisition, the sensor will be affected by the noise pollution of the surrounding environment and the other muscles of the subjects, so there is a certain amount of noise. Noise will reduce the accuracy of signal analysis. This paper uses a Chebyshev low-pass filter in experiments, the frequency of pass and reject band were 55Hz and 90Hz, respectively. The power frequency interference of mean filtering to sEMG signals acquisition is a periodic signal. Whole period filtering is proposed. The calculation equation of the filtered data of the j th point after the whole period mean filtering of sEMG signals is as in (1).
where X j is the filtered data of the j th point after filtering the whole period mean value of sEMG signals. The calculation equation of the summation point K is as in (2).
where K is the summation point of the whole period average filter, M is the whole period number in the observation signal. f s is the sampling frequency of sEMG signals. f g is the frequency of power frequency electric noise, with 50Hz.

2.Signal segmentation
To extract the feature information contained in sEMG signals, the method of window segmentation analyzes the relationship between the signal and action, and after it is amplified and filtered, the original sEMG signals data needs to be segmented. Lei [16] used the method of moving window to integrate the square of a signal in a short period of time to extract the energy. By setting the size of the threshold artificially, the extracted energy value was compared with the threshold value to determine the action conversion point. This method was affected by the size of the time window, and the time window could not be too large, otherwise the accuracy of the action conversion point recognition would be affected.
According to waveforms and test records of the original sEMG signals, it is corresponding to a period of independent behavior that is selected as the research object to extract the time-frequency and time-frequency characteristics. The envelopes of sEMG signals processed by a Chebyshev lowpass filter for a right-hand click (G2) are shown in Fig. 5. To see the amplitude waveform clearly, the analysis of sEMG signals uses the discrete Fourier transform as implemented in Hilbert, and finally the upper and lower envelopes of the processed sequence are returned as the magnitude of sEMG signals. In the legends of right-hand gesture G2, EMG1, EMG2, EMG3 and EMG4 represent the envelopes of processed sEMG signals of the muscle of thenar, muscle of hypothenar, extensor digitorum, and flexor digitorum superficialis, respectively.

B. FEATURES EXTRACTION OF INDEPENDENT GESTURE SEMG SIGNALS 1) TIME-DOMAIN FEATURES EXTRACTION
The features of sEMG signals are the description of the structure attribute of the specific signals of each channel. Common research methods include time-domain feature, frequencydomain feature and time-frequency-domain feature analysis. In this paper, for the sEMG signals processing of independent gestures, the values of MAV and VAR and the envelope of time-domain signal are mainly used, and different waveforms of sEMG signals of different gestures are obtained. By calculating the relationship between VAR and MAV, the weights of sEMG signals of muscles with different gestures can be obtained, which can provide the weights of the input neurons for the next step of the LSTM calculation.
The MAV can reflect the average intensity of sEMG signals, and indirectly reflect the intensity of muscle movement in this period. For a series of segmented data obtained by the preprocessing of sliding window described above, the MAV of each segment of data can be expressed as in (3).
where k represents the kth segment in the original sEMG signals segment data series, x i is the ith original data in the segment data. W is the sliding window width, and the calculated MAV is the absolute mean value of the sEMG signal of this segment. The VAR reflects the degree of deviation between the sEMG signal and its average value, and its value can represent the clustering characteristics of a certain action of sEMG signal. The smaller the variance is, the smaller the amplitude fluctuation of the signal is, and the better the clustering, the larger the variance is. In contrast, the larger the amplitude fluctuation of the same signal is, the worse the clustering performance is. The calculation formula of VAR is as in (4).
Since the collected data are discrete data, the maximum value point is calculated by the difference equation [17]. On the basis of the integration of the original data of each channel, the wave envelope is extracted.
where y (i) is the difference absolute mean sequence of sEMG signals, b i and a i are the filter parameters, which can be determined according to the sampling frequency f s = 1024Hz and designed according to attenuation δ = − 3 dB. The spectrum characteristics of the Chebyshev I filter are shown in Fig. 6.

2) FREQUENCY-DOMAIN FEATURES EXTRACTION
Early frequency domain analysis is usually used in the research of the mechanism of sEMG signals, the detection VOLUME 8, 2020 of sports pathology and the analysis of muscle fatigue [18].
The main methods are power spectrum (PS) analysis, cepstrum (CS) analysis, mean frequency and median frequency [19], [20]. Fast Fourier transform (FFT), discrete Fourier transform (DFTT) and short time Fourier transform (STFT) are often used in frequency domain analysis. In this paper, FFT is used to calculate the energy of sEMG signals and extract the frequency-domain Pcele features of gestures, as shown in Fig. 7.
In this paper, the power spectra of one-handed and two-handed gestures are calculated and counted respectively, and the energy consumption and total power of sEMG signals of each muscle of independent gestures are obtained.

3) TIME-FREQUENCY DOMAIN FEATURE EXTRACTION
The low-frequency and high-frequency decomposition coefficients of sEMG signals can be obtained by wavelet, which is more significant than the original signal. It is widely used in the field of gesture recognition [21]. In this paper, Daubechies 3-order basis wavelet is used to decompose the processed sEMG signals into two layers, and two relative energy coefficients representing different frequency components are extracted to form a feature sample with 4-space.
Wavelet theory and wavelet transform (WT) are signal analysis tools. As a method combining time-domain and frequency-domain features, the important idea of its theory is to utilize the method of expansion and translation to calculate displacement r using a function ψ(t), known as the base wavelet. It does inner product with the signal f (t) to be analyzed under different scales a.
The equivalent expression of the frequency domain of wavelet transforms is as in (7).

C. INTEGRATED GESTURE PREDICTION METHOD
To predict the energy consumption of integrated gestures more accurately, according to the time continuous characteristics of sEMG signals, the gradient descent of ideal loss function is obtained by LSTM. Using LSTM to train 9 kinds of independent gestures, the error value is minimum, which proves LSTM model has more stable and useful value. The average values of mean squared error (MSE), root mean squared error (RMSE) and mean absolute error (MAE) in the training group are 0.0315, 0.1246 and 0.0963, respectively, and the average error functions of the test group are 0.0235, 0.1024 and 0.0797, respectively. The trained model is connected in series to predict the energy consumption of the integrated gesture. RNN is an important branch of neural networks, which can be used to deal with time series data. Its parameters can be trained by back propagation through a time (BPTT) algorithm [22]. Different from the input and output (I/O) with a fixed dimension of a feedforward neural network, the neurons in RNN have self-feedback, which can be easily extended to a longer sequence. Most RNN can also handle a sequence with a variable length. At time t, the output of RNN not only depends on the weight of the network and the input of the current time but also depends on time t-1 and all previous times. The design of LSTM networks aims to solve the longdistance dependence problem in the time series model. Three important gate control functions are re-introduced into the memory cell module of the hidden layer. The control functions of three gates are realized by their functions, namely, input gate, forget gate, and output gate.
(1) The input gate i t controls the input process of the current time information, which includes the update process of the current time information completed by the input gate. The input of a time on the hidden layer is added to the current state to realize the dependence of the time sequence information. The calculation method for the update process of current time information is shown in (8).
(2) The forget gate f t is used to control the proportion of the time span of input information. When real-time order information passes the forget gate, forget gate uses sigmoid function σ to control the superposition of hidden layer h t−1 and input layer x t . The output expression f t of forget gate is shown in (9).
(3) The output gate o t controls the output and the time sequence information returned to the hidden layer before the output of memory unit information. The calculation method of output gate is shown in (10).
CandidateC t is used to conclude new knowledge, to be stored in the cell state. The value ofC t is connected to the function matrix W c to be trained, the memory matrix and bias term to be trained b c , as shown in formula (11). The memory matrix includes the previous moment h t−1 and the current time Cell state C t is equal to the sum of the dot product of forgetting gate and long-term memory of last moment C t−1 , plus the dot product of the input gate i t and candidateC t . That is, the superposition process of the output at one time and the current input on the hidden layer, as shown in (12).
Hidden state h t is the product of output gate o t and the tanh function of cell state C t . It belongs to short-term memory, that is, information returned to the hidden layer.

A. FATIGUE ANALYSIS OF INDEPENDENT BEHAVIOR GESTURES
In this experiment, the time-domain feature extraction of MAV, VAR and envelope, and power spectrum feature extraction of sEMG signals of each muscle of nine groups of independent gestures were carried out as the basis of a comprehensive gesture analysis. Fig. 8 shows the characteristics of MAV and VAR in the time domain, and Fig. 9 displays the envelope diagram in the time domain. Fig. 10 shows the comparison of power characteristics in the frequency domain. The time-domain MAV of one-handed gestures shows that the thenar muscle of one-handed gestures based on smartphone is most used, and this trend in VAR is more obvious, as shown in Fig. 8 (a). The sEMG signal amplitudes of G7 and G8 based on PCs are more uniform. The extensor digitorum of left hand is used more frequently. The thenar muscle of right hand of G5 is more used, and the hypothenar muscle utilization in G6 is slightly higher than that of the extensor digitorum of left hand. The thenar muscle of the right hand, the thenar muscle and the extensor digitorum of the left hand of G9 are used more frequently. The VAR diagram highlights the differences.
The window signal filtered by the G1-G9 gesture is shown in Fig. 9. The time-domain waveform is represented by the envelope and processed via the Chebyshev low-pass filter. Because of the difference of the window length of each signal, the time length of the horizontal axis depends on the window length. For intuitive comparison, the longitudinal axis, namely, the amplitude axis, uses the axial scale of 100 ∼ 1650uV. The sampling frequency is 1024Hz. The legends of EMG1, EMG2, EMG3, EMG4, EMG5, EMG6 and EMG7 represent sEMG signals of the right-hand thenar muscle, right-hand hypothenar muscle, right-hand extensor digitorum, right-hand flexor digitorum superficialis, left-hand thenar muscle, left-hand hypothenar muscle, and left-hand extensor digitorum, respectively.
The envelopes of G1-lower and G3-lower show that the thenar muscle is more used. The sEMG signals of G1-middle and G3-middle are lower. The amplitudes of G7 and G8 are the lowest, both lower than 200 µV, followed by G4, G5 and G6. The thenar muscle of right hand of G5 is more used, and the hypothenar muscle of left hand of G6 is more used. The amplitude of sEMG signals in G9 are the largest, followed by G2.
The trend of the power spectrum in the frequency domain is basically consistent with that of VAR in the time domain, as shown in Fig. 10. In one-handed gestures, the thumb is the most used finger. G1-upper and G2 have higher energy consumption. The energy consumption of G9 is the highest.
As shown in Fig. 11, the highest total energy consumption of the nine independent gestures is the input gesture G9 based on PCs, which consumes 42436.91 dB. It is also the highest energy consumption of two-handed gestures. The highest energy consumption of one-handed gestures is G2 based on the smartphone, which is 27276.4 dB, followed by the one-handed gesture G1-upper 23759.4 dB in the upper area of the screen. The lowest energy consumption of independent gestures based on smartphones is the G1-middle of a righthand click on the middle area of the screen is 1700.76 dB. The one-handed mouse clicks G7 and mouse middle button slide G8 based on PCs are the gestures with the lowest global energy consumption, only 1522.07 dB and 1194.58 dB respectively.
1. Analysis of independent gestures sEMG signals of the smartphone (1) Separate screen sEMG signals analysis of one-handed independent behavior G1-lower. G1 is a handedness click gesture. When the right-hand clicks in the lower area of a screen, it is mainly operated by a thumb. The waveform and amplitude of the muscle of thenar are the highest with 140.58 µV, and the energy consumption is up to 6182 dB. The four fingers mainly support the smartphone. The amplitudes of the extensor digitorum and flexor digitorum superficialis are both 35 µV, and the muscle of hypothenar is rarely used with only 21 µV, as shown in Fig. 9 (a).
G1-middle. The gesture uses a right-hand click in the middle area of the screen, the extensor digitorum, flexor digitorum superficialis and muscle of hypothenar stably support the smartphone, which reduces the difficulty of the thumb click. Therefore, the amplitude of sEMG signals of the muscle of thenar is relatively low with only 51 µV, and the energy consumption is only 746 dB. When clicking, the peak value of the amplitude is the highest with 248 µV, as shown in Fig. 9(b).
G1-upper. When the upper part of the screen is clicked, the amplitude of thenar muscle is higher, the average value is 221.35 µV, and the energy consumption is 13816 dB. The muscle of hypothenar and four fingers support the thumb to reach the operation position in the difficult area. The sEMG signals of them have large energy consumption, which are 2708 dB, 3580 dB and 3656 dB respectively. The peak values of the muscle of thenar and extensor digitorum are up to 500 µV. Compared with the middle region, the average amplitude is three or four times that.
The total energy consumptions of the three screen areas of gesture G1 are 6889.66 dB, 1700.76 dB and 23759.4 dB respectively, indicating the upper area of the screen is not suitable for click operation. The muscle energy consumption of G1-upper is the largest, showing that G1-upper is not suitable for long-term use. The gesture or NUI design should avoid setting operation buttons at the top of the screen. It is convenient to click in the lower area, and the fingers can  put in force. However, when one hand holds the smartphone to control the lower area of the screen, this requires the operation of the hands and fingers, which leads to the energy consumption of the sEMG signals at a medium level. The middle region with the lowest energy consumption, is the most suitable location for one-handed gestures.
G3-lower. G3 is a gesture of vertical drag with onehandedness. The sEMG signals envelope diagram of the right hand in the lower area of the smartphone screen shows that the amplitude of the muscle of thenar is the highest, with the MAV 134.35 µV, which is 2-3 times that of the other muscle groups. It shows that the thumb is used more to operate, which is also the main energy consumption muscle group with 4796dB. The quadriceps and hypothenar muscles are mainly used to support smartphones. The MAV amplitudes of hypothenar and flexor digitorum superficialis are both 55 µV, and the energy consumptions are 581 dB and 763 dB respectively. The extensor digitorum is less used, as shown in Fig. 9 (e). The total energy consumption is 6390.71 dB.
G3-middle. When operating in the middle area of the smartphone, the thumb is the main executive organ of the drag action. The amplitude of the muscle of thenar is still large, its average value is 164 µV, and the energy consumption is up to 6430dB. The amplitude of the hypothenar muscle remains high. Its waveform corresponds to that of the muscle of thenar, and the energy consumption is 843 dB. When the users swim up and down with one hand in order to reach a higher screen position with the thumb, the muscle of hypothenar will contract and strengthen the support for the smartphone, with the MAV amplitude of 56 µV, while, the four fingers move less. As shown in Fig. 9 (f), the total energy consumption is 7998.87 dB.
G3-upper. To reach the upper area of the screen, the users will move the position of right hand from the lower to the middle or upper position of the smartphone. When the thumb presses the upper left corner area of the screen, the maximum peak value of the muscle of thenar is 275 µV, the energy consumption is 3951 dB, and the total energy consumption is 5262.82 dB, which is nearly the same energy consumption of G3-lower. The muscle of hypothenar also contracts to support the thumb touch to the difficult-to-reach area, with a peak value of 260 µV and energy consumption of 618 dB, as shown in Fig. 9 (g).
It can be concluded that the buttons of one-handed vertical drag gestures are suitable to be set at the bottom of the screen, which can minimize the operation energy consumption.
(2) Analysis of the sEMG signals of one-handed independent gestures.
G2. The use scenario of a horizontal drag by handedness mainly includes connecting the phone of an Android or switching the current application software of iOS, which can be completed in the lower area of screen, so only the sEMG signals in the lower area is tested. When the thumb crosses the screen horizontally from left to right, the envelope shape of sEMG signals of thenar muscle is parabola. The amplitude of the thumb is as high as 235.65 µV, and the maximum energy consumption is 17465 dB. When the thumb is at the right side of the screen, the grip strength of the thumb and palm is concentrated to one side of the smartphone, which affects the control. At this time, the muscle of hypothenar contracts and strengthens support, and the sEMG signals are significantly improved. The amplitude is increased to 600 µV, and the energy consumption is 6728 dB, as shown in Fig. 9(d).
G2 is the second highest energy consumption of a one-handed gesture, twice as much energy as G1-lower. That indicates that the horizontal drag gesture on the lower part of the screen is not suitable for high-frequency and long-term use. Android and iOS systems have also changed the slide answer to a click answer.
(3) Analysis of sEMG signals of two-handed independent behavior.
G4. Two-handed click gesture. The user grasps the smartphone with left hand and clicks the screen with a righthand finger. The results show that the trend of sEMG signals envelopes of the hypothenar muscle, extensor digitorum and flexor digitorum superficialis are all parabolas. The peak values are 175 µV, 170 µV and 150 µV respectively, and the energy consumptions are 235 dB, 617 dB and 252 dB respectively. The signal amplitude of the extensor digitorum muscle of the left hand is 80 µV, and the energy consumption is up to 1432 dB, indicating that the left four fingers play a major supporting role.
The energy consumption of G4 is 5367.16dB, which is less than that of G1-upper and G1-lower. This shows that the energy consumption of two-handed is less than that of one-handed. Only the energy consumption of the G1-middle gesture is less than that of G4, which proves that the one-handed gesture with the same function only has  the advantage of low energy efficiency in the best position. G4 has the highest energy consumption of two-handed independent gestures, indicating that the energy consumption of a one-handed or two-handed click is higher than that of a drag.
G5. Because the left hand supports the smartphone and the right hand operates it, it is not limited by a one-handed operation, so the sEMG signals are not analyzed in the split screen. When operated two-handed, the MAV amplitudes of the thenar and hypothenar muscles of the left-hand are 20 µV, and 21 µV respectively, and the energy consumptions are 86 dB and 123 dB respectively. The amplitude of the extensor digitorum is high, up to 68 µV, and the energy consumption is 999 dB. It can be seen that the energy consumption of the four fingers is higher when the left-hand grasps the smartphone. When the thumb of the right hand crosses the screen, the signal amplitude of the thenar muscle of the right-hand is 111.21 µV, and the energy consumption is 3368 dB. The use of the four fingers of the right hand is relatively less, as shown in Fig. 9 (i).
The total energy consumption of G5 is 3035.65dB, which is the lowest energy consumption of two-handed independent gestures. It is more comfortable than a two-handed click and vertical drag. Compared to G2, which are both horizontal drag gestures, the energy consumption of G5 is one tenth of that of G2, which indicates that G5 has the advantage of low energy consumption.
G6. The gesture of two-handed vertical drag uses the left hand grasping the smartphone and the right finger drags. The amplitudes of the extensor digitorum and flexor digitorum superficialis are 39 µV and 33 µV respectively, and the energy consumptions are 396 dB and 266 dB. The signal of the hypothenar muscle of the left-hand is increased significantly, with MAV of 37 µV and energy consumption of 1914 dB. It shows that the left palm support is strengthened after dragging. The maximum amplitude of extensor digitorum is 80 µV and the energy consumption is 1041 dB, which indicates that the left four fingers consume more energy in this gesture.
The total energy consumption of G6 is 4051.44 dB, which is between G4 and G5, indicating that it has some advantages in two-handed gestures. Compared with the vertical drag of one-handed gestures of G3-lower, the energy consumption of G6 is smaller, which indicates that the two-handed gesture is more suitable than that of one-handed.
2. Analysis of independent gesture sEMG signals of PC In this experiment, the standard desktop keyboard is used, and the envelopes of sEMG signals are detected by the user randomly inputting text information (without a mouse) as is shown in Fig. 9.
G7. The mouse is operated by a mouse click gesture with right-hand, and Lenovo standard mouse is used for tests. The muscle of thenar controls the mouse movement, with a higher MAV with 58 µV. The hypothenar muscle supports with MAV amplitude of 40 µV. The MAV amplitudes of the extensor digitorum and flexor digitorum superficialis are 29 µV and 27 µV respectively. The results show that the amplitude of the extensor digitorum muscle and flexor digitorum superficialis increase when the index finger mouse clicks, and the energy consumptions are 192 dB and 190 dB respectively. The thenar and hypothenar muscles, which can control the mouse by the palm, have high energy consumption of 738 dB and 402 dB respectively, as shown in Fig. 9 (k).
G8. The middle mouse button rolling gesture is a gesture independently completed by the right-hand. When using the mouse, the right thumb, ring finger and little finger of the subjects hold the mouse, and the index finger and middle finger are placed on the left and right mouse buttons respectively. The index finger of the right-hand is used when using the middle mouse button's pulley to scroll. The sEMG signals amplitude of the right-hand thenar muscle is the highest, and the MAV amplitude is 58 µV, indicating that the muscle of thenar supports the palm. The maximum energy consumption is 766 dB. Second, the amplitudes of the flexor digitorum superficialis and extensor digitorum are 27 µV and 30 µV respectively, and the energy consumptions of them are 201 dB and 171 dB, respectively. The peak value of the flexor digitorum superficialis is up to 100 µV, indicating that the energy consumption of the four fingers is relatively high, as shown in Fig. 9 (l).
G7 and G8 are not only the gestures with the least energy consumption of PC, but also the gestures with the least energy consumption of independent gestures. Compared with natural interactive gestures, G7 and G8 initiate interaction through third-party control, which has less energy consumption in one-handed gestures and has obvious advantages of low energy consumption.
G9. The sEMG signals amplitudes of the left-hand thenar muscle and extensor digitorum of keyboard input gesture are higher, which indicates that the energy consumptions of lefthand thumb and four fingers are larger in typing. The MAV amplitudes are 194 µV and 176 µV respectively, and the energy consumptions are 19588dB and 8399dB respectively. The MAV amplitude of the muscle of hypothenar is 175 µV and the energy consumption is 9474 dB, indicating that the right hand is the main support. The energy consumption of the right-hand four fingers is less than that of the left-hand with 1322 dB and 1325 dB, respectively, as shown in Fig. 9 (m). This is affected to some extent by the distribution of keyboard letters and personal typing habits.

B. FATIGUE ANALYSIS OF COMPREHENSIVE BEHAVIOR GESTURE
The specific process of comparative training will be discussed in SEMG-based Neural Network Predication Models Selection of Gesture Fatigue and Datasets Optimization. The nonlinear neural network and fuzzy neural network are tried in the selection of neural network models, and abandoned because of the high error values. By comparing the prediction error values of BP neural network, RNN and LSTM, the experimental results show that LSTM model is more suitable for gesture fatigue prediction. Wavelet decomposition can extract the high-frequency and low-frequency coefficients of sEMG, which is widely used in gesture recognition field. The processed sEMG signals and wavelet decomposition coefficients are tested as the feature sets of the model. It is found that the processed sEMG signals are appropriate for using as the training set fatigue degree of one-hand gesture. It is better to use wavelet decomposition coefficients as data set to predict the high-dimensional sEMG signals of two-hand gestures.
Because there is no obvious waveform window to be segmented in the integrated gesture simulation of daily use scenarios, 10000 sets of sEMG signals segments of the whole signal are uniformly picked up to extract the sEMG signals and wavelet decomposition coefficients as the analysis objects. Because there are many kinds of gestures and complex muscle groups, it is difficult to judge energy consumption by frequency-domain calculation. Therefore, the energy consumption of integrated gesture is predicted by LSTM models of 9 independent gestures trained previously. The integrated gesture browsing information is one-handed four channel gesture, and its data set is the most suitable for using sEMG signals. The training models of G1, G2, G3 are connected in series to make prediction. Series models of G7 and G8 are selected for PC-based browsing gesture prediction. Playing games and typing are both two-hand gestures, which have 7-dimensional input signals. It is suitable to use wavelet decomposition coefficients of sEMG signals as data sets. Playing games and typing on smartphones use series models of G4, G5 and G6 to predict. PC-based games and typing use prediction training model of independent gesture G9. The prediction results are shown in Table 2.

1) ANALYSIS OF INTEGRATED GESTURES sEMG SIGNALS OF SMARTPHONES
The integrated gestures based on smartphone browsing information can be divided into these independent gestures: righthand horizontal drag, right-hand vertical drag and right-hand click. Therefore, the LSTM one-handed training models of G1 (lower, media, upper), G2, G3 (lower, media, upper) VOLUME 8, 2020 are concatenated to predict. Using the trained LSTM prediction model, after inverse normalization, the total energy consumption of 10000 groups of consecutive signal segments based on the integrated gesture of browsing information behavior of smartphone is predicted with −1.61×10 16 dB.
The integrated behavior of playing games and inputting information can be decomposed into independent gestures: two-handed click, two-handed horizontal and vertical drag. The LSTM training models of G4, G5 and G6 in series are used for prediction. The total energy consumption of 10000 consecutive sEMG signals segments for playing game via LSTM prediction and inverse normalization is −1.74×10 14 dB. The total energy consumption of 10000 consecutive signal for typing segments is −9.70×10 12 dB.
It can be seen that in the comprehensive behavior of a smartphone terminal, the sEMG signals energy consumption of playing games is the highest, followed by information input, and the energy consumption of browsing information is the lowest.

2) ANALYSIS OF INTEGRATED GESTURES SEMG SIGNALS OF PCs
The integrated gestures based on browsing information on PCs, can be divided into the independent gestures of onehanded mouse click and middle mouse button scrolling. The LSTM one-handed training models of G7 and G8 are connected in series to predict. The total energy consumption of 10000 consecutive signal segments based on PC integrated gesture of browsing information behavior is −4.14×10 15 dB.
The comprehensive behavior of playing games and typing can be divided into these independent gestures: right-hand click mouse, right-hand scrolling middle mouse button, and keyboard input. The serial LSTM training models of G7, G8 and G9 are used for prediction. The comprehensive energy consumption of playing games is −4.43×10 15 dB. The total energy consumption of typing is −3.34×10 15 dB.
It can be seen that input information is the most energy consuming comprehensive behavior, followed by browsing information and computer games.

V. CONCLUSION
This paper discusses the energy consumption of three user behavior gestures based on actual use scenarios. To predict the fatigue degree more accurately, the three gestures are decomposed into nine independent gestures, and their timedomain and frequency-domain characteristics are analyzed. It is more objective to evaluate the fatigue degree through quantitative analysis of sEMG signals and envelope waveform. The prediction and evaluation model of integrated gestures is established. The processed sEMG signals and the wavelet decomposition coefficients of nine independent gestures are used as the input variables of neural networks.
Innovations: 1. The method of subjective evaluation of fatigue degree in perceptual engineering is improved, and the monotonous evaluation method of sEMG signals is improved and optimized. 2. A more accurate solution for fatigue prediction of complex integrated gestures is proposed. The integrated gestures are divided into simple independent gestures for analysis, and then the neural network is used to predict in series. 3. It is found that LSTM is the most suitable model for fatigue prediction. The model is accurate and reliable. 4. The prediction results provide a reference for gesture design related fields.
(1) NUI design: according to different gesture use scenarios, the middle and lower part of the screen is the most suitable position for one-handed interactive gestures with the least fatigue. The user can operate the NUI conveniently while holding the smartphone. The bottom of the screen is a sub convenient area with less fatigue, which can be easily reached by fingers by one-handed gestures. However, the operative buttons shall not be designed on the side far to the handedness. The upper area of the screen has the highest fatigue and is the difficult area of one-handed operation. Therefore, the middle and lower part of the NUI are the most suitable areas for arranging interactive buttons, which can limit user gesture fatigue effectively. When operated by two hands, the positions of screen buttons are no longer limited.
(2) Gesture setting: according to the experimental verification, the overall energy consumption of the two-handed independent gestures based on smartphones is lower than that of the one-handed gestures. The two-handed horizontal drag is better than vertical, and drag is better than click gestures. When only one-handed gestures are limited by the environment, it is recommended to give priority to the click gestures at the bottom of the screen, followed by the vertical and horizontal drag gestures in the middle of the screen. Therefore, in gesture design, fatigue can also be reduced by using dragging instead of clicking interactions.
The energy consumption of mouse click and slide is lower than that of smartphone. Therefore, it is better to develop PCs client of the same app. Users would like to choose PC operation first under the allowed environment.
(3) App settings: gesture fatigue indicates that when designing a smartphone or PC human interaction interface, not only the user's visual processes, but also the impact of the comprehensive comfort of gesture operation should be considered. It is necessary to avoid repeated operation of single gestures for a long time and to reduce muscle static injury. Different carriers with various sizes have different operation modes and operation areas. One-handed gestures are more suitable for simple behaviors such as browsing information. When designing internet applications, the buttons of NUI should be arranged in the easy area that the thumb can reach. The two-handed gestures are suitable for activities requiring frequent operation such as playing games, during which the non-handedness holds the smartphone and handedness operates it. The button layout should also be suitable for the operation habits of the left and right thumbs. It should not only be set in the easy area of the thumb, but also be set appropriately according to the distinguished flexibility of the left and right thumbs. In addition to games, people who use smartphones in sports also have different experiences in the NUI operation process. For example, when using smartphones in different scenes such as kitchen, washroom, waiting room, etc., the design of the operation time of smartphones and operation mode of Apps directly influences safety, attention and other needs, which should also be considered in gesture design. He is currently a Professor and a Doctoral Supervisor with the Department of Industrial Design, and the Director of the Industrial Design Research Institute, Mechanical Engineering, Shandong University, China. He is also a member of information and interactive design professional committee of China Industrial Design Association, a Senior Member of Industrial Design Society of China Mechanical Engineering Association, the Director of Shandong publishing Artists Association, and a member of Shandong Artists Association. He has led and competed more than seven projects funded by National or Shandong province, and has published two monographs and seven articles cited by SCI/CSSCI/EI. His research interest includes humanmachine interaction.
YAN LIU received the Ph.D. degree in 2016.
She is currently a Professor and a Doctoral Supervisor with the Department of Industrial Design, and the Vice Director of the Industrial Design Research Institute, Mechanical Engineering, Shandong University, China. She have presided over and competed ten projects funded by National or Shandong province, and published three monographs and 18 articles accepted by SCI/CSSCI/EI. She honored with ten National Awards.
JIAHUI NIU received the B.S. degree in mechanical design and manufacturing and its automation and the M.S. degree in mechanical manufacturing from Shandong University, China, in 2016 and 2019, respectively, where she is currently pursuing the Ph.D. degree in mechanical engineering.
Her research interests include the highspeed machining mechanism, the development of finite element simulation technology of machining, and the research of machining surface quality. VOLUME 8, 2020