Recognition Punches in Karate Using Acceleration Sensors and Convolution Neural Networks

Coaches and athletes need to understand the kinematics and dynamics of karate kicks to improve the training process and results. The research was aimed at studying the automatic recognition of punches in karate using only linear acceleration sensors. Accelerometers were part of the Inertial Measurement Units (IMUs), which were attached to the left and right wrist of the athlete. To develop a model of punches, highly qualified athletes with 3–7 years of karate experience participated in the research. We analyzed the acceleration fields of various karate punches: Yun Tsuki, Mawashi Tsuki, Age of Tsuki, Uraken. We have proposed more straightforward approach to extracting features without calculating their statistical characteristics. To solve the classification problem, we have used various architectures of convolutional neural networks: multilayer perceptron, 1- and 2-dimension Convolution Networks. Since the recognition of punches was carried out in the conditions of a shadow fight, in addition to the recognition of punches, another output parameter was introduced – movement without punches. Studies have shown a high level of punch recognition based on the developed models. The multi-class accuracy value is 0.96, and the average F1 value is 0.97 for five different punch classes. Thus, the proposed approach is more suitable for practical implementation in automatic learning systems.


I. INTRODUCTION
Karate is a traditional Japanese martial art. However, this Japanese martial art has gained popularity all over the world. Sports competitions of national and world level are held in karate. The popularity of karate as a sport is growing and, in this regard, the methods of training karate athletes are increasingly becoming scientific in nature. To develop effective training techniques, coaches need to understand the kinematics and dynamics of karate punches [1]- [6]. Therefore, our research was aimed to analyze the velocity fields of punches in karate, as well as to develop and analyze various models of artificial neural networks for recognizing punches.
Sports in the modern world is a socially significant element, and, therefore, the technical and technological aspects of social process research are essential, including the study The associate editor coordinating the review of this manuscript and approving it for publication was Paolo Crippa . of machine learning technologies. Some features of the use of neural networks in socially oriented processes and algorithms for optimizing the search for a sufficient and optimal solution to the posed problem are given in [7]- [11].
To solve the problems of the study, we have used inertial measurement units (IMUs), which included an accelerometer and a gyroscope. IMUs were attached to the wrists of karate athletes. We have used the IMUs because in sports and martial arts, they proved to be an effective tool for analyzing the kinematics and biomechanics of human movements [12].

A. REVIEW OF CURRENT RESEARCH
In [13], studies of the acceleration and speed of punches were carried out using IMUS that were installed on the wrists of boxers. The accelerometers in this study had a large range -200g (g is the acceleration of gravity = 9.8 m / s2), but the acceleration graphs show that the maximum acceleration was about 25g. In addition, this acceleration corresponded to the final phase of the punch when the athlete's fist stopped abruptly, and this led to a large negative acceleration. This allows us to conclude that for studies of the kinematics of punches in martial arts, it is possible to limit the measurement range to 16-25g. In [13], it was found that the speed of punches of male athletes was 8.1±1.4 m/s for jab-out punches, and 7.7 ± 1.5 m/s for cross-out punches. The women had the following results: 6.6 = 1.6 m / s (job-out), 5.7 = 1.5 m / s (cross-out).
The authors of the work [14] investigated the difference between the biomechanics of punches of elite and novice boxers based on IMUs which in the amount of 17 pieces were installed on the body of boxers. IMUs had an accelerometer measurement limit of 18g; they included an accelerometer, gyroscope, magnetometer. Since the IMUs were installed on each body segment, the contribution of the body segments to the punching technique of boxers was determined. In both groups (elite and novice athletes) the elbow contributed the most to the cross-out technique and the shoulder contributed the most to the hook and uppercut.
In [15], the analysis of the kinematics of boxers ' punches using accelerometers was carried out in conjunction with videography. The authors searched for the correlation of postures and fields of acceleration of blows with the fatigue of the athletes. The graphs of punch accelerations given in [15] show that the maximum values are in the range of 20-40 m / s2, which also allows us to choose an IMU for experiments with a measurement limit of up to 16g. In [15], it is stated that a large number of degrees of freedom of human hands does not allow us to draw unambiguous conclusions about the kinematics of blows, so videography was additionally required. It can be noted that in this work the magnetometer and gyroscope which are usually included in the IMU were not used; perhaps their use could lead to the fact that videography would not be needed.
Also, various techniques of artificial neural networks (ANN) are used to analyze the kinematics of punches in martial arts, which can also help in conditions of lack of data. The advantages of ANN have led to the fact that they are actively used in sports and martial arts [2]. For example, the authors in [13] concluded that according to the accelerometer data, it was difficult to find the time when the boxer's hand begins to return after a punch. It can be assumed that the use of ANN methods could deal with this problem.
In [16], the ANN in the form of a multilayer perceptron was developed for the purpose of automating the data collection of boxers' punches. The input data for ANN was the IMU data that was attached to the boxers' wrist. The accuracy of punch recognition ranged from 87.2 ± 5.4 \% to 95.33 ± 2.51\%.
In [17], six different deep machine learning models for recognizing boxers' punches were investigated. The IMUs were installed in two versions: 1 -the IMUs were attached to both wrists; 2 -the IMUs were attached to both wrists and the third thoracic vertebra. The accuracy of the impact prediction was for version 1 -0.90 ± 0.12, for version 2-0.87 ± 0.09. For version 1, the support vector machine (SVM) model worked best (accuracy = 0.96), for version 2the multi-layer perceptron neural network (MLP-NN) model (accuracy = 0.98) did.
Not many works are devoted to the analysis of punches in karate based on IMUs and ANN. And so far, no research has been conducted on a specific karate punch which is called Uraken in Japanese (a punch is made from the inside out).

B. CURRENT CHALLENGES AND SUGGESTED APPROACHES
Punches in karate are distinguished by a complex kinematic pattern. It is impossible to search for the most effective training methods without knowledge of the biomechanical features of punching movements. Therefore, it is required to study the kinematic parameters of the punch -speed, and acceleration.
The next problem in the field of martial arts is the development of a punch model. The effectiveness of motor actions fulfillment is determined by the degree of their kinematic and dynamic structure closeness to the most effective punch model.
However, the development of a model of punches in karate requires the inclusion of many factors into this model (kinematics and dynamics of punch, body position, time phases of the punch, features of the athlete's physique, etc.), which are still difficult to combine into the unified model. Therefore, at present, the development of punch models based on deep learning technologies has been actively evolving [2], [12], [13], [16]- [20].
The work structure includes abstract, keywords, and five sections. Section 1 contains an introduction to the research topic. Section 2 presents the materials and methods used in the study. Section 3 describes the results of the study. Section 4 describes the discussion of the results of the work. In section 5 the conclusions are presented.

A. PARTICIPANTS
The study involved sixteen healthy participants (n=16), 12 men, 4 women, aged 22±3 years, weighing = 70±14 kg, height = 165±21 cm, with 3-7 years of experience in karate. Ethical approval was granted by the Human Research 76 Ethics Committee at Financial University under the Government of the Russian Federation.

B. MATERIALS
The design of the experiment can be seen in Figure 1.
On the wrists of the athletes, devices were fixed which included a microcontroller, IMUs, and Bluetooth modules. Athletes punched in shadow fight mode. The IMUs (accelerometer and gyroscope) data was initially transmitted via the Bluetooth channel to the android device. On the android device, the data was saved as files for each type of punch. This data was then processed on the computer. In order to label and identify each punch for develop models of the The data acquisition device ( Figure 2) was a 50 × 20×10 mm box containing three modules -the microcontroller stm32f103 [21], IMU MPU6050, and Bluetooth module HC-05 with a BC417143 [22].   Figure 3 also shows the directions of the acceleration axes and the angular velocity of the gyroscope.
We did not use gyroscope and magnetometer measurements and did not calculate the angles and position of the sensor. We used only linear acceleration sensors measurements.
To record the session, we used Xiaomi Redmi 7 camera with 1980 × 1080 resolution 30 fps. Video analysis was used for labeling ground actual punches. To record data, we used Bluetooth Serial Terminal Android application. In Figure 4-7, the red arrow shows the approximate trajectories of the punches.
The design of the data acquisition device was developed as a result of the review of work in the field of obtaining and processing data from sensors installed on a person and showing the movement of body parts in space [23]- [40].
Also, for the development of the device, the works in which the data was processed using convolutional neural networks were analyzed [41]- [55]. Measured data was packed to dataset X: every sample has 3 columns (x, y, z acceleration). Train/validation random splitting was made with 10:1 proportion for each class. Histogram of classes samples distribution are in Figure 8.
Data preprocessing was conducted with python 3.7 packages: numpy, sklearn. Visualization was made with matplotlib; Neural Net models were built with tensorflow.     Figure 9.
In experiments 4 models took part: • Multilayer perceptron; • 1-dimension convolution network; • 2-dimension convolution network; • 2-dimension convolution network with additional layers. Multiclass Accuracy used as classification metric for all classes:  N_T -number of true classified punch, N -total number of punches.
Precision (P), recall (R), and F1-score were used as classification metrics for single classes: N_TP -number of true positive classified punch, N_FP -number of false positive classified punches.
Neural network models were developed using Python software, the full program code along with the dataset is freely available in the GitHub repository [56]. The pseudocode of the program implements generalized approaches for the developed models.  In Tables 2-5, ReLu, Sigmoid are the activation functions used on these layers. Batch normalization means applying a transformation that maintains the average output value close to 0 and the standard deviation of the output close to 1. Glob-alAveragePooling2D applies an average pooling of spatial dimensions until each spatial dimension is unified and leaves the other dimensions unchanged.
The confusion matrix is in Figure 14.
During the training process, we can see the difference between train and validation accuracy. We propose that the linear model is overfitting. To avoid this, we try a more complicated model -1D convolution network for time series classification from [57].   activations. Optimizer is Adam, learning rate = 2e −3 and batch size 64.
After 100 epochs of training, we have 0.81 training and 0.62 validation accuracy. Classification metrics are in Table 7.
The confusion matrix is in Figure 15.
During the training process, we observe small train accuracy, unstable validation accuracy, and big loss. We suppose that 1D convolution model is unsuitable for punch classification. So, we try to use feature combination and model with 2D convolution layers.   activations. Optimizer is Adam, learning rate = 2e −3 and batch size 64.
After 100 epochs of training, we have 0.97 training and 0.93 validation accuracy. Classification metrics are in Table 8.    The confusion matrix is in Figure 16. During the training process we can see a much better train and validation accuracy, so we try a deeper model with 3 2D convolution layers.    with size (2, 32), other layers have 64, 64, and 96 (2, 2) size kernels. Batch normalization and ReLu activations are also used. Optimizer is Adam, learning rate = 2e −3 and batch size 64.
After 100 epochs of training, we have 0.98 training and 0.96 validation accuracy. Classification metrics are in Table 9.
The confusion matrix is in Figure 17.
During the training process, we can see better train and validation accuracy, and small loss. We try more layers, but this does not significantly improve classification metrics.

IV. DISCUSSION
Having the field of accelerations for different types of a blows, it is possible to set the task of the analysis of the   technology of performance of each of punches. For more independent analysis of the value of accelerations, we will divide into free-fall acceleration of g = 9.8 m/c 2 , the data thereby obtained will become dimensionless. To improve the quality of the analysis, we will integrate the calculated values of accelerations. We approximate acceleration on the interval [(n-2)T, n T] using the parabola on three values a(n-2), a(n-1), a(n). The difference equation has the appearance: V (n) = V (n-1) + T [5/12a (n) +8/12a (n-1)-1/12a (n-2)]. Thus, we will receive dependence of the speed of blow on dimensionless time. In works [13]- [15] several sensors for the analysis of movements were used. In our research to analyze the technology of punches' performance, we have used a three-axis accelerometer to draw qualitative conclusions on each type of punches equipment. Having constructed schedules of dependence of speeds, it was noted that for the same punches they significantly differ depending on the hand on which measurements were made. At the same time for the same hand of speeds dependence for identical types of blows are similar, but don't coincide. As dependences of speeds for a particular hand and particular punch don't coincide, but are close, average dependences of speeds on the number of measurements for each type of punch, for the left hand and right hand were found. Measurements were performed using the accelerometer fixed on a hand. In Figure 3 the illustration for the left hand with the indication of the directions of a coordinate is presented. When fixing an accelerometer on the first hand, the abscissa axis was directed along a forearm towards the athlete.
Let's consider concrete types of blows.

A. KINEMATIC ANALYSIS OF YUN TSUKI
In Figures 18, 19 dependences of a projection of speed of a hand to an abscissa axis (on the left) and an axis of ordinates (on the right) from measurement time are presented. Let's consider the schedule on the left. For the right hand, the schedule has a minimum between two maxima, for the left hand, a maximum is located between two minima. From presented on the left part schedules it is visible that the punch consists of several phases, namely, a swing phase (inverse to the main driving) before the first crossing of an abscissa axis on graphics, phases of increase in speed of a hand for punch with the subsequent delay (the second extremum on graphics), transition to a phase of back motion (the second crossing of an abscissa axis) with the subsequent delay (the third extremum), and transition to ''residual'' driving in the direction of the main punch (the third crossing of an abscissa axis). From schedules of the speed of a hand for this athlete, it is visible that the maximum speed of swing by the right hand on axis X is less than the same size for the left hand and swing by the right hand shorter, than swing by the left hand. However, the maximum speed in the fissile phase of a blow for the left hand is more, than for the right one, but it is reached a little later, than the moment of achievement of the maximum speed in the corresponding phase by the right hand. Thus, more intensive swing by the left hand gives high maximum speed in a projection to axis X in the main phase of the blow.   It is possible to note that the time of the beginning of back motion for the right and left hand almost coincide, as well as the time of the beginning of residual driving in the direction of the blow, but the delay of back motion by the left hand is sharper and occurs slightly earlier (the third extremum on schedules). Analyzing the change of a projection of speed to axis Y, it is possible to note that there is no swing on an axis of ordinates, but the speed maximum modulo for the right hand is less, than for the left hand and is reached by the right hand earlier, than left hand. Further, the phase of movement delay of a hand down with the subsequent insignificant increase in speed follows on axis Y.

B. KINEMATIC ANALYSIS OF MAWASHI TSUKI
In Figures 20, 21, dependences of projections of speed to axes X (on the left) and Y (on the right) from measurement time are presented.
From the analysis of the left graph, it is visible that the pith punch of swing to axis X isn't present. On axis X speed at first grows and then decreases almost to zero. Change of a projection of speed to axis Y in Figure 21 shows that along this axis the swing for both hands, approximately identical, takes place. The maximum projection of speed to axis Y for the right hand is more than for the left hand. After achievement of the maximum projection of speed to an axis of ordinates delay of driving in this direction (in fact down) with reaching some constant value begins. It is possible to note that the final projection of speed to axis Y in the right hand is more than a similar size for the left hand.

C. KINEMATIC ANALYSIS OF AGE TSUKI
In Figures 22, 23, dependences of projections of speed to axes X (on the left) and Y (on the right) from measurement time for this type of blow are presented. From the left graph, it is visible that in an initial phase there is a swing on axes X (back motion), both the left hand and right hand, but swing by the left hand is more intensive and more long-lived.
In the following phase a set of the maximum speed on an abscissa axis of the right hand occurs quicker, than left hand, the maximum value of a projection of speed to axis X is approximately identical to both hands. Further, the breaking phase before crossing the zero line which comes to an end earlier at a blow with the right hand follows. The short phase of back motion comes to an end with a transition approximately at the same time for both hands (most left hand crossing the zero line) in the last phase of the punch. The  final projection of speed to axis X for the left hand is more than the same size for the right hand. Analyzing schedules for a speed projection to an axis of ordinates we will note swing existence (upstroke) both the right and the left hand. Swing on this axis of the right hand is more intensive and lasts longer than swing by the left hand. In the following phase of a set of the maximum speed down the maximum speed, the left hand is more, than the maximum speed of the right hand (minima about the 50th measurement), but it is reached a little later. Further, the phase of decrease of vertical speed, and right hand follows, having slowed down driving down, even begins to move up (the second crossing of the zero line of the graph), but then moves down, finishing with a small vertical speed. The repeated swing was made by the right hand. The left hand, having slowed down vertical driving, subsequently accelerated, having finished with a good vertical speed. Summarizing this type of punch by the athlete, we will note that this punch by the left hand turns out more intensive, than the right hand one.

D. KINEMATIC ANALYSIS OF URAKEN
In Figures 24, 25, dependences of projections of speed to axes X (on the left) and Z (on the right) from measurement time for this type of punch are presented. Let's consider dependences of a projection of speed to axis X. It is possible to note that on this axis there is no swing, advance is made in the beginning and after the achievement of the maximum speed there is a delay, and further short the driving site back after crossing the zero line. The maximum speed along this axis is more at a blow by the right hand. Analyzing the change of a projection of speed to axis Z, it is easy to notice that in an initial phase the swing which is more intensive for the left hand becomes. Then the driving counteracting the attack of the opponent the same hand becomes, and the maximum speed for the left hand is more than for right one, after delaying along this axis the whipping driving in inverse, from within outside, the direction is made. And the whipping driving of right is insignificant. Further, the hand begins driving inside and to itself.
Summing up the result of the analysis of the considered punches made by the athlete, we will note that in the presence of swing in an initial phase the maximum speed of blow of subjects is more the swing is more. VOLUME 9, 2021 In work [13] the dependence of the speed of blow on the experience of the athlete is established, but the speed difference for different hands isn't established. In our work, thanks to the separate analysis of the kinematics of each hand the difference in speeds for different hands is established. This difference is caused by the difference in technology of realization of blow-by each hand. In work [14] differences were found in the technology of realization of blows by athletes with different experiences. In our research, the dependence between different phases of realization of blow is established. It can be useful for trainers to improve the technology of realization of punches.  Other authors who conducted research on the recognition and classification of punches did not use such a class as ''movement without punches''.
The multilayer perceptron, despite its simplicity, showed good results. The best F1 score is 0.95 for U-punch, worst is 0.86 for N-punch. In [16], studies were conducted on the recognition of punches of boxers using the MLP. The authors [16] obtained a recognition level of 92.93 ± 4.33% for highly qualified boxers.
The difference between train and validation accuracy is the result of model overfitting. Increasing the number of training samples will solve this problem. 1D convolution model from [57] works very badly and is not suitable for punch recognition. Train accuracy is about 0.8, but validation accuracy is only 0.65 and very unstable. Worst F1 score is 0.11 for MT-punch, best F1 is 0.81 for No-punch class. Loss after 100 epochs training is only about 0.3. As we proposed, 2D convolution model with 2 conv layers works better. Metrics are like on MLP: best F1 is 0.93 for YT-punch and worst is 0.90 for U-punch class. A little gap between training and validation accuracy curves is told about some overfitting, so we tested deeper conv model with 4 layers.
In [18], the movements of fencers were studied. In this work, models were developed that combine the input data received from the IMU and Kinect. Then, the input data was preprocessed based on Dynamic Time Warping (DTW) and Support Vector Machine (SVM). After preprocessing, the data was processed in MLP. The accuracy of the obtained recognition models varied for different types of movements from 87% to 99%.
2D convolution model with 3 conv layer shows best result: 0.97 validation accuracy. Best F1-score 0.99 for YT-punch class, worst 0.90 for AT-punch class. Comparing with MLP, we achieved better classification metrics and shift invariant model, based on 2D convolution.
One of the best results in the classification of punches in boxing were shown by the machine learning algorithms Linear Regression and Support Vector Machines [12]. With it was possible to achieve a multiclass accuracy of 0.96 and an average F1 of about 0.95. In our study, we used convolutional neural networks to classify karate strikes. The best result was shown by a 2D convolutional architecture with three layers, which allowed achieving similar results: 0.96 accuracy and 0.97 average F1. In the work [17], the data obtained using IMU were used. In these studies, IMUs were attached to the wrists of boxers. Various deep learning methods, including MLP, were used to recognize impacts. For the experiment configuration, when IMUs were attached to the boxer's wrists, the accuracy was 0.98.
Unlike [12], we used a larger dataset -1700 samples for training and 212 for testing. We used only the results of measuring linear acceleration along the x, y, z axes, did not use measurements of angular acceleration and magnetic field and did not calculate the angles of the sensor position. We used raw data without statistical processing -we did not calculate the mean, standard deviation, min max, etc.

V. CONCLUSION
The research purpose included the study of kinematics and the development of various neural network models of such karate punches as Yun Tsuki, Mawashi Tsuki, Age Tsuki, Uraken. The kinematics of the Uraken has still been little studied since this punch is not used in all martial arts. Also, a type of classification was introduced -''movement without punches,'' which is also not found in studies, although this approach will help distinguish punches from feints and movements in the future.
We believe that our proposed approach is more suitable for practical implementation and implementation in the final product. The results of studies of punches' kinematics will help deepen the understanding of the biomechanics of karate punches, which will allow scientists to develop the theory of punch, and coaches and athletes to learn the correct technique more effectively. The work of trainers relates to the constant measurement of the parameters of punches; the recognition of punches allows these routine operations to be automated.
Karate punches have complex kinematics and dynamics, so the developed model was limited to the study of single punches. That is, the model did not include combinations of punches, feints, and movements of athletes. Also, the model did not include gyroscope and magnetometer data, which probably limits its applicability and accuracy due to complex impact trajectories. All these limitations are areas of future research. The subject of these studies should answer a lot of topical questions: -development of an optimal, unified punch model (the study should also answer the question: is it possible to create such a model, or can we only create limited models for certain conditions); -development of models for recognizing punches in combinations; -recognition of punch in real sparring, when two athletes interact; -inclusion of input parameters such as gyroscope and magnetometer data into the model; -creating punch models in the joint study of punch video capture and kinematic data received from the IMUs.

ACKNOWLEDGMENT
The author would like to thank athletes, coaches, and the leadership of the He has investigated the influence of software and hardware on the metrological characteristics of measuring systems. Since September 2019, he has been a Researcher with the Department of Communication Systems and Robotic Systems, where he research in the field of artificial intelligence and data analysis and the guidance of students' scientific work (computer vision). Since February 2021, he has been a Senior Educator at the Financial University under the Government of the Russian Federation. His research interests include algorithms and data structures and deep learning. KONSTANTIN BUBLIKOV was graduated from the Faculty of Nonlinear Processes, Saratov State National Research University, in 2014. He is currently a Researcher at the Institute of Electrical Engineering, Slovak Academy of Sciences. He is the author of over 20 articles. His current research interests include electronics, materials research, and robotics. VOLUME 9, 2021