Real-Time Detection of Gait Events by Recurrent Neural Networks

This paper proposes a gait detection model that can recognize important gait events in real time. Human gaits are periodic, with each gait cycle containing the important gait events of mid-swing, heel-strike (HS), and toe-off. The correct identification of different gait patterns caused by stroke or progressive neurodegenerative Parkinson’s disease could ensure that the patients receive appropriate treatment and rehabilitation strategies. However, online detection of gait events can be challenging because each person has their own walking patterns and speeds. This paper applies recurrent neural networks (RNNs) to develop a model that can instantly detect important gait events in any subject. We collected clinical gait data and used them to develop an RNN model for real-time detection of HS. The model correctly recognized HS events with an average success rate of 98.84% and an average delay of 0.024 s in the laboratory environment. We then applied the model to three different groups of subjects: healthy elderly subjects, stroke patients, and patients with Parkinson’s disease. The developed RNN model also correctly recognized HS events in all three groups with an average accuracy of more than 99.65%, even though the subjects had very different walking patterns. In the future, the developed gait detection model can be integrated with real-time rehabilitation systems to provide patients with repetitive guidance using clinician-determined cures for enhanced clinical gait training.


I. INTRODUCTION
Human gaits are periodic motions, where a complete gait cycle can be defined as the period from the initial ground contact of the stance leg to the next initial ground contact of the same leg, as shown in Fig. 1. Normally, three important gait events occur in one gait cycle: the mid-swing (MS), heel-strike (HS), and toe-off (TO) events [1], [2]. Correct labeling of these events allows researchers to estimate gait performance and gait abnormalities. For example, healthy subjects tend to spend 60% of the gait cycle in the stance phase (from HS to TO) and 40% in the swing phase (from TO The associate editor coordinating the review of this manuscript and approving it for publication was Laura Celentano . to HS). However, as the global population ages, the number of people suffering from neurodegenerative and orthopedic disorders is anticipated to increase appreciably. Patients with neurological or orthopedic diseases often display gait disturbances that lead to slow walking speeds and increased risk of falls and that ultimately affect the quality of life [3]- [5].
Gait and balance are the most affected motor characteristics in patients with stroke, traumatic brain injury, or Parkinson's disease (PD) [3]- [5]. Stroke or traumatic brain injury leads to a hemiplegic gait characterized by asymmetry and reduced weight-bearing on the paretic limb due to motor weakness and spasticity caused by brain lesions [3], [5]. In patients with PD, the degeneration of dopaminergic neurons in the substantia nigra can lead to other abnormal gait VOLUME 9, 2021 This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/ patterns, such as stooped posture, shuffling gait, freezing of gait, festination, and reduced step length and height [4]. Each of these different patient groups requires specific rehabilitation strategies to improve lower limb motion and restore locomotion function. For instance, rehabilitation approaches for stroke and traumatic brain injury mainly emphasize improving weight-bearing on the affected limb and reducing compensation patterns [6]. By contrast, gait rehabilitation for PD typically aims to increase the step length and improve dynamic posture control to avoid falling, especially when the patient is attempting to turn around [7]. Gait disorders are also common in patients with knee or hip osteoarthritis, a painful degenerative joint disease characterized by a loss of cartilage and changes in bone structure [8]. Increased dynamic joint load, particularly during walking, is a contributing factor to the disease progression of knee or hip osteoarthritis, and gait modification is a conservative strategy frequently used in the clinic to reduce knee or hip joint loads and to overcome pain [9], [10]. For all these disorders, early detection and differentiation of individual gait patterns could assist in ensuring that patients in today's aging society receive appropriate treatments and rehabilitation strategies.
Conventional gait rehabilitation uses hands-on techniques, including symmetrical weight bearing, weight shifting, and strengthening exercises, accompanied by manual and verbal cueing/feedback by the physical therapists to guide movements and facilitate normal gait patterns [6], [7], [11]. Therapists use clinical observation to identify the parameters of gait dysfunction and to determine the most appropriate gait training strategy for each patient. However, this treatment is labor intensive and places high demands on the attention and skill of the treating therapists. The conventional gait rehabilitation strategies also do not provide quantitative information on abnormal gait parameters, and the determination of the intervention strategy for the underlying clinical manifestation may be limited by the therapist's clinical expertise and experience. For these reasons, technology-based gait intervention strategies are important for identifying gait events in real time to provide correct feedback to the patients and to reduce workload and physical strain on the therapists.
Knowledge of gait events is therefore a key point in medical intervention during rehabilitation. Fast and accurate identification of gait phases is critical for achieving efficient control of lower limb motion and enhancing intervention effectiveness. For example, Wang et al. [12] analyzed neurodevelopmental treatment (NDT) processes and found that therapists tended to cue the subject's anterior superior iliac spine (ASIS) when they observed the subject's HS on the opposite side. NDT controls the subjects' joints but lets them intentionally drive their bodies, thereby allowing them to have the feeling of walking. However, traditional NDT training is very labor intensive and time consuming for therapists; therefore, Wang et al. [12], [15] designed a stationary NDT trainer that could automatically repeat the therapists' intervention patterns (i.e., pulling the user's ASIS when detecting the user's HS events on the opposite side).
The NDT intervention is carried out at the key points of the gait; therefore, identifying gait events in real time is important. For this reason, Wang et al. [12], [15] also incorporated a motion capture system to detect the HS. The idea was extended in [16], where a movable NDT trainer provided users with visual feedback as they walked at their preferred speeds. A gait detection algorithm was also developed that could identify the gait events from the signals obtained by inertial measurement units (IMUs). The algorithm could automatically update parameters in real time for different users who might have varying walking patterns and speeds. However, even the few seconds required to update the algorithm parameters could result in degradation of the rehabilitation effects during this period. The aim of the present study was therefore to apply neural networks to develop a gait detection model that could instantly identify important gait events and patterns in real time.
Gait changes can be monitored using optical motion capture systems, such as VZ4000 [17] and VICON [18]. These systems have been frequently applied for measuring motions via optical markers attached to the subjects and high-speed cameras that record the positions of these markers. However, these optical systems can be very expensive, and their measurements are restricted to special locations. Hence, much research has addressed the combined use of IMUs to obtain gait information and artificial intelligence to detect gait events. For example, Williamson and Andrews [19] applied machine learning to detect real-time gait events using accelerometers and achieved an accuracy of 86%. Aminian et al. [20] developed a model to estimate stride length and velocity by integrating the angular velocity obtained from gyroscopes. Bebek et al. [21] developed a personal micro-navigation system that could detect the zero-velocity duration to reset the accumulated integration errors from accelerometers and gyroscopes; the system was capable of achieving an average position error of 4 m at the end of half-hour walks. Norrdine et al. [22] developed a pedestrian navigation system that applied a foot-mounted permanent magnet to detect HS events. Mazilu et al. [23] proposed multi-feature fusion to recognize freezing of gaits and achieved an accuracy of 95% and a mean detection latency of 0.34 s. Dolatabadi et al. [24] combined two machine learning methods and could discriminate normal and pathological gait patterns with an accuracy of 90%. Li et al. [25] used a dynamic time-warping algorithm to distinguish stroke gaits and achieved an accuracy of 94%. Sun et al. [26] further applied the gait acceleration data from wearable IMUs to recognize the identities of elderly users, with an average success rate of 96.7%. Smartphone sensors have now also been applied, as technologies advanced. For instance, Ashraf et al. [27] applied a smartphone to measure the geomagnetic field for indoor localization using Convolutional Neural Networks. Qiu et al. [28] proposed a sensor fusion algorithm that applied quaternion notation and an extended Kalman filter to fuse the inertial and magnetic sensors to distinguish stance phases.
In the present paper, we applied IMUs to collect experimental gait data and then used those data to develop a gait detection model using a recurrent neural network. The model successfully detected the HS with an accuracy of 98.84% and a mean detection latency of 0.6 sample (0.024 s). This model, when applied to three different groups (healthy subjects, stroke patients, and patients with PD), successfully detected the HS in real time with an average accuracy of 99.65% and an average detection latency of 0.69 sample (0.028 s), even though the subject groups had very different walking patterns. Compared to the existing literature, the main contribution of this paper is that the recurrent neural network (RNN) proposed for the development of the models can successfully detect gait events in real time with high accuracy, even in subjects with very different walking patterns.
This paper is organized as follows: Section II introduces the gait detection model. We applied IMUs to collect gait data and used these experimental data to develop an RNN model that can identify HS events in real time.
In Section III, we apply the developed RNN model to the three different subject groups. The results confirm that the proposed model can successfully recognize the HS event, even though the subjects have varying walking patterns. Finally, we draw conclusions in Section IV.

II. THE RNN MODEL FOR GAIT DETECTION
We measured the gait data and applied recurrent neural networks (RNNs) to develop a gait detection model that can identify HS in real time. This section describes the development of the DNN model. We selected the HS events because they are the key points in clinical NDT rehabilitation [12], [15], where the therapists tend to cue the subject's pelvis when the subject's foot touches the ground on the opposite side.
Developing a robust model that can correctly identify the HS in real time for all users is challenging, because different persons have various walking patterns and speeds. Therefore, we developed an RNN model to detect HS events. First, we recruited five healthy subjects; their data are illustrated in Table 1. We applied the APDM OPAL system [29] with wearable IMUs to detect the subjects' kinematic data with a sampling rate of 25Hz. Two IMUs were implemented on their legs, as shown in Fig. 2, and a rehab gaiter with a 2 kg mass was applied to limit the subject's joint movement on one knee to increase the gait varieties. Each subject was asked to walk down a 20 m corridor four times. The IMUs recorded three axial accelerations and three axial angular   velocities for each leg. The IMU data of these subjects are shown in the Appendix. As an example, subject H2's angular velocity on the sagittal plane is shown in Fig. 3, where the following gait events can be distinguished [20], [30], [31]: the MS usually takes place when the angular velocity reaches its global maximum in one gait cycle, the HS usually happens with the first negative peak after the MS, and the TO usually occurs with a negative peak before the next MS. In addition, the constrained leg tends to show more abnormal trembling and vibration compared to the unconstrained leg.
We applied the measured gait data to build an RNN model for instant identification of a HS event. As shown in Fig. 4, the model was developed in two stages: the training stage and the validation stage. In the training stage, we applied the six-axial IMU data and marked the HS events to train an RNN model. In the validation part, we applied the trained model to detect the HS events in real time and evaluated the model performance.  Because each person has different walking patterns and walks at varying speeds, we first normalized the measured IMU data with a mean of zero and a deviation of one. We then segmented the normalized data with a sliding window consisting of fifty samples of six-axial data as the model input: as illustrated in Fig. 5, where the window was moved by one sample each time, with an overlap of 98%. The output was represented as follows: where each element was marked as 1 for a HS or 0 for a non-HS event. RNN is suitable for predicting gait events because human gaits are regular and periodic. We applied a special form of RNN, called long-short-term memory (LSTM), to construct the gait detection model. Compared to the traditional RNN, the LSTM uses long-term memory to enhance confidence in the output. The model structure is shown in Fig. 5, in which the model input X ∈ R 50×6 is obtained from the IMU sensors and is used to derive the output Y at each step. The output Y is then sent to a fully-connected layer W to produce the layer output Z ∈ R 50×1 . The sigmoid function is then applied to calculate the prediction output P = σ (Z ) ∈ R 50×1 .
In the training process, the HS events are marked offline, with P(50) = 1, to train the model parameters. In the validation process, we applied the trained model to estimate whether the model can correctly recognize HS events in real time. We gave an input data X ∈ R 50×6 to the model with a sampling rate of 25 Hz; the current gait was labeled as a HS if P(50) was greater than a threshold value. We then compared the results with the HS events marked offline to evaluate the model performance.
We applied binary cross-entropy as a loss function [32] because cross-entropy can quantify the similarity between the model output and the label events. In addition, simultaneously applying the cross-entropy as the loss function and the sigmoid function as the activation function to the output layer can avoid learning rate decreases in the gradient descent [33]. We also selected Rmsprop [34] as the optimizer of the RNN model because it was designed specifically for training RNNs and could solve the problem of early termination in deep learning.
The RNN model was trained and validated using leaveone-out cross validation (LOOCV). Specifically, we built five RNN models, where each model was trained by four subjects' gait data and then verified using the remaining subject's data. In the training process, we selected 500 samples (i.e., batch size = 500) each time to train the model to update weights, and we repeated the training sixty times (epochs = 60). We also avoided overfitting by adding a dropout rate of 0.2 to the LSTM layer to give each neuron a 20% probability of being deleted.
We applied two performance indexes, the Positive Predictive Value (PPV) and the Mean Absolute Error (MAE), to evaluate the model performance. The PPV is defined as follows: where N is the number of total steps and n represents the number of HS events detected by the model. The MAE is defined as the time intervals between the HS detected by the model and the HS marked offline. The validation results of the RNN models are illustrated in Table 2. Each model was given more than three thousand samples that included 130-199 HS events. As shown in Table 2, all models were able to recognize the HS events. The average PPV was 98.69% and 98.99% on the constrained and the unconstrained sides, respectively. The average MAE was 0.60 sample (0.024 s) and 0.58 sample (0.0232 s) at the constrained and the unconstrained sides, respectively. Based on these results, the proposed RNN model was deemed successful at detecting HS events in real time.

III. MODEL TESTS
The performance of the developed RNN models was tested by three groups of subjects that differed from the participants in the model training and validation processes. The information about these subjects is listed in Table 3, where E1-E5 are elderly healthy subjects, S1-S5 are stroke patients, and PD1-PD5 are patients with PD. All test subjects signed informed consent forms approved by the Institutional Review Board of National Taiwan University Hospital (IRB number: 202012017RINA, 201510EM011) before joining the study. The IMU data for these subjects are shown in the Appendix. The subjects' angular velocity responses on the sagittal plane are also shown in Fig. 6-9, where the star symbols indicate the HS events detected by the RNN model. Note that these subjects have very different walking patterns. For example, healthy subjects have smooth and regular gait cycles, but show some irregular responses on the constrained side. The elderly healthy subjects tend to have slow walking speeds, while a stroke patient's paretic leg might show more abnormal trembles and vibrations than the leg on the   non-paretic side. The patients with PD therefore showed gait cycles characterized by trembles and vibrations, and they had shorter swing phases because of a festinating gait. This VOLUME 9, 2021 FIGURE 6. Angular velocities and labeled HS of the healthy subjects with a rehab gaiter. caused a shorter swing phase but a faster speed and easily led to falling forward.
The detection results are illustrated in Tables 4-6. Table 4 shows that the developed RNN models perfectly detect the HS events of the healthy elderly subjects in real time with an average latency of 0.75 sample (0.03 s). Table 5 shows that the RNN models perfectly recognize the HS events of the stroke subjects' non-paretic sides in real time, with an average latency of 0.78 sample (0.0312 s). By contrast, on the paretic side, the average detection accuracy is slightly reduced to 98.23%, with an average latency of 0.89 sample (0.0356 s). This difference reflects the fact that stroke survivors usually have a decreased stance phase and a prolonged swing phase on the paretic side [3]. At foot contact, the correct HS is also often lost on the paretic side due to a decreased strength of control of the ankle dorsiflexor muscles or to spasticity of the calf muscles; this is referred to as foot drop at initial contact. This abnormal gait can lead to a toe-to-heel gait pattern and cause compensatory movement patterns. Table 6 shows that the RNN model can perfectly detect the HS events of the PD subjects' affected sides in real time, with an average latency of 0.62 sample (0.0248 s). On the dominant side, the average detection accuracy is slightly reduced to 99.69%, with an average latency of 0.35 sample (0.014 s). Although the basic gait patterns are preserved in patients with PD, these patients usually demonstrate a reduced stride length and walking velocity during ambulation. In advanced stages of the disease, patients with PD tend to strike the ground simultaneously with the heel and the forefoot, resulting in a flat foot strike [35].
These tests confirmed that the developed RNN models are able to correctly recognize the HS events of different subjects who might have very different walking patterns. The correct detection of the initial contact events is of vital importance for designing the most suitable gait training programs for different subject groups.

IV. CONCLUSION
This paper proposed a gait detection model that can recognize HS events in real time. Identification of gait events can help clinicians to evaluate gait performance and to make decisions about medication and rehabilitation strategies. We recognized that online detection of gait events can be challenging; therefore, we applied IMUs to collect experimental gait data and applied those data to develop an RNN model that can recognize HS events in real time. We applied the LOOCV method and showed that the RNN models can detect HS events in real time with an average accuracy of 98.84% and an average latency of 0.024 s. We also applied the model to three groups of subjects with quite different gaits: healthy elderly subjects, stroke patients, and patients with PD. Our results verified that the RNN models can successfully detect HS events with an average accuracy of more than 99.65% and an average delay of 0.028 s, even in subjects with very different walking patterns. In the future, the developed model can be extended to recognize other gait events, such as MS and TO. This real-time gait detection model can be integrated with rehabilitation strategies to provide patients with cliniciandetermined cues regarding gait performance, thereby delivering a greater number of skilled repetitions during gait training. In this paper, we applied the commercial OPAL IMU system, which has high sampling rates and resolutions. The smartphone IMU system might also be compatible in the future, as technologies advance. Further studies that apply this technology are recommended to enhance clinical gait training in selected neurological and orthopedic populations.

APPENDIX
The gait data in this paper is available at: http://140.112.14.7/~sic/PaperMaterial/Gait_RNN_2021 .zip. In the Ph.D. degree, she investigated the molecular mechanisms of LRRK2 mutations in neurons by using a Drosophila model systems. She received the neurological residency training from the National Taiwan University Hospital (NTUH). She received a Postdoctoral Fellowship training from the Center for Applied Neurogenetics, The University of British Columbia, Canada. She is currently a Clinical Professor with the Department of Neurology, National Taiwan University Hospital. Her main academic interests include genetic and molecular biology studies of Parkinson's disease and related neurodegenerative disorders. She investigated the molecular mechanisms of mutations of Parkinson's disease causative genes in neuronal degeneration by using cellular and animal model systems. She is also investigating the gut-brain axis mechanism in the pathogenesis of neurodegeneration. She is an Executive Member of Taiwan Movement Disorders Society, a Representative Member of the Leadership Program of the International Movement Disorder Society. She is also a Committee Member of the Educational Committee of the MDS-AOS Section, and an Active Member of Evidence-Based Medicine (EMB) and Basic Neuroscience Committees of the International Movement Disorders Society. VOLUME 9, 2021