Marker-Less Monitoring Protocol to Analyze Biomechanical Joint Metrics During Pedaling

Marker-less systems are becoming popular to detect a human skeleton in an image automatically. However, these systems have difficulties in tracking points when part of the body is hidden, or there is an artifact that does not belong to the subject (e.g., a bicycle). We present a low-cost tracking system combined with economic force-measurement sensors that allows the calculation of individual joint moments and powers affordable for anybody. The system integrates OpenPose (deep-learning based C++ library to detect human skeletons in an image) in a system of two webcams, to record videos of a cyclist, and seven resistive sensors to measure forces at the pedals and the saddle. OpenPose identifies the skeleton candidate using a convolution neural network. A corrective algorithm was written to automatically detect the hip, knee, ankle, metatarsal and heel points from webcam-recorded motions, which overcomes the limitations of the marker-less system. Then, with the information of external forces, an inverse dynamics analysis is applied in OpenSim to calculate the joint moments and powers at the hip, knee, and ankle joints. The results show that the obtained moments have similar shapes and trends compared to the literature values. Therefore, this represents a low-cost method that could be used to estimate relevant joint kinematics and dynamics, and consequently follow up or improve cycling training plans.


I. INTRODUCTION
A detailed biomechanical joint analysis can be used as a clinical decision-making tool to treat the individual in complex cases, such as during a specific sport training to improve the performance or in clinical evaluation to avoid joint pain. Both experimental kinematics (coordinates, velocities and accelerations) and external forces are required to perform an inverse dynamics analysis and calculate joint moments or powers. The most popular methods to calculate the joint angles are based on tracking the trajectory of joint angles and positions [1] or on measurements with inertial measurement units [2]. External forces between the bicycle and the cyclist, such as pedal and saddle contact forces [3], can be calculated using force sensors.
The associate editor coordinating the review of this manuscript and approving it for publication was Francesco Mercaldo .
There exist just a few instrumented systems to measure pedal forces in the market, e.g., Smartfit PowerForce (Radlabor GmbH, Freiburg, Germany) or I-Crankset (Sensix, Poitiers, France). Most existing systems in the literature are built in a research laboratory by assembling commercial force transducers [4]- [9], which are normally quite expensive. These systems are mainly bicycle specific, which cannot be used indistinctively for road and mountain bikes [10]. Some studies rely on commercial optical cameras [7] to capture the motion, and others use marker-less commercial systems, less expensive than a set of optical cameras, but still relying on commercial software [11], [12].
Marker-less motion tracking systems are promising low-cost methods that are gaining ground in different fields, not only in sports biomechanics [13] but also in other areas such as surgery [14] or neurological rehabilitation [15]. However, there are still some challenges to be faced to obtain better accuracy and robustness [13], especially when there are occlusions on the camera images due to interactions of humans with other equipment (such as a bicycle) [16]. Recently, there have been efforts to combine marker-less motion capture software with machine learning [17]- [20]. But when some parts of the model are hidden or different parts of the model are confused by overlapping and having the same colors, tracking errors can still appear. If algorithms can post-process the data and correct tracking errors, these methods could be used in such situations, like pedaling. Accordingly, the main contribution of this study is the integration of the open-source OpenPose library [18], which can superimpose 2D-skeletons over images of people, with an algorithm able to correct tracking errors, and a measuring system to acquire saddle and pedal forces while pedaling. We present how the error correction strategy works for different trials and a brief analysis of the main biomechanical outcomes (joint angles, moments and powers).

A. MEASURING SYSTEM
The measuring system consists of force resistive sensors to obtain the interaction forces between the cyclist and the bicycle, and two webcams to get the video of the movement. Seven force resistive Flexiforce A201 sensorsmax 445 N -(Tekscan, South Boston, USA), previously calibrated, were used to measure contact forces. Three sensors were placed on the saddle and two at each pedal using wood supports (FIGURE 1, a). These supports were built to ensure that normal forces were transmitted only through the sensors. Two locking bolts were used to prevent the upper part of the support from falling when the subject is not in contact with the pedal (FIGURE 1, b and c). The data of the sensors were acquired using an Arduino DUE and processed in MAT-LAB R2019a. The voltage data were converted to force using polynomial curves from the calibration, and filtered with a lower-pass Butterworth filter at 3 Hz.
Two webcams c922 Pro Stream -HD 720p, 60 Hz -(Logitech, Lausanne, Switzerland) were placed at the lateral sides of the cyclist, capturing the motion of the pelvis and the legs of the subject. The webcam data were synchronized with the Arduino data by the trigger of two LEDs. VideoReader MATLAB tool was used to acquire the pixels of the image at each frame. OpenPose C++ library [18] was used to capture the motion of the subject while pedaling. This library allows the user to obtain the points of 25 human landmarks (mainly joints, FIGURE 2a) by the use of a trained neural network, which relies on OpenCV [17]. For this study, a MEX file was written so that from a matrix of three dimensions containing the RGB colors of each pixel from any image, it returns the locations of the 25 points. However, OpenPose is intended to capture the joints of both legs at the same time, and with the webcams placed on the sides of the cyclist, one leg will always be hidden, which can introduce errors in identifying the legs. To correct this issue, we formulated a corrective algorithm implemented using MATLAB. The MEX file and the MATLAB algorithm are available at https://github.com/gilserrancoli/capture_2Dcycling.

B. SKELETON TRACKING AND RECONSTRUCTION
The algorithm starts processing the first frame of the video, and it shows the tracked joint locations, asking the user to select the hip, knee, ankle, front foot (fifth metatarsal), and heel points (FIGURE 3). At this point, the segment lengths (in pixels) and the angles among foot segments are calculated, and the side of the body (right or left) is automatically identified. Then, from the second frame, the points of all joints are tracked, and only the ones from the selected leg are shown.
During this process, the position of the tracked points could be wrong, since OpenPose could fail to identify one or more points, or to recognize one limb. Therefore, we use the following process (FIGURE 5): 1. Initial identification. First, OpenPose identifies all points of all skeletons found in the image (FIGURE 2a). Only hip, knee, ankle, and foot points (metatarsal and heel), which are in a distance lower than 25 pixels from the previous frame, are taken into account. A robust identification and prediction of the joint positions are applied based on distance consistency and trajectory estimation, as explained in the following. Because of the cyclic nature of the motion we are studying, we consider two different scenarios according to the amount of data available: a. If frame ≤ 20, all five points are predicted, assuming that they are following a spline (FIGURE 4a). b. If frame > 20, all five points are predicted, assuming that they are following an ellipsoid trajectory (FIGURE 4b). This process takes advantage of the fact that based on our framerate video capture, after around 20 frames (0.3s approximately), the cyclist will have performed more than one cycle. Therefore, we assume a cyclic motion, and we approximate the position of points by an ellipsoid trajectory and segment angles by a sinusoid function. After these two scenarios, we can correct or identify the positions and angles of the full motion capture. 2. Point choice. From all candidate points, we found the best option based on the errors of the segment lengths and the predicted angles between these segments. For the worst case, when no skeleton prediction is available, we assume the hip point is the same as in the previous frame. Then, the algorithm computes knee and ankle points using the prediction of the absolute angles of the femur and tibia, and heel and front points are predicted depending on the scenario: spline (if frame ≤ 20, FIGURE 4a) or ellipsoid trajectories (if frame >20, FIGURE 4b). At this stage, we have a tracked value for all the joints. Then, the algorithm uses the length and the predicted values of the angles to validate these new positions.

Corrections based on angles and segment lengths.
If the length of the femur segment differs by more than 20% of its value, the knee point is predicted according to the femur angle, and a new ankle point is predicted based on the tibia angle and the length of the tibia in the first frame. If the error between the predicted and the current value of the absolute angle of the femur is higher than 20 • , the algorithm recalculates the knee point using the predicted femur angle, and the ankle point using the predicted tibia angle. Then, it recalculates the heel and front points, assuming that they follow an ellipsoid trajectory. If instead, the error between the predicted and the current value of the absolute angle of the tibia is higher than 20 • , it recalculates the ankle point using the predicted tibia angle and the points at the foot are recalculated accordingly. In case the tibia angle difference between two consecutive frames is higher than 20 • , the ankle point is predicted based on the cyclic trajectory (FIGURE 4a or b). Most critical points are the points at the metatarsals and the heel. Therefore, if the metatarsal or the heel points are not found, the missing point is estimated based on the evolution of the angle of the foot sole. If neither heel nor metatarsal points are found, these points are estimated based on the idea that the distances between pair points of the foot (ankle-heel, ankle-front, front-heel) are kept constant. If the distance between the ankle and the front foot is higher than 40% of the original value, front and heel points are predicted using the evolution of the angle of the foot sole. If the distance between the ankle and the heel points is higher than 40% of the original value, front and heel points are also predicted using the angle of the foot sole. If this angle is higher than 50 • (unlikely when pedaling), we also correct its value using the appropriate prediction and the fact that the distances of the foot segments are kept constant.

C. INVERSE KINEMATICS AND DYNAMICS
Experimental lengths of the segments were used to scale a generic OpenSim [21] model of the cyclist, with 6 degrees of freedom -dofs -(right and left hip, knee, and ankle joints). The OpenSim model contains the masses and moments of inertia of all segments of the skeleton (FIGURE 2b). Relative angles between segments were calculated from the trajectory of the joints, previously low-pass filtered at 3 Hz (3rd order Butterworth filter). The angle curves were parameterized with bsplines (10 knots per second) and derived to obtain the angular velocities and accelerations. The torso was considered to remain in a vertical position. An inverse dynamics analysis was applied in OpenSim to calculate the joint moments from the kinematics (joint angles) and the external forces (normal forces at the seat and pedals). Basically, joint moments (τ joint ) are obtained from the following relation (equation of motion): where M is the mass matrix, C and G are the vectors corresponding to the centrifugal and gravity terms computed by OpenSim from the joint coordinates q, velocitiesq and accelerationsq; τ cont are the known generalized forces due to the contacts (pedals and saddle) and τ joint are the joint moments at the hip, knee and ankle.

D. SYSTEM EVALUATION
To assess the performance of the algorithm, we processed the videos of five subjects (gender: three men and two women, mass: 64.2 ± 9.3 kg, height: 169.8 ± 2.2 cm) pedaling on a bicycle roller (In Ride 100, B'Twin, Lille, France) at three different pedaling velocities: 111.2 ± 15.0 rpm (high velocity, HV), 71.6 ± 3.6 rpm (self-selected velocity, SV) and 40.0 ± 4.7 rpm (low velocity, LV). They were asked to cycle for about ten cycles at each velocity and with two different resistances (level 1 minimum -LR -and level 7 maximum -HR -of the bicycle roller). For each trial, we evaluated the number and the type of corrections that the algorithm performed per 100 frames.
To assess the applicability of the method, i.e., the calculation of joint moments and powers, we acquired and analyzed the data for six different subjects (gender: four men and two women, mass: 71.2 ± 16.9 kg, height: 171.3 ± 6.6 cm) at the two levels of resistance mentioned above while pedaling at 76.3 ± 1.9 rpm. The data were split into crank cycles and averaged over all data (three cycles per subject). The zero angle was considered when the pedal is at the highest point of its circular trajectory (top dead center).
To assess the accuracy of the measured positions and joint angles, we compared the kinematics results of six cycling trials (at HV, SV and LV, and HR and LR) with the ones obtained with a manual point identification for two individuals (subject 1: mass 80 kg, height 180 cm, subject 2: mass 60 kg, height 168 cm). Root mean square differences (RMSD) are reported for the point trajectories and the joint angles.

III. RESULTS
The results of the motion tracking show that OpenPose overall worked better at the lowest velocities (LV in FIGURE 6) since the algorithm had to correct the position of the landmark points a lower number of times. The algorithm had to adjust the wrong positions mainly at foot points (metatarsals and heel). The types of errors with most occurrences were due to the errors tracking the front foot points  (metatarsals) and due to the distance errors between the ankle and the metatarsal points, with values of 7.42 ± 9.92 and 7.21 ± 9.82 occurrences per 100 iterations respectively.
The most distal points (front foot points), which overall also have the widest range of motion, had the highest position errors (FIGURE 7). Mean front foot RMSD values were 1.9 cm and 3.0 cm for Subject 1 and 2, respectively, whereas the mean hip RMSD values were 1.0 cm and 1.2 cm. The angles at the most distal joints (ankles) also were the ones with the highest RMSD values (mean values of 6.1 • for both right and left ankle angles in Subject 1, and 11.4 • and 11.3 • for right and left ankle angles, respectively, in Subject 2).
Kinematics and dynamics results are consistent in terms of the biomechanics function of the anatomical joints (FIGURE 9). For example, in terms of joint kinematics, the leg has the maximum extension around the bottom dead center and the maximum flexion around the top dead center. In terms of joint dynamics, the knee moment increases in the first half of the cycle (when the pedal rotates from the top dead center, i.e., when the knee has the highest flexion, to the bottom dead center, i.e. when the knee is around the maximum extension). The knee peak moments are at 38 and 43% of the cycle for the high and low resistance, respectively, and the ankle peak moments are at 33 and 38% of the cycle for high and low resistance, respectively. The maximum hip power is around the 3 o'clock orientation of the pedals. Knee power oscillates between negative and positive values, having the closest values to zero when the pedal is in the vertical position. Ankle power is overall positive during the first half of the cycle and decreases to negative values in the second half.
The power exerted by the pedals (FIGURE 8) reaches its peak during the first half of the cycle (between the 3 o'clock orientation and the bottom dead center of the pedals). The values of the peaks are 145.4 W and 193.7 W for the low and high resistance trials, respectively.
In terms of computational speed, the evaluation of the MEX function to extract the position of the landmarks in one frame takes 14.1 s in a regular laptop (Intel Core i7-6700HQ @2.6 GHz).

IV. DISCUSSION
The pedaling measuring system presented in this study represents a low-cost option to measure joint moments and joint power. The hardware mainly consists of only two webcams, seven force resistive sensors, and one Arduino DUE microcontroller to acquire the data. The algorithm relies on the OpenPose C++ library to obtain the landmarks of the body and on custom MATLAB code to face up the possible confusions that could appear when identifying the points. The routine can predict the positions of the points based on the trajectory of previous frames (hip, knee, ankle, front foot, and heel) and can correct the positions if needed. As expected, the results showed that the probability of missing a landmark point increases with the pedaling velocity.
The developed motion capture algorithm can correct the errors of the point positions that OpenPose can have, especially due to the confusion produced by having one leg hidden by the other (since both legs are in parallel planes to the camera) or by light issues. We provide the code able to correct the positions when these issues appear. Additionally, in contrast to other software [13], the code is available on an open-source basis. This algorithm (described in the Methods section) is based on the knowledge that: a) the segment lengths will remain constant throughout the cycle; b) the shape of the foot (angles between the pairs of vectors ankle-metatarsals, ankle-heel and metatarsals-heel) will also remain constant; and c) the time evolution of the trajectory of the points will be continuous.
One of the main benefits of the system is that it allows performing a complete dynamics analysis (including upper extremities if required), since it measures forces at the pedals and saddle, unlike other systems presented in the literature, which measure only forces at the pedals [5], [8], [22]. The instrumentation used is also more affordable than in other studies [3]. Another advantage of this method is that it can be installed on any bicycle. Once the sensors are calibrated, and the bicycle is instrumented, it requires minimal effort by the user to extract the joint kinematics and dynamics.
The accuracy of the kinematics results calculated as the RMSD between the measured data and the obtained with manual trajectorization (mean RMSD < 3 • for hip, < 5 • for knee and <11.5 • for ankle) was comparable to other studies. Castelli et al. [23] obtained the highest error at the hip when analyzing 2D gait kinematics with a silhouette tracking algorithm (mean RMSD = 6.1 • ). Ceseracciu et al. [24] reported overall higher values for mean RMSD during gait (hip 17.6 • , knee 11.8 • and ankle 7.2 • ). Corazza et al. [25] reported lower values than in our study (< 4 • ); however the computation time was of the order of hours.
The example of kinematics and dynamics analyses presented in this study is consistent with the literature, in terms of joint angles, moments [9], [11], [12], [26] and powers [6], [8], [27]. For example, in terms of joint angles, the magnitude and shape of hip and knee angles and the shape of the ankle angle are similar to Martin and Nichols [28], reproducing the typical joint angle pattern observed during pedaling. In terms of joint dynamics, the hip moment has a similar pattern as in other studies [12], [29], with the maximum extensor moment around 3 o'clock orientation of the pedal. Knee moment starts the cycle being in extension and it decreases during the first half (around 20% of the cycle). Its maximum is around half of the cycle (to get the maximum impulse against the pedal) and then it increases again until extension at the end of the cycle. The magnitude of the moment is more similar to [12] than [9], [11], [26], due to the chosen resistance. Ankle moment remains in plantarflexion throughout the cycle, being the maximum just before the half of the cycle for the same reason.
The pattern of hip powers are similar to the ones from the literature. Hip power is mainly positive during the first half, showing the maximum peak at 3 o'clock orientation of the pedals. The pattern is similar to Martin and Nichols [28] and McDaniel et al. [27], though our peak powers are 165 W and 193 W for lower and high resistances while in those references are between 400 and 1000 W. Knee power in the literature shows two positive peaks, one around 25% and the other around 75% of the cycle. In our case, the first peak is much lower than the second. The shape of the ankle power is also similar to the literature, having the positive peak around 90 • (at the propulsive phase [30]), a slightly later in our case. There are differences in terms of magnitude, since our peaks are lower than 100 W both for the knee and the ankle, whereas it reaches 600 W and 400 W for the knee and ankle respectively in [8]. These differences can be explained due to the variations in the angular velocities (around 75 rpm in our case versus 120 rpm in [8]) and also in the resistance, measured indirectly by the crank power. In our case, the crank power did not reach 300 W, whereas in [8], it reached 1600 W.
The developed system has some limitations. It is a marker-less system, and these systems currently still have some inherent limitations [14], [15]. It is estimated that the accuracy of the results given by OpenPose can be lower than 30 mm or less in case the algorithm corrects the tracking errors [31]. The maximum sample frequency of the webcams used in our experiments is 60 Hz, which could be a limitation when recording high speeds (> 120 rpm). In that case, a camera able to record at a higher frequency should be used. In terms of measured forces, we neglected tangential pedal forces, which means we lose information to calculate joint moments and powers. However, the feet were not attached to the pedals (therefore, the tangential forces will not be as high as compared to when using clips), and according to literature values [9], tangential forces only represent around 10% of the normal forces at the peak value. In the presented prototype, we did not have sensors at the handlebar, as in [3]; therefore, the subject cannot use the handlebar. For this reason, in all our measurements, the subjects were in a straight torso pose, without having any external forces apart from those at the saddle and pedals. Nevertheless, if the goal was to capture the dynamics of a professional sport cyclist, for example, and handlebar forces were available, these forces could be appended to the dynamics analysis. In terms of computational efficiency, the evaluation of the OpenPose MEX function takes 14.1 s; however, as mentioned in the literature [18], OpenPose could work in real-time when executed on a graphical processing unit (GPU).
In conclusion, the integration of the marker-less capture system, the algorithm to correct the tracking errors, and the measurement of the external contact forces provide a low-cost system to obtain joint moments and powers of each joint of the body. This system could be used to do a follow-up of the joint loads while pedaling for clinical evaluations, monitor treatment outcomes or sports training. GIL   His current research interests include biomechanical analysis, signal processing, machine learning, and advanced manufacturing processes and systems.
JORDI TORNER RIBÉ received the Ph.D. degree in multimedia engineering from the Universitat Politècnica de Catalunya, Barcelona, Spain. He has worked in companies, such as Bayer AG and TNS Sofres. He has taught with the ICE-Institute of Educational Sciences. He is currently an Associate Professor with the Universitat Politècnica de Catalunya. He is also the Director of the Multimedia Applications Laboratory (LAM), Universitat Politècnica de Catalunya. He has participated or participates in several international higher education projects and research projects funded in national and international competitive tenders. He has participated in several master's and postgraduate courses. His research interests include virtual reality, engineering design, web design and programming, biomedical and rehabilitation apps, multimedia, and e-learning. He is a member of the UNESCO Chair of Higher Education Management, Universitat Politècnica de Catalunya. VOLUME 8, 2020 SAMIR KANAAN-IZQUIERDO received the Ph.D. degree in biomedical engineering from the Universitat Politècnica de Catalunya, in 2017. He is currently a Full Time Lecturer with the Universitat Politècnica de Catalunya. He has participated in ten national and European research projects. His research interests include machine learning, especially in multiview methods, neural networks and their applications to engineering, and biomedical problems.
ANTONI SUSÍN received the Ph.D. degree in applied mathematics from the University of Barcelona, in 1993. He has been a Visiting Professor with the University of Minnesota, Minneapolis, in 1992, Universidad Autónoma Metropolitana, Mexico, in 1995, the University of California, Irvine, in 2002, the University of Zürich, Zürich, in 2010, Addis Ababa University, Ethiopia, in 2012, and Seoul National University, in 2017. He is currently an Associate Professor with the Universitat Politècnica de Catalunya. He is also the Head of the Dynamic Simulation Laboratory, ViRVIG Research Group, Barcelona. His major research interests include numerical methods and physically-based simulation applied to medical images and biomechanics.