Continuous A-Mode Ultrasound-Based Prediction of Transfemoral Amputee Prosthesis Kinematics Across Different Ambulation Tasks

Objective: Volitional control systems for powered prostheses require the detection of user intent to operate in real life scenarios. Ambulation mode classification has been proposed to address this issue. However, these approaches introduce discrete labels to the otherwise continuous task that is ambulation. An alternative approach is to provide users with direct, voluntary control of the powered prosthesis motion. Surface electromyography (EMG) sensors have been proposed for this task, but poor signal-to-noise ratios and crosstalk from neighboring muscles limit performance. B-mode ultrasound can address some of these issues at the cost of reduced clinical viability due to the substantial increase in size, weight, and cost. Thus, there is an unmet need for a lightweight, portable neural system that can effectively detect the movement intention of individuals with lower-limb amputation. Methods: In this study, we show that a small and lightweight A-mode ultrasound system can continuously predict prosthesis joint kinematics in seven individuals with transfemoral amputation across different ambulation tasks. Features from the A-mode ultrasound signals were mapped to the user's prosthesis kinematics via an artificial neural network. Results: Predictions on testing ambulation circuit trials resulted in a mean normalized RMSE across different ambulation modes of 8.7 ± 3.1%, 4.6 ± 2.5%, 7.2 ± 1.8%, and 4.6 ± 2.4% for knee position, knee velocity, ankle position, and ankle velocity, respectively. Conclusion and Significance: This study lays the foundation for future applications of A-mode ultrasound for volitional control of powered prostheses during a variety of daily ambulation tasks.


I. INTRODUCTION
R OBOTIC prostheses have the potential to overcome the limitations of passive devices [1], [2] by replicating the kinetic and kinematic motion of the missing biological limb using their embedded actuators [3], [4], [5].In laboratory settings, these powered prostheses have shown the ability to improve ambulation [6], [7], [8] and to enable activities that are difficult or impossible with passive devices, such as crossing over obstacles [9] or ascending stairs with a natural gait pattern [10], [11], [12].However, to be viable in real life, powered prostheses must be able to promptly adapt to a changing environment and the user's intent to move.
Researchers have proposed using high-level control strategies aiming to classify the user's intended ambulation mode online.These classifiers are typically based on onboard mechanical sensors [13], [14], such as inertial measurement units (IMU) and load cells, or a combination of mechanical sensors and neuromuscular signals, such as electromyography [15], [16] and sonomyography [17], [18].The application of vision and depth sensing have also resulted in improved environment and ambulation mode classification [19], [20], [21].Although these studies show high levels of accuracy, classification is not perfect.A misclassification of the intended ambulation mode can be dangerous for the user, as even a single misinterpreted stride could lead to a motion of the prosthesis that is far from the desired trajectory that would allow the user to safely negotiate their environment.Therefore, there are still open questions related to the use of classification-based controllers in real life.
Another open challenge in powered prosthetics is the adaptation to the environment and the user intent within a defined ambulation mode.Controllers specific to an ambulation mode define the desired torque [22], [23], impedance [4], [12], or position [9], [24] of the powered prosthesis at the joint level by reacting to the user's movements.Some of these activityspecific controllers break down the gait cycle using a discrete number of phases [4], [22], [25], others use a continuous phase evolution [24], [26].A few of these controllers have some intrinsic adaptation capabilities.For example, researches have shown continuous adaptation to walking speed/cadence [22], [24], [27], foot clearance [9], incline [12], [28], and step height [10].Although these methods provide some adaptation to the environment and the user, there is not yet a direct control system that works seamlessly across a variety of ambulation modes.
Researchers have proposed optimization approaches using the residual limb motion to enable unified controllers [9], [10].Even if successful, relying on mechanical sensors and residual limb motion alone will still not provide users with direct volitional control.This issue can be addressed by adding a sensing modality that monitors the user's neuromuscular signals, which directly tracks the user's intention to move.Volitional control using EMG has been demonstrated in non-weightbearing settings [29], [30] and in weightbearing settings within a specific ambulation mode [31], [32].Moreover, one recent study has shown that shared EMG control enables seamless, voluntary transition between different ambulation modes, including walking, squats, lunges, and sit-to-stand transitions under various loads and conditions [33].Despite the promising results of EMG control for powered prostheses, this approach is limited by a poor signal-to-noise ratio and a lack of muscle specificity [34].A better neuromuscular sensing modality could improve the performance of volitional controllers for powered prostheses.
Muscle ultrasound, also known as sonomyography, is an alternative sensing method that can track neuromuscular signals while providing better muscle specificity and additional muscle activation information from deep-seated muscles compared to using EMG.Comparative studies have demonstrated improvement in ambulation mode classification [35] and robustness against muscle fatigue during hand gesture recognition [36].Two types of ultrasound modes have been used in lower-limb studies: B-mode and A-mode ultrasound.B-mode ultrasound generates a 2D sonomyography image revealing the underlying muscle bellies.In lower limb studies with able-bodied subjects, changes in these ultrasound images have been used for the continuous classification of ambulation modes [35], and the estimation of lower-limb kinematics [37], [38] and kinetics [39].B-mode ultrasound-based measurements of muscle fatigue [40] and muscle force [41] have also been incorporated into the control of lower-limb hybrid exoskeletons [42].Furthermore, 3D imaging techniques utilizing B-mode ultrasound have proven successful in accurately detecting joint position [43], [44].
A-mode ultrasound returns a 1D array, representative of echogenicity along depth [45].It is an appealing solution to achieve volitional control in powered prostheses due to its lightweight and compact design.In transradial amputees, Amode ultrasound has shown success in gesture recognition and wrist rotation estimation [46].Among transfemoral amputees, A-mode ultrasound has been used for continuous ambulation mode classification [17] and for kinematic prediction during level ground walking [18].These prior works illustrate the promise of the A-mode ultrasound sensing modality, but in order to translate the technology to everyday use applications, it must also be able to recognize the user's intent across a range of activities.
In this study, we assess, for the first time, the capability of A-mode ultrasound sensing to predict the prosthesis kinematics of individuals with transfemoral amputation across different ambulation modes.Specifically, we tested the hypothesis that the prosthesis joint position and velocity can be continuously estimated through A-mode ultrasound throughout different types of activities.This hypothesis was tested by recording residual limb sonomyography from seven transfemoral amputee subjects as they completed a circuit that included static tasks, such as quiet sitting and standing, as well as dynamic tasks, such as sit-to-stand transitions, level and inclined walking, turning, and stair climbing.By attempting to predict the prosthesis kinematics across various ambulation modes and transitions, this study provides the foundation for future studies using A-mode ultrasounds for volitional control of powered prostheses in real life.

A. A-Mode Ultrasound System
A portable A-mode ultrasound system (Fig. 1(a)) with 4 transducers was used for this study [45].The system recorded the ultrasound signals at 80 Hz, reading between the 4 transducers sequentially to reduce crosstalk.Each signal transmitted by the system consists of a set of 997 datapoints corresponding to tissue echogenicity at varying penetration depths of the ultrasound.Given a total penetration depth of 3.94 cm, each datapoint corresponds approximately .04 mm of tissue depth.Due to alternating between the 4 transducers every 12.5 ms, 50 ms are required to update the information from all four channels.Custom 3D printed thermoplastic polyurethane (TPU) sensor holders prevented the sensors from tilting and moving, minimized disruptions to socket stability, and minimized socket discomfort during use.A 3-cell lithium-ion battery powered the A-mode ultrasound system.A 3D printed case contained the electronics and battery and was strapped to the ipsilateral side of the subject.The total weight of the system, battery, and case was 440 g.

B. Experimental Setup
Seven transfemoral amputees participated in this study.The sex, age, and socket type of the subjects are reported in Table I.The study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board of The University of Utah (Protocol #00103197, approved 06/16/2021).Informed consent was obtained from all subjects involved in the study.Study participants consented in writing to the use of photos and videos of the experiment.
At the beginning of the experimental session, we placed the ultrasound transducers on the user's residual limb while they were seated with their socket and passive prosthesis off.Two sensors were located on the anterior side of the residual limb to target the quadriceps and two sensors were placed on the posterior side of the thigh to target the hamstrings (Fig. 1(b) and (c)), resulting in each muscle group being sampled at 40 Hz.We positioned the transducers near the muscle belly of the target muscle and then moved the pair of sensors until the peaks of the 1D ultrasound signal were clearly defined and changes in the Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.signal were observed when the user contracted their muscle.We applied ultrasound gel to the determined sensor locations.Kinesiology tape held the sensors and the custom 3D printed sensor holders in place (Fig. 1(b) and (c)).Then, subjects donned their passive prosthesis and walked around the ambulation circuit to ensure comfort.If the user reported physical discomfort or issues with socket suction, we repositioned the sensors, repeating the protocol described above.
After the ultrasound sensor placement was finalized, the subjects donned an IMU-based motion capture system (Xsens MVN, Enschede, Netherlands) and performed the calibration procedure [47].The motion capture system matched the ultrasound system's sampling frequency of 80 Hz.An ethernet cable, long enough to allow for safe ambulation throughout the experimental circuit, transferred the recorded A-mode ultrasound signals to a laptop.A DAQ system (National Instruments USB-6001) served as an intermediary between the laptop and the Xsens motion capture system, synchronizing the ultrasound data and IMU data recording.
After completing the experiment preparation, the subjects performed the data acquisition protocol by walking in an ambulation circuit (Fig. 2).The ambulation circuit included level-ground walking, stair ascent, stair descent, ramp ascent, ramp descent, sitting, standing, sit-to-stand, stand-to-sit, and turning.Each subject walked in the ambulation circuit 20 times, following the path shown in Fig. 2. Each trial started with the subject in a sitting position.Once prompted, subjects would stand up and Fig. 2. Ambulation circuit and subject path through one trial.Subjects started in a sitting position (red dot).They then stood up and walked through the ambulation circuit, which included ascending four steps, descending two steps, and descending a ramp.Subjects then turned around (red x) and proceeded to go through the ambulation circuit in reverse order to end again in a sitting position (red dot).
start walking towards a set of stairs, which they ascended.Then, they walked towards a second set of steps that they descended.From the second set of steps, the subjects walked towards a ramp, which they descended.After the ramp, subjects walked towards the mid-point of the circuit, where they turned around and stood until prompted by the experimenter to continue walking.Once prompted, the subjects went through the ambulation circuit in reverse order.They walked towards the ramp and ascended it.Afterwards, they walked towards the second set of steps to ascend them, turned, and walked towards the first set of steps to descend them.Subjects then walked towards the starting point.Once they reached the chair, they turned around and sat down.Each trial ended when the subject was seated and resting on the Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.chair.The ramp included in the ambulation circuit had a rise of 1 inch for every 12 inches in horizontal distance, which is the steepest slope compatible with the Americans with Disabilities Act Standards for Accessible Design [48].Both sets of steps had a stair height of 7 inches, although the first set included four steps, while the second set only included 2 steps.During each trial, an experimenter clicked a button connected to the DAQ to indicate when the subject transitioned between different ambulation modes.The Xsens system has been shown to perform consistently for over 90 minutes, which is much longer than our acquisition sessions [49].Moreover, the position and heading were zeroed in between trials to minimize any potential time-related drift in our data.

C. Raw Data Processing
We imported the A-mode ultrasound data and the joint kinematics data into MATLAB (Mathworks, Natick, MA, USA) and performed feature reduction as described in our previous lower-limb work [17], [18] and previous upper-limb studies [50] (Fig. 3).First, we rectified the raw ultrasound signal, which the ultrasound system represents as a set of 997 datapoints corresponding to tissue echogenicity at increasing penetration depths.Then, we used a moving average filter, with a window size of 77 datapoints and a step-size of 1, to calculate the envelope of the rectified signal.The envelope of the signal was reduced to 960 by removing the 37 datapoints pertaining to the deeper end of the signal.We then segmented the envelope into 48 windows of 20 datapoints each, with no overlap between neighboring windows.The mean of each window served as a feature for our machine learning input, leading to 48 features per channel.With all 4 channels, we have a total of 192 features for our ultrasound feature set.
The joint kinematics imported from the Xsens motion capture system also included the thigh kinematics of the residual limb.The thigh kinematics, along with the ankle and knee kinematics used as labels, did not go through further processing once they were imported.This included angular thigh position, velocity, and acceleration, whose values were used directly as IMU-based features in comparative models.

D. Machine Learning
We mapped the ultrasound features of each subject to their passive prosthesis joint kinematics through a regression neural network.The feature input included a total of 192 features, which was the combination of the 48 features from 4 A-mode ultrasound channels.The feature set was updated at 80 Hz, 48 features at a time, resulting in a unique feature set corresponding to the kinematics of the leg at each timestep.Although overlapping features within sequential feature sets exist due to the iterative measurement from the four channels, this overlap is contained within individual ambulation circuit trials and do not overlap between training and testing trials.For each subject, individual models were trained for knee position, knee velocity, ankle position, and ankle velocity.A regression neural network made up of three 10-node fully connected layers and three rectified linear unit (ReLU) activation layers was used for each model.With the additional output regression layer, this results in a model with 4290 FLOPs/prediction, which could be adequate for the translation of this technique online.For each subject, we used the first 17 repetitions of the ambulation circuit for training and the last three repetitions for testing.The neural network was trained using ultrasound and prosthesis kinematics data from across the entire ambulation circuit.However, we segmented the data from each lap of the ambulation circuit into 21 sections, with each section corresponding to a different ambulation mode (Fig. 4).The transitions between ambulation modes were recorded during the experiment using a clicker.The ambulation mode labels were not used in our models and were only used for the purpose of evaluating the accuracy of the prediction within each ambulation mode.We used the trained models and the ultrasound features from the testing set to generate predictions for the kinematics of the user's prosthesis (knee position, knee velocity, ankle position, ankle velocity).Training time on a laptop using an Intel(R) Core (TM) i7-8650U processor was 464 ± 69 s across the 7 subjects.Training data varied between 81049 and 120192 frames between subjects, likely due to different ambulation speeds.The average prediction time utilizing the same hardware was 158312 ± 18015 inferences per second.
We developed two additional machine learning models to estimate joint variables as a comparison to the ultrasound-based model.This included an IMU-based model trained solely on ipsilateral thigh kinematics, and a combined model utilizing both the ipsilateral thigh kinematics and the A-mode ultrasound features.Both models used the same machine learning framework as the ultrasound-based model, albeit with a different number of inputs.The inputs to the IMU-based model consisted of the angular thigh position, thigh velocity, and thigh acceleration.While the dimensionality of the IMU feature set is significantly smaller, the IMU-based inputs are heavily processed and can be related with more certainty to the residual limb motion.On the other hand, it is not yet clear how the uncovered ultrasound features relate to the residual limb motion; therefore, a higher dimensionality is appropriate.The combined model used both the ultrasound features and the thigh kinematics values as inputs, resulting in 195 inputs.The predicted variables for these two models are the same as those for the ultrasound-based model and were filtered in the same manner as detailed in the subsequent section.The predictions from these additional models provided a means in which to evaluate our ultrasound-based predictions and discern the advantages and limitations of A-mode ultrasound sensing throughout the different activities.

E. Filtering and Outcome Measures
Before calculating outcome measures, we filtered the resulting predictions from the three predictive models using a 1 st order Butterworth filter with a cutoff frequency of 6 Hz.While filtering does introduce a time-delay that affects prediction accuracy, our previous work utilizing a similar method for kinematic prediction has demonstrated that the introduced delay is suitable for intent recognition [18].After filtering, we calculated the root mean square error (RMSE) of the four predicted joint variables for each ambulation mode separately.We then normalized the RMSE values for each ambulation mode by the range of each joint variable across the entire circuit.We decided to use the range across the entire ambulation circuit, because using the ambulation-specific range would have exaggerated the error of static activities and incorrectly shown a higher accuracy for the dynamic tasks.Mean ambulation mode-specific errors presented in the article were calculated by taking the average of the normalized RMSE values of the four predicted variables within the ambulation mode.Mean variable-specific error values were calculated by taking the mean of the normalized RMSE values across the 10 ambulation modes, as opposed to across the frames of the entire ambulation circuit.We also assessed the relationship between the measured error and the proportion of frames present for each ambulation mode to ensure that the disproportionate representation of each activity throughout our ambulation circuit did not influence our prediction.
To test for statistically significant differences between tested conditions, we performed a two-way ANOVA on RMSE values.Before doing so, we checked the normality of our error values using the Shapiro-Wilk test and a series of Q-Q plots.The ANOVA checked for a main effect due to the ambulation mode (e.g., walking, sitting) and the feature set (e.g., ultrasoundbased, IMU-based, combined), as well as for an interaction effect between these two effects.When appropriate, t-tests with Tukey-Kramer corrections were conducted between the RMSE values of all three models: the ultrasound-based model, the IMU-based model, and the combined model.RMSE values were compared instead of the normalized RMSE values to preserve the independence of observations for the ANOVA.

A. Normalized RMSE
The normalized RMSE between the predicted and recorded joint kinematics provides a term of comparison for the accuracy of the A-mode ultrasound-based models.Fig. 5(a) shows the group mean normalized RMSE and subject-specific mean normalized RMSE across the different ambulation modes.The subject-specific values in Fig. 5(a) are calculated as the average of the normalized error across the four different joint variable predictions.The observed normalized RMSE for the different ambulation modes were as follows: Sitting (5.2 ± 6.0%), Sit-to-Stand (5.9 ± 3.3%), Standing (3.0 ± 2.0%), Stand-to-Sit (7.5 ± 3.0%), Level Walking (6.5 ± 0.6%), Stair Ascent (5.9 ± 2.4%), Stair Descent (8.5 ± 3.1%), Ramp Ascent (7.7 ± 1.0%), Ramp Descent (8.2 ± 0.8%), and Turning (4.5 ± 1.7%).Fig. 5(b) shows the overall mean normalized RMSE and subject-specific results for the different joint variable predictions.In this case, the subject-specific results in Fig. 5(b) are calculated as the average of the normalized error across the 10 different ambulation modes.The overall normalized RMSE for prosthesis knee position, knee velocity, ankle position and ankle velocity were 8.7 ± 3.1%, 4.6 ± 2.5%, 7.2 ± 1.8%, and 4.6 ± 2.4%, respectively.Ambulation-specific normalized RMSE values are reported in Table S1 while the non-normalized RMSE values are reported in Table S2.

B. Kinematics
The shape of the predicted joint trajectory provides a qualitative measure of the proposed prediction method.Fig. 6 showcases one representative test trial that captures the typical prediction for the four different joint variables.The different ambulation modes are indicated using shaded regions in Fig. 6.Because separate models were trained for the four predicted variables, the kinematic predictions of position and velocity are independent-i.e., the predicted joint velocities are not the derivatives of the predicted joint positions.All four joint variable predictions qualitatively match the shape of the measured prosthesis kinematics throughout every section of the ambulation circuit, including transitions.However, some inconsistencies do appear throughout most trials and subjects.During the dynamic sections of the ambulation circuit, the largest source of error comes from the predicted kinematics failing to reach the maxima and minima of the measured kinematics.In contrast, two sources of error appear during static activities and support phases.One is an offset error, where the static pattern is accurately predicted but offset from the measured kinematics.An example of this type of error is shown in Fig. 6 for the ankle position at the initial sitting position, and for knee position at the final sitting position.The second source of error is inaccurately predicting changes in the kinematics when the measured kinematic variables do not change.An example of this type of error is shown in Fig. 6 for knee position in the initial sitting position, both cases of stair ascent, turning, and standing.Knee velocity experiences the same type of error during stair ascent and turning, while ankle position experiences it during the stand-to-sit transition.This pattern is common in all other trials across most subjects.Further examples depicting the predicted kinematics from other trials and subjects are shown in Fig. S1. and Fig. S2.

C. Error and Ambulation Mode Frequency
The ambulation circuit used for this study included 10 different ambulation modes.However, certain ambulation modes were more frequently encountered than others.For example, there were more instances of level-ground walking than stair descent.To assess the effect of the disproportionate number of frames between the different ambulation modes, we examined the relationship between the error and the proportion of frames for each ambulation mode.This relationship is depicted in Fig. 7, where the vertical axis corresponds to the same normalized RMSE values and standard deviations presented in Fig. 5, and where the horizontal axis corresponds to the mean proportion of frames for each ambulation mode across subjects.The average proportion of frames for each ambulation mode is as follows: Sitting (11.4 ± 1.5%), Sit-to-Stand (3.1 ± 0.5%), Standing (7.7 ± 1.0%), Stand-to-Sit (3.8 ± 0.5%), Level Walking (39.3 ± 3.8%), Stair Ascent (8.8 ± 1.9%), Stair Descent (7.9 ± 1.6%), Ramp Ascent (4.9 ± 0.3%), Ramp Descent (5.2 ± 0.3%), and Turning (7.7 ± 0.9%).While level walking was encountered most often throughout the circuit, there is no clear trend between the normalized RMSE values and ambulation mode frequency.Thus, the differences in prediction accuracy between ambulation modes is not likely to be an artifact of the ambulation circuit design.

D. Predictive Model Comparison
We compared the performance of the ultrasound-based model, IMU-based model, and combined model in predicting the four  joint variables across the different ambulation modes (Fig. 8).The corresponding RMSE values presented in Fig. 8 are detailed in Table S2, along with their standard deviations.Results from the two-way ANOVAs are detailed in Table S3 for all four joint variables.P-values for all t-tests are detailed in Table S4, with statistically significant p-values (p < 0.05) also marked in Fig. 8.The two-way ANOVA for the ankle position and velocity found a main effect of ambulation mode and predictive model, but no interaction effect.On the other hand, the ANOVA revealed an interaction effect between predictive model and ambulation mode for both knee position and knee velocity.The t-tests performed between the ultrasound-based model and the IMUbased model showed significant differences for some ambulation modes in both the knee position (p < 0.05) and the knee velocity (p < 0.05) but not for the ankle position or velocity.
Significant differences in knee position accuracy were observed between ambulation modes and predictive models.During sitting, sit-to-stand, and stand-to-sit, the ultrasound-based model resulted in an error that was 7. Comparisons between the combined model and the ultrasound-based model resulted in significant differences between knee position during sitting, sit-to-stand, stand-to-sit, and turning.In these ambulation modes, the combined model resulted in an RMSE that was 7.

IV. DISCUSSION
The goal of this study was to test the hypothesis that A-mode ultrasound can continuously predict the prosthesis kinematics across different ambulation modes in individuals with transfemoral amputations.Results from seven individuals with a transfemoral amputation who walked in an ambulation circuit that included sitting, standing, level and inclined walking, turning, and stairs, show that prosthesis kinematics can be estimated using only A-mode ultrasound.With the ultrasound-based model, overall prediction RMSE was between 4.6 and 8.7% (Fig. 5(b)) and ambulation mode specific RMSE was between 3.0-8.5% (Fig. 5(a)).This error approximates the variability observed in prosthesis kinematics in daily life [51], [52].Therefore, this study suggests that A-mode ultrasound could be used to predict prosthesis kinematic for the control a powered prosthesis across different ambulation modes.
The breakdown of the normalized RMSE by joint variable (Table S1) shows that across every ambulation mode, the position error is generally higher than the velocity error.However, we should not conclude that predicting velocity is better than predicting position because there is a fundamental difference in prosthesis behavior between position and velocity control.Table S1 also suggests that the ultrasound-based model achieves better accuracy in predicting the ankle kinematics than the knee kinematics.This result is likely due to the lower ankle mobility and range of motion, due to subjects using their passive prosthesis.As powered ankle prostheses have a wider range of motion than that observed in this study, this result may not generalize to powered ankle prostheses.Due to these differences, the result of this study should not be used as the sole metric to inform online control of powered prostheses.
Statistical analysis shows that the performance of the joint variable prediction depends on the ambulation mode (Fig. 8).For example, the knee velocity error is significantly higher during stair descent and ramp ascent than walking.We believe that the higher prediction error observed for some of the dynamic activities is due to the higher variability in how subjects performed these activities, which has been demonstrated before in uncommon non-walking activities among transfemoral amputees [53].Not surprisingly, higher variability occurred in ambulation modes such as stair descent, which subjects do not encounter often in the real world.Interestingly, the knee position error is comparable to the errors of other joint variable predictions during the dynamic activities, but shows considerably higher error during sitting, sit-to-stand, and stand-to-sit.This result could be due to the fact that all the velocity variables reach zero (i.e., the same value) whenever the subject is at rest.In contrast, the knee position varied greatly during sitting while the muscle activation did not change, due to both anthropometry (e.g., subject height) and subjective preference (e.g., subjects can sit with their prosthesis being flexed slightly inward or extended outward).The higher variability observed in sitting resulted in higher variability in the transitions between sitting and standing and vice versa.Notably, this variability affected the knee position but not the ankle position, as the passive carbon fiber foot used by the subjects (Table I) does not visibly deflect when sitting.Thus, the task variability could explain why the accuracy of prediction is generally higher for the knee velocity, ankle position, and ankle velocity than the knee position during sitting and standing.
The performance of the ultrasound-based predictions can be further analyzed by observing the measured and predicted kinematics (Fig. 6).For all joint variables, there was a mismatch between the predicted and the measured maxima and minima.This "clipping" effect could be due to the neural network optimizing across the entire dataset, which, in turn, may lead to a reduced performance for the underrepresented points at the maxima and minima.Wider networks and additional features, such as the derivative of the ultrasound features were explored in pilot analyses but were not implemented because no significant difference in performance was noted.
Visual analysis of the kinematic predictions shows an offset type error in the knee position predictions for static activities like quiet sitting and standing.This offset can be explained by kinematic differences in static poses which do not correlate with muscle activity, as discussed previously.In contrast, the predictions of knee velocity and ankle velocity (Fig. 6) consistently match during rest, as there is little variability in velocity during these static activities.Further evidence of this type of prediction error in static tasks is provided in Fig. S1.
Another consistent error that the ultrasound-based model encountered was predicting a change in the kinematics while the measured kinematics remained constant.This error is apparent in Fig. 6 for knee position in the initial sitting position, for both knee variables during stair ascent, standing, and turning, and for the ankle variables during the stand-to-sit activity.Further examples of this type of error can be found in the trials shown in Fig. S1. and Fig. S2.In all these instances, the measured prosthesis is relatively still, yet the ultrasound-based models predict movement.A likely explanation of this result is that for these movements there is little to no correlation between the user's muscle contractions and the prosthesis kinematics.In other words, the user could be contracting the residual-limb muscles without causing a movement of the prosthesis.For example, as the targeted muscles are also connected to the hip, any muscle deformation due to the subjects adjusting their posture could be interpreted as knee or ankle movement.During stair ascent, users could be activating their residual limb muscles to pull the relatively motionless passive prosthesis behind them (Video S1).During standing, variations in the muscle activity can be due to subjects shifting their weight or balancing, but the alignment of the knee prosthesis is such that the knee joint would rest fully extended against the end stop.Overall, the visual inspection of the kinematics predictions suggests that ultrasound-based models may be better suited for the prediction of velocity than position, mostly due to the offset-type error in static tasks.Therefore, exploring velocity control techniques might hold more promise for future online studies.
Comparisons of the A-mode ultrasound-based predictions against the IMU-based predictions and the predictions from the combined model show the benefits and limitations of these approaches.The ultrasound-based model performs significantly better than the IMU-based model in predicting knee position during stair descent, and in predicting knee velocity during walking, stair descent, and ramp descent (Fig. 8).The only ambulation tasks in which the ultrasound-based model underperforms compared to the IMU-based model is in the prediction of knee position during sitting, sit-to-stand and stand-to-sit.We believe that the ultrasound-based model could be underperforming in these tasks because the muscle activity does not correlate with the movement of the prosthesis.For example, subjects can contract their muscles without affecting the knee position when sitting, and can rest their prosthesis at different positions while seated.Thus, our results suggest that the ultrasound-based model may be better suited for dynamic activities that demand coordination between the residual limb and prosthesis kinematics, leading to a closer correlation between the ultrasound features and intended prosthesis movement.
The combination of ultrasound and IMU sensing appears to improve the prediction performance.Not surprisingly, for the tasks in which the ultrasound-based model underperformed the IMU-based model, the combined model was able to match the accuracy of the IMU-based model.This result suggests that the added ultrasound features did not hinder the prediction accuracy in these settings.Moreover, the addition of ultrasound visibly improves the performance of the IMU-based model in predicting knee kinematics during walking, stair descent, ramp ascent and ramp ascent when compared to the IMU-based model (Fig. 8).The combined model outperformed the IMU-based model in predicting ankle position during ramp descent, but there are no significant differences in ankle velocity between the three models.The superior performance of the combined model suggests that the ultrasound data contains pertinent information to the user's movement that cannot be provided by the user's residual limb kinematics as measured by the IMU.Notably, when comparing the ultrasound-based model and the combined model, there are improvements only in the knee position and ankle position predictions, but there is no significant difference in the knee and ankle velocity predictions.This result suggests that A-mode ultrasound alone could predict the user's desired joint velocity with sufficient accuracy, making it a viable technology for intent recognition across multiple ambulation modes.
Despite substantial differences in frequency of occurrence between different ambulation modes, the performance of the predictive models does not differ between ambulation modes.For example, the error for level walking falls near the middle of the other ambulation-specific errors (Fig. 7).Moreover, there is no trend in the remaining cluster of ambulation modes, which suggests that the proportion of training frames does not influence the error.Observing the trends in the ambulation-specific error for the four joint variables in Fig. 8 further corroborates that the Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.
frequency of occurrence of the ambulation mode does not affect prediction accuracy of our models.
The performance of the ultrasound-based models could have also been affected by muscle fatigue, as our models were tested against the last three trials throughout all subjects.Although fatigue was not measured in this study, it is possible that subjects not accustomed to continuous movement or to performing certain activities present in our ambulation circuit grew tired by the end of the data acquisition.Whether this led to muscle fatigue or the adoption of energy conserving ambulation strategies, it is possible that the muscle deformation patterns changed during the latter part of the acquisition, affecting the ultrasound features used for our predictions.The results of this study suggest that the ultrasound-based prediction strategy is robust against different ambulation modes.However, future studies should focus on measuring its robustness against prolonged use or multi-day use cases.
Both EMG and B-mode ultrasound have been used in ablebodied subjects to predict lower-limb kinematics across various ambulation modes.In level-ground walking, ramp ascent, ramp descent, stair ascent, stair descent, EMG has been used to predict knee position with a normalized RMSE of 11.7%, 13.0%, 11.2%, 19.8%, and 17.0%, respectively, and similarly to predict knee velocity with a normalized RMSE of 15.6%, 14.4%, 12.4%, 19.0%, 18.9%, and 16.1% [37].Across those same activities, B-mode ultrasound has been used to predict knee position with a normalized RMSE of 7.6%, 3.9%, 7.6%, 12.7%, and 9.9% respectively, and to predict knee velocity with a normalized RMSE of 11.3% 9.6%, 10.7%, 13.0%, and 12.3% [37].In comparison, this study (Table S1) showed a knee position normalized RMSE of 7.3%, 7.3%, 9.4%, 5.0%, and 12.3% for walking, ramp ascent, ramp descent, stair ascent, and stair descent, respectively, as well as an knee velocity normalized RMSE of 6.4%, 7.2%, 8.0%, 3.2%, and 7.2%.Interestingly, our kinematic predictions during walking improved from our previous experimental setup, which had resulted in a mean normalized RMSE of 9.0 ± 3.1%, 7.3 ± 1.6%, 8.3 ± 2.3%, and 10.0 ± 2.5% for the knee position, knee velocity, ankle position, and ankle, respectively [18].While the same machine learning structure was used in the two studies, the amount of training data greatly increased for this study, possibly resulting in the improved predictions for walking.Furthermore, this improvement could imply that ambulationspecific performance, in this case walking, is not necessarily sacrificed with the inclusion of data from additional ambulation modes.Comparing these results to that of prior studies is quite challenging because of the lack of studies with individuals with amputations.Transfemoral amputees using passive prostheses use a different gait pattern in stairs, resulting in substantially different joint trajectories compared to healthy biological legs.Walking and ramps provide a better comparison, as they are performed using the same gait pattern.These ambulation modes show that A-mode ultrasound achieves comparable results in knee position and better tracking of knee velocity than EMG and B-mode ultrasound in healthy subjects.However, further studies are necessary to avoid confounding factors and properly quantify the differences between different sensing modalities.
In summary, incorporating A-mode ultrasound sensing in the residual limb of transfemoral amputees allowed for the accurate prediction of their lower-limb prosthesis kinematics across a variety of ambulation modes.The performance of this technology suggests that the sensing modality could be widely translated online for different activities.However, further online experiments are necessary to confirm this indication.Moreover, it is possible that some of the limitations of A-mode ultrasound sensing could be addressed by creating a shared online controller that considers the type of ambulation task (e.g., dynamic vs. static) to prevent the errors encountered in this offline study.Furthermore, an online experiment is necessary to measure if the current machine learning model with a prediction frequency of 80 Hz is adequate for recognizing user intent online.Finally, while this study tackled the regression problem of predicting the kinematics across different ambulation modes offline, future studies could explore other means to interpret and utilize the A-mode ultrasound sensing, such as recognizing joint torque through musculoskeletal models or other feature extraction techniques.

V. CONCLUSION
Seamless and accurate detection of user movement intention across different ambulation tasks is necessary for powered prostheses to work in real life.Volitional control of powered prosthesis movements could lead to fine adaptation of the prosthesis function to different environments and user intentions.This study lays the foundation for ultrasound-based volitional powered prosthesis controllers by demonstrating that A-mode ultrasound can be integrated with the sockets of transfemoral amputees to continuously predict the prosthesis knee and ankle kinematics across different ambulation modes.This study further suggests that classifying the ambulation mode intended by the user may not be necessary.Future studies will focus on translating our A-mode machine learning system for online control of powered prostheses.

APPENDIX
Table S1.Ambulation-Specific normalized RMSE across Prediction Models Table S2.Ambulation-Specific RMSE across Prediction Models Table S3.Two-way ANOVA of RMSE values across predictive models and ambulation modes Table S4.p-Values between RMSE values from different predictive models (t-tests with Tukey-Kramer corrections) Fig. S1.Additional representative testing trial showcasing predicted kinematics.

Fig. 1 .
Fig. 1.A-mode ultrasound system and sensor placement.(a) 4-channel A-mode ultrasound system.(b) Anterior placement.(c) Posterior placement.(d) Representative subject wearing both the A-mode ultrasound system and Xsens IMU-based motion capture system.

Fig. 3 .
Fig.3.Feature reduction.For each channel, the ultrasound system transmits a 1D signal (blue) comprised of 997 datapoints that correspond to tissue echogenicity along penetration depth.The signal is first rectified (yellow).A moving average (green) of the rectified signal is taken.The moving average is then segmented into 48 windows (bar plot) of 20 data points.The average of each segment serves as an input feature (purple circle) to our neural network.

Fig. 4 .
Fig. 4. Sample joint kinematics and A-mode ultrasound recordings throughout a single ambulation circuit trial, where different ambulation modes are denoted by the different colored shaded regions, and transitions are marked by vertical dashed lines.

Fig. 6 .
Fig. 6.Representative test trial demonstrating measured kinematics (black) and predicted kinematics from the ultrasound-based model (blue) for knee position, knee velocity, ankle position, and ankle velocity.Different ambulation modes are denoted by shaded regions and letter labels at the top of figure.

Fig. 7 .
Fig. 7. Ambulation-specific normalized RMSE relative to proportion of training frames for the ultrasound-based model.
4 deg, 4.6 deg, and 4.9 deg greater than the error resulting from the IMU-based model.During stair descent, the knee position prediction from the ultrasound-based model was 7.5 deg more accurate than the prediction from the IMU-based model.Moreover, significant differences in the knee velocity predictions were observed in walking, stair descent, and ramp descent, where the RMSE for the ultrasound-based model was 18.7 deg/s, 26.8 deg/s, and 25.3 deg/s less than the one from the IMU-based model.For all other ambulation modes and joint predictions, no statistical difference was found.
5 deg, 5.0 deg, 5.6 deg, and 2.9 deg lower than the ultrasound-based model.In predicting ankle position, the combined model resulted in a RMSE that was 0.7 deg less during turning.No significant differences were found in the velocity variables between the ultrasound-based model and the combined model.

Fig. 8 .
Fig. 8. Performance of ultrasound-based model (red), IMU-based model (dark grey), and combined model (light grey) in predicting knee position, knee velocity, ankle position, and ankle velocity throughout 10 different activities.

Fig. S2 .
Second additional representative testing trial showcasing predicted kinematics.Video S1.Video of one subject performing one trial of the ambulation circuitACKNOWLEDGMENTAuthors would also like to thank Kaitlin Rabe, Grace Hunt, and Sarah Hood for helping with human studies.