Toward Real-Time Muscle Force Inference and Device Control via Optical-Flow-Tracked Muscle Deformation

,

Abstract-Despite the utility of musculoskeletal dynamics modeling, there exists no safe, noninvasive method of measuring in vivo muscle output force in real time -limiting both biomechanical insight into dexterous motion and intuitive control of assistive devices. In this paper, we demonstrate that muscle deformation constitutes a promising, yet unexplored signal from which to 1) infer such forces and 2) build novel device control schemes. Through a case study of the elbow joint on a preliminary cohort of 10 subjects, we show that muscle deformation (specifically, thickness change of the brachioradialis, as measured via ultrasound and tracked via optical flow) correlates well with elbow output force to an extent comparable with standard surface electromyography (sEMG) activation during varied isometric elbow contraction. We then show that, given real-time visual feedback, subjects can readily perform a trajectory tracking task using this deformation signal, and that they largely prefer this method to a comparable sEMG-based control scheme and perform the tracking task with similar accuracy. Together, these contributions illustrate muscle deformation's potential utility for both biomechanical study of individual muscle dynamics and device control, in a manner that -thanks to, unlike sEMG, the localized nature of the signal and its tight mechanistic coupling to output force -is readily extensible to multiple muscles and device degrees of freedom. To enable such future extensions, all modeling, tracking, and visualization software described in this paper, as well as all raw and processed data, have been made available on SimTK as part of the OpenArm project (https://simtk.org/projects/openarm) for general research use.

I. INTRODUCTION
Despite decades of study, noninvasive, in vivo, real-time measurement of muscle forces remains an open problem in the biomechanics community. Without good models of muscle force output during natural movement, our understanding of how humans execute dexterous motions is fundamentally limited, as is our ability to safely modify or replace this execution using assistive devices and to accurately characterize and treat musculoskeletal pathology.
Historically, muscle forces have either been computed using full-body modeling frameworks like OpenSim [2] and AnyBody [3] -which account for little physiological variation, enable only limited real-time computation, and make strong optimization-based assumptions about force distribution across synergists -or using surface electromyography (sEMG), a sensing modality that measures the neurological input to the musculoskeletal system, not the resultant output forces, and fundamentally does not allow for direct muscle force inference [4]. While these methodologies have resulted in impressive advances in motion modeling and device control, our ability to both understand and replicate dexterous motions -particularly for the upper limb -remains severely limited.
As a complementary technology to optimization-based modeling and sEMG measurement, we propose muscle deformation (as measured via ultrasound) as a class of signals that is both more directly representative of muscle output force and easier to spatially localize than sEMG. In an early version of this work, we showed that several simple measures of muscle deformation (cross-sectional area, thickness, etc.) correlate well with output joint force, and are readily trackable via optical flow [1]; here, we leverage a novel real-time tracking and visualization system to corroborate those findings on an expanded subject cohort and a refined collection of elbow flexion trajectories. We also present a proof-of-concept system for real-time trajectory tracking with the deformation signal, representing an important step toward deformationbased device control.
The contributions of this paper are as follows: • a novel experimental platform, consisting of both networked hardware and real-time optical-flow-based muscle contour tracking software, that enables the simultaneous, time-synced collection and display of joint force, muscle activation, and muscle deformation data, facilitating both force-deformation-activation correlation analysis and the execution of gamelike trajectory tracking tasks; • a novel, quantitative description of the correlation relationship between deformation of the brachioradialis muscle and output force at the elbow joint during isometric contraction, shown to be consistent with simultaneous sEMG measurements of biceps brachii activation, in a preliminary cohort of subjects; • evidence that this preliminary subject cohort can intentionally modulate this muscle deformation to perform a trajectory tracking task, shown to compare favorably with an sEMG-based controller in an evaluation of tracking performance and preferences; and • an open-source time series data set, including simultaneous ultrasound, sEMG, and elbow force data, collected during both correlation and trajectory tracking trials, for future study and modeling. The data and software associated with all of the above contributions have been made available for general research use as part of the OpenArm project on SimTK (https://simtk.org/projects/openarm), which also hosts complementary research and data sets examining 3D (static) muscle shape under various conditions [5,6], as well as data and code from early versions of this work [1].

II. MOTIVATIONS & RELATED WORK
Although finding a noninvasive measure of individual muscle forces remains a core challenge inhibiting our understanding of human motion [7], no single sensing modality or analysis framework has emerged as a dominant measurement solution. In the context of current biomechanics research, we first argue that muscle deformation constitutes a promising signal from which to infer these muscle forces. We then introduce and motivate 1) this paper's preliminary data set as a source of initial insights into the time series behavior of deformation, and 2) our selected trajectory tracking task as a first step toward physical device control.

A. Deformation as a Measure of Output Force
While the human musculoskeletal system is highly complex, geometrically irregular, and dominated by the physics of various nonlinear materials, the core mechanism underlying human movement is straightforward: muscles ratchet together (largely via the actin-myosin cross-bridge cycle [8], though other proteins like titin are thought to play a role as well [9]), inducing a length change along the line of action, which pulls the attached (roughly elastic) tendons, which then impart the force to the skeleton. Under the (mild) assumption that muscles are isovolumetric [10,11], this length change by definition induces a shape change, or deformation, in the activated muscle.
Several isolated studies have established a correlation between muscle activation and shape change [12][13][14][15][16][17], including our own previous work examining the full 3D extent of the biceps brachii under static (but kinematically varied) elbow loading [5]. This prior study showed evidence that such deformation -measured as muscle cross-sectional area (CSA) or thickness changes -is readily observable via ultrasound, though the appearance of the muscle cross section varies drastically with both sensor location and kinematic configuration. This signal complexity is the result not only of the nonuniform material properties of a given muscle-tendon unit, but of its contact dynamics with surrounding structures. This complexity means that interpreting the deformation signal is not always obvious: while some locations along the arm show a reliable CSA or thickness increase corresponding to increased output force, others may show a decrease or no reliable change, and this varies based on which precise parameterization of deformation is used. Despite these challenges, the deformation signal has several advantages as compared with electrode-based measures like sEMG: first, it is localizable and thus attributable to a particular muscle, and second, it can be measured from both deep and superficial muscles (given, for example, an ultrasound transducer with sufficient depth resolution).
A simpler approach is to measure the force-generating muscle-tendon unit length change directly, in either the muscle or the tendon [18][19][20], or to measure muscles' [21] or tendons' [22] vibrational behavior to infer strain ("mechanomyography"). These approaches are complementary, but limited: several muscles of interest may be attached to the same tendon, and these attached tendons -or, in many cases, aponeuroses -may have complex geometry that makes establishing a single "length" value difficult or impossible, and vibrational measurements are often corrupted by ambient noise and difficult to localize. The muscle deformation we examine in this paper can be thought of as an amplified, higher-dimensional version of this length signal, which intrinsically incorporates additional information about factors like contact dynamics and fascia elasticity.

B. Brachioradialis Measurement via Ultrasound for Proofof-Concept Force-Deformation Correlation
As an initial study of the force-deformation relationship, we examine deformation of the brachioradialis (one of several elbow flexors) and its relationship to elbow force output. As in previous studies [5,6], we focus on isometric elbow flexion as a proof-of-concept motion, as it is a comparatively simple joint (with only one degree of freedom and only a few muscles) that is relevant to the upper-limb modeling cases for which this research may be especially applicable. Also as before, we measure this deformation via 2D B-mode ultrasound, a technology that is safe, portable, and provides a relatively clear image of the fascia between muscles.
To allow collection of time-varying data, we restrict ourselves to a single 2D ultrasound frame collected from the same location along the arm at all time points. We target the brachioradialis for analysis because it is the smallest of the elbow flexors and its cross-section largely fits within the frame of a single ultrasound scan for many subjects (unlike the biceps or brachialis, for which substantial portions of the cross-section cannot always be captured in a single frame under varied force conditions). Building on earlier versions of this work [1], which identified cross-sectional brachioradialis muscle thickness as a particularly forcecorrelated and reliably trackable deformation measure (despite the muscle's comparatively small size as compared with other flexors), we leverage this thickness signal both to evaluate force-deformation correlation (section IV) and as our control signal for performing trajectory tracking tasks (section V).
Note also that although the ultimate goal of this research is to relate measures of muscle deformation to individual muscle forces, for the purposes of this preliminary (noninvasive) study, we restrict our analyses to net joint output force. * This research is interesting precisely because there are no readily available individual force measures with which to compare our data, and we believe that the deformation-net-force correlations presented here, along with the insights in section II-A regarding the physiological causes of deformation, constitute a powerful case that this signal is a promising candidate for individual muscle force measurement. In the future, we aim to probe this claim both empirically (e.g., via invasive animal study that enables muscle-tendon unit isolation and/or comparison with tendon-tapping force inference methods [22]) and through enhanced modeling (e.g., fitting Hill-type muscle models and examining their predictive power).

C. Real-Time Trajectory Tracking as a Proof-of-Concept Control Task
Building on this muscle force-deformation relationship, we examine the feasibility of leveraging the brachioradialis deformation signal in a device control task. We see the deformation signal as particularly promising due to its spatial localizability: as discussed above, deformation is by definition attributable to a particular muscle, providing both direct information about muscle force output and theoretically allowing for the simultaneous extraction of multiple independent signals from both deep and superficial muscles to control different device degrees of freedom in a straightforward manner.
Ultrasound-based ("sonomyographic") control, while not new [23][24][25], is underexplored: although preliminary control has been demonstrated on a single-degree-of-freedom prosthetic hand [24], as well as on more complex devices by leveraging learning-based image processing and gesture classification [23,25], these techniques have not seen wide adoption, are rarely evaluated against state-of-the-art sEMG control systems, and are inhibited by poor understanding of the underlying musculoskeletal dynamics. This lack of underlying biomechanical knowledge inhibits not only generalizability, but device safety and efficacy: as we begin constructing devices that physically modify human movement, understanding the resultant musculoskeletal forces will be essential both to avoid injury and to ensure that, for example, rehabilitative devices induce therapeutic exertions.
Considering these safety and efficacy concerns, we demonstrate a first step toward the use of deformation as a device control signal with a proof-of-concept case study in which subjects completed a trajectory tracking task by modulating either ultrasound-measured deformation or sEMG-measured activation. In addition to the preliminary evaluation of tracking success and user preferences contained in this document, we provide a full suite of contact force, activation, and deformation data collected during these trials to the wider community as a resource for ongoing study of the associated neuromuscular dynamical relationships.

III. MUSCLE DEFORMATION & ACTIVATION TRACKING SYSTEM
To accomplish both objectives of this paper -evaluating force-deformation correlation during isometric contraction at the elbow (section IV), then leveraging these signals for control during a trajectory tracking task (section V) -we developed a novel experimental platform to enable simultaneous collection and optional real-time display of joint force, muscle thickness (as measured via ultrasound), and muscle activation (as measured via sEMG) during varied isometric elbow flexion. † This section details, first, hardware aspects of this system that allow for collection of these data under consistent kinematic conditions, and second, the signal processing software used to extract, calibrate, display, and record these signals over time.

A. Hardware Setup
The hardware platform was designed for data collection from the right arm of a subject seated comfortably upright, feet planted, right upper arm comfortably adducted (vertical), elbow flexed 90 • , forearm fully supinated, with the elbow supported by a static jig from below, as shown in Fig. 1. The right wrist was firmly strapped into a brace mounted to a 6-channel force-torque sensor (ATI Mini45, ATI Industrial Automation, Apex, NC, USA), which was in turn mounted to the end effector of a 7-degree-of-freedom robot arm (KUKA LBR iiwa 14 R820, KUKA AG, Augsburg, Germany). ‡ Subjects pressed upward on this sensor to generate a measure of "output joint force" at the elbow; while this measured contact force was inherently confounded by contributions from other linked joints, subjects were instructed to exert force using only elbow flexion motion, and the wrist was immobilized, with the force sensor attached near the proximal edge of the palm, to isolate the elbow joint as much as possible.
To gather muscle activation and deformation data, the subject's right arm was instrumented with an Arduino-driven MyoWare sEMG system (Advancer Technologies, Raleigh, NC, USA) set to a single consistent gain and a 3-12 MHz linear ultrasound transducer (L3-12 NGS, eZono AG, Jena, Germany) attached to its corresponding ultrasound unit (eZono 4000, eZono AG, Jena, Germany). Surface EMG electrodes (Red Dot 2560, 3M, St. Paul, MN, USA) were placed in a differential configuration on the biceps brachii, with the two signaling electrodes placed with 40 mm proximaldistal center-to-center separation (adhesive edges abutting) and the top electrode roughly centered on the lateral belly of the muscle, and the grounding electrode placed on the acromion. The ultrasound transducer was manually placed perpendicular to the lower arm such that the brachioradialis cross section was maximally in frame and held in place by an adjustable foam and neoprene cuff.
Note that ultrasound and sEMG sensors were placed to target different muscles: the brachioradialis and the biceps, respectively. While this inherently limits any correlation insights we might make between deformation and activation values (i.e., our analyses should be understood in the context of the multi-muscle elbow flexion motion, rather than statements about the force-deformation-activation relationship of an individual muscle), this configuration was chosen Experimental setup for the collection of time series force, ultrasound, and surface electromyography (sEMG) data during constrained isometric elbow flexion. Setup includes ultrasound probe (a) attached securely to user's forearm with cuff (b); surface electromyography (sEMG) electrodes (c); wrist brace (d) through which subject transmits force to attached to force-torque sensor (e), in turn held stable by KUKA robot (f); elbow stabilizing jig (g); goal and sensor trajectory display (h) for real time visual feedback for subject self-assessment; and real-time ultrasound thickness tracking data (i) for continuous experimenter system status monitoring. This system allows subjects to precisely follow a specified force trajectory to enable study of force-deformation correlation under varied trajectory types (section IV) and to perform trajectory tracking tasks using experimental deformation-and activation-based signals (section V).
to allow simultaneous collection while preventing sensors from physically interfering with one another (a challenge due to the brachioradialis's comparatively small size, which -as discussed in section II-B -allowed more complete deformation analysis but prevented simultaneous ultrasound and sEMG recording). Earlier work [1] also found stronger sEMG signals from the biceps than the brachioradialis, such that this configuration provides a more competitive baseline against which to evaluate deformation data in terms of both strength of correlation and use as a control signal.

B. Signal Tracking & Display
During data collection, the subject faced a large monitor, which displayed two or more signal streams: a time series goal trajectory, and either force (to enable consistent force output for correlation analysis), ultrasound-extracted deformation/thickness (to evaluate the feasibility of deformationbased trajectory tracking), and/or sEMG-extracted activation (as a baseline against which to evaluate deformation-based trajectory tracking control), each normalized to the subject's  Fig. 2. Still frame of optical-flow-based brachioradialis thickness tracking system. Points were tracked along the superficial (red) and deep (blue) fascial surfaces of the brachioradialis, and thickness was reported as the vertical (superficial-to-deep, green-to-green) distance between the center (mean) of each cluster. A line connecting each cluster center was also displayed (magenta) to allow for easy observation of particle drift. strength capabilities. Our methods for signal extraction, calibration, and display are outlined below, and full details can be found in the released codebase.
1) Tracking Muscle Thickness via Optical Flow: Drawing on previous tracking successes [1], brachioradialis thickness was tracked over time via the standard iterative Lucas-Kanade method of optical flow estimation [26] as implemented in the OpenCV Python library [27]. Specifically, at the start of each trial, while the subject was instructed to remain still, 10 points were manually selected along both the top and bottom (i.e., superficial and deep) surfaces of the brachioradialis muscle fascia, forming two clusters of points. These points were used to define the vertices of two polygons from which contours were extracted [28]; these contour points, as shown in Fig. 2, were then tracked over time (on a bilaterally-filtered version of each image [29] to suppress speckle noise) at a best-effort frame rate § of 1 kHz, and the thickness value calculated at each iteration as the vertical (superficial-to-deep) distance between the mean location of tracked points in each cluster.
To prevent points from drifting away from the selected surfaces (a frequent challenge in optical flow estimation), we leveraged the knowledge that the fascial segment selected by each point cluster should remain intact -i.e., points within a cluster should remain at similar positions relative to each other. Thus, during each iteration, if the average squared distance from tracked points to their cluster center exceeded a specified distance (here, 200 px 2 , or approximately 3 mm 2 ), all tracked point locations were reset to their initial locations. In practice, these resets happened rarely for most subjects and are noted in released data.
2) Signal Processing & Calibration: Force, ultrasoundextracted thickness, and sEMG-measured activation were sampled for both recording and display at a best-effort rate of 1 kHz. Activation values (measured as the difference in signal value across the two sEMG electrodes) were smoothed via a 250-point (0.25 s) moving average filter.
To normalize force, deformation, and activation signal traces s(t) to a subject's strength, at the beginning of each trial, subjects were first instructed to remain still for several seconds, then to press upward with maximum possible contraction force for several seconds; mean minimum and maximum signal values were then calculated over the final 200 samples (approximately 0.2 s) at each condition as s min and s max , respectively.
Normalized signal tracess(t) were then calculated for both display and recording as During each trial, one or more of these normalized traces s(t) was displayed alongside a goal trajectory, which the subject was then instructed to track by modulating elbow flexion force, as detailed in sections IV-A and V-A below. Note that regardless of display, all goal, force, deformation, and activation signals, both raw and processed, were collected for all trials.

IV. CORRELATION OF MUSCLE DEFORMATION WITH JOINT FORCE
In this section, we present preliminary data, collected via the platform detailed above, indicating that brachioradialis muscle deformation (i.e., thickness change) correlates with output force at the elbow during varied isometric contraction and is consistent with simultaneous sEMG data. We first outline our subject cohort and collection procedure, then present preliminary time series data alongside qualitative and quantitative analysis of the force-deformation relationship. Lastly, we comment on study limitations and how these preliminary analyses suggest future research directions.

A. Data Set Collection
As an exploratory data set, simultaneous force, deformation, and activation signals were collected from a preliminary cohort of subjects using the platform described above in section III as they tracked a specified elbow flexion force trajectory with visual feedback. Details of this subject cohort and collection procedure are outlined below.
1) Subject Biometric Data & Consent: Data were collected from the right arm of 10 subjects (7 female, 3 male, 9 righthanded, 1 left-handed, age 25.6 ± 0.966, mass 61.7 ± 10.5 kg, height 1.69 ± 0.0742 m, body mass index 21.5 ± 2.89), hereafter denoted Sub1-Sub10. ¶ All subjects were healthy, with a wide variety of exercise regimes, body types, and familiarity with nonstandard computer interfaces. The study protocol was approved by the University of California Institutional Review Board for human protection and privacy under Protocol ID 2016-01-8261 (first approved 4 April 2016) and written informed consent was obtained from each subject. ¶ Statistics are reported as mean ± standard deviation. For additional demographic data, broken down by subject, see the full open-source data release.
2) Trial Specification: After being strapped into the data collection system outlined in section III-A and instrumented with all relevant sensors, each subject performed three 90 s tracking trials (to enable preliminary analysis without imposing extensive fatigue): one unstructured trial to familiarize them with the system, followed by two trials intended for correlation analysis, the latter of which is plotted and analyzed below.
The initial familiarization trial -from which no actual data was analyzed for publication -was designed to both familiarize subjects with the system and assure investigators that all sensors were behaving as expected. After initializing the thickness tracking system and performing min/max calibration (as outlined in sections III-B.1 and III-B.2, respectively), the subject's monitor was set to display all three normalized force, thickness, and activation traces, as well as the goal trajectory to be used in future trials. Subjects were then instructed to freely modulate elbow flexion to get a sense for how the various signals changed with force output, though they were not yet asked to perform the tracking task.
During the two data collection trials, the same tracking initialization and min/max calibration was performed, but this time, the subject's monitor displayed only the goal and normalized force trajectories. The subject was instructed to match the force trace to the goal trace -scaled to the subject's force generation capability and detailed belowby modulating isometric elbow flexion force.
3) Goal Trajectory Elements: To evaluate correlation across various types of force exertion, both sustained and quickly varying, the same 90 s goal trajectory containing these varied elements and shown in Fig. 3 was used for all trials. Specifically, the trajectory contained the following elements in sequence: • sustained flexion at 0.25, 0.5, and 0.75 of maximum force capability, with intervening rest phases; • a slow ramp in flexion from 0 to 0.75 of maximum force capability, then a slow ramp back to 0, after briefly sustaining; • arbitrary, quickly varying step changes in flexion; and • a sine wave ranging from 0.25 to 0.75 of maximum force capability. Force-deformation and force-activation correlation were evaluated across both the entire trajectory and individual elements, as detailed below.

B. Correlation & Evaluation
In the following analyses, we use the Pearson correlation, applied directly to the synchronized data streams, to assess the viability of using our candidate deformation measures to infer output force, alongside or as an alternative to sEMG. We first examine an illustrative time series, then discuss how our assertions translate across across subjects and trajectory types.
1) An Illustrative Time Series: Fig. 3 shows representative trajectory data from a single trial (specifically, that of Sub1). In this series, and in general, brachioradialis thickness deformation correlates comparably with sEMG activation (though this varies with both subject and trajectory type, as discussed below). This exemplar illustrates two important data set qualities: first, that subjects proved impressively skilled at following goal trajectories given visual feedback, yielding both the sustained and quickly varying signal types we sought for comparison; and second, that while our optical-flow-based tracking system exhibited some drift over time, thickness was largely well-tracked for most subjects throughout the full 90 s duration of each trial.
Lastly, this time series illustrates an artifact observed in our ultrasound tracking data at all trials -namely, a "stair step" quality not present in force or sEMG data, in which values are sustained and then jump suddenly. We attribute this quality to the limitations of our data collection system: observed brachioradialis thickness changes were generally less than 4 mm from relaxation to full exertion (and often substantially smaller), corresponding to changes of fewer than 30 pixels in our (relatively low resolution) ultrasound image, resulting in choppy optical flow tracking that must be accounted for in our analyses.
2) Correlation by Subject: Fig. 4 shows the strength of deformation and activation correlation with force for each of our 10 subjects. Though force-activation correlation is higher than force-deformation for most subjects, the latter shows consistent (moderate to strong) correlation across most subjects, with many showing a correlation magnitude of around 0.7 or higher -an even higher magnitude than that found in earlier versions of this work [1] -even though subjects vary significantly in terms of muscle morphology, as illustrated in Fig. 5, suggesting a common underlying biological mechanism.
At the same time, two subjects -Sub7 and Sub9 -fail to show the same thickness-force correlation. While more principled analysis is needed to tease out the exact reasons for this lack of correlation, experimenters noted during collection that the observed deformation appeared qualitatively different, as illustrated in Fig. 5: unlike most subjects, for which vertical (deep-superficial) expansion was observed, widening the brachioradialis contour, these subjects showed substantial lateral motion, in which fibers appeared to slide side to side, but the deep and superficial fascial surfaces seemed to move very little. Given that this different motion paradigm appeared  Fig. 4. Correlation of muscle deformation (blue) and activation (orange) signals with elbow output force across all subjects. Despite substantial differences in morphology (illustrated below in Fig. 5), most subjectsaside from Sub7 and Sub9, who displayed morphological quirks that resulted in poor signal quality, as discussed in section IV-B.2 -showed moderate to strong correlation between deformation and output force. in multiple subjects, and could be the result of a quirk of morphology, a function of sensor placement, or differing elbow flexion strategy, we see other deformation measures (e.g., localized fiber motion in any direction) as worthy of future study.
3) Correlation by Trajectory Type: Generally, as shown in Fig. 6, both deformation and activation correlate well with force across all four trajectory types (sustained, ramp, step, and sine, as outlined above in section IV-A.3), with activation showing indistinguishable, and high, levels of correlation across all trajectory types. Force-deformation correlation was particularly high for ramp and step conditions, though lower and more variable for sustained and sine conditions. Based on qualitative examination of the data, we theorize that these lower values are the result of limitations in our tracking software, rather than an underlying physiological mechanism: specifically, we observe first, that under quick, dramatic force/thickness changes like those at the start and end of each trial, the optically tracked points fail to fully remain on the fascia, and thus do not return to baseline, and second, that these points generally drift over time, perhaps impacting the final (sine) section of the trajectory most dramatically. Nevertheless, deformation -even as measured by this limited, drifting, proof-of-concept system, which we are working to refine to address these issues -is consistently correlated with output force, showing promise for use in device control or motion analysis under both fast and slow movement conditions.

C. Study Limitations & Future Directions in Force-Deformation Modeling
The sections above constitute preliminary analysis on a limited data set, which we aim to further expand with targeted repeated trials (for more rigorous statistical analysis), additional subjects (of varying age and ability), additional force conditions (e.g., non-isometric, natural/unconstrained motion), additional ultrasound views (e.g., longitudinal), a more streamlined (and higher resolution) ultrasound probe and support cuff to support collection of (less choppy) ultrasound and sEMG data from the same muscle, and an improved (e.g., multi-channel) sEMG system for more equitable ultrasound comparison (especially important as we begin to examine dynamic motions, during which electrodes placed on the skin may slide relative to underlying structures). Such enhancements will allow for expanded understanding of the results above (including the impact of age on the deformation signal) and of phenomena not yet explored (e.g., temporal and spatial sEMG-deformation relationships, the impact of fatigue).
A further limitation of this correlation analysis is our assumption that force, activation, and deformation signals occurred simultaneously: we might have achieved better correlation by accounting for the multi-millisecond electromechanical delay expected between sEMG-measured activation and output force [30]. We aim to incorporate this delay into future correlation and modeling analyses (and even leverage our data to study this delay, which is variable and remains poorly characterized).
In addition to these data quality and modeling enhancements, we also seek an understanding of the biological mechanisms underlying our deformation measures. This is a significant analytical challenge, as shown visually in Fig. 5's illustrative frames: a single muscle cross section, without accompanying 3D shape data, is difficult or impossible to interpret, and simple deformation measures like those analyzed above barely scratch the surface of its architectural nuance. Informed by the correlations we observe in this paper, we are addressing these challenges from two complementary perspectives. First, we are constructing and analyzing full 3D muscle images to allow for improved interpretation of 2D cross-sections, including what low-dimensional deformation signals are most correlated with force and where they are best observed relative to the underlying skeleton under different kinematic configurations [5,6]. Second, we are working to track more sophisticated deformation signals, beyond the thickness value used in this paper, to enable the exploration of new and higher-dimensional deformation signals (e.g., statistical shape changes, dense fiber motion).

V. DEFORMATION-& ACTIVATION-BASED TRAJECTORY
TRACKING In this section, we present a proof-of-concept study in which subjects performed a trajectory tracking task using the thickness deformation signal examined above -showing evidence that not only does deformation correlate with joint force, it can be used for real-time device control, and is even, in some cases, preferred by the user over sEMG-based control. We first outline the trajectory tracking task and data collection procedure, then examine subjects' tracking performance and reported preferences. Lastly, we discuss future expansions and improvements to the tracking system and applications to physical device control.

A. Tracking Data Collection
The same cohort of subjects (section IV-A.1) was asked to evaluate two trajectory tracking control schemes (identified to each subject as "mode 1" and "mode 2"); each subject was only informed that each controller would be using some combination of sEMG and ultrasound data, with no additional details. In fact, these trajectories were much simpler than this statement implies: the two modes were simply the normalized thickness/deformation and normalized and meanfiltered activation measures extracted from ultrasound and sEMG data, respectively, and presented to the subject in a randomized order.
For each trial, the subject was instructed to perform the same trajectory tracking task described above in sections IV-A.2-IV-A.3, with identical tracking initialization and cali-  ) and activation (orange, bottom) tracking trials, alongside specified goal trajectories (black, dotted). Subjects were able to complete the tracking task qualitatively well using both signal traces, despite drift in the deformation tracking system that sometimes made returning the system to baseline difficult (as is evident to the right of this trajectory), and uncontrolled, high-frequency oscillations in the activation signal.
bration steps, except that instead of the ground truth force trajectory, the monitor displayed only the normalized thickness/deformation or normalized and mean-filtered activation trace, alongside the same goal trace used in correlation analysis. The subject executed the full tracking task using each signal twice, and the second trial was used in tracking performance evaluation, to enable familiarization without undue fatigue. Each subject was then asked to complete a questionnaire, with both Likert scale and free-form response elements, to evaluate their controller preferences.

B. Quantitative Tracking Performance
In the following analyses, we evaluate tracking performance of an exemplar subject, as well as across subjects and trajectory types.
1) Illustrative Trajectories: Fig. 7 shows time series tracking performance of subject Sub1 using each candidate signal. This subject and most others were able to complete the trajectory tracking task qualitatively well using both thickness/deformation and activation signal traces.
On the other hand, these data highlight the drift that occurs in our thickness tracking system, which we also observed in the correlation analyses in section IV-B. While most subjects could compensate for this drift somewhat when guided by visual feedback, they were often unable to return the signal to baseline at later stages of each trial (a challenge reflected in subjects' written feedback, as discussed below in section V-C). Activation-based tracking displayed no such drift, but showed many undesirable spikes and oscillations about the goal trajectory. In future iterations of this work, we will seek to ameliorate this noise with more aggressive mean filtering, though this will come at a cost of responsivity.
2) Tracking Performance by Subject: As shown in Fig. 8, most subjects completed the tracking task with reasonably low RMS error using either sensor -generally performing best using thickness/deformation-based control, underscoring the value of this novel control paradigm. Perhaps unsurprisingly, Sub7 (a subject discussed in section IV-B.2 as having irregular anatomy) was the exception to this success, and was largely unable to track the trajectory at all. Sub9, on the other hand, despite similarly irregular anatomy, was nevertheless able to adapt to the tracking system and achieve similarly low error to that of other subjects whose muscle deformation  Fig. 8. Tracking error during muscle deformation (blue) and activation (orange) trajectory tracking tasks across all subjects. With the exception of Sub7 -whose irregular anatomy prevented almost any deformationbased control -subjects were largely able to achieve better, or at worst comparable, performance when performing deformation-based control, as compared with our baseline activation-based system.  Fig. 9. Tracking error during muscle deformation (blue) and activation (orange) trajectory tracking tasks across various trajectory types and in aggregate, with noted standard deviation across subjects. During all but sine trajectories -which we again theorize were impacted by the drift in our deformation tracking system -subjects were able to consistently achieve lower tracking error using the deformation signal. more reliably correlated with force. These outliers underscore the importance of continuing to examine additional possible deformation signals as we work to build generalizable control systems, but also the promising adaptability of human users to suboptimal control signals in this space.
Subject preferences when asked to evaluate deformationand activation-based tracking tasks separately (top) and in head-to-head comparisons (bottom) on a 7-point Likert scale. Subjects largely found the deformation-based tracking task easier and perceived the deformation signal to better match their output force, though they rated the activationbased tracker as more responsive, and most preferred the deformation-based tracker overall. Full survey questions and responses are included with the open-source data release.
to activation-based tracking -at which subjects performed comparably regardless of trajectory type, aside from slightly higher and more variable error during the step condition, perhaps due to the combined challenge of modulating the activation signal both quickly and to arbitrary levelssubjects consistently achieved lower error using the thickness/deformation signal. The one exception to this improved performance was during the sine portion of the trajectory; we theorize that this diminished performance is at least partially due to the drift observed in our tracking software (and discussed in section IV-B.3), though more exploration is needed to validate this claim (e.g., temporally rearranging trajectory elements). Fig. 10 illustrates subjects' survey-reported tracking signal preferences, when evaluated both separately and comparatively. In both cases, subjects reported finding the thickness/deformation trajectory easier to control, and felt it better matched their perceived force trajectory, but that the activation trajectory was more responsive. Most subjects preferred the deformation-based controller overall -evidence that deformation-based tracking is not only feasible, but can be made intuitive for users.

C. Quantitative & Qualitative Tracking Preferences
In free-form responses, subjects noted many of the same qualitative characteristics of each controller that were observed by experimenters: many found that the deformation-based tracker was "better for maintaining a steady trajectory" but found it "hard to reduce the signal to 'at rest' level"; conversely, they noted the "quick response" of the activationbased tracker but found it "hard to maintain constant force" and "hard to not overshoot." One subject even noted that the two tracking schemes "essentially had opposite issues" and explicitly suggested sensor fusion. These comments -which were largely consistent with experimenters' own observations -will be used to inform future improvements to each tracking system in isolation (e.g., more aggressive low-pass filtering of the activation signal, improved tracking software to address drift and step artifacts) and in new, sensor-fused systems, as discussed below.
Subjects' full survey responses are included with the opensource data release.

D. System Limitations & Future Directions in Trajectory Tracking & Control
The promising results of this preliminary trajectory tracking study -in terms of both tracking accuracy and subject preferences -constitute strong evidence that it's possible, and even intuitive, to perform control with deformationbased signals. We are currently working to adapt this work to the control of physical devices, and in particular, to enable natural control of multiple degrees of freedom by extracting (highly localizable) deformation signals from multiple muscles simultaneously. Such expansions to real control applications will also require hardware enhancements, including the use of wearable ultrasound devices, to which this work should readily translate, many of which are under development [31][32][33].
At the same time, the limitations of this study -namely, the slower response time and drift associated with optical-flowtracked-deformation-based control -suggest compelling new control approaches leveraging both deformation and activation signals. We are currently exploring a number of control schemes that combine these signals to exploit the strengths of each (e.g., using baseline sEMG signals as a trigger to reset tracked ultrasound points, or formulating a consensus approach to maintain responsiveness while avoiding erroneous motion).

VI. CONCLUSIONS AND FUTURE WORK
In this paper, we have shown that a simple measure of muscle deformation -thickness change over time -is correlated with output force, and that individuals can leverage this signal to perform a trajectory tracking task. This set of studies represents the first step toward real-time measurement and understanding of the force-deformation relationship and a proof of concept that insight into this aspect of musculoskeletal dynamics can aid in the construction of intuitive control schemes. As we begin to build complex, humaninterfacing devices, continuing to expand our understanding of the underlying biomechanical system will be key -not only to extract intuitive control signals, but to maintain safety of the human user and avoid inducing pathological or injurious motion.
The SimTK OpenArm project constitutes a first effort at this type of holistic approach to the muscle force-deformation characterization and control problem, and we encourage researchers in the wider biomechanics, robotics, neuroscience, and vision communities to utilize and contribute to this project.