Evaluation of a 3D Printed Soft Sensor for Measuring Fingertip Interaction Forces

—Current force sensors used to capture ﬁngertip interaction forces lack compliance to the ﬁngertip tissue resultingin the loss of touch sensationof the user. 3D printing offers the possibility to create personalized soft sensing structures. This work evaluates a 3D printed soft sensor that measures normal and shear interactions forces based on the deformations of the thumb and index ﬁngertips of 7 subjects using an instrumented object. Due to the use of (carbon doped) thermoplasticmaterials, the signalsprovidedby these sensing structures suffer from nonlinearities. Therefore, two compensation models, based on a neural network and recurrent neural network analogousto an electrical model are used to compensate for the nonlinear effects. The performance of the sensors was analysed using the normalized cross-correlationand the root-mean-square error. The output of the force sensors are highly correlated with the applied shear and normal force components. When paired with compensation models the correlation and error of the sensor output can be further improved. These results indicate that the proposed ﬂexible ﬁngertip interaction force sensors have a high potential for future applications.

Adding information about normal and shear interaction forces at the fingertips during grasping tasks to the kinematic measures, is expected to highly improve the clinical relevance of these measurement systems, since these systems can provide more insight in movement control and effectiveness, allowing for further study on grasp control deficits in stroke subjects [4].
One approach to study the interaction grasping forces is by means of sensorized objects. These devices consist of faces built around the core of a force/torque transducer [5], [6] or have a shell of multiple piezoresistive force sensors [7], [8]. Using a matrix of force sensors allows for freedom of sensor placement and the recording of individual finger interaction forces. Although these sensorized objects give a reliable measure of the interaction forces, they are limited by the number of grasping strategies that can be used to manipulate the object and therefore do not represent a wide variety of daily tasks with ordinary objects. The second approach is by mounting force sensors on the fingertips. Such sensors have been developed e.g. by Brookhuis et al [9] and demonstrated by Kortier et al. [2] and Battaglia et al. [10]. Although, these interaction force sensors provide detailed information and can capture a wide range of grasping tasks, these sensors are not ideal to use on the fingertips since they are not compliant to the shape and stiffness of the fingertips resulting in poor sensor to skin attachment and the loss of touch sensation of the user [2] limiting the usefulness and reliability [4]. The compliance of force sensors can be increased in various ways, including the use of flexible circuits, soft and flexible packaging [11], the combination of soft and compliant polymers with flexible conductors such as conductive textiles [12], conductive liquids and inks [13]- [15] or by the use of conductive flexible polymers [16], [17]. The emerging developments in 3D printing sensors and the increasing availability of soft, flexible and conductive materials [18]- [20] make this technology a suitable candidate for the development of compliant fingertip sensors [14], [17]. The ability for local manufacturing i.e. distributed manufacturing and the easy to fabricate and customize sensors offered by 3D printing makes this technology an ultimate candidate to fabricate personalized sensing structures [19].
This work presents the evaluation of soft 3D printed fingertip sensors that use mechanical deformation of the soft tissue around the fingertip bone (distal phalanx), during grasping tasks. A preliminary version of this sensor was introduced and investigated using a finite element model and mechanical measurement setup in [17]. That work showed that the output of these sensors is related to the design of the sensing structure and to the position of the strain gauges. Hysteresis and other nonlinearities in the sensor output are associated with the used carbon infused thermoplastic polyurethane materials. Moreover, the gauge factor of these materials is dependent on the amount of strain, the strain rate and time.
This work introduces an adapted sensor design that allows for an improved fit on fingertips and evaluates the sensor for the first time on human fingertips using an instrumented object as reference. Additionally, this works aims to improve the signal quality of the 3D printed sensor with the use of strategies that compensate for the nonlinear effects caused by the material properties and geometry of the fingertip sensor. Figure 1 shows the design of the fingertip sensing structure. This design is based on an earlier concept of the finger sensor published by Wolterink et al. [17]. The presented sensor concept uses the mechanical deformation of the finger tissue around the bone of the fingertip (distal phalanx). This deformation is sensed by a flexible strap with two embedded strain gauges, wrapped around the fingertip [17]. To improve the fit on human fingertips the sensing strap is designed in an arch shaped geometry ( Figure 2). The ends of the strap mount to a mounting plate that fits on top of the nail and can be changed in length according to the subjects finger size. The strap has a width of 15 mm and a thickness of 0.6 mm and is made from non-conductive TPU (NinjaFlex, Fenner Drives, Manheim, PA, USA). The embedded strain gauges have a thickness of 0.20 mm and are made of carbon doped conductive TPU (PI-ETPU, Palmiga Innovation, Jonstorp, Sweden). The properties of both materials are listed in Table I. To allow for easy electrical connection the ends of the strain gauges have connection pads that reach out of the strap.

III. METHODS A. Fabrication
The fingertip sensing structures were printed in one go using a Diabase H-Series Hybrid (Diabase Engineering, USA) multimaterial FDM 3D printer. Control and printing settings were handled by the slicer software (Simplify3D, Inc., USA). The layer height of the structure was set to 100 µm, resulting in a strain gauge with a thickness of two layers. Subsequently, the structures were annealed overnight at 80 • C in an oven. Electrical interfacing to the strain gauges was made by melting a fine stranded copper wire into the connection pads. The resistance measured between the contact pads of the strain gauges is around 1.8 k.

B. Setup
To validate the performance of the 3D printed fingertip sensors during grasping tasks, two identical force sensors were used. One sensor was placed on the thumb and the second sensor was placed on the index finger of the subject. The two strain gauges of each force sensor were read out using voltage divider circuits with 2.2 k resistors as shown in Figure 3. Each channel represents one strain gauge and was read out using the auxiliary inputs of a TMSi Saga amplifier (TMSi, Oldenzaal, The Netherlands) with a sample frequency of 1 kHz. The same amplifier was used to provide a voltage of 5 V to the voltage divider circuit.  The reference force was provided by an instrumented object shown in Figure 4. This object was composed of two 6-axis force-torque (FT) sensors (K6D27 50N/1Nm, ME-Meßsysteme GmbH, Hennigsdorf, Germany) mounted at two opposite sides of a stainless steel disk. The FT sensors were read out at a sample frequency of 150 Hz using two synchronized amplifiers (GSV-8DS, ME-Meßsysteme GmbH, Hennigsdorf, Germany). The metal bases of the force sensors, where the fingertips interact with the object, were covered by a 3D printed plastic disk with a diameter of 30 mm for insulation purposes. The distance between the two interaction interfaces was 75 mm. The total weight of the instrumented object was 346 g. A synchronization pulse was provided by a Xsens Awinda Docking Station (Xsens, Enschede, The Netherlands) connected to a personal computer via a USB connection. This pulse was captured by the Saga amplifier and triggers the FT-sensor amplifiers that were all connected to the same computer via a USB connection.

C. Protocol
Seven healthy adult (18+) volunteers participated in the measurements that were approved by the faculty ethics committee, all participants gave written informed consent prior to the start of the experiment. Two 3D printed force sensors were placed on the right hand thumb and index finger of the subjects. The subjects were instructed to perform five recording sessions (tasks 1 to 5) of 3 minutes were the subjects picked up and placed the instrumented object multiple times between the sensorized thumb and index finger at the subject preferred speed. During task 1, 2 and 4 the subjects were asked to pick and place the object multiple times while performing random rotating and shaking movements in random directions, while the instrumented object was being pinched between their thumb and index finger. Task 3 and 5 involved a controlled movement where the subjects were instructed to only pick up and place the object repeatedly for 3 minutes. Between task 3 and 4 the sensors were doffed and re-donned on the fingertips to study the repeatability of the sensors after doffing and donning.

D. Data Preparation
All further data processing was performed offline using Matlab (Mathworks Inc., Natick MA, USA). The raw data of the FT sensors used in the instrumented object were converted to the normal (F n ) and shear (F x ) force components using the calibration matrix supplied by the manufacturer. To remove any bias force the mean over the first second, which does not contain any interaction forces, was subtracted of the whole recording. The same procedures were repeated to remove the bias voltage of each channel of the 3D printed fingertip sensors. Next, both the forces from the instrumented object and the voltage over the strain gauges of the fingertip sensors were low-pass filtered at a cutoff frequency of 10 Hz, using Matlab's lowpass function, since human motion generally takes place a lower movement rates [4]. Subsequently, the data from the fingertip sensors was re-sampled at 150 Hz using Matlab's resample function to match the data of the FT sensor. Finally, the recordings of each task were segmented to remove the idle parts before and after each task.
To estimate the force output of the printed sensors, a linear model was fitted using the data set of task 1 for each finger and subject. The fit was performed using Matlab's polyfit function set to a first order polynomial. The normal force sensor output (F n ) was given by the sum of the two strain gauges. The shear force (F x ) was given by the difference between the two strain gauges. These models were consequently used to provide an estimate of the normal and shear force of the other performed tasks.

E. Nonlinearity Compensation
3D printed resistive strain sensors fabricated using soft polymer materials have shown to suffer from strong nonlinear behaviour [17], [19]. To be able to use the finger sensor to its full potential two independent compensation strategies are applied: a neural network approach and a modified Power-Law (mPL) model.
The first compensation strategy involves a dynamic neural network (two-layer feed-forward network), where past values are used to predict future values. These networks can be trained by using a training set with a sufficient number of samples of the two fingertip sensor strain gauges and a target data set. The target dataset contains the applied normal (F n ) and shear (F x ) force components of the FT sensor. This network was trained using the Matlab neural network

time series toolbox (ntstool) with a nonlinear input-output function [23]
The second compensation model is based on the work of Schouten and Kosmas et al. [24], [25] and is referred to as the modified Power-Law (mPL) model. The mPL model estimates the hysteresis and creep behaviours using a recurrent neural network that is analogous to a the forward Euler simulation of an electrical circuit. This network also consists of two layers, see Figure 6. The first layer uses a hyperbolic sine f (x) = P S e x P T − e −x P T as activation function. The output layer uses a 4th order polynomial g(x) = g 0 + g 1 x . . . g m x m . Both weights and the parameters of both activation functions are included in the optimization. The used fitting method is based on Matlab's optimization algorithm for obtaining a function's global minimum [26]. The global problem is defined as multiple local optimization problems by means of GlobalSearch, with the algorithm's default configuration [27] and the fmincon method configured with an interior-point algorithm [28] as described by Kosmas et al. [25].
1) Model Training and Validation: Two types of training and validation were performed, to investigate if a generic inter subject model for the sensor is sufficient or whether each subject needs an individual model (intra subject model). Inter subject models are based on a model trained on the task 1 data set of subject 1 and were validated using all tasks of all subjects. Intra subject models involve models trained on task 1 of each individual subject and are only validated by the tasks performed by the same individual subject. For both the thumb and the index finger sensor a separate compensation model was trained to avoid for potential differences in sensor size or printing inaccuracies. In all situations the forces obtained form the FT sensors were used as target data.
The Matlab neural network time series toolbox (ntstool) with a nonlinear input-output was used to train the neural networks. The input data consisted of 2 input signals, each representing one strain gauge of the sensor. F n and F x of the FT sensor represent the two elements of the target data. The network consists of 10 hidden neurons and 50 delays. This number is found as a good balance between training time and performance.
Two separate modified Power-Law (mPL) compensation models were trained, the first model estimates the normal force F n using the sum of the two strain gauges as input. The second model estimates the shear force F x using the differential signal between the two strain gauges. The weights w 0 , w 1 , bias V b , weights P T and P S of the hyperbolic sine function [25], along with the coefficients g 0 . . . g m , were estimated using the experimental data for training and validation. The initial parameters before fitting are shown in Table II. To speed up the training the data was down-sampled by a factor 2, which resulted in a time step t of 13.3 ms.

F. Statistiscs
The correlation between the output of the models and the validation data was determined by the normalized crosscorrelation (NCC). Since the model input and validation data are synchronized, the NCC values were taken at zero lag for further analysis. The error between the model output and validation data was calculated using the root-mean-square error (RMSE). The Wilcoxon signed rank test was used to determine whether the NCC and RMSE values between tasks, validated using the compensation models differ or not. This measure was furthermore used to validate differences in performance between the applied compensation models. The Wilcoxon signed rank test was chosen due to the expected non-normal distributed data and small sample size.

IV. RESULTS
All seven participants completed the experimental protocol. The performance data of the second random movement (task 2) of subject 5 was discarded due to a malfunction in the synchronization between the finger sensors and the instrumented object. The finger sensor straps of the thumb were to their maximum size (3) for subjects 1,2,3,6 and to medium (2) for subjects 4,5,7, the straps for the index finger were set to medium length (2) for all subjects. Figure 7 shows parts of the training and validation data recorded at the thumb, the graphs on the left represent normal forces and the graphs at the right represent the shear forces. The low pass filtered output of the strain gauges of task 3 performed by subject 7 is shown in Figure 7a,b. The same fragment is shown again in the bottom graphs (e,f) after the compensation models trained on task 1 of subject 1 (c,d), are applied.   Figure 7 including the output provided by a linear fit performed on task 1. Figure 8 shows the output of a linear fit performed on task 1 of each subject for the normal force (F z ). This fit is determined by the sum of both strain gauges (Ch1 + Ch2). In the shear force (F x ) situation this fit is made using the difference between the two strain gauges (Ch1 -Ch2).
Since no significant difference (P < 0.05) in normalized cross-correlation (NCC) at zero lag at the training sets of the thumb and index finger for both the neural net and the modified Power-Law model (mPL) were found, the performance of the thumb and index finger were grouped in further analysis to double the number of measurements. Each finger sensor (thumb and index) is still compensated with the compensation model for that sensor. Figure 9 shows the box plots of the NCC at zero lag for both normal force (F z ) and the shear force (F x ) data obtained from each task of the non compensated data (Linear fit) and analysed using a neural network (Neural Inter/Neural Intra) or the modfied Power-Law model (mPL Inter/mPL Intra). The suffix 'inter' refers to models trained on subject 1 task 1 and applied on all subjects, suffix 'intra' refers to models trained on each individual subject on task 1. Figure 9 shows the box plots of the NCC at zero lag for both normal force (F z ) and the shear force (F x ) data obtained from each task of the non compensated data (Linear fit) and analysed using a neural network (Neural Inter/Neural Intra) or the modfied Power-Law model (mPL Inter/mPL Intra). The suffix 'inter' refers to models trained on subject 1 task 1 and applied on all subjects, suffix 'intra' refers to models trained on each individual subject on task 1.
The significance of the differences between the data presented in Figure 9 is shown in Figures 11 to 13 (Appendix). Figure 11 shows the significance of the differences of the NCC between the linear fit and the compensation models. Figure 12 shows the significance of the differences between compensation models using the same analysis method and Figure 13 shows this significance of the differences between the performed tasks. Figure 10 shows the root-mean-square error (RMSE) for both normal force (F z ) and the shear force (F x ). The graph clearly shows an order of magnitude lower RMSE for the sensor output compensated with the mPL models with respect to the linear fit and the output compensated using a neural network. The corresponding P-values of the differences between the linear fit and the models are shown in Figure 14 in Appendix . Figure 15 in Appendix shows the statistical power between the compensation models and Figure 16 between the tasks.

V. DISCUSSION A. Nonlinearity Compensation
The time domain data shown in Figures 7 and 8 indicate a high correlation between the target data provided by the instrumented object and the sensor output. As shown by Figure 9 the use of compensation models further increases the correlation of the printed fingertip sensors with the reference force, this effect was mostly significant for the mPL models as shown in Figure 11.
Although, the linear fit shows a high correlation with the reference signal, Figure 8 shows a large offset between the reference force and output of the linear fit. This offset is mainly caused by three factors: lower amplitude, drift and relaxation time. The last effect is strongly visible at the situation where the applied force rapidly decreases and is a typical behaviour for the used conductive polymer composite materials [17], [20], [29]. The relaxation time and drift are largely eliminated with the use of the neural networks and mPL models. As shown in Figure 10 the used mPL models provide a significant ( p ≤ 0.01) lower root-mean-square error (RMSE) and therefore these models seem to be the best compensation strategy for this sensor and experiment.
Notwithstanding the fact that the performed analysis allows to make a comparison between the performance of neural networks with respect to the mPL model, no conclusions should be drawn regarding the performance of these type of models. Both compensation models used in this work are preliminary versions using a limited number of parameters and provide many opportunities for improvements.
The mPL model with a single cell is only able to capture hysteresis behaviour described with local memory, since the model's global memory arises as a collective property from the multiple capacitor states when the number of cells is expanded (n cell > 1) [30]. Whether the hysteresis exhibited by the sensors can be described with local or global memory is yet to be evaluated. To evaluate this in a more systematic way, a generated system input on the finger sensor using a mechanical setup, such as performed on a previous design shown in [17] is needed.
In case of the neural network, care needs to be taken with increasing the number of neurons since the models fitted on normal force data seem to suffer from overfitting. In terms of training speed the neural network needed a computation time of around 4 to 6 minutes to reach satisfying training results. The mPL model used in this work reached this point usually around one minute. These model fitting times are obtained using a third generation (2.6 GHz Dual-Core) Intel Core i5, and highly vary depending on the used amount of training data, parameters, fitting iterations and hardware performance.

B. Normal Force Performance
As shown in Figure 9 and 12 the training data (task 1) in the normal force direction, is higher correlated when the models are trained for each individual subject (intra subject). The same effect can be seen in Figure 10 and 15, where intra subject models show a lower RMSE. This suggests that the compensation models are optimized to individual users. However, in the validation tasks (task 2 to 5) this effect becomes less prominent and significant. The inter mPL model shows a significantly higher median NCC value on all tasks compared to the neural network for the normal force data. Furthermore, the RMSE of the mPL model is an order of magnitude lower. This all shows that for estimation using a generic sensor model, the mPL model is in this situation, the best option to obtain the normal force component of the 3D printed finger sensors.

C. Shear Force Performance
In the shear force direction the mPL models show a larger spreading in the NCC values compared to the neural network. This might be due to the higher number of parameters and the ability to train the neural networks using the two separate inputs provided by the strain gauges of the finger sensor, whereas the input for the mPL model is only the difference between the two strain gauges. However, differential measurements were taken since these can highly increase the linearity of the response for 3D printed piezoresistive sensors [31].
The RMSE in Figure 15 shows an opposite trend; the median RMSE of the mPL results is an order of magnitude lower compared to the linear fit and neural networks and in the same range as the RMSE of the normal force. However, most of the signal power is in the normal component, as shown in Figure 7 where the amplitude of the normal force components is around one order of magnitude higher with respect to the shear force components. Therefore, the relative errors in the shear force situation are an order of magnitude higher with respect to the normal force situation. This could result in lower NCC values in some occasions in the shear force components, resulting in a higher variation and spread in NCC values.

D. Influence of Re-Donning
In the situation where re-donning the 3D printed sensors affects the performance of the sensors one might expect a lower NCC and a larger RMSE in the tasks performed after re-donning the sensors. If only one compensation model is used for multiple subjects, the sensors are re-donned between subsequent subjects and could therefore be a factor reducing the performance. However, the results do not provide an evident conclusion on the effect of re-donning the force sensors. Figures 12 and 15 do not show a consequent strong statistical power ( p ≤ 0.05) of differences between the inter and intra subject fitted compensation models. Figures 13 and Figure 16 do show small significant differences in some situations between the tasks before and after re-donning (task 2-4 and 3-5), indicating possible performance change caused by re-donning of the sensors. However, the controlled movements before and after re-donning the sensors (task 3 and 5) show a significant decrease of the RMSE after re-donning for the linear fit and mPL model trained intra subject. For these reasons, the effect of re-donning is not evident and in case a general inter subject fitted compensation model is used, no notable negative effects of re-donning are expected.

E. Limitations
The current setup does not record the temperature of the fingertip sensors. Previous studies have shown a temperature coefficient of resistance (TCR) of the used carbon doped conductive TPU to be around 0.002 • C −1 in the used temperature range of this application [32]. Due to the expected relative slow change in temperature, the effects might occur as drift in the normal force estimation. The effect of temperature will not be of influence on the shear force since this signal is mainly based on the difference between the two strain gauges.
Besides the used thermoplastic material in the finger sensor, the human fingertip tissue is also a highly nonlinear material with characteristics comparable to viscoelastic materials [33], [34]. Since the skin tissue is part of the sensing structure these characteristics will be present in the raw sensors output, but potentially are compensated by the presented compensation models.
Despite the possibility offered by 3D printing to develop custom sized sensors fitted to each individual subject's fingertip circumference, this preliminary study uses a strap-like structure to change the size of the fingertip sensor. Since this strap only has three discrete length configurations, the initial strain of this sensor around the subjects fingertip might vary. However, analysis between the inter and intra trained compensation models shows no clear evidence this effect negatively affects the NCC and/or RMSE.
The used instrumented object only contained two 6-axis force-torque sensors, allowing the current experimental setup to only validate pinch grasps, whereas in all day grasping activities the complexity of grasps usually involves more fingertips. Furthermore, in many all day human grasping activities more areas than only the fingertips, such as the phalanges or the hand palm, are also involved by supporting the object.
The contact area of the force sensors on the instrumented object was relatively small. Although the subjects were instructed to carefully place their complete fingertip on the force sensor area, in some situations it could not be avoided that the subjects only placed a part of their fingertip on the object. Resulting in discrepancies between the force sensors of the instrumented object and the finger sensors.
In some occasions, accidental fingertip interactions with other objects such as the table or the subjects own fingers occurred. These interaction recordings occur in the data sets of the printed sensors and are absent in the validation data provided by the instrumented object, reducing the NCC and increasing the RMSE values since there is no correlating validation signal.
It is assumed that the shear component of the 3D printed finger sensor is perpendicular to the F x -component of the instrumented object. However, the angle between these two components might vary with each grasp. In future research these errors could be compensated by using small inertial sensors placed on the fingertip nails such as described in [3], [4], to estimate the orientation of the shear component with respect tot the instrumented object or gravity.
Since the subjects are asked to lift an object, the performed tasks have a large coupling between shear and normal force component due to the inherent functioning of the 3D printed sensor. This coupling could lead to an unwanted trained correlation in the neural networks. To avoid this issue the subjects are asked to shake and rotate the object during the training tasks to generate decoupled forces. However, applying forces to a fixed reference sensor could help to increase the amount of decoupled data.
The soft fingertip sensors are capable of detecting the shear force component; Figure 8 clearly shows the increase in shear force component when the instrumented object is lifted. These signals are of high value for the development of grasp detection algorithms that are of relevance for assessment of upper limb impairments [4]. One of the potential applications of these sensing structures is the study of grasp control deficits in stroke patients. In these situations measuring the relative change in grasping force will already be a valuable addition. Therefore, the sometimes relativity large absolute error of the force as measured by the fingertip force sensors, is not necessarily a limitation.
This study has demonstrated a working concept of a soft 3D printed piezoresistive fingertip force sensor. Although the signal output of these sensors is far from perfect relative to commercially available solutions, the compliance of the sensors to the fingertip makes these sensors useful for clinical and all day use. Furthermore, 3D printing of force sensing structures offers benefits over commercial sensors since the Fig. 11. P-value between the linear fit and the compensation models as shown in Figure 9.  Figure 9 between compensation models at each task. Fig. 13. P-value showing the significant differences between the NCC value at zero lag of the performed tasks in Figure 9. Task 1 is the training data in all situations. fabrication technology allows for easy customization. This study does not investigate the users touch sensation, although it is expected that the use of compliant materials to the finger tissue and the open structure at the fingertips allows for improved touch sensation compared to currently available stiff sensors. These benefits highly increase the potential for adoption of this sensing technology.

VI. CONCLUSION
This work demonstrates the potential of 3D printed flexible fingertip force sensors to be used in practical applications.  Figure 10 between compensation models at each task. The correlation and the error between the output of the force sensors and the applied shear and normal force components can be highly improved with the use of models that compensate the nonlinear characteristics of the used sensors.
The signal quality of the fingertip sensors might be improved in the future since the used compensations models are relatively simple and are still in an early development phase. Nevertheless, the current state of these sensors might already be sufficient for elementary grasp detection applications that use the relative change in force. 3D printed fingertip sensors provide many benefits over commercial sensors due to their compliance and the ability for the users to keep their touch sensation. Bert-Jan F. van Beijnum (Member, IEEE) received the M.Sc. and Ph.D. degrees in electrical engineering from the University of Twente, Enschede, The Netherlands. He is currently an Associate Professor with the Biomedical Signals and Systems Research Group, University of Twente. His research is embedded in projects of the Techmed Centre and the Digital Society Institute. His research focus is on smart monitoring, analysis and decision support of human movement, physiological processes for the remote monitoring and management of chronic conditions and sports applications by combining data and signal analysis, modeling, and machine learning.