Neural-Network-Based Tactile Perception System Using Ultrahigh-Resolution Tactile Sensor

In this study, we developed the first tactile perception system for sensory evaluation based on a microelectromechanical systems (MEMS) tactile sensor with an ultrahigh resolution exceeding than that of a human fingertip. Sensory evaluation was performed on 17 fabrics using a semantic differential method with six evaluation words such as “smooth”. Tactile signals were obtained at a spatial resolution of 1 µm; the total data length of each fabric was 300 mm. The tactile perception for sensory evaluation was realized with a convolutional neural network as a regression model. The performance of the system was evaluated using data not used for training as unknown fabric. First, we obtained the relationship of the mean squared error (MSE) to the input data length <inline-formula><tex-math notation="LaTeX">${\bm{L}}$</tex-math></inline-formula>. The MSE was 0.27 at <inline-formula><tex-math notation="LaTeX">${\bm{L }} = {\bm{\ }}$</tex-math></inline-formula>300 mm. Then, the sensory evaluation and model estimated scores were compared; 89.2% of the evaluation words were successfully predicted at <inline-formula><tex-math notation="LaTeX">${\bm{L }} = {\bm{\ }}$</tex-math></inline-formula>300 mm. A system that enables the quantitative comparison of the tactile sensation of new fabrics with existing fabrics has been realized. In addition, the region of the fabric affects each tactile sensation visualized by a heatmap, which can lead to a design policy for achieving the ideal product tactile sensation.

evaluation for every new product development.Therefore, a quantitative tactile evaluation system is needed to replace this sensory evaluation.Various research results have been reported towards the development of a regression system for tactile perception.The information to use for regression can be broadly classified into the following two approaches: a model that uses information that can be measured with existing equipment, such as material constants [1] and surface shape [2], [3], and systems that use vibrations obtained from sliding on the target surface using tactile sensors [4], [5], [6], [7].Regression methods can be broadly classified into those using linear regression [1], [2], [3], [4], [5], [8] or nonlinear regression such as machine learning [6], [7].The latest research showed that a tactile perception system combining tactile sensors and a fully connected neural network can predict 68.2% of sensory evaluation scores caused by shape differences in a uniform material, even for unknown samples [7].To apply these tactile perception systems to the industry level, the use of sensory evaluation results from samples that have different material properties and shapes is necessary.On the other hand, in the tactile classification problem using machine learning based on neural networks [9], [10], [11], it is reported that tactile sensor improvement such as increasing the number of channels of information [10], and improving the spatial resolution and the sensitivity [11] can contribute to an improvement of classification accuracy for textures that are simultaneously different in terms of materials and shapes.Since the core of the neural network is feature extraction and both regression and classification models are realized with almost identical architectures, the above methods are considered effective approaches for tactile perception as regression problems too.Under these circumstances, our group has developed a tactile sensor that mimics a single fingerprint on a human fingertip [12], [13].The tactile sensor can measure surface shape and frictional force at the same point at the same time from contacting object and has an ultrahigh resolution for force, displacement, and horizontal spatial, which are more than that of a human fingertip [14], [15].In this study, an industry-level tactile perception system to estimate the tactile sensation of unknown fabrics using an ultrahigh-resolution tactile sensor is reported for the first time.Sensory evaluation was conducted by Unicharm Corporation, Japan, on 17 types of woven and nonwoven fabrics based on the evaluation important criteria for fabrics used in diapers development.The tactile signal of the same fabrics was measured using our tactile sensor.The regression model was created using a neural network to link human sensation and sensor signals.
The perception system's performance was evaluated using the measured signal of an unknown material, that is, fabric that was not used in the training of the network.As an application of the tactile perception system, we propose a design method to improve the tactile sensation by visualizing the contribution to the tactile score on a heat map.

A. Evaluation Samples and Method
Sensory evaluations were performed by Unicharm Corporation.Subjects were informed about the protection of personal information and their consent for its handling was obtained.The evaluation sample consists of 17 different fabrics: three types of fabric used for underwear, seven types of fabric used for clothing, and seven types of nonwoven fabric used for disposable diapers.The description of each fabric and the number of times the sensory evaluation was conducted is shown in Table I.The evaluation was conducted in two parts as follows: the first test included sample numbers 1-7 and 11-13 with 20 subjects, and the second test included sample numbers 1, 3, 4, 8-9, and 14-17 evaluated by 10 of the 20 subjects from the first test.The six words used in the sensory evaluation are listed in Table II.Each word was chosen for the following reasons as important for fabrics to be used in diapers:"smooth", "fluffy", "warm", and "thicker" (A-D) were employed as indices for expressing skin feel, "periodic" (E) was selected as an indicator of whether a nonwoven fabric feels like a woven cloth structure, and "preferable for underwear" (F) was adopted as a direct evaluation score to use a fabric for underwear.The Japanese words in Table II were used in the actual test.The evaluation employed a semantic differential  method with a five-step scale.Each word was scored from 1 (not applicable at all) to 5 (very applicable).The results were provided by Unicharm to Kagawa University after obtaining ethical approval for the study.

B. Results of Sensory Evaluation for 17 Fabrics
The results of the sensory evaluation are presented in Fig. 1, where each chart shows the mean score and standard deviation (SD) for each sample and each sensory evaluation word.Two points can be inferred from the results.First, there is a correlation between the "preferable for underwear" and "smooth" evaluation criteria.Second, there is a correlation among the "fluffy", "warm", and "thicker" evaluation criteria.

III. MEASUREMENT OF FABRICS USING TACTILE SENSOR
Fig. 2 shows photographs and the working principle of the tactile sensor that we developed [12].The sensor substrate is a p-type silicon on insulator with a 50-µm-thick device layer.The sensor structure consists of a sensing tip, a support spring, and an n-type piezoresistive sensing circuit on the support spring.The sensor is designed to mimic a human fingerprint.The sensing tip indicated in yellow in Fig. 2 has a diameter of 500 µm and Fig. 2. Ultrahigh-resolution MEMS tactile sensor [12]. is supported by a silicon spring with a spring constant of 0.05 mN/µm.These values are in accordance with our experimental results from human fingerprints.As shown in the inset of Fig. 2, when the sensor is scanned on an object, the sensor can obtain the vibration of the sensor tip as surface shape and frictional force on the basis of the voltage change of the detection circuit.The sensor has a force resolution of 9.9 µN and a displacement resolution of 0.17 µm, which are higher than the 131.2 µN [14] and 0.3 µm [15], respectively, of humans.The sensor signal corresponding to fingerprint vibration is considered to contain feature values of "smooth", "fluffy", "thicker", and "periodic".Also, since "warm" is affected by the air thickness as an insulating layer between the finger and the fabric, the feature value of "warm" is also contained in the shape signal.In addition, it is assumed that "preferable for is affected by each of the above factors.Therefore, it is possible to regress the sensory evaluation results using the sensor signal as input data.17 fabrics, the same ones from the sensory evaluation, were measured using the tactile sensor to create a training/test dataset.Each fabric was fixed on an x-axis stage, the sensor was fixed to the y-axis stage to control the contact force to the fabric, and the sensor scanned the fabric by moving the x-axis stage.The contact force between the sensor tip and the cloth was set to an average of about 0.2 mN, which corresponds to an experimentally obtained average contact pressure of 0.01 MPa between the finger and the fabric.Since the minimum element of the fabric is a fiber of 10 µm diameter, the spatial sampling rate in the scanning direction is controlled to 1000 samples/mm at a 1 mm/s scanning velocity and a 1000 SPS sampling rate of the datalogger.Fig. 3 shows an example of the obtained signal.It is confirmed that surface shapes and frictional forces are obtained from each fabric.The total scanning length of each fabric is 300 mm.Therefore, the total data length of the dataset is 5100 mm.

A. Architecture
The regression model used to connect the sensory evaluation and tactile signals was trained on the PyTorch 1.11.0+cu115 and Python 3.10.4environment.Fig. 4 shows the architecture of the convolutional neural network used in this study.The input tensor size is data length L × 2 channels (surface shape and frictional force), and the output vector has six components, with each component corresponding to a sensory evaluation score.First, to treat the different dimensions of each channel (length for surface shape, force for frictional force) equally, they are made dimensionless through standardization (mean: 0 and SD: 1).Then, feature extraction is performed through the use of four convolutional blocks.Each convolutional block consists of a convolutional layer (Conv1D), batch normalization (BatchNorm1d), and a rectified linear unit (ReLU).In the first two blocks, convolutions with five different filter sizes were performed in parallel to focus on different spatial frequency bands, and in the second two blocks, convolutions with a single filter size were performed.The number of output channel of the final blocks is six, which is equal to the output vector size.Finally, the feature value of each channel from the final block is averaged by a global average pooling layer (GAP).
Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

B. Training
The model was trained using the mean score of the sensory evaluation as the correct value.Table III shows the parameters used for model training in this study.The learning rate was reduced stepwise to ensure adequate loss convergence.17 models were trained to evaluate the generalization performance for unknown fabrics by cross-validation.Specifically, one of the 17 fabric data types was excluded as test data, assuming it to be an unknown fabric, and the remaining data of 16 fabric types were used as training data.Performance evaluation was then performed on the 17 models, with data from each unknown fabric serving as test data.

A. Model Performance Dependence on Input Data Length L
All evaluations were performed on 17 cross-validation models with the input of data of an unknown object, that is, fabric not used during training.The features just before the GAP in this model correspond to each sensory evaluation score at each location of the fabric, and the final model output is the average across the entire data length.Just as humans need to touch an object for a longer time or distance to identify slight differences in the objects, the model in this study requires a sufficient data Fig. 6.All sensory evaluation scores (colored) and estimated values using each cross-validation model (white).Each score is mean ± SD ( * * p < 0.01), and input data length L is 10 mm.Evaluation words are A: smooth, B: fluffy, C: warm, D: thicker, E: periodic, F: good for underwear.length for the averages of the features to converge.Fig. 5 shows the relationship between the mean squared error (MSE) and the input data length L. In each model, the loss converges with increasing input data length.The average MSE (L = 300 mm) of the 17 models was calculated to be 0.27, which serves as a performance index for the model in this study.

B. Accuracy of Estimation Sensory Evaluation Score
Fig. 6 shows the sensory evaluation scores and the estimated scores obtained from each model trained excluding fabrics for prediction, using an input data length of 10 mm.To determine whether the model sufficiently predicted the sensory evaluation scores, a threshold for allowable error should be defined.As in the past cases where sensory evaluation results were regressed on a neural network [7], if we assume that a two-tailed T-test result of p ≥ 0.01 is an adequate prediction, 80.4% at L = 10 mm and 94.1% at L = 100 mm of the evaluation words are adequately  predicted.However, since the system needs to predict the average value obtained by sensory evaluation within the original variability for fabric development applications, the threshold in this study was defined as the standard deviation.The accuracy of the model was calculated as the percentage of cases where the mean ± SD of the model's estimate for a given sample's evaluation word fell within the mean ± SD of the sensory evaluation results.The accuracy was 58.8% for L = 10 mm and increased as the data length increased, similar to the trend observed in Fig. 5, reaching 89.2% for L = 300 mm.These results suggest that it is possible to quantitatively evaluate the tactile sensation of unknown fabrics based on existing sensory evaluations.

C. Statistical Validation
All of the results presented below are based on predictions made using a data length L of 300 mm, which produced the best accuracy within the dataset.Fig. 7 shows a plot of the sensory evaluation values output by the model against the average of the sensory evaluations.The regression ability of the tactile perception system in this study was R = 0.914, and the root mean square error (RMSE) and mean absolute error (MAE) were 0.480 and 0.369, respectively (as shown in Fig. 7).We then compared the variability between human sensory evaluations and the model's predictions.Fig. 8 is a box-and-whisker plot showing the deviation between the sensory evaluation results and the model's predictions, each calculated on the basis of the average value of the human sensory evaluation.P-values were calculated using a two-tailed T-test.All of the sensory evaluation words showed high p-values above 0.84.These results indicate that the tactile perception system that combines high-resolution tactile sensor signals and convolutional neural networks is functioning properly.

D. Visualization of Tactile Sensation Using Heatmap
One potential application of the tactile regression model is to aid in the design of desired tactile sensations.The features extracted just before the GAP layer correspond to the sensory Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.evaluation scores at each position.By applying class activation mapping (CAM) [16] to this network structure, it is possible to visualize local tactile sensations on fabrics.Fig. 9 shows the measured waveforms of three fabrics: no. 3 (Underwear C) as the smoothest fabric, no. 12 (Diaper B) as the fabric with intermediate smoothness, and no.14 (Diaper D) as the least smooth fabric, along with the heatmap of each sensory evaluation word.For all evaluation words, darker colors correspond to locations with higher scores.In (a), the three indicators of smoothness, periodicity, and desirability for underwear are colored more extensively.In (c), some of the indicators are not colored (neurons are not firing), but even the low-score fabric is partially judged to be smooth.If we aim to develop fabrics that are smoother than No. 14, this heat map can be used to provide guidelines for developing a smoother fabric, such as improving the fabric in regions that are colored in the heat map.

VI. CONCLUSION
In this study, we developed a tactile perception system that connects sensory evaluation results, using six indices for 17 clothing fabrics, with measurement data from ultrahighresolution tactile sensors.The system's prediction accuracy can be improved by increasing the input data length, just as humans can better identify an object with a longer touch duration or distance.With an input data length of 300 mm, the evaluation results showed an achieved MSE of 0.27, R of 0.914, and 89.2% accuracy for each sensory word.The system enables a quantitative comparison of the tactile sensations of new and existing fabrics.If a higher estimation accuracy is desired, the Multi-tactile Scanner [17], which can easily measure and evaluate the surface texture of a small area in real time, can be used to increase the data length.Finally, we discussed further potential applications of this system, including the use of CAM-based heatmaps to visualize good fabric regions and clarify improvement strategies.These results are expected to contribute to the development of materials with superior tactile sensation and the elucidation of the physical factors that constitute tactile sensation as an unconventional tactile analysis tool.

Fig. 5 .
Fig. 5. Relationship between MSE and input data length L on each crossvalidation model (color) and average (black).

Fig. 7 .
Fig. 7. Relationship between actual and predicted scores using each crossvalidation model when input data length L = 300 mm.

Fig. 8 .
Fig. 8. Deviation from the mean in sensory evaluation score.Human: result of sensory evaluation (n = 300).Estimation: estimated score using each crossvalidation model (n = 17) when input data length L is 300 mm.P-values were calculated by a two-tailed T-test.

Fig. 9 .
Fig. 9. Heatmap to vision of important region of fabric.(a) Underwear C as the most "smooth" fabric, (b) diaper B as the fabric with intermediate "smooth", and (c) Diaper D as the least "smooth" fabric.

TABLE I SAMPLES
USED FOR SENSORY EVALUATION

TABLE II WORDS
USED IN THE SENSORY EVALUATION TEST