Sphygmopalpation using Tactile Robotic Fingers Reveals Fundamental Arterial Pulse Patterns

Sphygmopalpation at specific locations of human wrists has been used as a medical diagnostics technique in China since the Han Dynasty (202 BC - 220 AD) and it is now generally accepted that traditional Chinese medicine (TCM) doctors are able to decipher at least 28 fundamental pulse patterns among all patients using their fingertips. However, unlike collecting EEG (electroencephalography), ECG (electrocardiography), and EMG (electromyography) signals, there is no standardization on how the arterial pulse waves from the TCM sphygmopalpation methods should be digitalized and analyzed. We have developed a pulse sensing platform for studying and digitalizing arterial pulse patterns via a TCM approach. This platform consists of a robotic hand with three pressure-feedback-controlled robotic fingers (each with 4×6 sensing pixel arrays) for pulse measurement and an artificial neural network (ANN) for pulse pattern recognition. Data analyses reveal that 3 types of consistent pulse patterns, i.e., “HUA” (滑), “XI” (細), and “CHEN” (沉) – key fundamental pulse patterns described by TCM doctors – could be identified in a selected group of subjects. The classification rates are 99.1% in the training process and 97.4% in testing result for these 3 basic pulse patterns. The results will lead to further development of a high-level artificial intelligence system incorporating knowledge from TCM – the robotics finger system could become a standard clinical equipment for digitalizing and visualizing human arterial pulses.


I. INTRODUCTION
Traditional Chinese Medicine (TCM) has been used for healthcare in China for more than two thousand years. TCM physicians use four diagnostic methods including inspection, auscultation and olfaction, inquiry, and palpation to collect clinical information in order to make diagnosis for the constitution and syndrome pattern recognition. TCM sphygmopalpation (TCMS) [1], a combination of human arterial pulse sensing and diagnosis, has been used by TCM physicians since the Han Dynasty (202 BC -220 AD). Different than Western medicine practitioners, which use the palpation for estimating cardiovascular functions based on the pulse rates and rhythm, experienced Chinese medicine practitioners (CMPs) can use their fingers' sensations and their own experience to draw conclusions about patients' holistic health status. According to the theory of TCM [1] and classical TCM concepts recorded in an ancient Chinese masterpiece called "Mai Jing" ("The Pulse Classic") [2], arterial pulses detected at three different locations (i.e., "CUN" (寸), "GUAN" (關) and "CHI" (尺) of both wrists reflect the health conditions of the internal organs, i.e., if there is any change of physiological states in the internal organs and the related functions, the characteristics of the pulse will be affected, forming its unique diagnostic basis.
Experienced TCM physicians have developed advanced skills to sense the changes of the pulse patterns for their diagnosis. Although there are many written rules and wellproven records of the success of TCMS, the communication of the corresponding knowledge and skill is still based on individual understanding and experience, which needs a scientific verification. In contrast to electrocardiography (ECG), which already has a standard data acquisition procedure [3], the development of TCMS standardization and arterial pulse digitization is important for reliable and consistent diagnosis. Therefore, it is extremely critical to study TCM arterial pulses by a scientific, quantifiable, and reliable approach. It is also important to collect correct pulse signals based on TCM theories and methodologies. Hence, an ideal measurement system for mimicking the fingers of CMPs must have the ability to provide different levels of pressure, measure arterial pulses precisely at the "CUN", "GUAN" and "CHI" points, and classify at least 28 arterial pulse characteristics according to TCM theory [4] [5], as shown in Fig. 1. For the past few decades, vigorous academic TCM research efforts in China [6] - [9], Taiwan [10] - [12] and Hong Kong [10] [11], as well as other locations [15] [16], have pursued the development of useful instrumentations to capture the arterial pulse at the human wrist. These machines have often exhibited design issues that hinder their development. Some of them, for example [17], simply used a single pressure sensor for arterial pulse recording and diagnosis without considering the position where the pulse should be taken. Other machines [18] - [21] recorded pulses from different positions but still omitted the importance of the pressure applied on the wrists. However, in TCMS, a conclusive pulse diagnosis can be achieved only by analyzing all pulses measured from the three key positions (CUN/GUAN/CHI) and their variations under different applied pressures.
In addition to the development of instruments for pulse measurement, pulse pattern classification and analysis are also important for standardizing TCMS. The most common and effective technique is the use of artificial neural networks (ANNs), which is a machine learning method. Four important design parameters influence the performance of an ANN: data source, the size of the training sets, input features and output targets, and network structure [22] [23]. Some ANNs have been developed for TCM (e.g., see [24] - [27]) and diagnostics using medical images (e.g., see [28] - [33]). The source of the data collection was either a simple pulse monitoring system [34] [35] or arterial pulse reference books [36] [37]. The performance of these algorithms is restricted due to the limited amount and range of data obtained from a single patient. Additionally, none of these networks has addressed the influence of the arterial pulse under different applied pressures, i.e., these networks were developed without strictly following the written rules of TCMS; therefore, their performance may be far from that expected of TCMS.
In this paper, we present our development of a novel pulse sensing platform (PSP) that can record and classify human arterial pulses via the TCMS approach. This platform can be divided into two major parts: a palpation robotic hand (PRH), which consists of three robotic fingers for pulse measurement, and a dedicated control and signal processing algorithm for pulse data filtering and classification. The developed system can eventually adopt and learn from different practitioners who may belong to different disciplines of TCM and may have different interpretations of arterial pulses. For instance, the TCM disciplines for arterial pulse-based diagnosis can be divided into at least two methods: simultaneously palpation with 3 fingers (which is used to verify the entire trend of body state), and palpation with only one finger (which is used to identify the unique characteristics of viscera and bowels) [38]. With further development and big data analysis, this system can provide a conclusive pulse diagnosis and will benefit the development of more reliable and accessible TCM by providing quantifiable sphygmopalpation arterial pulse information. All arterial pulse data analyzed in this study was obtained by our designed palpation robotic hand (PRH), as shown in Fig.  2(a). The movement and the applied force (Fr) of the finger, as shown in Figure 2(b), are achieved by the force balance between a metal string (Ts) and a restoring spring (Fk). A more detailed discussion about the machine designs can be found in the reference [39] [40]. The machine consists of three robotic fingers, as shown in Fig. 2(c), that are driven by three individual driving-torque motors via metal strings. Each fingertip is curved similarly to a human finger and mounted with a flexible capacitive sensor. The sensors are custommade sensors from Pressure Profile System, US [41]. Each sensor has 4×6 sensing elements with an element size of 2mm×2mm. The working range of each element is from 0 to 9 psi with a repeatability of 0.7%. The resultant pressure can be calculated by averaging either all 24 element readings or a selection of individual sensing elements. One of the major advantages is that an array of sensors will give better position tolerance, which reduces the positional accuracy requirements of the robotic fingers. Moreover, the sensor array can give both temporal and spatial information on a pulse. To show the actual temporal arterial pulse picked up at 3 locations (i.e., "CUN", "GUAN" and "CHI", called CGC in this paper) under 3 applied pressures (i.e., "FU", "ZHONG" and "CHEN", called FZC in this paper), the selected data were plotted using 3D color contour maps in Result section.

A. DATA ACQUISITION
In addition, three unique features of the machine design ensure the accuracy and repeatability of CGC pulse measurement positions on the subject's wrist. First, interfinger distances can be manually adjusted. This ability is an extremely important feature since CGC positions vary among individuals. For example, a person with a shorter forearm will have CGC positions closer together. Second, three LED lasers are used to show the expected finger positions on the subject's wrist. The task, then, is to align these 3 laser spots with the CGC positions on the wrist. Third, each finger can be individually actuated and can maintain a precise applied pressure depending on the surface topography of the skin and the various biological tissues and bones underneath.

B. DATA LABELLING
Professor Jiangang SHEN from the School of Chinese Medicine of the University of Hong Kong, a TCM expert, conducted sphygmopalpation and performed pulse pattern diagnosis on 15 volunteers. Their pulses were recorded using our PRH and processed as mentioned above. Out of the 15 volunteers, 3 volunteers were specifically selected to go through a comprehensive pulse measurement using the PSP system due to the consistency of their pulses and were confirmed to have no cardiac abnormalities in the past; these 3 volunteers also showed different distinguishable waveforms, which were diagnosed by Professor SHEN as having "HUA", "XI" and "CHEN" pulse patterns. In our study, for each person, we collected 5 minutes of pulse data at each CGC position and FZC pressure. As the sampling rate of the tactile sensor is 50 Hz, there are 15,000 data points (50 Hz×60 sec×5 mins) per CGC position and FZC pressure. Thus, we obtained a total of 135,000 (15,000×9) data points, approximately 3,375 pulses per subject (assuming an average of 40 data points per pulse, which depends on the period of the pulse generated from every subject).

C. MACHINE CONTROL AND SIGNAL PROCESSING
During palpation, CMPs usually adjust their fingertip pressures to collect additional arterial pulse information. They apply three levels of fingertip pressures, namely, FZC. In this paper, we calibrated these 3 pressures based on CMP recommendations and set them to FU=0.5 psi (25.9 mmHg), ZHONG=1.0 psi (51.7 mmHg) and CHEN=2.0 psi (103.4 mmHg). Arterial pulses were recorded in at least 1-minute intervals. The PSP is designed to follow the technique described by a skilled CMP. Hence, the control logic and signal processing flow of the PSP are designed as shown in Figure 2(d). The fingers are actuated by applying voltages to the torque motors. The fingertip sensor is used as an input into the feedback control loop for real-time monitoring of the fingertip pressure. Once the designated pressure is reached, the sensors start recording pulse signals from the CGC positions. Additionally, pulses will be taken at the FZC pressures. Therefore, a total of 9 arterial pulse signals per hand will be obtained for every subject. Fig. 3(a) shows 3 of these signals recorded at the same location (i.e., "GUAN") under FZC pressures. Together with a CMP's diagnosis, these pulses will be input into the signal processing unit for further waveform preprocessing and classification. In the signal processing unit, two signal pre-processing steps are required to generate an input set for machine learning algorithms: baseline wandering correction, which removes undesired noise, and feature extraction, which defines a small number of feature points representing a single arterial pulse pattern. Then, the processed data will be input into the ANN, which will be discussed later, for machine learning and pulse classification. In our system, the average resultant pressure at a particular time instance is calculated by taking the pressure average among all 24 sensing elements and subtracting the applied pressure (Pfingertip), as shown in the equation below: where Pavg is the averaged pulse pressure and pi is the pressure reading of the i-th sensing element over a tactile sensor. As shown in Fig. 3(b), the measured arterial pulses have baseline drift problems. There are two main reasons for the baseline fluctuation during measurement: natural movement of the volunteer's hand and body due to breathing and natural variation in the systolic and diastolic pressure. This baseline drift makes pattern recognition more difficult. Therefore, either the training matrix must include training vectors over a wide spread of baseline pressures, or the raw data must be processed to provide a standard baseline to the neural network. In our proposed method, the preprocessing of raw data was chosen to reduce the complexity of the input layer of the network for the classification of pulse patterns. Hence, the raw data were filtered and rearranged. First, we applied a fast Fourier transform (FFT) to the raw data and obtained the frequency spectral distribution, which shows the amplitude of pressure as a function of frequency. As shown in Fig. 3(c), the FFT reveals that most of the signal energy lies within the low frequency ranges, especially below 2 Hz. As mentioned, this low-frequency signal may be due to the volunteer's physical movement and normal variation in the systolic and diastolic pressure. The other two peak frequencies are the systolic and diastolic peaks of the pulse. If we apply classical wave theory, the beat frequency of the pulse will be the same as the difference between these 2 peak frequencies. The signal without baseline wandering, obtained by the removal of low frequency noise is shown in Fig. 3(d). There are 24 sensing elements on each of the tactile sensor (S1, S2 and S3) to capture arterial pressure: (2) where n = 1, 2, 3; p is the recorded pressure (psi). In order to utilize the coverage of the sensing array to mimic TCM practitioners performing sphygmopalpation, the vertical sensing plane on each sensor with strongest signal energy is chosen which is (pi, 3) in our system: where t is the data acquisition time; i = 1, 2, …, 6.
The detailed procedure for obtaining the 3D contour maps is shown in Fig. 4(a). Firstly, a sensing plane with strongest signal intensity on each sensor, pressed on radial artery, was chosen and rearranged as 18 sensing locations to represent the locations in TCM palpation (namely, CUN, GUAN and CHI). The data from these sensing locations was plotted against times to show the pressure distribution under particular applied forces. Hence, three different 3D contour maps are presented under applied forces -"FU", "ZHONG" and "CHEN", as shown in Fig. 4(b).

A. ARTERIAL PULSE SIGNALS CLASSIFICATION IN TRIAL EXPERIMENTS USING UNSUPERVISED LEARNING
To validate the PRH's capability as well as the consistency of the collected pulse data, we performed 2 trial runs with ~11month time separating the experimentsthe 1st trial run had 7 volunteers and the 2nd trial run had 8 volunteers (with no overlapping volunteers). All volunteers have no record of having cardiac diseases. Professor Jiangang SHEN from the University of Hong Kong, a TCM expert, conducted sphygmopalpation and performed pulse pattern diagnosis for all volunteers. We collected arterial pulse data from the left wrist of all 15 volunteers, using the middle finger (corresponding to GUAN point on the wrist) of our PRH with an applied pressure of 1.0 psi (51.7 mmHg). To perform unsupervised learning, the maximum and minimum pressure, and the time interval between each turning points (in total four feature points: P1, t1, P2, t2), which form one input dataset, were extracted from each cycle of the collected sequence of pulse signals. Table I shows the clustering result of the 2 trial runs data using conventional k-means algorithm [42]. The values (within 0 and 1) in these tables represent the percentage of pulse datasets from a volunteer that fall into a particular cluster. A value of greater than 0.5 suggests that a volunteer belongs to a certain cluster. The k-means clustering result (i.e., Silhouette score = 0.5936 for K=3 clusters), which only left hand's data are selected, showed that the 15 volunteers can be classified into 3 clusters as shown in Table II

B. ARTERIAL PULSE WAVEFORM CLASSIFICATION USING DEEP LEARNING ALGORITHM
After validating the machine capability in classifying arterial pulse signals via the unsupervised learning approach, we collected other dataset consisting of 3D pulse wave records from three subjects to mimic the sphygmopalation via a TCM approach. For each subject, we have conducted 2 experiments of applying 3 different pressure variation (i.e., "FU", "ZHONG" and "CHEN", as defined in Methods section) at the 3 left wrist locations (i.e., "CUN", "GUAN", and "CHI" points, as defined in Fig. 1. For each applied pressure level, 5 minutes of the pulse wave data were collected from the 3 left wrist locations of each subject. This dataset was divided into training set, validation set, and testing set, in the ratio of 516:219:66. The convolutional neural network (CNN) was trained to detect 3 pulse patterns: the "HUA" (滑), "XI" (細), and "CHEN" (沉) patterns as diagnosed by a TCM expert. These three types of pulse patterns are displayed in Fig. 6.

C. FORMULATIONS OF INPUT METRICS FOR DEEP LEARNING
The collected 3D contour maps in Fig. 6 consist most of the information, including spatial, temporal and pressure information of one's arterial pulse. To classify such a high dimensional data, deep learning is the most suitable tool. Hence, convolutional neural network (CNN), commonly used for images, was chosen as the analytic tool to classify pulse waveforms from the 3 volunteers. Before feeding the 3D contour maps into the CNN, we converted them into 2D images, i.e., we call these 2D images as "X-ray" images of human pulses as they may allow CMPs to decipher patients' health status in the future, as shown in Fig. 7.
The procedures of generating the "X-ray" images of pulse information constructed based on the sensing data from 3 locations and 3 applied pressures as described by TCM doctors are discussed below. The 3D color contour maps of an arterial pulse in Fig. 7(a), which were taken at CGZ locations under FZC applied pressures (by using the process described in Fig.  4) were projected in the time-location plane to form a 2D color map, as shown in Fig. 7(b). Subsequently, the 2D images were segmented in the time-axis to crop a 18 pixels-height (axis of sensing location) and 1.08 s-width image sequentially to form a 54-by-54 pixels image (height: 18 pixels × 3 applied pressures = 54 pixels; width: 1.08 s × 50 Hz = 54 pixels), as shown in Fig. 7(c). Then, the image was reshaped (Fig. 7(d)) and converted into grayscale (Fig 7(e)) respectively to form an input image for the CNN. Reshaping the image into this size (i.e., same number of pixels of height and width) can simplify the structure of the CNN. Therefore, an input image generated (after applying the procedures described from Fig. 7(a) -Fig.  7(e)) contains the diagnosed arterial pulse signals of 3 locations × 3 applied pressures × 1.08 time-duration. Examples of the 3 detected pulse patterns which were formed to be the input samples of the database, i.e., patterns "HUA", "XI" and "CHEN", are shown in Fig. 7(f), 7(g), and 7(h), respectively.

D. DEEP LEARNING ARCHITECTURE AND NETWORK PERFORMANCE
The rows and columns of a 2D image, which is 54-by-54 pixels as discussed above, represent the selected sensing location (i.e., sensing location 0 to 17 on the selected plane as shown in Fig. 4) and time (i.e., 0.02 s (sampling frequency of our sensor) × 54 pixels = 1.08 s), respectively. These converted grayscale images were treated as inputs for the CNN, as shown in Fig. 7(f) -(h), which has 3 outputs for classifying 3 different waveforms using the architecture listed in Table II. The pulses collected from the 3 volunteers were converted into a total of 801 grayscale images. Each volunteer's pulse data were converted into 267 grayscale images, and were divided randomly into three sets, i.e., training, validation and testing sets, in the ratio of 516:219:66, respectively. These images are the input for the neural network.
The network consists of two sets of convolutional and average pooling layers, followed by a flattening convolutional layer, then three fully connected layers. The output layer is fed into a fully connected layer with a SoftMax activation which was used because the classes are not mutually exclusive (i.e., two or more classes may occur in the same exam). The input for the CNN is a 54×54 grayscale image which passes through the first convolutional layer with 6 feature maps with Kernel size of 5×5 and a Stride of 1. The image dimensions changes from 54×54×1 to 50×50×6. Then an average pooling layer is applied with a filter size 2×2 and a stride of 2. The resulting image dimensions are reduced to 25×25×6. For the second convolutional layer, it has 16 feature maps with Kernel size of 6×6 and a stride of 1 and the resulting image dimensions are changed to 20×20×16. The fourth layer is again an average pooling layer with filter size 2×2 and a stride of 2. This layer is the same as the second average polling layer, except it has 16 feature maps so the output is reduced to 10×10×16. The fifth layer is a fully connected convolutional layer with 120 feature maps each of size 1×1. Each of the 120 units in this layer is connected to all the 1600 nodes (10×10×16) in the fourth layer. The sixth layer is a fully connected layer with 84 units. Finally, there is a fully connected SoftMax output layer with 3 possible values corresponding to the 3 pulse patterns, namely, "HUA", "XI" and "CHEN". During the training process, 30 epochs and 1500 iterations were reached in total for the final model, as shown in Fig. 8. The training accuracy of our CNN designed for arterial pulse pattern recognition were 99.1% when it reached the final iteration less than 30 seconds.  After completing the training process, the trained network was subsequently used to predict a testing set consisting of 66 images, which was selected randomly from the 801 grayscale imagesthis testing dataset consists of 66 samples from three subjects diagnosed by experienced practitioner in TCM. Table  III shows the performance of the CNN on the testing set. As shown, these three types of pulse patterns were classified with high performance, i.e., with F1 scores above 96% and specificity indexes over 97%. Apart from using deep learning to classify the pulse signals acquired by our TCM-based robotic palpation platform, existing classical methods (i.e., applying features extractions on the acquired signals followed by patterns classification with machine learning algorithms) can also be used to analyze the collected data. For example, the turning points of the 3 patterns of pulse signals in the inset of Fig. 6 were extracted as features to form an input set, i.e., 36-by-1 matrix (4 turning points × 3 locations × 3 applied pressures). Subsequently, the input set was fed into a Levenberg-Marquardt backpropagation neural network (36-150-3 network structure) to classify these 3 patterns, which yield an average classification rate of 94.8 % in training and 92.7 % in testing, respectively. We note here that the overall classification rates of using feature extractions as inputs to ANN could be significantly lower than our proposed data analysis method (i.e., using greyscale "X-ray images"). For example, in the testing process, the classification rates of these 3 patterns were 100 %, 78.7 %, and 100 % for "HUA", "XI" and "CHEN", respectively. Even though the average classification rate of using the classical methods (92.7%) is not remarkably lower than using our proposed method (97.4%), a significant drop was observed for the classification rate of the pulse pattern for "XI", i.e., only 78.7 %. This is because the classical method can only analyze the pulse patterns as an one-dimensional data, i.e., time series analysis. Hence, the classical network could only successfully classify patterns with a small variance of pressure amplitude in time axis, and is not suitable for multiclass analysis. On the other hand, our proposed method could incorporate both temporal and spatial information during pulse pattern classification, which is also the logic of diagnosis adopted by TCM practitioners during their pulse palpation of patients. Therefore, using our proposed method, each class could be classified with a relatively high classification rate. In the future, as more pulse patterns and data will be analyzed and hence the difference of feature points among each pattern will be reduced. Accordingly, our proposed approach will be more suitable to handle the increasingly more complex data structures required to decipher 28 to 29 TCM pulse patterns than classical methods.

IV. DISCUSSION AND CONCLUSION
We have demonstrated an end-to-end arterial PSP for pulse classification of 3 unique pulse patterns (i.e., "HUA", "XI", and "CHEN") based on the TCM approach. The platform consists of 3 robotic sensing fingers for arterial pulse measurement, which mimics a CMP performing palpation, and an CNN for pulse classification. The pulse diagnostics platforms previously reported by other research groups or marketed commercially exhibit one or more of the following imperfections: 1) a single channel for data acquisition, 2) low sensitivity and 3) rigid sensors, 4) lack of control of the applied force, and 5) lack of an intelligent data analysis system. TCM practitioners rely on 3 fingers and 3 applied pressures for palpation to enrich the acquisition of pulse information; hence, our developed platform mimics the data acquisition process of TCM practitioners by featuring up to three tactile sensing channels (each with 4×6 sensing pixel arrays) for recording data, pressure-feedback-controlled robotic fingers, and machine learning algorithms. We have shown that CNN is superior to other machine learning models in pulse pattern classification because of its capability in tackling complex data structures. To quantify human pulse in diagnostic applications, we proposed a methodology of obtaining "Xray" image of pulse information constructed based on the sensing data from 3 locations (CGC; defined in Materials and Methods section) and 3 applied pressures (FZC; defined in Materials and Methods section), which contains all arterial pulse information in both spatial and temporal spans, and which could be used as an input to a deep learning algorithm. Compared to conventional approaches that use 1D time-series information of pressure pulses collected from a single wrist point, the proposed "X-ray" images preserve more dynamic pulse information. Our preliminary results show that this platform can classify 3 pulse wave patterns with validation accuracy of 99.1% in training and prediction accuracy of 97.4% in testing.
With the high classification rate and further improvement of the developed platform as well as the algorithms, we aim to digitalize and classify the basic 28 pulse characteristics described in TCM. This classification will help to minimize differences in TCM pulse diagnosis due to human variance by providing scientific-based pulse data information. The platform presented here can potentially become a standard clinical equipment as a low-cost and non-invasive diagnostic system for monitoring patient health status. Our long-term goal is to develop a databased transmission of TCM knowledge, which was previously conveyed only by word of mouth or written notes.