Slip Detection With a Biomimetic Tactile Sensor

Slip detection helps to prevent robotic hands from dropping grasped objects and would thus enable complex object manipulation. Here we present a method of detecting slip with a biomimetic optical tactile sensor—the TacTip—that operates by measuring the positions of internal pins embedded in its compliant skin. We investigate whether local pin movement is a strong signal of slip. Accurate and robust discrimination between static and slipping objects is obtained with a support vector machine (accuracy 99.88%). We then demonstrate performance on a task in which a slipping object must be caught. For fast reaction times, a modified TacTip is made for high-speed data collection. Performance of the slip detection method is then validated under several test conditions, including varying the speed at which slip onset occurs and using novel shaped objects. The proposed methods should apply to tactile sensors that can detect the local velocities of surface movement. The sensor and slip detection methods are also well-suited for integration onto robotic hands for deploying slip control under manipulation.

They also highlighted the importance of slip detection in allowing successful manipulation. Despite many studies being performed since, slip detection is yet to be commonplace on commercially available robotic hands [6]- [9]. The development of a slip detection method with minimal data preprocessing and fast classification speed would allow for slip to be checked with each data collection cycle, which means processing power could be dedicated to computationally intensive tasks such as in-hand manipulation. Ideally, such a method would require minimal training and be robust to changes in experimental conditions, so individual tactile sensors could be easily calibrated to perform slip detection accurately.
The aim of this study is to develop a simple, fast and robust classification method to detect slip using a biomimetic optical tactile sensor-the TacTip [10], [11]. Previous work with the TacTip has focused mainly on object perception and exploration [12]- [14], and thus the addition of slip detection capabilities will enable more applications, particularly in relation to its integration with robotic hands [11]. The TacTip operates by measuring the positions of internal pins embedded in its compliant skin surface, analogous to mechanoreceptors embedded around the dermal papillae in our fingertip skin. Here we propose that pin motion is a strong signal of slip. We validate this proposition by showing accurate and robust discrimination between static and slipping objects using a support vector machine (SVM) applied to pin velocity data.
To ensure the sensor is suited for rapid slip detection, we modify the existing TacTip [11] to utilize a high-speed camera system capable of recording at 120 FPS. A rig is also designed to allow data collection and testing to occur autonomously (Fig. 1). The performance of the method is validated under several test condition, including varying the speed at which slip onset occurs (as in real slip scenarios). Several differently shaped objects are used to demonstrate generalisation to different object geometries. A simple procedure to measure the slipping distance and determine whether a test is successful is also presented. The conclusion is that an SVM applied to pin velocities from the TacTip optical tactile sensor is a robust method of detecting slip. Thus, we can detect slip in real time, react to secure the object and minimise the slipping distance.

II. BACKGROUND AND RELATED WORK
The focus of this study is to detect in real time when an object is slipping and react to prevent it from being dropped, so minimising slipping distance. Since Howe & Cutkosky's early work many other sensors of slip have been developed. Melchiorri (2000) combined the data from a force/torque sensor and a tactile sensor to detect slip and only required knowledge This work is licensed under a Creative Commons Attribution 3.0 License. For more information, see http://creativecommons.org/licenses/by/3.0/ Fig. 1. The rig used to collect data of a slipping object. An object is secured to a clear perspex rectangle which moves inside a low friction rail.
of the coefficient of friction [15]. Hirai et al. (2010) used a simulated sensor consisting of virtual cantilevers connected by springs to model how slip would cause the sensor to behave [16]. They then made a sensor which consisted of a series of ridges on semicyclindrical surface to test the model and detect slip. Kondo et al. (2011) created a fabric of woven electroconductive yarns whose electrical resistance depends on the yarn stretch [17]. At the moment immediately prior to slip, the extension is at a maximum; therefore, slip is detected when a sudden change in resistance occurs.
The sensors discussed above were specially developed for detecting slip. Recent work has involved adding slip functionality to existing sensors with minimal modification. Veiga et al. (2015) used a BioTac tactile sensor and compared SVMs with random forests to detect slip [6]. Seven differently shaped objects-including a box and a ball-were used to collect data to train classifiers which were able to predict slip as well as detect it. Tests demonstrated high success for each object depending on the classifier used; however, no single classifier performed well across all objects. Meier et al. (2016) also used the BioTac to detect slip but used a convolutional neural network as the classifier [18].
Optical tactile sensors have also been used to detect slip. Yuan et al. (2015) used a GelSight sensor which has markers over a clear gel and calculated the entropy of the marker distribution to detect slip [9]. They showed that slip occurs at high entropy levels, but did not progress to a system that reacts to the onset of slip. That work was improved upon by Dong et al. (2017) who used shear, contact area and relative displacement between the sensor and an object to detect slip [19]. They used this slip detection capability to determine the required grip strength when picking up a various objects with a one-degree-of-freedom gripper.
Slip detection has not previously been performed on the Tac-Tip. Instead the focus has been on shape detection, such as following the contours of various unseen 2D objects [12] and accurate (sub-mm) localisation [11], [14], [20]. The TacTip has also been integrated on robotic grippers. Ward-Cherrier et al.  to manipulate cylinders held in a pinch grasp [21]. These capabilities suggest that the TacTip would be effective at slip detection, which is the purpose of this study to confirm.

A. Details of Tactile Sensor and Data Collection 1) Tactile Sensor:
This study is conducted using the TacTip; a biomimetic tactile sensor originally presented by Chorley et al. (2009) [10]. The TacTip is a 3D-printed optical tactile sensor containing 127 pins arranged in a hexagonal pattern on the inside of a hemisphere constructed from Tango Black+, a rubber-like material. The tip is filled with silicone gel (RTV27905) and sealed with a clear acrylic lens. This makes it compliant but allows it to rapidly reform its shape and reduces hysteresis. A camera is focused on the pins and records their positions when the TacTip contacts an object.
For this study we customised the sensor to use an ELP USB camera module (model ELP-USBFHD01M-L21) which runs at 120 frames per second (FPS), significantly faster than the previous camera used (Livecam, 30 FPS). A new mount was developed to contain the camera module that eased integration into the existing TacTip sensor design (Fig. 2). For all experiments here, the TacTip is mounted on a six degree-of-freedom robotic arm (UR5, Universal Robots).
2) Rig for Slip Experiments: To perform a rigorous analysis of slip, we designed a custom apparatus for automated collection of slip detection data. The rig was designed with several criteria in mind: (i) to allow an object to fall under gravity with minimal friction; (ii) to allow any experiments to be repeatable with high precision; (iii) to have the experiments performed entirely without human input after initial setup; and (iv) to allow a variety of object shapes and weights to be tested.
The apparatus utilised a low friction rail system with a slider to which multiple objects can be attached (Fig. 1). Four tactile stimuli with different radii of curvature were made to represent differently-shaped objects (radii of curvature: 20, 40 and 80 mm and the original flat surface), which can be easily interchanged for testing. Because the objects remain secured to the rail when dropped, the entire data collection procedure can be performed autonomously. 3) Data Collection: The TacTip was pressed against the test stimulus to secure it, then was raised to a default position. The arm was then retracted at a speed of 0.1 mms −1 until the object fell. The arm was next moved to the bottom of the rail to retrieve the object and return it to its original position. The UR5's maximum repeatability of ± 0.1 mm means that the data collection runs are consistent and can be performed quickly and safely. All of the data for training was collected using the flat stimulus. The data collection routine is performed 40 times for each test condition.

4) Data Preprocessing:
The images from the camera are processed using Python OpenCV. The images are captured, thresholded and a contour detection algorithm used to detect the pin centres, giving coordinates (x, y) in pixels. Fig. 3 shows the raw camera image and the result of pin detection, with each red marker on a pin centre. The list of all 127 pin positions is then passed to MATLAB for analysis. The pin positions are the sole output from the sensor and are the basis for any analysis performed. When the sensor contacts a surface the changes in pin positions are used as the tactile information.

B. Classifier Training and Experiment Description
1) Data Processing: The challenge for this work is to develop a classification method that is robust to changes in arm movement speed and object shape. As slip involves an object moving relative to the sensor, having a time-dependent component is useful for robustness. For this work the difference in position of each pin between consecutive frames-the pin velocity-was chosen. Velocity was chosen because a change in velocity will always be present when an object starts to slip and it is easily inferred from the pin positions. When holding an object, the surface of the TacTip stretches downwards due to the object weight. When that object begins to slip, the coefficient of friction changes from a static to kinetic value that is generally lower. Hence, the frictional forces decrease and the elastic force from the stretched sensor surface causes it to move back towards its non-deformed shape, shifting the pins upwards. This is evident in the vector field of the pin velocities, which has a characteristic alignment at the moment slip occurs (see Fig. 4(b)). This pattern allows us to determine which frames correspond to slip for training the classifier.
The vector of pin velocities is collected in Cartesian coordinates (Δx i , Δy i ) for 1 ≤ i ≤ 127, and transformed by: i + Δy 2 i and Δθ i = arctan(Δy i /Δx i ). The angular component (Δθ i ) of the pins is shifted to have a mean of zero with respect to the x-axis of the camera, removing all angular dependence of the data. This yields a 254-dimensional time-series input (Δr i , Δθ i ) to the classification method.
2) Classification Method: Support Vector Machines (SVM) were chosen as the classification method as they satisfy several criteria: (i) fast classification speed; (ii) ability to handle highly dimensional data (up to 254 dimensions for this work); and (iii) fast to train over small data sets.
SVMs are a binary classification method that works by effectively plotting (labelled) training data and drawing a hyperplane which best separates the two classes of the data. New data is plotted and is classified according to which side of the hyperplane it falls on. For further detail see [22]. Not all classes are able to be separated by a hyperplane; however, non-linear SVMs can be trained to transform the data so the classes can be linearly separated. Here we use a non-linear kernel k( x i , x j ) to transform the data as A Gaussian kernel was used as the non-linear kernel for this study, given by Both linear and non-linear (Gaussian) kernels were tested with parameters chosen-for the non-linear case-using Bayesian optimisation (using the MATLAB function fitcsvm). Two parameters were optimised: the Kernel Scale (γ in equation (2)); and the box constraint, which sets the cost of a training data point being on the wrong side of the dividing hyperplane. Binary classification splits the training data into two classes. These classes will be referred to as 'slip' and 'static'. The ground truth labels were set by manually looking at the vector fields for each frame and identifying the frames with the alignment of vectors consistent with slip.
3) Experimental Validation: The first set of experiments tested the SVMs on the data collected from the automated slip experiment (details in Section 3A.3). This 'offline' testing compares the output of a classifier to the ground truth labels for the data. Two analyses were performed to identify the best method: (i) a comparison between a linear and non-linear kernels; and (ii) the effect of varying the number of frames in the training data 'slip' class. This second test is important because the initiallyclear slip signal becomes noisy after only a few frames (see Fig. 4(c)). The classifiers were trained on half of the collected data with the remaining data used for testing.
Online testing involves using a classifier to detect slip occurring on a real object in real time. Online testing was performed using the best-performing classifier from the offline tests. The online testing involved securing the object in the manner used when collecting training data, however, when slip is detected the UR5 reacts to secure the slipping object.
Three online tests were performed. Firstly, the flat stimulus was tested under the same conditions as the data collection, with the robotic arm retracted at 0.1 mms −1 . Secondly, the speed of  the retraction was varied. Finally, the three curved stimuli were tested to determine the ability to detect slip of differently-shaped objects (Fig. 5).
To determine whether a test was a success an ArUco marker was attached to each stimulus, with a camera (Canon EOS 600D) recording its position (Fig. 6) [23]. The ArUco marker is used to record the height of the stimulus: it thus determines whether the stimulus has been caught or dropped when slip begins, and how far the stimulus fell before being caught. Calibration is achieved by lifting an object 50 mm at the start of each test to calibrate the pixel-to-distance ratio.

A. Inspection of Slip Training Data
The training data demonstrates how drastic an effect a slip event has on the tactile data. Fig. 7 shows the pin displacements relative to their initial positions and the frame-to-frame pin velocities, over a typical data collection run. Each coloured line represents a different pin. There is a clear spike in the pin velocities at around the 1200th frame and a step-change in the pin positions, both of which strongly signal the onset of slip. This pin velocity spike is present in all pins. Therefore, slip affects the entire TacTip skin surface.
Given that we will apply an SVM classifier to the velocity of pin positions, Fig. 7 shows how noticeable the slip onset is and indicates that the classifier should work well. The width of the spike in the velocity data is approximately 0.2s (∼20 frames). Hence, to prevent a slipping object from falling, the data must be collected, classified and the robot moved in a very brief period of time.

B. Offline Validation of SVM Classifier for Slip Detection
Offline testing will allow us to obtain the best classifier, which will then be used to identify slip in real time. The first classifier comparison was between the use of a linear and a nonlinear (Gaussian) kernel. The dimensionality of both kernels was also varied. We compared using both frame-to-frame changes in magnitude and angles, (Δr i , Δθ i ), to sole changes in angle, Δθ i and sole changes in magnitude, Δr i , corresponding to 254, 127 and 127 dimensional data, respectively. For this first comparison, a total of six classifiers were tested, with the results expressed in terms of the percentage of correctly classified frames (summarized in Table I).
Two observations are drawn from Table I. The first is that using both the changes in magnitude and angle (Δr, Δθ) gives better performance than just magnitude or angular data. Both the magnitudes and angles are useful for slip classification, justifying the use of the transformation from Cartesian coordinates to magnitude and angle. As an aside, the magnitudes give better classification than the angular data on both tests.
The second observation is that the Gaussian kernel outperforms the linear kernel. All three of the classifiers trained with a Gaussian kernel outperform all three with the linear kernel. The best performing Gaussian SVM has a score 4% higher than the best scoring linear SVM method. Clearly a hyperplane is not able to effectively separate the two classes, but the use of an optimised non-linear kernel is sufficient to drastically increase the classification success.
The second set of offline tests involves varying the number of frames in the 'slip' class. Previously for training, every frame was assigned manually to the class 'slip' or 'static'. For the first test, all ground truth frames in the slip class were used to train the classifiers; however, the pattern becomes noisy after only a few frames (see Fig. 4(c)). Choosing the appropriate number of frames in the slip class would ensure that only the data with a clear signal indicating slip would be kept, which would give a better separation between the two classes.
Testing demonstrated that that using 4 frames in the slip class increases classification success from 99.11% to 99.88%. Therefore, using the cleanest slip signal allows the SVM to

TABLE I COMPARISON OF KERNEL TYPES AND DIMENSIONS USED IN THE CLASSIFIER
The classifiers using both magnitude and angular data perform best. The Gaussian kernel also performs notably better than the linear. Columns three and four represent the optimised parameters kernel scale (KS) and box constraint (BC). better discriminate between slipping and static objects, although this only marginally outperformed using 3 or 5 frames. For the remaining tests, only the 4-frame classifier in the 'slip' class is used. It should be noted that the classifier is able to detect more than the slip onset and can detect upwards of 10 frames after onset, albeit with lower accuracy.

C. Online Slip Detection & Object Recovery
The main goal of this research is to detect slip and react in real time. The offline analysis above (Section IVB) allowed us to identity the best-performing classifier, which not only detects slip with minimal error but also has a low false-positive rate.
A false positive in this case involves classifying static data as slipping and therefore reacting to an incorrect impulse. In most circumstances, this would be less costly than a false negativewhere an object would fall without the robot reacting-but reacting unnecessarily could still be harmful. This is why we use a success rate based on classifying both static and slip frames rather than only detecting the slip frames.
For online testing in real time, a similar experiment is used as for the offline tests: the TacTip is pressed horizontally against a stimulus until it was secured, lifts the stimulus, and then the arm is again slowly retracted. However, in the online test, when slip is detected, the arm moves immediately back to prevent the object from falling. To minimise the effect of false positives, two consecutive frames must be classified as slip before a response is triggered from the robot arm.
The initial online tests were performed with the flat stimulus, which was also used for data collection and analysis. Therefore, online and offline performance can compared. The speed at which the arm was retracted was varied to evaluate performance under different slipping conditions. Validation was achieved by tracking an ArUco marker (Section IIIb3); thus, we can determine whether an object has been successfully caught and calculate the falling distance (Fig. 8).
A test is deemed to be a success when there are two distinct plateaus separated by a clear drop in height. This means the object was held steady as the arm retracted (first plateau), began to slip (drop in height) and was then stopped by the arm (second plateau). The successes determined from tracking the ArUco marker and calculating the stopping distances are shown in Table III. Overall the systematic error in this method is estimated to be ±0.1 mm, with standard and systematic errors summed in quadrature to give the total error.
Eleven different arm retraction speeds varying from 0.1 to 20 mms −1 were performed. Each speed is tested twenty times (Table III). Seven of the eight lowest retraction speeds have a success rate of 100%, with the success rate of catching the object decreasing above 3 mms −1 retraction speed. The average success over the 220 performed tests is 95%.
There is a positive correlation between retraction speed and stopping distance, with the slowest speed giving a stopping distance of 15.5 mm and the fastest 26.9 mm. These results  demonstrate that the method developed offline (Section IVB) performs well in real time to secure a falling object; moreover, this performance is invariant to changes in arm retraction speed. Fig. 9 shows the height of the object over twenty test runs. In this case, each test was successful. The entire computational loop including image capture, processing and classification runs at 100 frames per second. Classification was clocked at 3 milliseconds, less than the time between frames. The second online test uses three other stimuli with different radii of curvature to test robustness to object shape. Each stimulus was tested twenty times to investigate how the classifier generalises to objects with different curvatures. The stimuli were mounted such that their curvature was aligned to the vertical axis, with retraction speed 0.1 mms −1 . Table III shows the classifier is able to detect slip on all the curved objects, with only 3 failures from the 60 tests performed. For two stimuli, the stopping distance is less than that for the flat stimulus; however, this is within error. Testing on these objects mimics a range of situations that would be encountered when deploying slip detection in real-world scenarios. We emphasise Fig. 9. Graph showing the height of an object during twenty tests. Using the success criteria described in Section IV.C it can be determined that each slip was successfully detected and the object prevented from falling. It also shows that there is a reasonable amount of variability in the stopping distance. that these curved objects are entirely novel to the classifier, which was trained only on a flat object.

V. DISCUSSION
In this study we presented a method of detecting slip using the TacTip: a biomimetic optical tactile sensor. The velocity of internal pins within the sensor was used as the input to a support vector machine (SVM) to detect slip. When slip occurs the velocity vectors of the pins display a clear alignment. Transforming the velocity data into magnitude and angular components allowed the SVM to reliably discriminate between slipping and stationary objects. This required minimal training data and preprocessing. Tests were carried out by securing an object and then allowing it to slip by retracting the sensor. When slip was detected the sensor was moved to secure the object, preventing it from being dropped. These tests achieved a success rate of up to 100% when the TacTip is retracted at low speeds. The classifier was also able to detect slip in novel curved objects and prevent them being dropped.
Fast classification speed coupled with camera upgrades and streamlined processing means that the TacTip tactile sensor is capable of running at 100fps, over three-fold faster than previous studies using the TacTip. Classification speed was a major factor in the choice of an SVM. In a study by Veiga et al. random forests were deemed to be marginally better at detecting slip than support vector machines [6]. However, here offline testing yielded comparable results between the two studies with a classification success of 99.88%. This demonstrates that the velocity profile of the entire surface of the TacTip is a reliable measure to detect slip.
We have only considered situations where an object slips due to gravity. The method may be susceptible to false positives when lifting an object and the sensor surface is stretching under a load. There are multiple other scenarios in which slip could occur such as slipping under external load and at multiple angles relative to the surface of the sensor. However, here the scope of our study encompassed training only over a single angle (downwards under gravity). Furthermore, we did not consider the re-detection of non-slip, which would be an interesting extension of the method to detect when slip has stopped so that a precise action to minimise slip can be chosen.
The ability of the classifier to cope with the variation of the arm retraction speed-when inducing slip-demonstrates a robustness to changes in experimental conditions. The classifier was trained on just one retraction speed, yet generalizes to over 220 tests with speed varied from 0.1-20 mms −1 with success rate of 95%. Veiga et al. (2015) achieved an average success of 74%; however, this was over a wider variety of different objects. The classifier was also able to detect slip over novel objects, despite only being trained on a single object. Different shapes deform the surface of the TacTip in distinct ways; however, by examining properties common to all slip events-here, the pin velocity distribution-it is possible to still reliably detect slip for differently curved objects. This finding relates to work by Dong et al., who were able detect slip over a wide range of objects [19]; however, that study considered just the required grip strength and did not involve responding to the onset of slip to minimise slipping distance. The minimal processing of our method allowed for slip to be detected in real time. Nevertheless, using slip to determine grip strength on an object would be an interesting future study.

VI. CONCLUSION
The aim of this work was to present a robust method of slip detection using the TacTip biomimetic optical tactile sensor. Using the velocity of each internal sensor pin, a support vector machine was able to reliably detect slip and was robust to novel object shapes and slip (retraction speed) profiles. We expect the proposed methods will apply to other array-based soft tactile sensors that are able to detect the local velocities of surface movement. Future work will be concerned with further integration of the TacTip technology onto robotic hands so that slip can be tested in a wider range of scenarios. We expect that the method will extend to be independent of slipping direction and material, which is important when performing multiple grasps. Other tests will include determining grasp strength and investigating slip with soft and fragile objects.