Tactile Manipulation With a TacThumb Integrated on the Open-Hand M2 Gripper

Tactile manipulation will be essential for automating industrial and service tasks currently done by humans. However, the application of tactile feedback to dexterous manipulation remains a challenging unsolved problem, with robot capabilities lagging far behind those of humans. Here, we present the tactile thumb (TacThumb): a cheap, robust, 3-D-printed optical tactile sensor integrated on the Yale GrabLab model M2 gripper. To test tactile manipulation capabilities, a cylinder is rolled along the TacThumb using the opposing nontactile finger. The tactile information permits localization of the test cylinder along the TacThumb to submillimeter accuracy over most of the movement range. In consequence, the M2 gripper can perform accurate inhand tactile manipulation, by providing information that can be used to control the location of the test object within the hand. Tactile manipulation is demonstrated by rolling cylinders with a range of diameters up and down the TacThumb along a target trajectory, using only tactile data to update its current position and move it toward a target. This model-free approach gives a demonstration of basic tactile manipulation without the need for a kinematic model of the hand, in a manner that should generalize to other tactile manipulation tasks.


Tactile Manipulation With a TacThumb Integrated on the Open-Hand M2 Gripper
Benjamin Ward-Cherrier, Student Member, IEEE, Luke Cramphorn, and Nathan F. Lepora, Member, IEEE Abstract-Tactile manipulation will be essential for automating industrial and service tasks currently done by humans.However, the application of tactile feedback to dexterous manipulation remains a challenging unsolved problem, with robot capabilities lagging far behind those of humans.Here, we present the tactile thumb (TacThumb): a cheap, robust, 3-D-printed optical tactile sensor integrated on the Yale GrabLab model M2 gripper.To test tactile manipulation capabilities, a cylinder is rolled along the TacThumb using the opposing nontactile finger.The tactile information permits localization of the test cylinder along the TacThumb to submillimeter accuracy over most of the movement range.In consequence, the M2 gripper can perform accurate inhand tactile manipulation, by providing information that can be used to control the location of the test object within the hand.Tactile manipulation is demonstrated by rolling cylinders with a range of diameters up and down the TacThumb along a target trajectory, using only tactile data to update its current position and move it toward a target.This model-free approach gives a demonstration of basic tactile manipulation without the need for a kinematic model of the hand, in a manner that should generalize to other tactile manipulation tasks.Index Terms-Force and tactile sensing, grippers and other endeffectors, dexterous manipulation.

I. INTRODUCTION
M ANY different kinds of tactile sensors and robot hands have been developed for manipulation purposes [1].Tactile sensors have been used for object recognition [2], improving grasp stability with force control [3] and/or slip detection [4], and object exploration through edge or surface following [5].Tactile servoing [6] has also been applied to a form of object manipulation on an industrial robot arm [7].However, the application of tactile feedback to dexterous manipulation (for example, moving an object to different positions within a robot hand) still remains a challenging and largely unsolved problem, with robot capabilities lagging far behind those of humans.
The aim of this study is to present the development of a 3d printed tactile sensor and its integration into an open-source 3d printed robot hand, as an inexpensive, customizable platform for investigating dexterous tactile manipulation.Our TacThumb (Tactile Thumb) is a 3d-printed optical tactile sensor adapted from the TacTip [8] designed for mounting (Fig. 1) on the Model M2 Gripper [9], an open-source, 3d-printed robotic gripper from the Yale Open-Hand project [10].The M2 Gripper allows a one-dimensional rolling motion along the TacThumb, which is ideal as a basic first venture into tactile manipulation.
A key design aspect of the TacThumb is that it uses an internal webcam to image deformations of a flexible contact pad.Recent work has shown that this design is capable of sub-millimetre accuracy in tactile perception, since it is well suited for tactile superresolution methods [11], [12] based on a Bayesian approach for active touch [13]- [15].Therefore, we use this active touch approach here to accurately move a grasped object along a desired trajectory within the robot hand, while maintaining a stable grasp.
To test tactile manipulation capabilities, a cylinder is rolled along the TacThumb using the opposing non-tactile finger.Tactile feedback permits localization of the test cylinder along the TacThumb to sub-millimeter accuracy over its entire 20 mm movement range.In consequence, the M2 gripper can perform accurate in-hand tactile manipulation, by providing tactile feedback to control the location of the test object within the hand.Tactile manipulation is demonstrated by rolling the cylinder up and down the TacThumb along a target trajectory, using only tactile data to update its current position and move it towards a target.This approach is model-free.It thus demonstrates basic tactile manipulation without the need for a complex kinematic model of the hand, in a manner that should generalize to other tactile manipulation tasks.

II. BACKGROUND AND RELATED WORK
Tactile sensing is an essential ingredient in human-robot interaction and fine manipulation [16].As such, a variety of This work is licensed under a Creative Commons Attribution 3.0 License.For more information, see http://creativecommons.org/licenses/by/3.0/different tactile sensor designs have been developed for use on robot hands (see [1], [17] for reviews).
Tactile sensors [18] and tactile skins [19], [20] have been applied primarily to slip detection and grasp improvement [21].Others can be used as force/torque detectors, as well as providing tactile perception, for instance to detect an object's geometrical shape.One approach for tactile perception is to require a model of the sensor in order to compute accurate force vectors and locate contact points, referred to as a modelbased solution [5], [18], [19], [21].We focus on a model-free approach, which relies on a training phase but requires no knowledge of the sensor itself.
Here we aim to develop a tactile sensor for model-free dexterous manipulation and integrate it in a robot hand.We name the sensor TacThumb, as its design is based on the TacTip (Tactile fingerTip), a 3d-printed optical tactile sensor developed at Bristol robotics laboratory [8], [12], [22].The TacTip's compliance is an attractive aspect for integration into robot hands, as it is known to improve grasping [23].Work has previously been done on miniaturising the TacTip for integration into an elumotion ELU-2 robot hand [22].The current letter takes a different approach by modelling the sensor on the existing M2 gripper [9] thumb and 3d-printing the TacThumb in one piece using multimaterial 3d-printing.
Although many different robot hands exist [24], the Yale Open-Hand designs stand out with their simplicity, low cost and performance in gripping tasks [10].The hand used here is the Model M2 gripper [9], a 3d-printed, 2-fingered fully actuated robot gripper with a fixed thumb and opposing movable finger.Our TacThumb sensor design naturally integrates with the M2 Gripper (replacing the fixed thumb) since it is also 3dprinted, inexpensive and robust.Using an optical sensor also avoids possible issues related to excessive wiring [1].
Tactile sensor arrays (Takktile [25]) of MEMS barometers have been previously integrated into the Yale i-HY Open-Hand [26], but not used to control the hand.Object recognition has been demonstrated with the TakkTile sensors integrated into the Yale model T42 Open-Hand in combination with motor information [27], with emphasis that open-loop passive perception was sufficient with limited tactile information (two of the five tactile array elements).
Our focus here is on closed-loop tactile control of an Open-Hand gripper.Our control algorithm is based on a probabilistic method for active tactile perception [12], [13], [15].This approach has been successfully applied to relocate the sensor relative to the object for improved perception, resulting in superresolved object localization [11], [12], [14].Here we apply it to in-hand tactile manipulation, using the robot hand to reposition the object along the sensor.

A. Hardware
The TacThumb is designed to fit on the Open-Hand M2 Gripper [9], a 3d-printed, 2-fingered open-source robot hand.The M2 gripper's 2 fingers are a fixed thumb (replaced by the TacThumb) and an opposing moving finger with 2 pivot joints controlled by an agonist/antagonist tendon pair [28].
1) TacThumb Fabrication and Functionality: The overall design and functionality of the TacThumb is based on the TacTip, a cheap, robust optical sensor developed at Bristol Robotics Laboratory [8].As such, it is made up of 4 main parts (Fig. 2): • The base: This is the part which comes into contact with objects.It is made of a hemispherical silicon rubber pad with small (∼ 1 mm dia.) white pins on its inside surface, separated from each other by 4 mm.The pad is filled with RTV27906 silicon gel which gives the TacThumb its compliance.• The lens: A 1 mm thick acrylic lens separates the base from fragile electronic components.• The LED circuit: A circuit of 12 surface mounted LEDs illuminate the pins on the pad's inside surface.• The lid: This part allows a camera to be mounted, which tracks the movement of the pins as the sensor is brought into contact with objects.The main innovation in the fabrication of the tactile thumb is in the finger pad which is made completely through multimaterial 3d-printing.The rigid plastic base and rubber pad are thus printed together in one piece, avoiding the need to cast or secure the rubber pad.The pad's inside surface has pins in a regular rectangular layout.At the tip of each pin a small amount of white plastic is 3d-printed.This shortens and simplifies fabrication, by eliminating the need to paint the pins (as in the TacTip sensor [8]).The hemispherical design of the pad is chosen both for stress distribution, and to maximize pin displacements across the TacThumb, thus improving tactile perception.The lens is laser cut from 1 mm acrylic and superglued to the base. 2 holes are cut in the lens, through which the pad is filled with RTV27906 silicon gel.The holes are then sealed with 3d-printed rubber plugs.
The lid is also 3d-printed in ABS plastic, and designed specifically for mounting the Microsoft Lifecam Cinema HD webcam.The webcam is disassembled and its circuit boards rearranged to be more compact and a lens is added to improve field of view.The modular nature of the TacThumb's design means this lid can be redesigned to mount a smaller or higher framerate webcam for future versions of the sensor.The Microsoft Cinema HD webcam is chosen for its low cost, relatively good quality and ease of use (plug and play).
The TacThumb design emphasises straightforward manufacture and assembly, and keeps the manufacturing process simple by 3d-printing the rubber as well as rigid plastic parts, eliminating the need for casting, securing and painting the rubber parts.As with the TacTip, the TacThumb is also low cost, stable, easy to assemble, and robust (its electronic parts are clearly separated from the contact area).
2) M2 Gripper Mount: Basic manipulation is demonstrated by mounting the finger on the Yale Open-Hand Model M2 Gripper [9] (Fig. 1).The M2 Gripper has a fixed thumb (replaced with our TacThumb), and an opposing finger controlled by an agonist and antagonist tendon pair.Each tendon is connected to a separate Dynamixel MX-28AT servo.Although underactuated fingers on other versions of the Open-Hand demonstrate good gripping performance [10], the fully actuated finger on the M2 Gripper allows for finer control, with objects being rolled up and down the fixed thumb.The tip of the moving finger is replaced with a more compliant round tip (also made of a silicon membrane filled with RTV27906 silicon gel) to improve grip during rolling.

B. Data Collection and Processing
1) Data Collection: The M2 Gripper is designed such that to roll objects down (towards the palm), constant tension is applied in the agonist tendon and gradually released in the antagonist, thus bending the finger.Whereas to roll up (away from the palm), tension is applied to the antagonist, and slowly released in the agonist to straighten the finger.The design of the M2 gripper is such that only a restricted range of motion along the TacThumb is possible for the objects considered (≈ 20 mm).Although this range is narrow compared to the 100 mm length of the TacThumb, it is sufficient for demonstrating tactile perception and manipulation performance.
Python scripts written by the GrabLab team (https://github.com/grablab/openhand-software) are used to control the individual dynamixel motors of the M2 gripper.Each motor has a pre-assigned position range (normalised from 0 to 1).
The position of the motors is saved as a starting location after gripping the object, and subsequent movements are calculated from that position, to allow for objects of different sizes to be grasped and manipulated.
Motors are calibrated to roll objects up and down, such that a step downward roll corresponds to releasing the antagonist motor by 0.05 on the normalised scale and tightening the agonist by 0.025 (vice-versa for an upwards roll).
This calibration results in a nearly linear motion of rolling a gripped object along the TacThumb, such that 40 increments move the object over a 20 mm range (Fig. 5) in approximately equal steps of 0.5 mm.This calibration is independently validated by measuring the motion of the rolled object with an optical tracking method that localizes the position of the object along the TacThumb (Fig. 6).
Note that changes to the calibration described cannot affect the performance of the sensor, and hence neither do they affect the tactile manipulation; however choosing this calibration enables us to obtain relatively regular hand movements, as well as providing a means of validation of the sensor's performance.
a) Training: During the training phase, a cylinder of 25 mm diameter (and subsequently 20 and 30 mm dia.) is gripped and rolled down the TacThumb (Fig. 5) according to the motor calibration described above, covering a range of 20 mm in 40 increments (Fig. 6).
b) Testing: We distinguish 2 forms of testing.Offline testing provides an analysis of localization accuracy and algorithm performance using cross-validation post data collection.Online testing adds a physical confirmation of the method's performance during robot operation.
For offline testing of manipulation performance, validation is attained using the training and test data sets used for offline characterization of passive perception performance (this data is collected while using the finger to move the object systematically over the entire range).Data is then sampled from the test set during the simulated manipulation task and the object's position shifted according to a target trajectory in the virtual environment (stepped movements from a center).
For online testing, the test data and hand are controlled in real-time using a closed loop between data capture (python, openCV), analysis (MATLAB) and the control algorithms (python) for the robot hand (Fig. 3).The same training set is used as for offline testing.
We have included a supplementary MPEG format video clip (40.9 MB in size), available at http://ieeexplore.ieee.org,which shows the training and online testing experiments being performed.
2) Data Pre-Processing: In both the validation and manipulation experiments, the sensor's webcam records images of the pins on the inside of the finger pad (Fig. 4) at approximately 20 fps.To track the xand y-coordinates of the pin centers, these images are captured, filtered and thresholded in opencv (http://opencv.org/).The center of the pins are then detected for each frame using contour detection (Fig. 4), and their xand y-cooordinates recorded.Each pin is identified based on its proximity to a default common set of pin positions; if no pin is detected within a radius of 2 mm from its default, then the position from the previous frame is used.

3) Passive Location Perception:
To passively localize an object along the TacThumb, probabilistic methods for localization are implemented that have been previously applied to spatial superresolution with the TacTip [14].We summarize briefly the main steps here, referring to refs [12], [14], [15] for the full technical details; the relation to biomimetic tactile perception is discussed in refs [29].A training data set is collected with each of 40 move increments treated as a distinct location class, with the tactile data at that location used to construct a likelihood model of the location along the TacThumb (using a histogram method).Given test data of unknown locations, this model can be used to determine the likelihood of which location class it originates from.
For offline testing of location perception along the TacThumb, a distinct test set is taken from the hand for performance validation, and a Monte-Carlo procedure used to randomly sample testing data from single moves to classify location (1000 iterations per location class).A location decision error at each location y is then given by the mean absolute error e loc (y) = |y − y dec | with the ensemble average • evaluated over all test runs with the same true location class.This decision corresponds to using just one increment of test data (equivalently the decision threshold has probability zero, in the terminology of [12], [14]).

4) Active Manipulation:
To actively manipulate an object, a simple control algorithm is implemented to extend the passive location perception [12], [14] to active control of object location.A desired location class along the TacThumb is set, and after estimating the object's location (Sec.III-B3) a command is sent to the hand to move to that location.By updating the target location over time, target trajectories can be defined, and the hand attempts to follow them based solely on tactile feedback.
During the manipulation, the location probabilities are updated after each move using Bayes rule [12], [14], [15], with the probabilities from the previous move treated as priors that are combined with the likelihood from the present move.In addition, the location classes of these priors are shifted by the move to remain aligned with the sensor (in an egocentric frame of reference).The posterior location probabilities are then used to estimate the most probable location class for use in repositioning the object during manipulation.For offline testing, validation of the manipulation performance is attained using the training and test data sets also used for offline characterization of the passive perception as a virtual environment.Data is then sampled from the test set during the simulated manipulation task and the object's position shifted according to a target trajectory in the virtual environment (stepped movements around a central fixation).
For online testing, the manipulation task is performed on the hand in real-time, implementing a closed-loop between data capture, analysis and control.We use the same training set as for the offline testing and the same target trajectory for manipulation, rolling the cylinder along the TacThumb based only on tactile data.

A. Inspection of Data
Training data are collected from the hand by rolling a 25 mm diameter cylinder down the TacThumb in 40 increments over a range of approximately 20 mm (∼ 0.5 mm between each position).The range of cylinder movement is reported as 0-20 mm, with 0 mm corresponding to the cylinder's initial position within the hand (15 mm from the tip of the TacThumb; Fig. 5).At each location, the webcam records 15 frames, and the x and  y-coordinates of each of the 26 pins are extracted from each frame.Multiple (15) frames are recorded at each location to reduce noise arising from the pin detection algorithm or small movements of the hand.
Figure 7 shows the data gathered during one such training run, with displacements in the x-direction (across the thumb) in panel A and in the y-direction (along the thumb) in panel B. The pins displayed on the right-hand side diagram of the sensor use the same colour-code so as to be identifiable.The step changes of pin displacements observed in both panels indicate a move to the next location.Panel B also clearly displays the rows of pins (displayed in a similar colour) moving in unison in the ydirection, giving complementary information about the location of the object.

B. Validation-Passive Perception of Cylinder
The TacThumb's performance on localization is first tested using a probabilistic approach for tactile perception.We call this step offline validation as it involves Monte Carlo sampling over a recorded training and test set (Sec.III-B3).
The average localization errors are displayed for each location along the TacThumb in Fig. 8. From this figure, it is clear that localization accuracy is variable along the TacThumb.The best performance is away from the tip of the thumb (at 5-20 mm movement range; 20-35 mm from the TacThumb tip), with   sensor is the spacing between pins on the TacThumb membrane, since each pin functions analogously to single elements in taxel-based devices [11], [12], [14].)This disparity in sensing accuracy along the TacThumb is one of the reasons one might want to manipulate objects, in order to obtain better tactile feedback from them.The next section presents an experiment in which the localization along the TacThumb is applied to a tactile manipulation task.

C. Manipulation
A simple control algorithm is implemented for tactile manipulation, which extends passive perception into active control of object location [13] (Sec.III-B4).
An initial offline testing of manipulation capability is implemented using the training and test data sets as a virtual environment for performing simulated manipulation of the cylinder.Data is sampled from the test set during the simulated manipulation task to have the hand move the object along a target trajectory, with offline hand movements calculated accordingly.We use a stepped trajectory where the hand first attempts to move the cylinder to the center of the location range (10 mm), then displaces it 5 mm towards the base of the hand, back to the center point, 5 mm towards the TacThumb tip, back to the center, 5 mm towards the base, and finally back to the center point again.
For offline testing, the target trajectory is successfully followed within the simulated environment using only tactile information (Fig. 9).The tracking is apparently perfect (0 mm deviation), which is consistent with the localization errors determined from passive perception along that region (5-15 mm) of the TacThumb.
For online testing, the manipulation task is performed on the hand in real-time using the same training set as for the offline testing and the same target trajectory for manipulation (Fig. 10).The hand initially manipulates the cylinder according to the target trajectory, finding the centre point and then displacing the object towards the base and back again.
As can be seen by comparing Figs. 9 and 10, online tracking is not as accurate as in the offline simulation; that being said, online performance is still highly accurate with errors typically below 1 mm.This difference illustrates the importance of going beyond simulation to physical embodiment to fully test an approach.

D. Manipulation of Multiple Objects
The manipulation experiment described in the previous section is repeated with 2 more distinct cylinders (dia.20 mm and 30 mm) in order to verify the performance of the TacThumb with a range of objects.Results for the online tracking of all three cylinders are displayed in Fig. 11.Tracking is successfully performed at an accuracy of ∼ 1 mm across the 20 mm range for all 3 cylinders.
As explained above, our model-free approach allows us to gather tactile data without a precise knowledge of the sensor or hand kinematics.Hence, here we have shown that the approach is also object-independent, since tracking is successful for all 3 cylinders.

V. DISCUSSION
In this study, we designed a TacThumb sensor that can be fully and easily integrated in the Open-Hand model M2 gripper.We also demonstrated this sensor is capable of superresolved perception in a large portion of the gripper's position range.Basic manipulation was then demonstrated by rolling the cylinder up and down the TacThumb along a target trajectory, using a model-free approach which considered only tactile feedback, without a precise knowledge of the M2 gripper's kinematics.
Contact detection and accurate localization are known to be essential for manipulation in robot hands [24].This is a key design feature of the TacThumb, as illustrated by the 40-fold superresolved acuity found in its central region.The nature of the TacThumbs sensing mechanism through pin deflection could also lend itself to detecting other important contact information like force as well as simultaneously identifying the objects being manipulated, as has been found useful in other studies [5], [27].Specifically, the Bayesian approach to active manipulation (Sec.IIIB) could be trained on contact data over normal or shear force, so that then force could be controlled by the gripper to manipulate the object.
In-hand tactile manipulation was successfully demonstrated with 3 distinct objects to sub-millimeter accuracy along desired trajectories.This was less accurate than offline simulated tactile manipulation with pre-recorded test data (which had perfect accuracy), but still highly accurate with errors typically below 1 mm.This illustrates that issues can arise with physical embodiment that do not appear during simulations.The TacThumb's design provides many advantages: it is cheap, robust, easily fabricated, while maintaining a relatively high accuracy (∼ 0.5 mm) over most of its surface, and having the potential to gather information related to forces and object shape.Because the sensor is 3d-printed, design improvements can be readily implemented and assessed.In future work, we will consider use of a smaller camera to further reduce the form factor of the TacThumb sensor, and also modify the opposing finger of the hand to have tactile sensing.

VI. CONCLUSION
Tactile manipulation was demonstrated with a new tactile sensor, the TacThumb, integrated into the M2 gripper robot hand.Manipulation was guided in a model-free way, without the need for a complex kinematic model of the hand.Superresolved manipulation performance was attained over multiple objects, with objects following in-hand trajectories to sub-millimetre accuracy.The combination of the TacThumb with the 3d printed Yale M2 gripper offers an inexpensive, customizable platform for investigating robust and accurate tactile manipulation.As such, this tactile gripper is an ideal device for investigating a variety of tactile manipulation tasks.

Fig. 2 .
Fig. 2. Design of the TacThumb.The pins on the inside of the base's rubber membrane are illuminated by the LED circuit.These pins are displaced during object contact, and are tracked by a Microsoft Cinema HD webcam mounted on the lid.

Fig. 3 .
Fig. 3. Experimental setup.Data is captured from the TacThumb in python OpenCV (http://opencv.org)and sent to Matlab for analysis.The hand's movements are controlled through python.

Fig. 4 .
Fig. 4. Images recorded by the TacThumb.The left panel shows the image captured.The right panel shows the image after pre-processing -red dots represent the pin centers identified by the pin detection algorithm.

Fig. 5 .
Fig. 5. Position range of the M2 gripper during data gathering.The left panel shows the starting position, the right panel shows the end position.

Fig. 6 .
Fig. 6.Distribution of location classes along the range of motion for all 3 objects considered (20 mm, 25 mm and 30 mm dia.cylinders).The distributions being approximately linear, we will consider an equal spacing of 0.5 mm between each location class.

Fig. 7 .
Fig. 7. Training data for the 25 mm diameter cylinder rolling along the TacThumb over a 20mm range.15 frames were recorded at each of the 40 incremental positions.Panel A shows pin displacements along the TacThumb, and panel B shows displacements across the TacThumb.Pins are coloured according to their position on the TacThumb, as illustrated on the right-most panel.

Fig. 8 .
Fig. 8. Localization error e loc dependency on location y along the TacThumb.Errors are averaged over 10 sets of testing data gathered offline from the robot hand.

Fig. 9 .
Fig. 9. Offline tracking.The target trajectory consists of a sequence of pre-set target locations.In the offline version, the actual trajectory corresponds to a shift along location classes in the test set.

Fig. 10 .
Fig. 10.Online tracking.The target trajectory consists of a sequence of preset target locations.The actual trajectory represents the movements of the M2 gripper, as it attempts to replicate the target trajectory based on tactile data.

Fig. 11 .
Fig. 11.Online tracking of multiple objects (20 mm, 25m mm and 30 mm diameter cylinders).Target trajectories are pre-set locations.The actual trajectory shows the movements of the robotic finger during manipulation.