Manipulation of Boltlike Fasteners Through Fingertip Tactile Perception in Robotic Assembly

Robust manipulation of mechanical parts in different grasping configurations is a challenging problem in autonomous robotic assembly that can be overcome by adopting suitable mechatronic solutions. This article proposes a tactile-sensor-based approach that exploits in-hand pose estimation and contact perception to compensate for unavoidable picking, placing, and insertion errors that may occur during task assembly execution under uncertain/perturbed conditions. The main objective of this work is to demonstrate how the use of tactile data, together with both machine learning and model-based methods, allows us to obtain an advanced system able to successfully complete a task that requires the manipulation of boltlike fasteners with different shapes and grasped in different poses. Experiments carried out using the proposed robotic system are reported for a specific assembly task in order to evaluate the effectiveness of the proposed solution. By means of suitable calibration procedures exploiting the same methods proposed here, the system can be easily adapted to different objects and shapes.


Manipulation of Boltlike Fasteners Through Fingertip Tactile Perception in Robotic Assembly
In autonomous assembly tasks [2], [3], [4], [5] where multiple components must be repeatedly grasped, transported, and put together by a robot, precise manipulation is particularly critical since the task success and the quality of the assembled product directly depend on the robot performance.One relevant example is the production lines of cars, where metal joints must be inserted or glued to nonmetal elements (such as shells or panels) in precise configurations in order to provide reliable support for the additional components [6].Although industrial robots are typically designed to guarantee positioning accuracy, their performance during assembly tasks can still be affected by different factors such as erroneous grasping poses, contact dynamics, and slipping during transportation or placing.An effective strategy to overcome these issues is the exploitation of additional sensors for the in-hand estimation of the object pose or contact to compensate for possible misalignments [7], [8], [9], [10].For instance, in [8], Nozu and Shimonomura proposed an integrated gripping system, endowed with a camera and force sensor, for tactile and visual in-hand localization of bolts to be assembled.Analogously, in [9], a two-camera system is deployed for the in-hand pose estimation of VGA connectors to be grasped and plugged by a robot manipulator.In this work, we address a similar problem by considering an industrial assembly scenario in which a lightweight robotic manipulator is tasked to glue boltlike fasteners on carbon-fiber monocoque cells of cars.Specifically, in the proposed case study, the robot helps the human operator by autonomously picking, inserting/placing, and gluing boltlike fasteners called bigheads (BHs) on the monocoque [11], [12], [13].A BH is a commercial fastener composed of a flat pierced disk (commonly indicated as base or head), which is the part of the item to be glued on the monocoque, and a threaded side (commonly indicated as thread or screw) that is used as a support for the additional mechanical components to be assembled during the following steps of the production line.As depicted in Fig. 1, BHs can be either inserted into prearranged holes (by-the-screw) or placed (by-the-base) in predefined points of the monocoque.In both cases, high degree of accuracy is required to fit the holes or to precisely place the items within the expected tolerance.Those insertion/placing tasks, which must be repeated hundreds of times for each monocoque (∼250 BHs must be applied on a typical vehicle monocoque), are particularly hard and tiresome for the operators; hence, a robotic solution is preferred.If, on the one hand, a robotic manipulator guarantees the accuracy and the repeatability needed to accomplish such high-precision repetitive task, on the other hand, the robot should also be able to estimate the in-hand pose of BHs in order to compensate for possible errors, which may affect the insertion/placing process.
Tactile sensors represent a key enabling technology to improve the robot capabilities in the execution of complex tasks.Designing of handling tools equipped with tactile sensing motivated the development of commercial solutions, e.g., BioTac, DigiTacts, Tekscan, Peratech, Contactile, and Gelsight.However, new solutions to improve the commercial ones are still proposed, since some features (e.g., performance, flexibility, compliance, and adaptability) are still not satisfactory.For example, recent papers report both the use of commercial solutions in novel applications (e.g., Gelsight in [14] and Contactile in [15]) and the development of new technological approaches (e.g., TacTip based on a camera integrated into a deformable layer [16], Gtac that integrates a piezoresistive layer and a layer of Hall effect sensors [17]).In the last two decades, several results about tactile sensors applied to manufacturing and daily life applications have been presented.Typical approaches are based on both machine learning and model-based techniques.These methods mainly aim to discriminate object properties or to implement control tasks.For instance, Liu et al. [18] proposed a solution to discriminate object curvature using a sparse tactile sensor array; Da Fonseca et al. [19] presented a combination of fuzzy logic controllers and machine learning methods to improve the grasp stability for under-actuated hands; Costanzo et al. [20], [21] proposed control strategies for object pivoting and object in-hand manipulation by using parallel grippers equipped with tactile sensors.The cited papers are examples of the wide available literature on this subject.A technology review on soft biomimetic tactile solutions able to reproduce humanlike capabilities can be found in [22] while Romeo and Zollo [23] review both sensor technologies and methodology approaches for slip detection in robotic applications.The methodologies for estimation of object properties based on tactile sensor data are considered in [24].Tactile sensing for dexterous manipulation can be deepened in [25], [26], and references therein.
In this article, a robotized framework is presented, based on a sensorized compliant fingertip for the tactile perception of the grasped components, for the insertion and the precise placing/gluing of BHs to carbon-fiber monocoque cells of cars.
The sensor allows the estimation of the in-hand pose of the BH, which can be used to compensate picking errors by adjusting the robot trajectory during the insertion/placing phases.The main objective of this work is to demonstrate that the tactile sensor, developed by some of the authors [27], can be used to tackle complex tasks by using both machine learning and model-based methods on the same tactile data, selecting the most suitable one depending on the specific problem to solve.As model-based techniques we intend methods that do not require the acquisition of an extensive training set and a subsequent training phase to determine a possibly large number of hyperparameters, that, on the contrary, are required by machine-learning techniques.Previous papers reported the effectiveness of the proposed technology for the shape recognition of grasped wires [28] and for the classification of wires on the basis of their diameters [29].The main contributions of this work are as follows.
1) A novel tactile-sensor-based approach to enhance object grasping, insertion, and positioning during the execution of complex assembly tasks is proposed.The method exploits both machine learning and model-based techniques, but in different phases of the task, to allow robot trajectory corrections based on in-hand pose estimation and contact perception.
2) The proposed approach is experimentally tested on a challenging industrial case study, in which several components must be either inserted or placed/glued to the carbon-fiber monocoque cells of cars with high accuracy.
The effectiveness and performance are evaluated in terms of errors and success rate by considering several repetitions of the same task.In particular, the requirements of the industrial case study are assigned in terms of positioning accuracy in the case of the positioning of BHs from the threaded side, with a target of 0.5 mm; for the positioning from the base side, the target success rate is of 90% to minimize the human intervention in the failures.Finally, when the BH is glued on the surface, the pressure should be high enough to allow the glue to penetrate all six holes of the BH base.
The rest of the article is organized as follows.Section II provides the description of the main hardware components of the proposed framework.Section III focuses on the calibration procedures for the different types of tasks (insertion and placing).In Section IV, the experimental setups are described and the performance of the proposed approach is discussed.Finally, Section V concludes this article and discusses future works.

II. SENSORIZED FINGERS
For the task execution a standard robotic system, composed of a Universal Robot UR5e and a Robotiq Hand-E gripper, has been equipped with a couple of custom fingers: The first one is sensorized with a tactile sensor and the second one is passive (see Fig. 2).The tactile sensor is composed of 25 photo-reflectors, distributed as a 5 × 5 matrix on a printed circuit board (PCB), assembled with a suitably designed mechanical grid and a silicone pad.The tactile map corresponds to the deformation of the deformable silicone layer detected by the optoelectronic sensing elements (commonly called "taxels").The taxels have a spatial resolution equal to 3.55 mm, by resulting in a sensitive area of about 21 × 21 mm 2 .The optical components integrate infrared LEDs and phototransistors optically matched.The driving current for LEDs is fixed by using an adjustable current source and the phototransistor signals are directly digitized by a microcontroller equipped with 12-b low-noise A/D channels.This allows us to reach a sampling frequency of 500 Hz for all the 25 signals of the tactile map.The deformable pad transduces the contacts with external objects into distributed deformations measured in 25 points, corresponding to the photoreflector positions.The measurements correspond to the voltages available on the phototransistors on the basis of the light emitted by LEDs and reflected by the bottom side of the deformable pad.The mechanical properties of the silicone allow to obtain a response time lower than 10 ms.The dimensions of the silicone pad ensure an intrinsic compliance to the tactile sensor, which makes it particularly suitable for applications involving interactions with the environment in the presence of small uncertainties in positioning.The passive finger presents the same mechanical parts, and in particular the same deformable pad, as the sensorized finger, but without any photoreflector.A suitably developed robot operating system (ROS) node interrogates the microcontroller via a serial interface to acquire the raw voltages from the tactile sensor.In a preliminary elaboration, the node computes the voltage offsets of all phototransistors when the sensor is in rest condition, as the mean value over the first 50 received samples.Then, the node makes available the unbiased voltages v i , with i = 1, . . ., 25 hereafter.The interested reader can find additional details about the hardware and the elaboration system software in [27].Fig. 3 reports the taxel distribution with respect to the reference frame, by highlighting the corresponding voltage signal v i of each taxel.

III. TASK-BASED APPROACH AND SYSTEM CALIBRATION
For the execution of the BH assembly task, the proposed approach is based on three main capabilities of the tactile sensor that require a suitable calibration: 1) estimation of the  BH pose when grasped from the thread, 2) estimation of the BH-environment contact force, and 3) estimation of the BH pose when grasped from the base.

A. Pose Estimation While Grasping the Thread
The correct execution of the considered task needs the grasping pose knowledge of the BH.To this aim, a suitable calibration procedure has been developed for the estimation of the M 6 BH pose after its grasping.The pose is defined with respect to the reference frame reported in Fig. 2. The estimated pose can be used to improve the positioning precision of the BH with respect to the workpiece, by adjusting the nominal target pose of the robot on the basis of the actual pose of the BH with respect to the robot end effector.
The sensor calibration aims at estimating the pose of the grasped BH from the tactile map signals.The relation between the tactile signals and the BH pose is nonlinear as clarified ahead.Therefore, a machine learning approach based on a feed-forward neural network has been adopted.In detail, since the robotic system integrates a parallel gripper and the BH is axial-symmetric, during its grasping, it is evident that, with respect to the reference frame (see Fig. 4), the z coordinate can be considered always equal to zero.Hence, the grasped BH has only 3 degrees-of-freedom (DOFs).By defining with Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.(x 0 , y 0 ) the position of the thread end, and with ϑ the angle between the BH axis and the y-axis, the BH pose is uniquely defined by the variables (x 0 , y 0 , ϑ), which have to be estimated from the tactile map.Model-based approaches could be based on image features extracted from the tactile map.Fig. 5 reports an example of a tactile map, corresponding to a grasp from the threaded side, where the radius of each circle is proportional to the taxel voltage v i .From the latter, it can be recognized how the position of the BH could be correlated to the centroid of a tactile map, which is a nonlinear function of the taxel voltages v i , as shown in (3); furthermore, its orientation could be correlated to the second-order moments of the image, again in a nonlinear way (see [30,Ch. 10]).This explains why the coordinates x 0 and y 0 , and the angle ϑ are estimated from the tactile signals by means of a suitably trained neural network.By indicating with NN(u u u; π π π), a generic neural network with input vector u u u and learned hyperparameter vector π π π, the BH pose x x x = (x o , y o , ϑ) is computed as being v v v = (v 1 , . . ., v 25 ) the vector of the 25 tactile signals.
For the neural network, the hyperparameter vector has to be learned from a suitable training set, constructed as described in Section III-D.The calibrated neural network will be used to estimate the BH pose in real time, at the same sampling frequency of the tactile data.However, the nominal robot trajectory is corrected by leveraging the pose of the grasped BH, estimated just before the approaching phase, and assuming this feature static, as detailed in the experiments.

B. Contact Indicator Estimation
The sensorized finger can be used to estimate a two component contact indicator related to the force generated along the grasping xy−plane when the grasped BH comes into contact with the environment.The sensor was not calibrated in terms of physical force to avoid the acquisition of another training set and the use of a reference sensor.Hence, a simple model structure is adopted to define the contact indicator with no hyperparameter.Although this indicator is not dimensionally homogeneous to a force, its two components allow us to quantify the magnitude and the direction, in the plane of the tactile sensor, of any possible contact between the BH and the environment.This information can be appropriately used for the robot programming, enabling a robust BH positioning despite potential uncertainties related to, e.g., initial grasping position of the BH, coordinates of final positioning, the estimation of grasped BH pose.The two contact indicator components, defined as I c = (I x , I y ), are estimated as the displacement of the tactile map centroid caused by an external contact, i.e., where (x c , y c ) and (x c0 , y c0 ) are the coordinates of the tactile map centroid with and without contact, respectively.In both cases, the coordinates are computed from voltage signals v i as where (x i , y i ) are the ith taxel coordinates with respect to the reference frame.As for the BH pose, also the contact indicator is estimated in real time, at the same sampling frequency as that of the tactile data.Notice that the two estimated indicator components correspond to the displacement of the tactile map centroid induced by an external force, hence they are inherently connected to the contact force, regardless of the orientation of the grasped BH.In addition, the two components are not used independently for contact detection.Instead, only their norm ||I c || is considered, by comparing it against a threshold.This is exploited during the approaching and release phases to detect the contact and control the robot trajectory, as detailed in the experiments.

C. Position Estimation While Grasping the Base
As detailed earlier, the task execution implies the correct positioning of BH grasped from both the threaded side and the base side.The thin size and the circular shape of the BH base make essential the use of a compliant interface during grasping, capable of adapting its shape to conform to the base.The use of a rigid finger would reduce the contact area to almost a single point, by making the grasping unstable.Instead, by using the tactile sensor it is possible to grasp the BH from the base side in a stable way, enough to perform the task.We assume that the BH grasping from the base side starts with the BH positioned in a suitably designed housing, which allows a grasp pose always with the threaded side vertically oriented and the base horizontally oriented, as shown in Fig. 6.The latter shows the two variables (x b , y b ) that uniquely define the BH base position with respect to the reference frame in this configuration.These coordinates are directly related to the centroid coordinates defined in (3).In this specific application, it can also be assumed that after grasping, the BH is unlikely to be touched by anything, thereby enabling us to consider its in-hand orientation as fixed.Indeed, in this case, no glue application is required on the BH to be inserted in a hole, since the glue is supposed to be already put on the surface around the hole.These two assumptions have been verified during the experiments in a lab-scale setting reproducing an environment similar to the real one (images of the setup can be found in Section IV-B and in the attached video).Differently from the problem of estimating the BH pose while grasped from the threaded side, in this case, the contact area is quite limited, and thus, only the position of the contact point on the BH base has to be determined.This allows us to avoid the usage of machine learning techniques and to adopt a sort of graybox approach by assuming that the relation to the coordinates of the tactile map centroid can be modeled through two polynomial functions f x and f y whose coefficients are identified during calibration.

D. Calibration and Testing for Pose Estimation From the Threaded Side
In order to calibrate the neural network defined in Section III-A, a BH has been repeatedly grasped in different poses and with different gripper closures from a specific holder designed to ensure almost the same starting position of the BH between each grasp (see Fig. 7).In detail, the training set has been built by grasping the BH threaded side end in the (x 0 , y 0 ) positions, corresponding to the grid of yellow dots reported in Fig. 3.The values, with respect to the reference frame, correspond to x 0 ∈ [−5, 5] mm and y 0 ∈ [1,6] mm with a step of 1 mm, for a total of 66 different positions.The covered area was selected to be compatible with the BH dimensions while avoiding grasping positions too close to the sensor pad edges.For each (x 0 , y 0 ) position, several grasping with different angles ϑ have been applied.Taking into account the footprint of the BH base with respect to the tactile sensor case, the reachable ϑ range is smaller when the y 0 coordinate is closer to the frame origin.In detail, for each y 0 from 1 mm to 6 mm, the acquired ranges were, respectively, } with a step of 1 • .By combining these values for the three variables (x 0 , y 0 , ϑ), a total of 2156 different BH poses have been acquired.For each pose, six different gripper closures have been used, and three full repetitions have been executed, resulting in a total of 38 808 samples in the training set.It is worth noting that this procedure for the training data acquisition is fully automatized and the time needed for the acquisition of a complete training dataset is less than 60 min.Moreover, as the approach utilizes the same robotic system used for the task execution, the costs in terms of hardware and time appear fully compatible with applications in real industrial settings.Of course, if a new BH (e.g., with different threaded, size, or weight) has to be manipulated, the procedure can be easily repeated in the same time.
These data have been used to train the neural network, as defined in Section III-A, with a structure consisting of a single hidden layer employing tansig(•) as the activation function.Alternative activation functions were tested, but no significant differences were observed.To optimize the hidden layer, several networks with incremental dimensions in terms of the number of neurons have been trained.The resulting networks have been then compared by considering their performances on data belonging to a testing set, acquired by grasping the BH with poses not included in the training data.In particular, the positions used for the testing set acquisition, depicted in cyan in Fig. 3, correspond to intermediate points with respect to those used for the training set (yellow dots): x 0 ∈ [−4.5, 4.5] mm and y 0 ∈ [1.5, 5.5] mm with a step of 1 mm, for a total of 50 different positions.As for the training set, for each y 0 from 1.5 mm to 5.5 mm, the angle ϑ acquired ranges were different, i.e., ϑ ∈ {[−5 } with a step of 1 • , for a total of 1550 different BH poses.In this case, a single repetition involving 6 different gripper closures has been executed, resulting in a total of 9300 testing samples.The comparison among trained networks has been made by computing the root mean squared error (RMSE) performance function on the testing data.Fig. 8 reports the result of the comparison, by highlighting that a number of neurons in the hidden layer equal to 12 represents the best choice for the network.The obtained RMSE values are considered satisfactory in terms of performance when compared with the testing data ranges outlined above.In particular, the different performance among x 0 and y 0 is due to the fact that the variation range for x 0 is about twice the variation range for y 0 .Moreover, the RMSE value obtained for ϑ is mainly related to configurations in which the angle varies in wider ranges (up to 50 • ).However, high values for the angle are uncommon in the real scenario; hence, the experimental results reported in the following section demonstrate good performance for the tackled task.The results for the BH pose estimation of the selected network on the testing data are reported in Fig. 9, showing the comparison between estimated (x 0 , y 0 , ϑ) and ground-truth values.Additional tests have been carried out to evaluate the accuracy of pose estimation in generic cases not included in the training set neither in the Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

TABLE I
ESTIMATION ERROR e p FOR GENERIC POSES FOR BH GRASPED FROM THREADED SIDE Fig. 8. RMSE nets performances for x 0 , y 0 , and ϑ computed on testing data with respect to hidden layer dimension.Fig. 9. Comparison of estimated positions (x 0 , y 0 ) and estimated orientations ϑ with ground-truth for testing data.
testing set.By using the same setup in Fig. 7, the BH has been grasped with r = 10 different new poses and the estimation error e p (p = 1, . . ., r) has been evaluated for each one.The results are reported in Table I, where (x 0 , y 0 , ϑ) are the actual values, and (x 0 , ŷ0 , θ) the estimated ones.

E. Testing for the Contact Indicator
Specific testing has been carried out to evaluate the sensitivity of the contact indicator defined in Section III-B.In detail, a BH grasped in a known pose has been positioned, from the base side, on a workbench whose vertical position was known with an uncertainty equal to ±5 mm.After an initial phase, where the robot aligns the BH base to the workbench plane at a nominal distance equal to 5 mm, the robot approaches the workbench with incremental displacements of 0.1 mm each in a 0.1 s time interval, by continuously monitoring the norm ||I c || of the contact indicator.This approaching phase ends when the monitored norm exceeds a predefined threshold whose value depends on the application needs.Fig. 10 reports the norm ||I c || during one of the described approaching experiments, where the workbench actual position was about 2 mm far from the initial position.As shown, the contact indicator norm ||I c || starts from a nonzero value that is strictly related to the weight of the BH, since (x c0 , y c0 ) are computed before raising the BH from its starting position.Then, the signal presents a rapid decrement during the initial contact with the workbench plane (see the time interval around 2 s) while approaching the final position.This is due to the compensation of weight by the table reaction.As the robot continues the approach phase, the signal begins to increase due to the applied pushing force.By comparing the signal with a threshold, the robot movement can be halted, and the BH can be released onto the workbench when the contact indicator reaches the threshold.This latter should exceed the value corresponding to the BH weight.Thus, to make the decision independent of this parameter, the signal is unbiased just before the approach phase starts, as detailed in Section IV-A.

F. Calibration for Position Estimation From the Base Side
Suitable sets of tactile data have been acquired in order to estimate the coefficients of the polynomial functions f x and f y in (4).For the estimation of f x coefficients, the BH has been grasped from the base side in 9 known positions along the x-axis, corresponding to the coordinates x b = {−8, −6, −4, −2, 0, 2, 4, 6, 8} mm and y b = 0.The centroid x c values have been computed according to (3).Then, the coefficients of a third-order interpolating polynomial function have been estimated by means of a standard least-squares method.Fig. 11

IV. EXPERIMENTS
The task in the industrial scenario is constituted by a sequence of two subtasks: 1) positioning of some BHs grasped from the threaded side and 2) insertion of some BHs grasped from the base side.In this section, experiments dedicated to these subtasks are reported and discussed separately.All experiments are implemented using a ROS network, with ROS nodes implementing the components proposed above; all the nodes are executed at a sampling frequency of 500 Hz.

A. Positioning From the Threaded Side
The first set of experiments aims to demonstrate how the pose estimation presented in Section III-A, together with the contact indicator in Section III-B, can be used to improve the accuracy for the BH positioning.The main objective is to improve the robustness of the positioning procedure in the presence of uncertainties relative to, e.g., BH initial pose, position of the BH grasping plane and position of the BH releasing plane with respect to the robot reference frame.The experiment consists of grasping the 4 BHs initially positioned in the right side of the workbench, and moving them from the start positions to the corresponding target positions on the left side of the working area.Both start and target positions present an area with a 1-mm grid and a circumference corresponding to the BH base to allow evaluation of initial and final positioning errors.In the figures presented in the following and in the video attached to the article, several views of the working area with start and target positions are provided.During the approaching and release phase, the contact indicator described in Section III-E has been suitably exploited.In particular, the contact indicator ||I c || value is unbiased just before the execution of the approaching phase to remove the initial value related to the BH weight.Then, the threshold value is empirically fixed as it depends on several features characterizing the real scenario: the BH grasped pose, the BH weight, the stiffness of the tactile sensor pad, the presence of the glue, and its viscosity.As also reported in the video attached to the article, without the glue a threshold of 0.4 is successfully used for all presented experiments with the BH grasped from the threaded side.In the presence of glue, the threshold is empirically increased to 1.2 in order to ensure that the glue penetrates well in the BH holes.The experiment has been repeated for different test cases (TC), which are as follows.
1) Nominal conditions: The BHs are placed in start positions with no errors [see Fig.  Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.

3) Perturbed conditions (translation errors) with correction:
As in TC2, the BHs are placed in start positions with random translation errors both along horizontal and vertical axes.Differently from TC2, the BHs are grasped and moved to target positions, where their poses are corrected before being released by using the information on actual poses estimated through the tactile data.4) Perturbed conditions (translation and rotation errors): The BHs are placed in start positions with random translation and rotation errors with respect to the nominal case [see Fig.
].Then, the BHs are grasped and moved to target positions, where they are released, without using information about the actual poses estimated by the tactile data.5) Perturbed conditions (translation and rotation errors) with correction: As in TC4, the BHs are placed in start positions with random translation/rotation errors.Differently from TC4, the BHs are grasped and moved to target positions, where their poses are corrected before being released by using the information on actual poses estimated via the tactile data.Fig. 12 showcases images of initial poses for the various TCs, providing a visual representation of the accounted uncertainties.These uncertainties affect positions (up to 4 mm) and orientations (up to 10 • ).Each TC has been repeated 10 times with randomized perturbations.The maximum positioning errors e MAX = max(e p ), p = 1, . . ., 10 obtained after the placement, along the horizontal and vertical axes relative to the nominal target position, are reported in Table II.
The results in TC1 are useful to quantify errors due to robot precision and the effectiveness of the automatic BH releasing on the basis of the contact indicator value an example of the indicator trend is reported in Fig. 13; the releasing of the BH is triggered as soon as the norm of the contact indicator overcomes the threshold.The obtained values allow also a comparison with the other TCs results.In all TCs, it is evident that the initial horizontal positioning errors are automatically reduced and the final ones are comparable with the nominal TC1, due to autocentering of the parallel gripper used in the task.By comparing TC2-TC3 and TC4-TC5, instead, the result of using the estimated poses to correct the BH position and orientation with a standard rototranslation (executed by the robot before  the releasing phase) is evident.The applied corrections allow a significant reduction of e MAX in all cases, by ensuring at least a halving and up to a complete zeroing of the errors.Fig. 14 shows some pictures of final BH positions obtained in the different TCs.Fig. 15 illustrates the whole configurations reached for all 4 BHs in a TC5 experiment.
The TC5 has been repeated simulating the operation of adding glue on the BH before the final position phase, as it happens in the real task (for details see the video attached to the article).For practical reasons, in this experiment, the glue has been substituted by an alternative material easy to remove but with similar viscosity.The objective is to test the robustness of the pose estimation algorithm also in the presence of interactions with the environment (e.g., the application of the glue).In this case, in fact, the neural network needs to work with input data slightly different with respect to those it was trained with.There are two main differences: 1) the added weight of the glue on the BH and 2) the possible change in the pose while the BH is already in contact with the silicon pad of the tactile sensor.The experiment has been repeated using start positions similar to those used in previous experiments.During these tests, mainly two different situations occurred depending on the intensity of external forces acting on the BH (i.e., in the proposed test, the amount of pressure the operator applies during the glue application).In the case of low pressure, the compliance given by the silicone pad allows the BH to keep its position with respect to the fingers [see Fig. 16(b)] while in the other case, the BH slips during the application of the glue, leading to a change in the BH pose [see Fig. 16(a)].Nevertheless, both these situations are handled by the pose estimation algorithm and the final target position is corrected as shown in Fig. 17 16) is reported.Therefore, the overall results show similar performances in terms of positioning errors with respect to the case without the application of the glue, as confirmed by the errors reported in the last section of Table II.In all cases when the tactile data are exploited, the application requirements in terms of positioning accuracy and glue-effective application have been correctly met.Fig. 18 reports the pictures from different points of view for all BHs after the final positioning.Further details of TC4 and TC5 (with and without viscous material) executions are reported in the attached video.

B. Positioning From the Base Side
The second set of experiments aims to demonstrate how the position estimation of the BH base presented in Section III-C can be used to improve the accuracy of the BH insertion.In all experiments described ahead, each BH is grasped from a start position, lifted up, aligned with the nominal coordinates of the target hole center, moved down slowly in order to initiate the insertion, and finally released before the complete insertion.Also for this subtask, different test cases have been considered to evaluate the effectiveness of the proposed strategy.In particular, with reference to Fig. 19, the following test cases (TCB) have been repeated.
1) Nominal conditions: 3 BHs with random threads (M 5 or M 6) are inserted in a rail with a center distance equal to 6.5 mm at 3 nominal start positions with no errors.The BHs are grasped in sequence, lifted up, and inserted in 3 target holes characterized by 3 different heights with respect to start positions and diameters equal to 12 mm, 12 mm, and 6 mm, respectively (see Fig. 19 for details).2) Perturbed conditions: The BHs are placed in start positions as described in TCB1 with random translation errors both along the rail and in height.Then, the BHs are grasped, lifted up, and inserted in target holes by Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.using nominal coordinates of hole centers, without using information about grasped positions estimated via tactile data (according to Section III-C).3) Perturbed conditions with correction: The BHs are placed with random translation errors as described in TCB2.Differently from TCB2, the BHs are grasped, lifted up, and inserted in target holes only after a position correction with respect to nominal coordinates of hole centers, executed by the robot based on the BH position estimated using tactile data.During the insertion phase, both the contact indicator defined in Section III-B and the end-effector vertical position z ee (i.e., the height of the sensor reference frame with respect to the workbench) are monitored.The robot moves each BH to target holes with displacement steps of 0.2 mm each in 0.1 s, and for each step, both the contact indicator and the robot's vertical position are evaluated.The contact indicator is monitored to evaluate possible undesired collisions with hole edges.In this case, the threshold has been empirically fixed to 0.2 and, in particular, if the contact indicator overcomes the threshold value, the task is considered failed due to contact with hole edges.In such a case, the robot moves the grasped BH in a known position where it is released in a waste container.If no contact occurs (i.e., the contact indicator is always below the threshold), when the robot reaches the vertical position fixed for the BH insertion, the gripper is opened and the BH is released into the hole.The task is considered successfully completed if the released BH appears correctly inserted in the target hole.Fig. 19 reports a picture of the setup used to execute this TCB; the 3 BHs appear placed in start positions (in the top of the picture), the first and the second target holes (with 12 mm diameter) are in the center and in the bottom right part of the picture, the third hole (with 6 mm diameter) is in the bottom left part of the picture, and the red waste container is on the left.Fig. 20 shows some pictures of BHs placed in start positions with perturbed conditions both along the rail and vertically.The nominal conditions correspond to the case where the BH base lies on the rail and the thread is aligned with the bold vertical line.Fig. 21 reports some   pictures of possible cases at the end of the task execution.The TCB1 has been repeated 20 times to verify that the setup accuracy allows us to obtain a 100% of success rate during the task execution in nominal conditions.Then, to evaluate the effectiveness of the proposed approach based on the use of tactile sensor information, the TCB2 and TCB3 have been repeated 20 times each and the results have been compared in terms of success rate.During the repetitions, in both cases, random perturbed conditions and random BHs (with M 5 or M 6 thread) have been used.The comparison has been made by computing the success rate separately for the holes with 12 mm diameter and the hole with 6 mm diameter.The results of Table III show how the improvement in the success rate is significant, especially with holes with little tolerance.In all cases, the target of a 90% success rate has been achieved.Examples of TCB2 and TCB3 executions are reported in the attached video.

V. CONCLUSION
An approach to boltlike object manipulation for robot assembly tasks that relies on fingertip tactile perception was proposed.The method combines an advanced grasping system, consisting in a parallel gripper equipped with optoelectronic tactile sensors, Authorized licensed use limited to the terms of the applicable license agreement with IEEE.Restrictions apply.and tactile data interpretation algorithms, suitably designed for in-hand pose estimation.The system was calibrated and experimentally tested for the execution of insertion from the threaded side and placement from the base side of BHs on a lab-scale mockup.The results show how the fingertip perception of the object pose and contact highly improves the success rate during the execution of insertion cases (with success rate always ≥ 95%) and the precision during the positioning of the base side (by reaching at least a halving and up to a complete zeroing of the positioning error).The main advantage of the proposed solution is that the sensing system is always the same and the data interpretation methods are based on machine learning and modelbased techniques easy to adapt to different cases.Hence, by means of suitable calibration procedures, the proposed method can be easily adapted to the manipulation of different objects, shapes, and grasping poses.A limitation is clearly related to the estimation of BH orientation during the grasp from the base side.For the considered scenario and under the discussed assumptions, ignoring the orientation did not represent a real problem.However, to improve the applicability of the proposed strategies to different objects also with small contact areas, alternative approaches will be evaluated in future works, such as to increase the sensor spatial resolution and/or to use a couple of sensorized fingers, by combining the two tactile maps.

Fig. 1 .
Fig. 1.Example of BHs glued to the monocoque in the two configurations.(a) Inserted.(b) Placed.

Fig. 2 .
Fig. 2. Sensorized (left) and passive (right) fingers with the reference frame: x-axis in red, y-axis in green, and z-axis in blue.

Fig. 3 .
Fig. 3. Taxel distribution and corresponding voltage signal v i .Yellow and cyan dots represent the grid of positions used for the acquisition of training and testing sets, respectively, with respect to taxels and reference frame.

Fig. 4 .
Fig. 4. Definition of variables related to the BH pose (threaded side) with respect to the finger reference frame defined in Fig. 2.

Fig. 5 .
Fig. 5. Sample tactile map in grasping the BH from the threaded side.

Fig. 6 .
Fig. 6.Definition of variables related to the BH position (base side) with respect to finger reference frame defined in Fig. 2.

Fig. 7 .
Fig. 7. Setup for the calibration and testing of pose estimation from the threaded side.(a) BH placed in the holder.(b) BH grasped with a known pose.
reports the estimated f x polynomial function.Similar data have been acquired to estimate f y by grasping the BH base in 9 known positions along the y-axis, corresponding to the coordinates y b = {−8, −6, −4, −2, 0, 2, 4, 6, 8} mm and x b = 0.The centroid y c values have been computed according to (3) and the coefficients of the third-order interpolating polynomial function have been estimated, by obtaining results similar to those shown for f x .

12
(a)], grasped, and moved to target positions, where they are released.2) Perturbed conditions (translation errors): The BHs are placed in start positions with random translation errors both along horizontal and vertical axes with respect to the nominal case [see Fig. 12(b)].Then, the BHs are grasped and moved to target positions, where they are released, without using information on actual poses estimated by the tactile data.

Fig. 13 .
Fig. 13.Contact indicator ||I c || value and corresponding threshold used during the experiments with the BH grasped from the threaded side.
(a) and (b), where the final placing, with the BH pose estimated after the operator action (corresponding to the two examples of Fig.

Fig. 19 .
Fig. 19.Setup used for the insertion of BHs grasped from the base.

Fig. 20 .
Fig. 20.Examples BHs prepared in perturbed conditions: Translations (a) along the rail and vertical, (b) along the rail to the right, and (c) to the left.

Fig. 21 .
Fig. 21.Examples of BHs at the end of task execution.(a) BH correctly inserted in a 12-mm target hole.(b) BH correctly inserted in the 6-mm target hole.(c) BH in a failed insertion case.

TABLE III SUCCESS
RATE FOR POSITIONING FROM BASE SIDE