End-to-end pixelwise surface normal estimation with convolutional neural networks and shape reconstruction using GelSight sensor

The GelSight tactile sensor is designed to measure the topography of the contact surface. Photometric Stereo algorithm is utilized to reconstruct the depth map of the surface, which, however, suffers from artifacts because of the ununiform illumination of the sensor. In this paper, we propose a method based on Convolutional neural networks (ConvNets) to estimate the surface normal from the color image captured by the GelSight sensor. The ConvNets, which combine the information of pixel values of red, green and blue (RGB) channels and pixel positions, are trained end-to-end and pixel-to-pixel. We test the model by measuring the geometry of two 3D printed polygon objects and demonstrate that the reconstructed depth maps have better accuracy than those with the photometric stereo method. To further extend the application of measuring topography, we combine the ConvNets with point cloud registration method and perform object shape reconstruction in 3D space by tracking the pose of the GelSight sensor. We successfully reconstruct the 3D point cloud of a 30.6 mm × 30.6 mm × 30.6 mm cube and the fine texture on the cube surface, with less than 1.0 mm error. The method is also utilized to accurately estimate the center axis position of a bottle cap by touching it at four different positions with step of 90.0 degs. This technique can be helpful for robots to explore and interact with the surrounding environment.