Toward Vision-Based High Sampling Interaction Force Estimation With Master Position and Orientation for Teleoperation

In this study, a vision-based high sampling rate interaction force estimation method is proposed for teleoperation systems that uses master position and orientation information without using physical sensors such as force/torque (F/T) or tactile sensors. The proposed method uses red-green-blue (RGB) images, six-axis robot pose and motor current data, gripper position and current data, as well as master position and orientation information as inputs without requiring force sensors. To estimate the interaction forces, a deep neural network composed of densely connected convolutional network (DenseNet) and long short-term memory (LSTM) is proposed. The database was created by operators using grip and picking motions to interact with 10 objects over a teleoperation system. In addition, we compared the proposed method with different deep learning networks that used different sets of inputs. The results show that the proposed model can estimate 1 kHz interaction force based on 60 Hz images and 1 kHz master inputs. Moreover, the results indicate that the master position and orientation information are useful in estimating the interaction force at a high sampling rate through the result of the change in the network input.