Catching Robot: Predicting the Trajectory of a Rolling Ball Using Transformer

Various tasks in robotics such as pick and place and catching flying/rolling objects have been studied in the literature. Previously, to accomplish such tasks, it was necessary to detect the position of the object using a Sobel detector, a marker, or a stereo method and then predict the trajectory of the object through the model-based Kalman filter. However, these existing studies are not practical, since with this detection method, only one type of object can be detected or additional equipments are required. In addition, to compute the Kalman filter, a measurement of the object’s position is essentially required, which may not be precise in various situations due to unmodeled noise. In this paper, we study the new framework of catching a rolling ball task in robotics using only machine learning techniques. Unlike previous approaches that rely on specified markers or stereo camera systems, our method uses a machine learning algorithm that can learn object positions and detect various sizes of balls using only one RGB camera without any markers. In our method, Convolutional Neural Network (CNN)-based models are applied to detect objects and the transformer model with an attention mechanism is applied for end-to-end trajectory prediction. We use the robotics simulator to efficiently train models and evaluate their performance directly in the real world. The experimental results of catching a rolling ball show that our framework is practical, and performs well in various sizes of balls. By using the proposed framework, the performance of the
$Gripper \, vicinity$
is 93.3% and the Catching success rate is 73.3%. In contrast, other baselines, such as CNN and long short-term memory (LSTM), show poor Gripper vicinity and success rates, with all criteria falling below 30%.