Real-Time Reconstruction of 3-D Tactile Motion Field via Multitask Learning

The 3-D motion field on the surface of the vision-based tactile sensor contains rich tactile information and serves as the foundation for many downstream tasks. However, achieving real-time and precise reconstruction of 3-D motion field is challenging. In this study, the 3-D motion field is decomposed into depth and 2-D motion field, and a multitask learning model is employed to simultaneously estimate both. The approach excels in accuracy and real-time performance (9.20 ms/frame). For depth reconstruction, the model achieves an average mean absolute error (MAE) of 0.062 mm. For 2-D motion field tracking, an effective structured marker tracking algorithm (SMTA) is introduced, and a marker tracking network (MaTnet) is constructed based on it. This network features dynamic sensing fields and a distance field (DF) auxiliary task, offering strong generalization, interpretability, and ease of training. It exhibits excellent tracking performance in various deformations, with an average error of 1.044 pixels (about 0.03 mm). Finally, the strong transferability of the multitask model is demonstrated, with a maximum decrease in depth reconstruction accuracy of 0.018 mm.