Transformer-Based Multi-modal Fusion for Automatic Recognition and Learning of Craniomaxillofacial Suturing Skills

Efficient and accurate surgical skills are the key to ensuring that patients receive good treatment results. Integrating the operating skills of expert surgeons into robots to achieve intelligent and autonomous operation has become a research hotspot. Accurate recognition and learning of expert surgeons’ operating behaviors are the basis for achieving this goal. Among them, microsurgical suturing, as a basic operation in many surgeries, is particularly important in craniomaxillofacial surgery. At present, most studies focus on using robotic surgical images or force information for skills recognition and learning. However, single-modality data is difficult to comprehensively and objectively analyze complex operating behaviors qualitatively and quantitatively. To overcome this limitation, this study designed a surgeon suturing skills data acquisition platform based on multi-sensor technology, and proposed a multimodal data fusion algorithm (Swin-CMT) combining Swin Transformer and Cross-Modal Transformer. The fusion network overcomes the influence of the dimensionality curse on early fusion and the neglect of the complementarity between multimodal streams in late fusion. Experimental results show that compared with other common algorithms, the proposed Swin-CMT performs well in recognizing key steps, with an accuracy of 94.59%, which is significantly better than the results based on single-modal data, providing a scientific basis for the learning and objective evaluation of surgical skills. The results of this study will also provide a promising methodological and technical foundation for automated skills learning and robotic autonomous suturing.