Cross Modality Knowledge Distillation Between A-Mode Ultrasound and Surface Electromyography

Surface electromyography (sEMG) and A-mode ultrasound (AUS) are two widely employed sensing modalities to detect muscle activities. By comparison, the AUS modality shows the characteristics of higher decoding accuracy than the sEMG modality. However, AUS is far less reliable than sEMG in actual long-term use. To resolve this contradiction, we considered leveraging AUS as a teacher to supervise sEMG training better and learning an augmented sEMG representation. First, a novel network architecture multibranch network with a diverse focus (MINDS) was proposed for gesture recognition, which was suitable for both sEMG and AUS modalities. Second, a cross modality knowledge distillation (CMKD) framework was proposed, to transfer the latent knowledge of AUS to sEMG through Kullback–Leibler divergence (KLD) loss. The gesture recognition accuracies were compared between MINDS and the existing networks. The experimental results demonstrated that MINDS outperforms other networks under both sEMG and AUS modalities. Furthermore, the feasibility of the CMKD framework was evaluated on the proposed MINDS and other existing networks. The results revealed that with knowledge distillation from AUS, the accuracy of the sEMG modality obtained a significant improvement, regardless of the employed network architecture. This work confirms the superiority of the proposed MINDS network and the feasibility of the proposed CMKD framework.