Sequential2.0: Self-supervised Speech Translation Based on FAT Model and Replaced Token Detection

This paper proposes a self-supervised speech translation model based on the FAT model for knowledge distillation, which is mainly applied to bilingual business scenarios such as cross-border goods and services marketing between China and the United States. Compared with the original model, our model reduces the inference time by 16.44% and the parameter size by 12.17%, while improving the BLEU score by 12.48%. The reduced parameter size is more conducive to the deployment of the model on edge devices such as embedded systems and mobile phones. Experimental results show that using the replaced token detection algorithm is better than traditional masked language modeling, and combining the BPE algorithm and PCA algorithm for teacher model hidden state encoding helps improve translation performance. Although our model achieves better results in the Chinese-English bilingual scenario, larger models such as Whisper perform better in multilingual scenarios. Future research will focus on the knowledge distillation and optimization of large language models in multilingual scenarios, and we look forward to more participation and discussion from researchers in this field.