Monte Carlo Self-Training for Speech Recognition

Self-training in the teacher-student framework generally suffers from the confirmation bias problem, where errors from the teacher are propagated to the student and hence get amplified with multiple iterations. In this paper, we present Monte Carlo Self-training where pseudo labels are generated by sampling from a teacher distribution, as a way to mitigate this problem. We show that Monte Carlo Self-training is an approximation to minimizing label sequence level cross entropy between student and teacher. In our experiments we find that Monte Carlo Self-training always outperforms beam decoder based self-training and is quite robust even when the initial teacher WER is high. We also show that label sampling allows formulating pseudo label confidence in a more natural way and we show that these confidence measures give further improvements in our unsupervised adaptation experiments especially when the initial teacher WER is very high.