End-to-End Speech Translation with Self-Contained Vocabulary Manipulation

In machine translation, vocabulary manipulation is a way to reduce the target vocabulary based on the source sentence and the word dictionary, which is effective to lower latency during inference for text translation in industrial application. But vocabulary manipulation is hard to apply to the end-to-end speech-text translation, because neither source text nor speech-to-target mapping is available. We introduce a method that avoids this dependence. Through learning the projection between sentence-level speech encoder output and final target vocabulary, the proposed method allows self-contained vocabulary manipulation without knowing source speech transcripts or dictionaries. Experimental results show that the proposed method speed up about 20% while keep the comparable translation quality.