Memory Efficient Neural Speech Synthesis Based on FastSpeech2 Using Attention Free Transformer

In this paper, an attention-free transformer is sug-gested for neural text-to-speech synthesis based on FastSpeech 2 which significantly reduces the memory footprint of the attention layers. FastSpeech 2 is a non-autoregressive transformer-based acoustic model which has been shown to be very effective in producing high quality synthetic speech. However, because of the type of attention mechanism used, the necessary memory footprint of a standard FastSpeech 2 is very high which is an issue in applications with memory constraints. The memory reductions using an attention-free transformer is proportional to the utterance duration. Compared to standard FastSpeech 2, for a 6 and 8 seconds utterances, the memory reduction using an attention-free transformer is over 30% and 45%, respectively, while, there is no significant quality regression. Moreover, the use of the suggested attention-free transformer in FastSpeech 2 results in more efficient training (-20%) and inference (-50%) time.