Task Vector Algebra for ASR Models

Vector representations of text and speech signals such as word2vec and wav2vec are used commonly in automatic speech recognition (ASR) and spoken language understanding systems. Recent results in natural language processing have proposed a task vector, defined as the difference vector between a model’s converged parameters and its initial parameters. Task vector algebra provides a simple and computationally-efficient way to solve several modeling problems, including model editing to reduce undesirable behavior, multi-tasking, and improving domain generalization. We apply task vectors to ASR models for the first time. Our experiments with Conformer-RNNT models trained on the SpeechStew corpora show that task vectors retain their scaling and multi-tasking applications. We propose two novel applications of task vectors to ASR. First, we show that task vectors can perform zero-shot adaptation of ASR models to unseen domains without using any supervised data. Second, we present a novel "task analogy" formulation that enables us to use models trained on high-resource tasks to improve performance on low-resource tasks. We also explore a technique to improve the performance of task vector arithmetic for ASR models.