A Dense Retrieval Model Training Method Combining Matryoshka Representation Learning and Knowledge Distillation

Recently, Retrieval-Augmented Generation (RAG) driven by Large Language Model (LLM) has gained significant attention. RAG-based systems retrieve information from a large document corpus, which is then used by LLM to generate more accurate text. However, LLM often require substantial computational resources, which can limit the size and performance of the retrieval models. Additionally, traditional models produce fixed-dimensional vectors that may not adapt well to downstream tasks. To address these challenges, this paper proposes a dense retrieval model training method that combines Matryoshka Representation Learning (MRL) and knowledge distillation, termed Multi-Dimensional Representation Distillation (MDRD). This method allows the model to learn representations layer by layer from high to low dimensions, and transfers the knowledge from the high-dimensional model to the low-dimensional one. MDRD enhances model performance and offers flexible dimension selection without incurring additional costs during inference and deployment. Additionally, a sample filtering algorithm is introduced to minimize the effect of false negatives, leading to better semantic representation. In experiments on the T2Ranking and DuReaderretrieval datasets, MDRD shows performance improvements of 2.05% and 0.62% in MRR@10, 0.71% and 0.08% in R@1, respectively. These improvements demonstrate that the proposed method is effective in enhancing retrieval performance.