Medical Document Embedding Enhancement with Heterogeneous Mixture-of-Experts

Retrieval-Augmented Generation (RAG) has emerged as a crucial technique to enhance the accuracy and reliability of large language models, particularly in specialized domains like medicine. However, the effectiveness of RAG heavily depends on the quality of text embeddings used for retrieval. In this paper, we introduce Med-MoE-Embed, a novel approach to improve medical text embeddings tasks. Med-MoE-Embed leverages a pretrained embedding backbone augmented with a trainable Mixture of Experts (MoE) network, allowing for efficient adaptation to specific medical subdomains and tasks. We design each expert to be a compact KANs or a MLP with heterogeneous activation functions such as GELU and SWIGLU. Furthermore, we propose a two-step fine-tuning process that optimizes expert training and selection, enhancing the model’s adaptability across various medical datasets. Our extensive evaluation focuses on RAG tasks in the medical domain, demonstrating significant improvements in retrieval accuracy and generation quality. Med-MoE-Embed mitigates the challenges of limited data accessibility and domain-specific requirements in the medical field, offering a versatile and efficient solution for enhancing embedding quality in medical natural language processing applications.