CORB-RAG: A Comprehensive Evaluation Benchmark for Retrieval-Augmented Generation Systems in the Chinese Telecommunications Operator Domain

Current evaluations of Retrieval-Augmented Generation (RAG) systems primarily focus on general datasets, lacking effective testing on domain-specific datasets, especially in application scenarios that requiring deep-seated domain knowledge. Additionally, there is limited research on customized retrieval strategies tailored for different domains, which directly impacts the response quality and efficiency of RAG systems. To address these issues, this paper proposes an evaluation benchmark for RAG systems in the Chinese telecommunications operator domain. Firstly, we expand the evaluation scope by not only utilizing general datasets but also constructing domainspecific datasets for telecom operators, enabling a more detailed analysis of RAG system performance and optimization potential. Secondly, in terms of generator evaluation, we go beyond existing benchmarks such as CRUD and Ralle by introducing advanced evaluation dimensions, including Contextual Relevance, Faithfulness, and Answer Relevance, providing a more comprehensive measure of content quality. Finally, we propose an innovative score-based hybrid retrieval strategy, which demonstrates superior performance in knowledge-based question-answering tasks within the telecom operator domain, significantly improving the accuracy and efficiency of information retrieval.