Generative Reader Optimization in the RAG-System

This paper proposes the simple methodology for optimizing a generative reader subsystem as part of a RAG QA-system. Described methodology can be used as a preliminary optimization that could help building the question-answering system of the reasonable quality in a short time period. The main principle of this methodology is the usage of the OpenAI 's GPT-4 generative model as a gold reference generative reader. Our methodology describes the procedure for generation of the synthetic dataset and defines main optimization metrics. Usage of synthetic datasets makes it possible to accurately control that all changes in the QA-system make its answers closer in average to the gold reference reader. Based on the developed methodology we performed the set of basic optimization experiments to quickly find the better configuration of the reader subsystem. These experiments made it possible to significantly improve the quality of the answers of the reader subsystem. As a result, we achieved relative improvement of the semantic similarity between the answers of our reader and the reference one based on GPT-4 for almost 50% by BLEURT scale and 26% by the SAS scale. Our methodology was verified on the QA-dataset in Russian language but it's also applicable to any other language without significant modifications.