RuSimulBench a Benchmark for Assessing the Quality of Language Models in Russian

The rapid advancement of Large Language Models (LLMs) has significantly improved natural language processing capabilities across multiple languages. However, evaluating their performance in languages with limited benchmarking resources, such as Russian, remains a challenge. In this work, we introduce a novel benchmarking framework designed to assess Russian LLMs along two critical dimensions: creativity and stability. Creativity is essential for generating diverse, original, and contextually appropriate responses, while stability ensures that models provide consistent outputs when faced with slight prompt variations.Our benchmark makes use of a unique and comprehensive approach, combining automated and human evaluated systems. We propose the stability coefficient for measuring the value of novel model’s prompts rewording and demonstrate the creative score, which claims value based on originality, diversity and coherence in reactions of silenced models. For thoroughness, we include several LLM architectures that were adapted to Russian language to be assessed against structured test cases derived from the MERA benchmark.The experimental results provide deep insights into the trade-offs between stability and creativity in Russian language models, highlighting strengths and areas for improvement in existing architectures. By offering a standardized evaluation approach, our benchmark contributes to the development of more reliable and effective Russian-language AI systems. Additionally, our findings can inform future research on enhancing LLM adaptability and robustness in low-resource languages.