Towards an Open NLI LLM-based System for KGs: A Case Study of Wikidata

The rise of large language models (LLMs) has significantly advanced information retrieval, yet challenges like the limitation of knowledge updating ability, lack of openness, and hallucination issues persist. To address these, Retrieval-Augmented Generation (RAG) has been introduced but remains limited in interpretability due to its reliance on vector-based representations. This paper presents a question-answering (QA) system using GraphRAG, a RAG system with knowledge graphs (KGs) as its base. We develop a natural language interface (NLI) for QA over Wikidata, a popular, open, and crowdsourced KG. Our approach employs LLM chaining, i.e., a paradigm that leverages multiple LLM calls sequentially, to generate SPARQL queries, with the aim of creating an open system that ensures transparency and allows direct inspection of its components. Utilizing an experimental research approach, we evaluated the generated SPARQL queries and found that incorporating a broader set of property candidates into the prompts significantly boosts performance, achieving a Jaccard similarity score of 0.7806. These findings demonstrate the system’s effectiveness in SPARQL query generation, highlighting its potential for further development. However, we consider the limitation of the LLM’s context window and the hallucination phenomenon as the major challenges that limit the system’s performance.