Comparing Accuracy and Consistency: LLMs vs. SOTA Deep Learning Models in Text-to-SQL

Text-to-SQL is one of the most prominent areas where Large Language Models (LLMs) have seen extensive application. While Text-to-SQL systems using LLMs and pre-trained deep learning models are evaluated using standard metrics such as Exact Match (EM), Execution Accuracy (EX), and Valid Efficiency Score (VES), they overlook a critical aspect: consistency. This paper introduces a Consistency metric that evaluates a model’s reliability in generating the same SQL query for repeated inputs. In the study, we performed a comparative analysis of the pre-trained deep learning models and large language models (LLMs) using the Spider dataset. The study highlights that the pre-trained deep learning models tend to be more accurate, efficient, and consistent compared to LLMs. LLMs are consistently incorrect, and consistency alone does not guarantee correctness. As the difficulty of SQL queries increases, LLMs struggle more with accuracy, while the pre-trained deep learning models remain more robust. The paper’s findings suggest that consistency, combined with existing metrics, is essential for improving Text-to-SQL systems.