MAVIDSQL: A Model-Agnostic Visualization for Interpretation and Diagnosis of Text-to-SQL Tasks

Significant advancements in semantic parsing for text-to-SQL tasks have been achieved through the employment of neural network models, such as LSTM, BERT, and T5. The exceptional performance of large language models, like ChatGPT, has been demonstrated in recent research, even in zero-shot scenarios. However, the inherent transparency of text-to-SQL models presents them as black boxes, concealing their inner workings from both developers and users, which complicates the diagnosis of potential error patterns. Despite the fact that numerous visual analysis studies have been conducted in natural language processing communities, scant attention has been paid to addressing the challenges of semantic parsing, specifically in text-to-SQL tasks. This limitation hinders the development of effective tools for model optimization and evaluation. This paper presents an interactive visual analysis tool, MAVIDSQL, to assist model developers and users in understanding and diagnosing text-to-SQL tasks. The system comprises three modules: the Model Manager, the Feature Extractor, and the Visualization Interface, which adopt a model-agnostic approach to diagnose potential errors and infer model decisions by analyzing input-output data, facilitating interactive visual analysis to identify error patterns and assess model performance. Two case studies and interviews with domain experts demonstrate the effectiveness of MAVIDSQL in facilitating the understanding of text-to-SQL Tasks and identifying potential errors.