AI-Assisted SQL Authoring at Industry Scale

SqlCompose brings generative AI into the data analytics domain. SQL is declarative, has formal table schemas, and is often written in a non-linear manner. We address each of these challenges and develop a set of models that shows the importance of each problem. We first develop an internal SQL benchmark to perform offline tests at Meta. We evaluate how well the Public Llama model performs. We attain a BLEU score of 53% and 24% for single- and multi-line predictions, respectively. This performance is consistent with prior works on imperative languages. We then fine-tune Llama on our internal data and database schemas. This model, SC-Schema, substantially outperforms Llama by 16 percentage points on BLEU score. SQL is often written with multiple sub queries and in a non-sequential manner. We build on our schema aware model to develop SqlCompose fill-in-the-middle (SC-FIM) which is also aware of the context before and after the line(s) that need to be generated. It outperforms SC-Schema by 35 percentage points. We also measure how often the models get the correct table names, and SC- FIM is able to do this 75% of the time, a major improvement over the other two models. Aside from our scientific research, we also roll out the SC- FIM version of SqlCompose at Meta. It is used on a weekly basis by over 10k users including data scientists and software engineers, less than 1 % of users have disabled SqlCompose. We use the feedback from users to improve SqlCompose. Interesting positive themes include completing tedious or repetitive SQL clauses, suggesting boilerplate coding, and eliminating the need to remember difficult SQL syntax. The most significant negative themes were table and column name hallucinations, which has been reduced with the release of SC-FIM. The SqlCompose models consistently outperform public and internal LLMs despite their smaller size (7 bn and 13 bn), which provides early indications that smaller specialist models can outperform larger general purpose models.