Unify: An Unstructured Data Analytics System

Despite unstructured data constituting over 80% of the information available today, no specialized analytics system currently exists to process this type of data. The lack of a predefined schema in unstructured data renders traditional structured query languages, such as SQL, unsuitable for unstructured data analytics. A seemingly straightforward solution is to utilize natural language for crafting queries, thereby making analytics more accessible to users without technical expertise. However, understanding and executing queries posed in natural language presents significant challenges. A brute-force approach involves relying on users to manually derive solutions, tapping into their knowledge and experience. However, this method of generating query plans by human intervention is neither scalable nor efficient. Therefore, a pertinent question arises: how can we automate unstructured data analytics? To address these challenges, this paper introduces Unify, an innovative system leveraging the capabilities of large language models (LLMs) to automatically generate, optimize, and execute query plans for unstructured data analytics, where queries are articulated in natural language. Unify initializes by defining common operators used in unstructured data analytics and creates both pre-programmed implementations and LLM-based implementations for physically executing these operators. It then guides LLMs to devise logical plans by methodically deconstructing queries into smaller steps, ensuring accurate logic by aligning with suitable operators. For translating a logical plan into an optimal physical plan, we further introduce a technique for physical plan optimization that employs a semantic cost model alongside semantic cardinality estimation. Comprehensive tests conducted on real-world datasets demonstrate that Unify can expedite query processing by up to $40 \times$, while preserving high accuracy, thus positioning Unify as an effective tool for largescale unstructured data analytics.