Inferring Relationships between Tabular Data and Topics using LLM for a Dataset Search Task

In the big data era, data is called new oil and essential things in both business and academic fields. Both private and public data are continuously increasing and it is expected that such data are used for innovation and business improvement. However, finding datasets for specific purposes becomes a new challenge with the increase in the variety of data. To support the finding data task, there are some studies for dataset search; however, they are immature because of the lack of appropriate evaluation datasets, which consist of sets of queries, datasets, and labels, such as other machine learning areas. One of the major obstacles is that annotating the labels to a pair of queries and datasets requires specialized expertise and a substantial investment of time and effort. In this study, we aim to automate the data annotation process using a large language model (LLM) to reduce the manual effort required for creating annotated datasets for dataset search tasks. We propose an annotation framework that consists of LLM annotation with table compression and multiple results aggregation mechanism. We evaluated the proposed framework by human annotated datasets previously published for the dataset search challenge. The evaluation results show that the proposed framework outperformed the baseline method, which input simply a specific number of head rows to LLM. Moreover, our qualitative analysis gives insights into improving LLM-based systems and replacing human annotators.