Spatial Data Discovery Using Small Language Model

The ubiquitous use of data in a plethora of use cases and different organizational dependence on it for various insight has made data a valuable resource across business and scientific domains. Data has attained much more significance than oil itself, unlike the British mathematician Clive Humby’s quote in 2006 “data is the new oil”. Every business and processes generate data and with the advent of AI and machine learning, all types of data have valuable use case attached. This also ensued a series of initiatives to clean, augment and provide data as data-products. This has given rise to businesses and solutions using inhouse data or data purchased from a dedicated data provider. Data consumers often face the challenge of inefficient data product discovery. More frequently than not they require active support from data product experts to point them to right dataset. The semantic meaning of the attributes and keywords of a data product is also heavily domain oriented. Data providers find it easy to update and augment metadata of a data set with details more feasible solution than modifying the dataset itself with attribute rename or adding composite attribute. The expected next step is that the metadata be self-explanatory enough to help in data cataloguing, but often we find consuming data scientist looking for help from data experts to make sense of the data in the context of the consumer’s use case. This research seeks to analyse if the role of a data expert can be reduced and a finetuned small language model can help a data consuming data scientist in discovering the right data product, based on the metadata of the data product. Overall, this research contributes to the advancement of data discovery techniques and highlights the potential of leveraging language models to streamline and democratize access to valuable data resources, thereby empowering organizations to make more informed and data-driven decisions.