Data Discovery as a Service for Data Lake

Data Lake (DL) is a low-cost big-data architecture designed to centralize an organization’s data, enabling the extraction of valuable insights and economic benefits. To effectively manage, describe, and discover these data assets, data catalog tools are essential. These tools utilize metadata to inventory data assets, allowing end-users to search, query, and locate datasets within the Data Lake. Existing data catalogs do not provide the necessary context or descriptive statistics of datasets, hindering the efficient and insightful utilization of data. Furthermore, it struggles to categorize data based on context or domain during ingestion and lack the necessary automation and scalability to support modern data stacks. This work proposes integrating Data Discovery as a Service(DDaaS) into Data Lake. The Data Discovery service, developed using Plotly-Dash, is an interactive web dashboard integrated into a data lake. DDaaS assists end-users in discovering structured datasets based on domain. Service provides descriptive statistics of data using interactive visualization and pre-computed statistical views to describe data. DDaaS reduces latency in locating related datasets and describes trends, patterns, and insights. To validate the proposed Data Discovery as a Service (DDaaS) features, a comparative study is conducted, corraborating its attributes with those of existing data catalog systems. This study aims to demonstrate DDaaS’s superiority in context-based categorization and the provision of descriptive statistics. Performance validation is executed by measuring the resource consumption of each service component, including metrics such as CPU usage, memory consumption, to ensure operational efficiency. Additionally, the efficacy of DDaaS is assessed by evaluating the latency of its descriptive statistics methods across datasets from diverse domains and varying record sizes. This evaluation ensures that DDaaS can deliver timely and accurate statistical descriptions, confirming its robustness and effectiveness in handling datasets of different natures and scales.