Reimagining Enterprise Data Management using Generative Artificial Intelligence

Enterprise Data Management (EDM) is a comprehensive approach encompassing data acquisition, profiling, standardization, quality assurance, and transformation, along with governance, to optimize the lifecycle of an organization’s data assets and facilitate meaningful analysis. The recent rise of Large Language Models (LLMs) and Generative Artificial Intelligence has fundamentally transformed data-related tasks. In this study, we delve into the integration of LLMs and Generative AI within EDM, proposing a collaborative approach that incorporates human input for computationally intensive processes. Our proposed pipeline strategically utilizes advanced Large Language Models to tackle challenges at each stage of the Enterprise Data Management process, significantly enhancing overall efficiency. Through the implementation of a case study in the pharmaceutical domain, we replicate conventional steps taken before creating business reports to enhance sales strategies, demonstrating that the proposed pipeline significantly reduces the time needed for the entire EDM process. Additionally, a comparison between various Open Source LLM models and OpenAI GPT-3.5 reveals GPT-3.5’s superior performance in code generation tasks, especially in scenarios requiring complex query generation. The proposed pipeline demonstrated satisfactory performance across various stages, including tasks such as source-totarget identification and primary key identification, as well as in other phases. The study concludes that the proposed pipeline, utilizing LLMs and Generative AI models, holds promising potential for optimizing and transforming data management workflows, providing efficient and effective solutions for organizations.