Universal Data Model as a Way to Build Multi-Paradigm Data Lakes

The paper focuses on data lakes building that combine all data from different models stored and processed across the enterprise in both OLAP and OLTP modes. As a test of this idea, an experiment was conducted to evaluate the performance of a multi-paradigm data lake built on a single SparkSQL-HDFS platform against 4 specialized DBMSs that contained the same data. The experiment showed that such a solution is possible, but further experimentation on a larger amount of data and using unstructured data is needed to confirm. As a further development of the idea of creating multi-paradigm data lakes on a single technological platform, the article proposes the concept of a universal data model. It is based on the archigraph structure supporting graph, tabular and multidimensional data representation, text documents, and Search Engine search index. Unlike the first multi-paradigm data lake experiment, the second one develops a metagraph DBMS as a unified technology platform. The architecture of the data lake management system built on the universal model basis is developed with its use, and the variant of representation of the archigraph data describing the lake structure in the metagraph DBMS is given. The article is furthered by, the paper briefly describes an ongoing project to build a data lake management system using a universal data model, and a planned experiment to evaluate the performance and scalability of the implemented data lake construction method.