Search Procedures Optimization in Digital Twin Data Lakes

The development of Data Lake technology is observed in many areas of smart manufacturing. Data lakes are used to create digital twins in the fuel and energy complex, energy, and metallurgy. The processes of obtaining new materials are no exception in this development direction of science and technology. In the context of the rapid growth of new data and the accumulation of chemical information obtained in the form of digital models of chemical compounds and materials, new information solutions and tools are in great demand, allowing for the search, analysis and visualization of available diverse data. The research is devoted to the development of methods for organizing known information about the structure and properties of compounds and materials, as well as software components for automating and increasing the productivity of search procedures. The issues of organizing the storage and processing of chemical data are considered, ways to improve the performance of search queries are shown, as well as displaying modifications of file structures in a relational database and in a distributed database based on Hadoop tools. File analysis and parsing highlights various properties of crystalline and molecular structures, organizing them into tree structures (JSON format), convenient for indexing and searching information in Data Lake of the “column family” type, which is a sparse matrix of search attributes. The results of accelerating search procedures for some search examples are presented, showing a significant acceleration in obtaining search query results.