DataLakeIO: A Connector Between Apache Beam and Data Lake

In the era of big data, Apache Beam is widely used because it provides a unified programming model and supports multiple data processing engines. At the same time, data lake is becoming more and more widely used. However, Apache Beam is not currently interconnected with the data lake. In this paper, we proposed DataLakeIO, a connector between Apache Beam and data lake. First, a Apache Beam pipeline is initialized. Then, Apache Beam ingests the data from the data source, converts it to Row type, and put it into pipeline. Then, Apache Beam passes the data and parameters to the write method of DataLakeIO. In this method, a Spark session is created based on these parameters, the data is converted into a Dataset, and finally stored in data lake. The process for Apache Beam to ingest data from data lake into its pipeline is similar. This connector supports the interconnection of Apache Beam with Delta Lake, Apache Hudi, and Apache Iceberg, which shows the extensibility for data lakes. The experimental result shows the efficiency of DataLakeIO.