Research of distributed ETL engine based on MAS and data partition

In order to improve the efficiency of ETL workflow executing, this paper presents a distributed ETL engine based on MAS and data partition technology and also researches the methods of partitioning the massive data stream in both horizontal and vertical ways. The engine referred to will partition an ETL workflow which meets the conditions of being partitioned into multiple sub workflows for parallel executing. Each of the sub workflow is executed by an agent, so that multiple agents could work together to complete the collaborative work. Experimental results show that this system has good scalability and could well improve the efficiency of ETL workflow executing.