Performance optimization of Learning Trees modules in the ECL repository

Multi-Node computation, also known as distributed computing, is a paradigm that allows for the efficient utilization of multiple interconnected nodes or machines to perform complex computational tasks. By dividing the workload and distributing it across multiple nodes, multi-node computation enables parallel processing, scalability, and improved performance for a wide range of applications. HPCC Systems® from LexisNexis Risk Solutions is a proven, open source solution for big data insights that can be implemented by businesses of all sizes. HPCC Systems offers a consistent data-centric programming language, two processing platforms and a single, complete end-to-end architecture for efficient processing. ECL is the Enterprise Control Language designed specifically for huge data projects using the HPCC Systems platform. It is extremely scalable and a powerful declarative data-processing language with native multi-processing capabilities and used for HPCC systems. Some core ECL functions, like the ones handling Learning Tree algorithms of Machine Learning Bundle are recursive in nature and hence high computational time is needed. This paper thus presents a solution to achieve optimized processing of ML learning tree algorithms in ECL through embedding python libraries. The proposed approach was tested on standard datasets and a significant decrease in computation time. For Real Estate Valuation Regression dataset the computation time was reduced from 11.371 seconds to 0.886 seconds using the proposed method showing 1324% improvement with no impact on the accuracy. Random Forests are also implemented using the proposed Decision Tree method allowing the users to specify the number of decision trees to be used in the forest.