Distributed information gain theoretic feature selector using spark

From past ten years, the area of Feature Selection is favorite among researchers because of the increasing size of the datasets. Several existing feature selection algorithms do not scale well with the large datasets. Sophisticated feature selection tools also do not work well with large datasets on low configuration machine. This work implements feature selection using spark cluster computing to have a scalable solution. In the proposed algorithm, large scale dataset is partitioned vertically into several small datasets. Spark distributes it in form of RDD (Resilient Distributed Dataset) among the machines in the cluster. The information gain based filter is applied to RDDs on each machine to evaluate ranks of features on multiple machines in parallel. The sub-results are collected and combined on master machine and attributes with high information gain are selected as relevant attributes. The program is run on different number of cores with different dataset sizes having different number of attributes and samples. Proposed approach shows good scalability and effective reduction in overall execution times with higher number of cores.