Feature Selection in Big Data using Filter Based Techniques

Today organizations are capturing and processing a very large amount of data that is informative which is further continuing to increase in an exponential manner. The analysis of data of this size to make a decision in decision making system often goes inaccurate and it computationally overburdens the system. The Hadoop is a working model which is based on a framework known as the Map-Reduce which efficiently process big data and reduce computation complexity. The methods of feature selection primarily aim at the elimination of irrelevant, redundant and noisy features that could otherwise bring down the performance of classification. But most of the traditional methods lack the level of scalability that is needed to cope with the various instances and bring about some successful results within a limited time. The work brings about feature selection that is based on the Minimal Redundancy Maximal Relevance (MRMR) along with the Information Gain (IG) which makes use of the paradigm known as the MapReduce for obtaining the feature subsets from all big datasets. The methods of feature selection were evaluated by making use of the classifiers such as the Naïve Bayes and the K-Nearest Neighbour (KNN). The results of the research show that this proposed method has achieved a good demonstration compared to the other methods.