Application of Preprocessing Techniques on the Dataset obtained from Education Sector to Predict Students’ Performance

Preprocessing technique in Data Mining process is used to transform the raw data into the valid data to increase the consistency of information which improves the model’s prediction accuracy. In this research article, five preprocessing techniques such as 1. Finding attributes of dataset with high percentage of missing values, 2. Finding attributes of dataset with a single unique value, 3. Finding highly correlated features using Pearson Correlation Coefficient method, 4. Finding features with zero importance features using Gradient Boosting Machine algorithm and 5. Finding features with low importance, are applied on the datasets collected from education sector. These preprocessing techniques are implemented on the dataset using Python programming language packages such pandas, numpy, matplotlib, seaborn, etc. For this dataset, we collected the data of students from university site related to four years Bachelor of Engineering – Computer Science and Engineering result analysis ledgers and prepared with the help of Microsoft Excel features and Mathematical functions. Initially the prepared dataset consists of 960 attributes and after applying these five preprocessing techniques, the final dataset contains 431 attributes after removing 529 attributes from original dataset. Sample Python code is also given in this research article.