Proposed Document Frequency technique for minimizing dataset in Web crawler

The explosive growth of webpage number on the Web has brought up some problems in the search process. One of these problems is that the general purpose search engines often return too many irrelevant results when users are searching for specific information on a given topic. Another problem is the massive increase in the number of pages to be indexed by Web search systems. In this research, two steps for Web Crawling are used to decrease these difficulties. First step is the feature selection for the datasets used. A proposed algorithm of feature selection, which uses the Document Frequency technique for the term in the category, is presented. Second step is Web page classification. Two famous techniques of Web page classification are used: (i) Support Vector Machine and (ii) Naïve Bayes Classifier. It is concluded that the proposed algorithm, using Document Frequency technique, reduces the redundancy during feature selection and increases accuracy during Web page classification. Complete evaluation is performed, in JAVA, to indicate the effectiveness of our proposed algorithm.