Enhancing Web Page Classification Through a Semantic-Aware and Efficient Focused Crawling Methodology

Focused crawlers are designed to search the web for pages on specific topics, aiming to gather relevant, preprocessed pages while ignoring unrelated ones. However, traditional focused crawlers often struggle with accurately categorizing web pages across multiple topics. The challenge stems from the vast amount of unstructured data found on web pages, making it difficult to classify them based on a particular subject. This study seeks to improve focused crawling by leveraging web page classification techniques. We propose a text classification model that combines the GloVe word vector model with the TF-IDF weighting method to enhance classification accuracy. This GloVe-based model is then used to help focused crawlers categorize web pages more effectively. We validated the proposed approach on 10 datasets, comparing its performance with traditional machine learning algorithms and other methods like Naive Bayes, Bag-of-Words, and Word2Vec. The results show that our model outperforms traditional algorithms by $7-12 \%$.