Unified Parsing Script using Machine Learning

Gathering noiseless data is an important phase in fields such as machine learning and target advertising and is also one of the most tedious processes. The evolution of the Internet has made it the largest source of data, in every domain. Web scraping can be defined as the process of extracting and parsing information from unstructured web pages. Advancement in web scraping has provided us with tools such as BeautifulSoup, lxml, and Selenium, which precisely extract information from webpages based on the knowledge of the HTML document to be parsed. Nevertheless, one major inconvenience associated with such primitive web scrapers is that they are site-dependent. They have to be adapted for each new website. In this paper, we propose a machine learning based approach that searches for patterns in the semantics of the webpages of a single domain, capable of acting as a site-independent web scraper. The strategy proposed involves an algorithm that utilizes the results of three independent machine learning models to extract key information from web pages belonging to a specific domain. The experimental analysis of the simulation demonstrates that the solution proposed enhances the process of web scraping by enabling a single web scraper to extract data accurately from multiple websites, thus improving productivity.