From Manual to Machine: How AI is Redefining Web Scraping for Superior Efficiency: A Literature Review

Web Scraping involves the use of bots for the purpose of extracting data from the online web. To extract such data, the web scraper must conduct at least 3 different steps, i.e., collect the necessary links from the web, extract data using the provided links, and save the information in a neatly organized CSV file or database. However, traditional methods to accomplish these steps have faced many difficulties, such as anti-scraping measures and website changes. To tackle these difficulties and to accomplish the steps in a more rapid yet advanced way, Artificial Intelligence (AI) can be utilized. As of mid-2024, the rapid development and improvement of AI has resulted in its integration into many businesses and educational institutions. The current state of AI is remarkably advanced that it can be integrated in about any field and sector available. In this paper, the researcher defined the term Web Scraping, Artificial Intelligence, Natural Language Processing, and Machine Learning, explained the use of AI in Web Scraping including its importance and potential, and the understanding of ethical web scraping practices. This literature review indicated that the integrated use of AI improves the performance of Web Scraping compared to traditional methods.