Web Scraping Approaches and their Performance on Modern Websites

When it comes to information, the internet is a gold mine. Whether you need data for your business, school, or personal use, you may uncover a wealth of information by performing an internet search. Web Scraping (WS) is a computerized method of obtaining big amounts of information from internet sites. The bulk of this information is in the form of unstructured HTML that would be transformed to structured data in a digital document before use in diverse applications. Although web scraping is not illegal, it might go against the ethics. This paper contains research-based findings of different methods of web scraping techniques used to extract data from websites. The various approaches used in this paper to obtain the results include requests library, selenium and other external libraries. Using the results obtained the findings are visualized through different graphs having contrasting parameters as measurements. Results obtained from our paper show that there is a lot of improvement needed to protect websites from web scraping bots.