Scraping Google Scholar Data Using Cloud Computing Techniques

The current revolution in Internet technologies has led to a tremendous amount of data on the world wide web (www). Dealing with such big data is not an easy task in terms of collection and analysis. Moreover, the unstructured nature of most of this data makes it more difficult to collect and analyze. Therefore, scraping and crawling techniques are considered the most common types of data collection from the web. In this paper, a scraper is developed based on two methods to scrape data from one of the most complicated platforms on the web, which is Google scholar. These two methods are Beautiful Soup and SERP API. The proposed scraper is divided into three parts: the web scraper finds the needed links on the internet, the data is extracted from the source links, and lastly the data is stored in a CSV file. For implementing the scraper, Python language is used. The results showed that the designed scraper is efficient in retrieving data from Google scholar. Also, it was found that Python is a suitable and efficient language for scraping necessary data from a particular website of interest due to the abundance of library resources.