Octobot - Web Scarping towards retrieving Google Scholar Data

Web crawling is widely used these days in many different applications which can efficiently crawl the internet to provide the user with relevant results by utilizing its core algorithm. Google Scholar is one of the very important websites that contains a huge data about researchers and their research output along with the different citation counts. In this paper we try to utilize we crawling to retrieve the Google scholar data for Applied Science University-Bahrain by developing a searching engine tailored for this purpose. To achieve that goal, several sorts of web crawlers, and their architecture is presented briefly. Moreover, looking through numerous existing web crawlers to study their benefits and drawbacks in order to determine which criteria this web crawler should address. In addition, assess reducing online crawling techniques. Based on prior work and accepted software comparison measures, an assessment framework for comparing online crawling solutions is designed and used to aid the process. The implemented Web Crawler (OCTOBOT) is presented and different usage scenarios were shown.