PyBot: An Algorithm for Web Crawling

PyBot is a Web Crawler developed in Python to crawl the Web using Breadth First Search (BFS). The success of the World Wide Web (WWW), which itself built on the open internet, has changed the way how human share and exchange information and ideas. With the explosive growth of the dynamic Web, users have to spend much time just to retrieve a small portion of the information from the Web. The birth of the search engines have made human lives easier by simply taking them to the resources they want. Web crawler is a program used by search engines to retrieve information from the World Wide Web in an automated manner.