Web crawlers: To detect security holes

Today, the web is all about the dynamic content; the information created whilst it is needed i.e. the resources are not readily available to the users. Then how it is possible that a web crawler finds a resource that is either protected by a session or hidden behind an authentication form? The query triggered to look-for the answers to the questions on web crawlers which are; what is a crawler? Why it's needed? How it works? Implementation of a typical crawler, How crawlers are categorized? For a comprehensive study on the existing web crawlers their comparative analysis on the basis of different attributes to find problems, from crawler and application perspective, research was conducted. Moreover, after discerning the grounds, architecture of a prototype that ensures and inspects the security of online assets in order to maintain the information security by finding where are the security holes that allow a web crawler to intrude in and fetch the location of a specific resource from a hidden database? Or what features are included in the crawlers that make them good enough to find hidden resources? This paper is an attempt to find out the answer to both questions in either way, to cope up with the security issues and vulnerabilities of online resources.