Considerations on the functions and importance of a web crawler

There are hundreds of millions of searches performed through search engines daily. The increasing importance of search engines represents the end of the traditional outbound marketing strategy, now replaced with the reorientation of companies to inbound marketing strategies, presently a part of human existence. In the present electronic age the consumer has real time information available on all products, whether this information is obtained from search engines from specialized web pages, forums or other sources. This makes search engines essential, alongside with the website - containing the desired information on a favorable position with respect to other web sites - positioning. The paper presents a set of considerations regarding the state of the art and future of web crawling.