Adaptive technique for web page change detection using multi-threaded crawlers

World Wide Web is getting dense as many new web pages and resources are created on a daily basis. Keeping track of the changes in the web content has become an immense challenge and is a research problem with a great significance. Even the search engines require to detect changes in the web to keep search indexes up to date. Numerous researches have been carried out on optimizing the change detection algorithms. This paper presents a methodology named Multi-Threaded Crawler for Change Detection of Web (MTCCDW), which is inspired from the producer-consumer problem. The suggested change detection process mainly analyses the performances and suggests a tread-based implementation process for the optimisation of the changed detection process. The experimental results show that the proposed methodology is capable of reducing the effective time to detect changes in a web page by 93.51%.