iDUSTER: Improved Method for Removing DUST Based on Efficient Multiple Sequence Alignment Technique

The recent advancement in web applications usage and access for sharing the digital information over the Internet is increasing the network traffic and hence web crawler logs. Such web crawler logs containing the different URLs with same contents means duplicate URLs which required extra resources for analyzing duplicate URLs from URL logs. In this paper we are presenting the efficient framework for detecting such duplicate URLs and removing them from web crawler logs by using efficient multiple sequence alignment technique. Our core functionality is based on recently proposed DUSTER method in which basic method of multiple sequence alignment is used; this is one of the limitations of DUSTER method. We are presenting the Improved DUSTER method in which recently presented efficient algorithm for multiple sequence alignment is presented in order to improve the performance of removing and detecting the duplicate URLs as compared to existing DUSTER method. IDUSTER method is focusing on deriving quality rules based on efficient proposed multi-sequence alignment method. The multiple sequence method is proposed to get small and more common normalization rules set. This method is based genetic algorithm.