An Algorithm of Detection Duplicate Information Based on Segment

It's a hot issue to detect and eliminate approximately duplicate records in data cleansing. Aiming at recognizing duplicate records of multi-language data, the segment strategy based on character features is proposed, and the algorithm of edit distance with variable weight is presented. The experiment results indicate that the segment time is small change along with data scale growing. It means the total running time of detecting duplicate records is not influenced by segment in the big data scale. The experiment results also indicate that the algorithm running efficiency and detect precision can be improved.