Text Duplicated-checking Algorithm Implementation Based on Natural Language Semantic Analysis

Natural language is a tool and way to express the interaction between computer and human (NATURAL) language. At present, the analysis ability of text semantics is relatively mature. The clustering algorithm, probability graph model algorithm, text mining algorithm and other processing methods have been successfully implemented. This paper is based on natural language processing text, combined with word2vec word vector conversion technology, through similarity calculation, using its semantic analysis ability to construct an optimized LDA model which refers to an importance sampling idea to extract topic words and use cosine similarity to calculate the repetition rate. We can achieve an ideal semantic text duplicate analysis effect by comparing it with the results of the LDA model before and after optimization.