Research on Index Optimization Method Based on Line Quantification in Topic Network Crawler

When using a Boolean model to match words to determine topic relevance in a topic network crawler, semantic information is easily lost, and the use of the one-hot representation can lead to problems such as “dimensional disaster” and “high sparsity”. This paper proposes an indexing method based on linear quantization. On the basis of the product quantization index, for each clustered quantization index region, a linear quantization strategy is used to further map the data vector into a secondary index region, thereby further reducing the time cost of word vector matching, optimizing the storage and retrieval mechanism of word vectors, meeting the needs of large-scale topic similarity comparison, and improving crawler efficiency and accuracy.