Comparative Analysis of Different Vectorizing Techniques for Document Similarity using Cosine Similarity

In this paper, multiple methods to vectorize documents were compared, and cosine similarities were calculated for the corresponding documents. Some of the vectorizing methods also consider the text's semantic meaning. The methods involve cosine similarity with algorithms like Bag of Words, Binary Bag of Words, Tf-Idf, Bidirectional Encoder Representations from Transformers, and Universal Sentence Encoder. Two important libraries to preprocess the text were used; these are NLTK and Genism. The Binary bag of words with Genism gave the best results of all the methods used. The dataset used involved around 2000 short news articles; these belonged to 5 categories.