A Deep Learning Approach for Semantic Similarity Prediction Between Question Pairs Using Siamese Network and Word Embedding Techniques

Question-answering platforms, such as Quora, Red-dit, and StackOverflow, have become immensely popular in the virtual community. Among these platforms, Quora stands out as a widely utilized and resourceful repository with over 300 million monthly visitors. However, the prevalence of similar questions with paraphrased content poses challenges for users in finding relevant answers. This paper proposes a deep learning approach for predicting the semantic similarity of question pairs to address the issue of duplicate questions. The proposed model employs a Siamese network-based Long Short-Term Memory (LSTM) architecture, utilizing word embedding vectors from various algorithms such as Glo Ve, Word2Vec, and FastText. Previous research on duplicate question identification is reviewed, with our approach overcoming drawbacks such as low-level semantic connections and text type incompatibility. After pre-processing the Siamese LSTM network compares the embeddings using L1 distance to determine question pair similarity. The model is trained and evaluated on the Quora Question Pair Classification dataset, and outperformed other state-of-the-art methods with an accuracy of 84.77% with the FastText word embedding.