Identifying Question Intent Similarity Using Computational Techniques

Paraphrase identification is becoming an important use case in the industry. From a business context, paraphrase identification is significantly helpful to improve the user experience of the platform. It is an active research area by companies like Quora, Stack Overflow, and other question-answering websites. One of the main issues that these companies face is the problem of repeating questions. If one is successfully able to capture and interpret the semantics of the sentence and the intent of the user, this would improve the consumer experience. Quora’s official dataset was used in the present research work. To find a question that was similar, we modeled the dataset of Quora question pairs. It was a Binary classification task and various approaches were explored and experimented with. Bag of Words was used for feature extraction. A traditional machine learning approach was followed and algorithms like XGBoost, RF, and decision tree were used. These models were evaluated based on their F-1 and accuracy scores. Feature engineering was performed that significantly improved the model accuracy. The decision tree model was able to achieve an accuracy of up to 78.8% while Random Forest and XGBoost had an accuracy of 76.2% and 75.4%.