Identifying Semantically Duplicate Questions Using Data Science Approach: A Quora Case Study

Identifying semantically identical questions on, Question and Answering\nsocial media platforms like Quora is exceptionally significant to ensure that\nthe quality and the quantity of content are presented to users, based on the\nintent of the question and thus enriching overall user experience. Detecting\nduplicate questions is a challenging problem because natural language is very\nexpressive, and a unique intent can be conveyed using different words, phrases,\nand sentence structuring. Machine learning and deep learning methods are known\nto have accomplished superior results over traditional natural language\nprocessing techniques in identifying similar texts. In this paper, taking Quora\nfor our case study, we explored and applied different machine learning and deep\nlearning techniques on the task of identifying duplicate questions on Quora's\ndataset. By using feature engineering, feature importance techniques, and\nexperimenting with seven selected machine learning classifiers, we demonstrated\nthat our models outperformed previous studies on this task. Xgboost model with\ncharacter level term frequency and inverse term frequency is our best machine\nlearning model that has also outperformed a few of the Deep learning baseline\nmodels. We applied deep learning techniques to model four different deep neural\nnetworks of multiple layers consisting of Glove embeddings, Long Short Term\nMemory, Convolution, Max pooling, Dense, Batch Normalization, Activation\nfunctions, and model merge. Our deep learning models achieved better accuracy\nthan machine learning models. Three out of four proposed architectures\noutperformed the accuracy from previous machine learning and deep learning\nresearch work, two out of four models outperformed accuracy from previous deep\nlearning study on Quora's question pair dataset, and our best model achieved\naccuracy of 85.82% which is close to Quora state of the art accuracy.\n

Paper

Similar papers

© 2026 NYSGPT2525 LLC