Text classification with word embedding regularization and soft similarity measure

Since the seminal work of Mikolov et al., word embeddings have become the\npreferred word representations for many natural language processing tasks.\nDocument similarity measures extracted from word embeddings, such as the soft\ncosine measure (SCM) and the Word Mover's Distance (WMD), were reported to\nachieve state-of-the-art performance on semantic text similarity and text\nclassification.\n Despite the strong performance of the WMD on text classification and semantic\ntext similarity, its super-cubic average time complexity is impractical. The\nSCM has quadratic worst-case time complexity, but its performance on text\nclassification has never been compared with the WMD. Recently, two word\nembedding regularization techniques were shown to reduce storage and memory\ncosts, and to improve training speed, document processing speed, and task\nperformance on word analogy, word similarity, and semantic text similarity.\nHowever, the effect of these techniques on text classification has not yet been\nstudied.\n In our work, we investigate the individual and joint effect of the two word\nembedding regularization techniques on the document processing speed and the\ntask performance of the SCM and the WMD on text classification. For evaluation,\nwe use the $k$NN classifier and six standard datasets: BBCSPORT, TWITTER,\nOHSUMED, REUTERS-21578, AMAZON, and 20NEWS.\n We show 39% average $k$NN test error reduction with regularized word\nembeddings compared to non-regularized word embeddings. We describe a practical\nprocedure for deriving such regularized embeddings through Cholesky\nfactorization. We also show that the SCM with regularized word embeddings\nsignificantly outperforms the WMD on text classification and is over 10,000\ntimes faster.\n