Representing Mixtures of Word Embeddings with Mixtures of Topic Embeddings

A topic model is often formulated as a generative model that explains how\neach word of a document is generated given a set of topics and\ndocument-specific topic proportions. It is focused on capturing the word\nco-occurrences in a document and hence often suffers from poor performance in\nanalyzing short documents. In addition, its parameter estimation often relies\non approximate posterior inference that is either not scalable or suffers from\nlarge approximation error. This paper introduces a new topic-modeling framework\nwhere each document is viewed as a set of word embedding vectors and each topic\nis modeled as an embedding vector in the same embedding space. Embedding the\nwords and topics in the same vector space, we define a method to measure the\nsemantic difference between the embedding vectors of the words of a document\nand these of the topics, and optimize the topic embeddings to minimize the\nexpected difference over all documents. Experiments on text analysis\ndemonstrate that the proposed method, which is amenable to mini-batch\nstochastic gradient descent based optimization and hence scalable to big\ncorpora, provides competitive performance in discovering more coherent and\ndiverse topics and extracting better document representations.\n