word2vec Skip-Gram with Negative Sampling is a Weighted Logistic PCA

We show that the skip-gram formulation of word2vec trained with negative sampling is equivalent to a weighted logistic PCA. This connection allows us to better understand the objective, compare it to other word embedding methods, and extend it to higher dimensional models.

cs.CL stat.ML Computer Science Mathematics

View graph Source

Paper

References (8)

01Explaining and Generalizing Skip-Gram through Exponential Family Principal Component AnalysisJason Eisner, Benjamin Van Durme, Ryan Cotterell et al.2017 · Conference of the European Chapter of the Association for Computational Linguistics · 28 citations

02Word Embedding Revisited: A New Representation Learning and Explicit Matrix Factorization PerspectiveYitan Li, Linli Xu, Fei Tian et al.2015 · International Joint Conference on Artificial Intelligence · 131 citations

03Neural Word Embedding as Implicit Matrix FactorizationOmer Levy, Yoav Goldberg2014 · Neural Information Processing Systems · 2.0k citations

04GloVe: Global Vectors for Word RepresentationJeffrey Pennington, R. Socher, Christopher D. Manning2014 · Conference on Empirical Methods in Natural Language Processing · 35k citations

05word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding methodYoav Goldberg, Omer Levy2014 · arXiv.org · 1.7k citations In Library

06Distributed Representations of Words and Phrases and their CompositionalityTomas Mikolov, I. Sutskever, Kai Chen et al.2013 · Neural Information Processing Systems · 35k citations In Library

07Collaborative Filtering for Implicit Feedback DatasetsYifan Hu, Y. Koren, C. Volinsky2008 · 2008 Eighth IEEE International Conference on Data Mining · 3.6k citations

08Logistic Matrix Factorization for Implicit Feedback DataChristopher C. Johnson2014 · 246 citations

Similar papers

word2vec Explained: deriving Mikolov et al.'s negative-sampling word-embedding methodYoav Goldberg, Omer Levy2014 · arXiv.org · 1.7k citations In Library

On SkipGram Word Embedding Models with Negative Sampling: Unified Framework and Impact of Noise DistributionsLiu, Dezhi, Richong Zhang, Ziqiao Wang2020 · arXiv (Cornell University) · 1 citation In Library

The strange geometry of skip-gram with negative samplingDavid Mimno, Laure Thompson2017 · Conference on Empirical Methods in Natural Language Processing · 121 citations In Library

Bayesian Neural Word EmbeddingOren Barkan2016 · AAAI Conference on Artificial Intelligence · 89 citations In Library

Second-order Co-occurrence Sensitivity of Skip-Gram with Negative SamplingDominik Schlechtweg, Cennet Oğuz, Sabine Schulte im Walde2019 In Library

基于强负采样的词嵌入优化算法(Word Embedding Optimization Based on Hard Negative Sampling)Yuchen Wang, Miaozhe Lin, Jiefan Zhan2020 In Library

Spectral Word Embedding with Negative SamplingBehrouz Haji Soleimani, S. Matwin2018 · AAAI Conference on Artificial Intelligence · 11 citations In Library

Review on Word2Vec Word Embedding Neural NetSoubraylu Sivakumar, Lakshmi Sarvani Videla, T. Rajesh Kumar et al.2020 · 2020 International Conference on Smart Electronics and Communication (ICOSEC) · 60 citations In Library

Improving Negative Sampling for Word Representation using Self-embedded FeaturesLong Chen, Fajie Yuan, J. Jose et al.2017 · Web Search and Data Mining · 48 citations In Library

Riemannian Optimization for Skip-Gram Negative SamplingA. Fonarev, Oleksii Hrinchuk, Gleb Gusev et al.2017 · Annual Meeting of the Association for Computational Linguistics · 11 citations In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal