Model Choices Influence Attributive Word Associations: A Semi-supervised Analysis of Static Word Embeddings

Static word embeddings encode word associations, extensively utilized in\ndownstream NLP tasks. Although prior studies have discussed the nature of such\nword associations in terms of biases and lexical regularities captured, the\nvariation in word associations based on the embedding training procedure\nremains in obscurity. This work aims to address this gap by assessing\nattributive word associations across five different static word embedding\narchitectures, analyzing the impact of the choice of the model architecture,\ncontext learning flavor and training corpora. Our approach utilizes a\nsemi-supervised clustering method to cluster annotated proper nouns and\nadjectives, based on their word embedding features, revealing underlying\nattributive word associations formed in the embedding space, without\nintroducing any confirmation bias. Our results reveal that the choice of the\ncontext learning flavor during embedding training (CBOW vs skip-gram) impacts\nthe word association distinguishability and word embeddings' sensitivity to\ndeviations in the training corpora. Moreover, it is empirically shown that even\nwhen trained over the same corpora, there is significant inter-model disparity\nand intra-model similarity in the encoded word associations across different\nword embedding models, portraying specific patterns in the way the embedding\nspace is created for each embedding architecture.\n