VAST, the Valence-Assessing Semantics Test, is a novel intrinsic evaluation\ntask for contextualized word embeddings (CWEs). VAST uses valence, the\nassociation of a word with pleasantness, to measure the correspondence of\nword-level LM semantics with widely used human judgments, and examines the\neffects of contextualization, tokenization, and LM-specific geometry. Because\nprior research has found that CWEs from GPT-2 perform poorly on other intrinsic\nevaluations, we select GPT-2 as our primary subject, and include results\nshowing that VAST is useful for 7 other LMs, and can be used in 7 languages.\nGPT-2 results show that the semantics of a word incorporate the semantics of\ncontext in layers closer to model output, such that VAST scores diverge between\nour contextual settings, ranging from Pearson's rho of .55 to .77 in layer 11.\nWe also show that multiply tokenized words are not semantically encoded until\nlayer 8, where they achieve Pearson's rho of .46, indicating the presence of an\nencoding process for multiply tokenized words which differs from that of singly\ntokenized words, for which rho is highest in layer 0. We find that a few\nneurons with values having greater magnitude than the rest mask word-level\nsemantics in GPT-2's top layer, but that word-level semantics can be recovered\nby nullifying non-semantic principal components: Pearson's rho in the top layer\nimproves from .32 to .76. After isolating semantics, we show the utility of\nVAST for understanding LM semantics via improvements over related work on four\nword similarity tasks, with a score of .50 on SimLex-999, better than the\nprevious best of .45 for GPT-2. Finally, we show that 8 of 10 WEAT bias tests,\nwhich compare differences in word embedding associations between groups of\nwords, exhibit more stereotype-congruent biases after isolating semantics,\nindicating that non-semantic structures in LMs also mask biases.\n