All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality

Similarity measures are a vital tool for understanding how language models\nrepresent and process language. Standard representational similarity measures\nsuch as cosine similarity and Euclidean distance have been successfully used in\nstatic word embedding models to understand how words cluster in semantic space.\nRecently, these measures have been applied to embeddings from contextualized\nmodels such as BERT and GPT-2. In this work, we call into question the\ninformativity of such measures for contextualized language models. We find that\na small number of rogue dimensions, often just 1-3, dominate these measures.\nMoreover, we find a striking mismatch between the dimensions that dominate\nsimilarity measures and those which are important to the behavior of the model.\nWe show that simple postprocessing techniques such as standardization are able\nto correct for rogue dimensions and reveal underlying representational quality.\nWe argue that accounting for rogue dimensions is essential for any\nsimilarity-based analysis of contextual language models.\n