Black-box probing models can reliably extract linguistic features like tense,\nnumber, and syntactic role from pretrained word representations. However, the\nmanner in which these features are encoded in representations remains poorly\nunderstood. We present a systematic study of the linear geometry of\ncontextualized word representations in ELMO and BERT. We show that a variety of\nlinguistic features (including structured dependency relationships) are encoded\nin low-dimensional subspaces. We then refine this geometric picture, showing\nthat there are hierarchical relations between the subspaces encoding general\nlinguistic categories and more specific ones, and that low-dimensional feature\nencodings are distributed rather than aligned to individual neurons. Finally,\nwe demonstrate that these linear subspaces are causally related to model\nbehavior, and can be used to perform fine-grained manipulation of BERT's output\ndistribution.\n