When classifying grammatical role, BERT doesn't care about word order... except when it matters

Because meaning can often be inferred from lexical semantics alone, word\norder is often a redundant cue in natural language. For example, the words\nchopped, chef, and onion are more likely used to convey "The chef chopped the\nonion," not "The onion chopped the chef." Recent work has shown large language\nmodels to be surprisingly word order invariant, but crucially has largely\nconsidered natural prototypical inputs, where compositional meaning mostly\nmatches lexical expectations. To overcome this confound, we probe grammatical\nrole representation in English BERT and GPT-2, on instances where lexical\nexpectations are not sufficient, and word order knowledge is necessary for\ncorrect classification. Such non-prototypical instances are naturally occurring\nEnglish sentences with inanimate subjects or animate objects, or sentences\nwhere we systematically swap the arguments to make sentences like "The onion\nchopped the chef". We find that, while early layer embeddings are largely\nlexical, word order is in fact crucial in defining the later-layer\nrepresentations of words in semantically non-prototypical positions. Our\nexperiments isolate the effect of word order on the contextualization process,\nand highlight how models use context in the uncommon, but critical, instances\nwhere it matters.\n