Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little

A possible explanation for the impressive performance of masked language\nmodel (MLM) pre-training is that such models have learned to represent the\nsyntactic structures prevalent in classical NLP pipelines. In this paper, we\npropose a different explanation: MLMs succeed on downstream tasks almost\nentirely due to their ability to model higher-order word co-occurrence\nstatistics. To demonstrate this, we pre-train MLMs on sentences with randomly\nshuffled word order, and show that these models still achieve high accuracy\nafter fine-tuning on many downstream tasks -- including on tasks specifically\ndesigned to be challenging for models that ignore word order. Our models\nperform surprisingly well according to some parametric syntactic probes,\nindicating possible deficiencies in how we test representations for syntactic\ninformation. Overall, our results show that purely distributional information\nlargely explains the success of pre-training, and underscore the importance of\ncurating challenging evaluation datasets that require deeper linguistic\nknowledge.\n