Examining the Inductive Bias of Neural Language Models with Artificial Languages

Since language models are used to model a wide variety of languages, it is\nnatural to ask whether the neural architectures used for the task have\ninductive biases towards modeling particular types of languages. Investigation\nof these biases has proved complicated due to the many variables that appear in\nthe experimental setup. Languages vary in many typological dimensions, and it\nis difficult to single out one or two to investigate without the others acting\nas confounders. We propose a novel method for investigating the inductive\nbiases of language models using artificial languages. These languages are\nconstructed to allow us to create parallel corpora across languages that differ\nonly in the typological feature being investigated, such as word order. We then\nuse them to train and test language models. This constitutes a fully controlled\ncausal framework, and demonstrates how grammar engineering can serve as a\nuseful tool for analyzing neural models. Using this method, we find that\ncommonly used neural architectures exhibit different inductive biases: LSTMs\ndisplay little preference with respect to word ordering, while transformers\ndisplay a clear preference for some orderings over others. Further, we find\nthat neither the inductive bias of the LSTM nor that of the transformer appears\nto reflect any tendencies that we see in attested natural languages.\n