Using Distributional Principles for the Semantic Study of Contextual Language Models

Many studies were recently done for investigating the properties of\ncontextual language models but surprisingly, only a few of them consider the\nproperties of these models in terms of semantic similarity. In this article, we\nfirst focus on these properties for English by exploiting the distributional\nprinciple of substitution as a probing mechanism in the controlled context of\nSemCor and WordNet paradigmatic relations. Then, we propose to adapt the same\nmethod to a more open setting for characterizing the differences between static\nand contextual language models.\n