A Comparative Study of Lexical Substitution Approaches based on Neural Language Models

Lexical substitution in context is an extremely powerful technology that can\nbe used as a backbone of various NLP applications, such as word sense\ninduction, lexical relation extraction, data augmentation, etc. In this paper,\nwe present a large-scale comparative study of popular neural language and\nmasked language models (LMs and MLMs), such as context2vec, ELMo, BERT, XLNet,\napplied to the task of lexical substitution. We show that already competitive\nresults achieved by SOTA LMs/MLMs can be further improved if information about\nthe target word is injected properly, and compare several target injection\nmethods. In addition, we provide analysis of the types of semantic relations\nbetween the target and substitutes generated by different models providing\ninsights into what kind of words are really generated or given by annotators as\nsubstitutes.\n