Contextualized Embeddings in Named-Entity Recognition: An Empirical Study on Generalization

Contextualized embeddings use unsupervised language model pretraining to\ncompute word representations depending on their context. This is intuitively\nuseful for generalization, especially in Named-Entity Recognition where it is\ncrucial to detect mentions never seen during training. However, standard\nEnglish benchmarks overestimate the importance of lexical over contextual\nfeatures because of an unrealistic lexical overlap between train and test\nmentions. In this paper, we perform an empirical analysis of the generalization\ncapabilities of state-of-the-art contextualized embeddings by separating\nmentions by novelty and with out-of-domain evaluation. We show that they are\nparticularly beneficial for unseen mentions detection, especially\nout-of-domain. For models trained on CoNLL03, language model contextualization\nleads to a +1.2% maximal relative micro-F1 score increase in-domain against\n+13% out-of-domain on the WNUT dataset\n