Mention Memory: incorporating textual knowledge into Transformers through entity mention attention

Natural language understanding tasks such as open-domain question answering\noften require retrieving and assimilating factual information from multiple\nsources. We propose to address this problem by integrating a semi-parametric\nrepresentation of a large text corpus into a Transformer model as a source of\nfactual knowledge. Specifically, our method represents knowledge with `mention\nmemory', a table of dense vector representations of every entity mention in a\ncorpus. The proposed model - TOME - is a Transformer that accesses the\ninformation through internal memory layers in which each entity mention in the\ninput passage attends to the mention memory. This approach enables synthesis of\nand reasoning over many disparate sources of information within a single\nTransformer model. In experiments using a memory of 150 million Wikipedia\nmentions, TOME achieves strong performance on several open-domain\nknowledge-intensive tasks, including the claim verification benchmarks HoVer\nand FEVER and several entity-based QA benchmarks. We also show that the model\nlearns to attend to informative mentions without any direct supervision.\nFinally we demonstrate that the model can generalize to new unseen entities by\nupdating the memory without retraining.\n