Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers

Following the major success of neural language models (LMs) such as BERT or\nGPT-2 on a variety of language understanding tasks, recent work focused on\ninjecting (structured) knowledge from external resources into these models.\nWhile on the one hand, joint pretraining (i.e., training from scratch, adding\nobjectives based on external knowledge to the primary LM objective) may be\nprohibitively computationally expensive, post-hoc fine-tuning on external\nknowledge, on the other hand, may lead to the catastrophic forgetting of\ndistributional knowledge. In this work, we investigate models for complementing\nthe distributional knowledge of BERT with conceptual knowledge from ConceptNet\nand its corresponding Open Mind Common Sense (OMCS) corpus, respectively, using\nadapter training. While overall results on the GLUE benchmark paint an\ninconclusive picture, a deeper analysis reveals that our adapter-based models\nsubstantially outperform BERT (up to 15-20 performance points) on inference\ntasks that require the type of conceptual knowledge explicitly present in\nConceptNet and OMCS. All code and experiments are open sourced under:\nhttps://github.com/wluper/retrograph .\n