KI-BERT: Infusing Knowledge Context for Better Language and Domain Understanding

Contextualized entity representations learned by state-of-the-art\ntransformer-based language models (TLMs) like BERT, GPT, T5, etc., leverage the\nattention mechanism to learn the data context from training data corpus.\nHowever, these models do not use the knowledge context. Knowledge context can\nbe understood as semantics about entities and their relationship with\nneighboring entities in knowledge graphs. We propose a novel and effective\ntechnique to infuse knowledge context from multiple knowledge graphs for\nconceptual and ambiguous entities into TLMs during fine-tuning. It projects\nknowledge graph embeddings in the homogeneous vector-space, introduces new\ntoken-types for entities, aligns entity position ids, and a selective attention\nmechanism. We take BERT as a baseline model and implement the\n"Knowledge-Infused BERT" by infusing knowledge context from ConceptNet and\nWordNet, which significantly outperforms BERT and other recent knowledge-aware\nBERT variants like ERNIE, SenseBERT, and BERT_CS over eight different subtasks\nof GLUE benchmark. The KI-BERT-base model even significantly outperforms\nBERT-large for domain-specific tasks like SciTail and academic subsets of QQP,\nQNLI, and MNLI.\n