Relational World Knowledge Representation in Contextual Language Models: A Review

Relational knowledge bases (KBs) are commonly used to represent world\nknowledge in machines. However, while advantageous for their high degree of\nprecision and interpretability, KBs are usually organized according to\nmanually-defined schemas, which limit their expressiveness and require\nsignificant human efforts to engineer and maintain. In this review, we take a\nnatural language processing perspective to these limitations, examining how\nthey may be addressed in part by training deep contextual language models (LMs)\nto internalize and express relational knowledge in more flexible forms. We\npropose to organize knowledge representation strategies in LMs by the level of\nKB supervision provided, from no KB supervision at all to entity- and\nrelation-level supervision. Our contributions are threefold: (1) We provide a\nhigh-level, extensible taxonomy for knowledge representation in LMs; (2) Within\nour taxonomy, we highlight notable models, evaluation tasks, and findings, in\norder to provide an up-to-date review of current knowledge representation\ncapabilities in LMs; and (3) We suggest future research directions that build\nupon the complementary aspects of LMs and KBs as knowledge representations.\n