NodePiece: Compositional and Parameter-Efficient Representations of Large Knowledge Graphs

Conventional representation learning algorithms for knowledge graphs (KG) map\neach entity to a unique embedding vector. Such a shallow lookup results in a\nlinear growth of memory consumption for storing the embedding matrix and incurs\nhigh computational costs when working with real-world KGs. Drawing parallels\nwith subword tokenization commonly used in NLP, we explore the landscape of\nmore parameter-efficient node embedding strategies with possibly sublinear\nmemory requirements. To this end, we propose NodePiece, an anchor-based\napproach to learn a fixed-size entity vocabulary. In NodePiece, a vocabulary of\nsubword/sub-entity units is constructed from anchor nodes in a graph with known\nrelation types. Given such a fixed-size vocabulary, it is possible to bootstrap\nan encoding and embedding for any entity, including those unseen during\ntraining. Experiments show that NodePiece performs competitively in node\nclassification, link prediction, and relation prediction tasks while retaining\nless than 10% of explicit nodes in a graph as anchors and often having 10x\nfewer parameters. To this end, we show that a NodePiece-enabled model\noutperforms existing shallow models on a large OGB WikiKG 2 graph having 70x\nfewer parameters.\n