UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the Unified Medical Language System Metathesaurus

Contextual word embedding models, such as BioBERT and Bio_ClinicalBERT, have\nachieved state-of-the-art results in biomedical natural language processing\ntasks by focusing their pre-training process on domain-specific corpora.\nHowever, such models do not take into consideration expert domain knowledge.\n In this work, we introduced UmlsBERT, a contextual embedding model that\nintegrates domain knowledge during the pre-training process via a novel\nknowledge augmentation strategy. More specifically, the augmentation on\nUmlsBERT with the Unified Medical Language System (UMLS) Metathesaurus was\nperformed in two ways: i) connecting words that have the same underlying\n`concept' in UMLS, and ii) leveraging semantic group knowledge in UMLS to\ncreate clinically meaningful input embeddings. By applying these two\nstrategies, UmlsBERT can encode clinical domain knowledge into word embeddings\nand outperform existing domain-specific models on common named-entity\nrecognition (NER) and clinical natural language inference clinical NLP tasks.\n