CancerBERT: a BERT model for Extracting Breast Cancer Phenotypes from Electronic Health Records

Objective: Accurate extraction of breast cancer patients’ phenotypes is important for clinical decision support and clinical research. This study developed and evaluated cancer domain pre-trained CancerBERT models for extracting breast cancer phenotypes from clinical texts. We also investigated the effect of customized cancer-related vocabulary on the performance of CancerBERT models. Materials and Methods: A cancer-related corpus from breast cancer patients was extracted from the electronic health records of a local hospital. We annotated named entities in 200 pathology reports and 50 clinical notes for eight cancer phenotypes for fine-tuning and evaluation. We kept pre-training the BlueBERT model on the cancer corpus with expanded vocabularies (using both term frequency-based and manually reviewed methods) to obtain CancerBERT models. The CancerBERT models were evaluated and compared with other baseline models on the cancer phenotype extraction task. Results: All CancerBERT models outperformed all other models on the cancer phenotyping NER task. Both CancerBERT models with customized vocabularies outperformed the CancerBERT with the original BERT vocabulary. The CancerBERT model with manually reviewed customized vocabulary achieved the best performance with macro-F1 scores equal to 0.876 (95% CI, 0.873-0.879) and 0.904 (95% CI, 0.902-0.906) for exact match and lenient match, respectively. Conclusions: The CancerBERT models were developed to extract the cancer phenotypes in clinical notes and pathology reports. The results validated that using customized vocabulary may further improve the performances of domain specific BERT models in clinical NLP tasks. The CancerBERT models developed in the study would further help clinical decision support. INTRODUCTION Breast cancer is one of the most prevalent and lethal cancers for women in the US. It is estimated that there will be about 250,000 patients diagnosed with breast cancer each year and around 40,000 deaths due to breast cancer [1]. The development of precision medicine has contributed new approaches to the better diagnosis, prognosis and treatments of breast cancers, with the ultimate goal of selecting optimal treatments for individual patients [2-4]. A representative example is the targeted therapy for breast cancer, which uses different medications to treat patients with different hormone receptors status, such as human epidermal growth factor receptor 2 (HER2) and estrogen receptor (ER). The application of precision medicine and its related translational research need the support of large amounts of cancerspecific patient clinical information. The widely adopted electronic health record systems (EHRs) are fundamental sources to provide longitudinal and multi-perspective patient clinical data, which includes patients’ demographics, lab results, disease progress, treatments, and outcomes. Some of these data are stored in the codified (structured) part of EHRs; however, a large amount of information is distributed in the narrative text data, such as the clinical notes and lab reports [5]. How to effectively extract the target information from the narrative text part of the EHRs data remains an important research topic [6-7]. Previous studies (detailed in Background section) focused mainly on the rule-based and conventional machine learning methods. The state-of-the-art language models such as bidirectional encoder representations from transformers (BERT) [8] have shown significant improvement in many NLP tasks; however, there is no cancer domain specific BERT model for downstream clinical NLP tasks, such as cancer phenotype extraction. In addition, there is no investigation of out-of-vocabulary (OOV) issue for BERT-based model in cancer domain. Objective To address these gaps in current status of the cancer phenotyping extraction, our contributions in this study include: 1) We developed and evaluated cancer domain specific BERT models (CancerBERT) that are able to extract comprehensive collections of breast cancer related phenotypes (i.e., Hormone receptor type, Hormone receptor status, Tumor size, Tumor site, Cancer grade, Histological type, Cancer laterality and Cancer stage) from both clinical notes and pathology reports in EHRs. Our CancerBERT models significantly outperformed other existing BERT-based models (e.g., BlueBERT, BioBERT, CharBERT) on our name entity recognition (NER) task to extract target cancer phenotypes for breast cancer patients. 2) We also evaluated different methods to address the OOV issue of original BERT models. Specifically, we used two approaches (domain knowledge-based and statistics-based) to generate and add additional cancer specific words that were missing in the original BERT vocabulary. We found that additional cancer specific words can further improve performance of the CancerBERT model on the NER task. Background Previous works have developed different approaches to extract information from narrative data in EHRs. Manual chart review is a feasible approach to extract the phenotypes from the clinical texts; however, it is time-consuming and not cost-effective [9-10]. Researchers have developed approaches based on natural language processing (NLP) to finish the task automatically. Before the deep learning era, established studies mainly focused on rule-based, traditional machine learning-based methods, depending on the characteristic of the data and specific tasks. Nguyen, et al. developed a rule-based pipeline in 2015 to extract cancer-related phenotypes, including histological type, cancer grade, primary site and laterality, from the textual contents of the pathology reports in EHRs [11]. The F1 scores for different variables range from 0.61 to 0.93, and a message producer/consumer module was integrated into the pipeline to enable the real-time processing of the reports. Yala, et al. developed a machine learning algorithm to classify phenotypes of breast cancers. N-grams were used as features and boosting algorithm was applied to do the classification of phenotype status. The performances were robust, with F1 scores ranging from 0.57 to 1 for different categories. However, the pipeline was designed for judging the status of phenotypes, mainly binary classification; no detail information in the text can be captured [12]. The DeepPhe software was developed in 2017 to extract cancer phenotypes from clinical records [13]. It could extract a wide range of breast cancer phenotypes from the EHRs through different approaches, such as, rules, domain knowledge bases, and machine learning methods. The inter-annotator agreement (IAA) of the DeepPhe range from 0.2 to 0.96 [13]. Qiu, et al. developed a convolutional neural network (CNN) model to extract the cancer primary site from pathology reports, and the CNN model outperformed the traditional frequency vector space approach with a micro-F score of 0.722 [14]. A coarse-to-fine multi-task CNN model was further proposed to extract the cancer primary site, laterality and grade from the pathology reports at the same time; this model obtained an F-1 score of 0.775 for extracting cancer primary site [15]. These studies focused mainly on the rule-based and conventional machine learning methods. The latest BERT-based models have been developed in recent years and show great advantages in NLP tasks compared to the traditional feature-based machine learning approaches [8]. For BERT-based models, previous works have shown that using biomedical domain specific text as training data can obtain better performance compared to models trained on generaldomain language for tasks related to the biomedical domain [16-19]. In clinical domain, studies have explored using the advanced BERT-based models to solve clinical information extraction tasks [16-17,20]. These studies are mainly focusing on testing the BERT model on clinical benchmark datasets, such as Informatics for Integrating Biology and the Bedside (i2b2), SemEval, MedSTS. Currently, only few studies have applied the advanced deep learning models include BERT to extract the cancer phenotypes for cancer patients. For example, BERT models were applied to extract the clinical information for breast cancer patients from Chinese clinical texts and achieved F1 scores of 0.786 to 1 for different clinical concepts [21]. It is known that Chinese and English are different, and there are no existing studies that have explored the BERT-based models to extract cancer phenotypes from clinical notes and pathology reports in English. Furthermore, most BERT-based models deal with the out-of-vocabulary (OOV) issue by tokenizing an unknown word into multiple sub-words that exist in the vocabulary. In this case, the sub-word representations may not capture the semantics of the whole word [22]. Several studies explored to improve the OOV issue by either using character-level word embedding [22-23] or building a brand-new domain specific vocabulary to best match the training corpus [24]. These studies obtained promising results in benchmark tasks, but they need to train the models from scratch, which need a huge training corpus and computing sources.