Bioformer: an efficient transformer language model for biomedical text mining

. Bioformer has been successfully deployed to PubTator Central providing gene annotations over 35 million PubMed abstracts and 5 million PubMed Central full-text articles. We make Bioformer publicly available via https://github.com/WGLab/bioformer, including pre-trained models, datasets, and instructions for downstream use.

cs.CL Topic Modeling Biomedical Text Mining and Ontologies Natural Language Processing Techniques Computer Science Medicine

View graph Source

Paper

References (43)

01Multi-label topic classification for COVID-19 literature with BioformerLi Fang, Kai Wang2022 · arXiv.org · 4 citations In Library

02Multi-label classification for biomedical literature: an overview of the BioCreative VII LitCovid Track for COVID-19 literature topic annotationsQingyu Chen, Alexis Allot, Robert Leaman et al.2022 · Database J. Biol. Databases Curation · 45 citations In Library

03Benchmarking Effectiveness and Efficiency of Deep Learning Models for Semantic Textual Similarity in the Clinical Domain: Validation StudyQingyu Chen, Alex Rankine, Yifan Peng et al.2021 · JMIR Medical Informatics · 15 citations

04Artificial Intelligence (AI) in Action: Addressing the COVID-19 Pandemic with Natural Language Processing (NLP)Qingyu Chen, Robert Leaman, Alexis Allot et al.2020 · Annual Review of Biomedical Data Science · 43 citations In Library

05Domain-Specific Language Model Pretraining for Biomedical Natural Language ProcessingYu Gu, Robert Tinn, Hao Cheng et al.2020 · ACM Trans. Comput. Heal. · 2.6k citations In Library

06MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited DevicesZhiqing Sun, Hongkun Yu, Xiaodan Song et al.2020 · Annual Meeting of the Association for Computational Linguistics · 1.1k citations In Library

07BioConceptVec: Creating and evaluating literature-based biomedical concept embeddings on a large scaleQingyu Chen, Kyubum Lee, Shankai Yan et al.2019 · PLoS Comput. Biol. · 56 citations In Library

08Deep learning in clinical natural language processing: a methodical reviewStephen T Wu, Kirk Roberts, Surabhi Datta et al.2019 · J. Am. Medical Informatics Assoc. · 443 citations

09HuggingFace's Transformers: State-of-the-art Natural Language ProcessingThomas Wolf, Lysandre Debut, Victor Sanh et al.2019 · arXiv.org · 4.2k citations In Library

10DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighterVictor Sanh, Lysandre Debut, Julien Chaumond et al.2019 · arXiv.org · 10k citations In Library

11Well-Read Students Learn Better: On the Importance of Pre-training Compact ModelsIulia Turc, Ming-Wei Chang, Kenton Lee et al.2019 · 777 citations

12RoBERTa: A Robustly Optimized BERT Pretraining ApproachYinhan Liu, Myle Ott, Naman Goyal et al.2019 · arXiv.org · 31k citations In Library

Scroll for more · 31 remaining

Similar papers

BioReader: a Retrieval-Enhanced Text-to-Text Transformer for Biomedical LiteratureGiacomo Frisoni, Miki Mizutani, Gianluca Moro et al.2022 · Conference on Empirical Methods in Natural Language Processing · 39 citations In Library

BioGPT: generative pre-trained transformer for biomedical text generation and miningRenqian Luo, Liai Sun, Yingce Xia et al.2022 · Briefings in Bioinformatics In Library

BioMedGPT: Open Multimodal Generative Pre-trained Transformer for BioMedicineYi Luo, Jiahuan Zhang, Siqi Fan et al.2023 · arXiv.org · 142 citations In Library

BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and MiningRenqian Luo, Liai Sun, Yingce Xia et al.2022 · Briefings Bioinform. · 1.5k citations In Library

BioBERT: a pre-trained biomedical language representation model for biomedical text miningJinhyuk Lee, WonJin Yoon, Sungdong Kim et al.2019 · Bioinform. · 7.6k citations In Library

BioFuse: an embedding fusion framework for biomedical foundation modelsMirza Nasir Hossain, David Harris-Birtill2026 · PLoS ONE · 0 citations In Library

Text Mining for Bioinformatics Using Biomedical LiteratureAndre Lamurias, Francisco M. Couto2019 · Encyclopedia of Bioinformatics and Computational Biology · 37 citations In Library

Applications of transformer-based language models in bioinformatics: a surveyShuang Zhang, Rui Fan, Yuti Liu et al.2023 · Bioinformatics Advances · 167 citations In Library

Foundation models for bioinformaticsZiyu Chen, Lin Wei, Ge Gao2024 · Quant. Biol. · 57 citations In Library

An Experimental Evaluation of Transformer-based Language Models in the Biomedical DomainPaul Grouchy, Shobhit Jain, Michael C. Liu et al.2020 · arXiv (Cornell University) · 4 citations In Library

Source of record

arXiv

44B

Welcome.

This panel sticks with you. Pick anything below and it opens right beside it, so you can dig through 60,000-plus records without ever losing your spot here.

Travel 44B

LibraryPapers, policy, standards, statute LabsEvery organization building AI ModelsIntelligence, price, and speed BenchmarksThe evaluation catalog IncidentsAI failures and harms SearchOne field across everything DashboardThe state of AI accountability in NY 44B RegistryThe Art. 44-B compliance portal