MedicalBERT: enhancing biomedical natural language processing using pretrained BERT-based model

Abstract

Recent advances in natural language processing (NLP) have been driven by pretrained language models like BERT, RoBERTa, T5, and GPT. These models excel at understanding complex texts, but biomedical literature, with its domain-specific terminology, poses challenges that models like Word2Vec and bidirectional long short-term memory (Bi-LSTM) can't fully address. GPT and T5, despite capturing context, fall short in tasks needing bidirectional understanding, unlike BERT. Addressing this, we proposed MedicalBERT, a pretrained BERT model trained on a large biomedical dataset and equipped with domain-specific vocabulary that enhances the comprehension of biomedical terminology. MedicalBERT model is further optimized and fine-tuned to address diverse tasks, including named entity recognition, relation extraction, question answering, sentence similarity, and document classification. Performance metrics such as the F1-score, accuracy, and Pearson correlation are employed to showcase the efficiency of our model in comparison to other BERT-based models such as BioBERT, SciBERT, and ClinicalBERT. MedicalBERT outperforms these models on most of the benchmarks, and surpasses the general-purpose BERT model by 5.67% on average across all the tasks evaluated respectively. This work also underscores the potential of leveraging pretrained BERT models for medical NLP tasks, demonstrating the effectiveness of transfer learning techniques in capturing domain-specific information.