Overview

Bio ClinicalBERT is a domain-specific language model designed to bridge the gap between general biomedical text and the nuanced, often messy reality of electronic health records (EHR). Unlike standard BERT, this model is further pre-trained on large-scale clinical corpora, making it significantly more effective at handling medical shorthand, clinical jargon, and the idiosyncratic structure of patient notes. For developers building healthcare applications, it is an ideal backbone for Named Entity Recognition (NER), clinical document classification, and patient outcome prediction. It integrates seamlessly into the Hugging Face ecosystem, allowing for rapid fine-tuning on downstream clinical tasks where general-purpose models typically struggle with vocabulary mismatch.

Highlights

Optimized for electronic health records and clinical notes
Superior performance on medical entity recognition tasks
Seamless integration via the Hugging Face Transformers library
MIT licensed for flexible commercial and research use
Reduced vocabulary mismatch compared to base BERT models

Usage

Install

# Install Hugging Face transformers
pip install transformers torch

SDK Usage

# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance

pip install -U huggingface_hub

CLI Download

Download the full repository

huggingface-cli download emilyalsentzer/Bio_ClinicalBERT

Download a single file to a local folder (e.g. config.json into ./dir)

huggingface-cli download emilyalsentzer/Bio_ClinicalBERT config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('emilyalsentzer/Bio_ClinicalBERT')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

pip install -U transformers torch

Load the model and run inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('emilyalsentzer/Bio_ClinicalBERT')
tokenizer = AutoTokenizer.from_pretrained('emilyalsentzer/Bio_ClinicalBERT')

Full Documentation

来源: HuggingFace

---
language: "en"
tags:

fill-mask

license: mit

---

ClinicalBERT - Bio + Clinical BERT Model

The Publicly Available Clinical BERT Embeddings paper contains four unique clinicalBERT models: initialized with BERT-Base (cased_L-12_H-768_A-12) or BioBERT (BioBERT-Base v1.0 + PubMed 200K + PMC 270K) & trained on either all MIMIC notes or only discharge summaries.

This model card describes the Bio+Clinical BERT model, which was initialized from BioBERT & trained on all MIMIC notes.

Pretraining Data

The Bio_ClinicalBERT model was trained on all notes from MIMIC III, a database containing electronic health records from ICU patients at the Beth Israel Hospital in Boston, MA. For more details on MIMIC, see here. All notes from the NOTEEVENTS table were included (~880M words).

Model Pretraining

Note Preprocessing

Each note in MIMIC was first split into sections using a rules-based section splitter (e.g. discharge summary notes were split into "History of Present Illness", "Family History", "Brief Hospital Course", etc. sections). Then each section was split into sentences using SciSpacy (en core sci md tokenizer).

Pretraining Procedures

The model was trained using code from Google's BERT repository on a GeForce GTX TITAN X 12 GB GPU. Model parameters were initialized with BioBERT (BioBERT-Base v1.0 + PubMed 200K + PMC 270K).

Pretraining Hyperparameters

We used a batch size of 32, a maximum sequence length of 128, and a learning rate of 5 · 10−5 for pre-training our models. The models trained on all MIMIC notes were trained for 150,000 steps. The dup factor for duplicating input data with different masks was set to 5. All other default parameters were used (specifically, masked language model probability = 0.15 and max predictions per sequence = 20).

How to use the model

Load the model via the transformers library:

code

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")

More Information

Refer to the original paper, Publicly Available Clinical BERT Embeddings (NAACL Clinical NLP Workshop 2019) for additional details and performance on NLI and NER tasks.

Questions?

Post a Github issue on the clinicalBERT repo or email [email protected] with any questions.