Bio ClinicalBERT
简介
核心亮点
- 基于临床真实病历训练，深谙医疗领域语义
- 擅长处理电子病历等非结构化医疗文档
- 适用于医疗 NER 和临床文本分类任务
- 兼容 BERT 生态，可快速迁移至下游场景
使用方法
# 安装 Hugging Face transformers
pip install transformers torch# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")Hugging Face 下载
我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。
操作指引：在下载前，请先通过如下命令安装 huggingface_hub：
pip install -U huggingface_hub命令行下载
下载完整模型库
huggingface-cli download emilyalsentzer/Bio_ClinicalBERT下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）
huggingface-cli download emilyalsentzer/Bio_ClinicalBERT config.json --local-dir ./dirSDK 下载
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('emilyalsentzer/Bio_ClinicalBERT')Git 下载
请确保 lfs 已经被正确安装
git lfs install
git clone https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT如果您希望跳过 lfs 大文件下载，可以使用如下命令
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/emilyalsentzer/Bio_ClinicalBERT模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。
PyTorch / Transformers 使用
安装 Transformers
pip install -U transformers torch模型加载和推理
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('emilyalsentzer/Bio_ClinicalBERT')
tokenizer = AutoTokenizer.from_pretrained('emilyalsentzer/Bio_ClinicalBERT')完整文档
---
language: "en"
tags:
- fill-mask
license: mit
---
ClinicalBERT - Bio + Clinical BERT Model
The Publicly Available Clinical BERT Embeddings paper contains four unique clinicalBERT models: initialized with BERT-Base (cased_L-12_H-768_A-12) or BioBERT (BioBERT-Base v1.0 + PubMed 200K + PMC 270K) & trained on either all MIMIC notes or only discharge summaries.
This model card describes the Bio+Clinical BERT model, which was initialized from BioBERT & trained on all MIMIC notes.
Pretraining Data
TheBio_ClinicalBERT model was trained on all notes from MIMIC III, a database containing electronic health records from ICU patients at the Beth Israel Hospital in Boston, MA. For more details on MIMIC, see here. All notes from the NOTEEVENTS table were included (~880M words).
Model Pretraining
Note Preprocessing
Each note in MIMIC was first split into sections using a rules-based section splitter (e.g. discharge summary notes were split into "History of Present Illness", "Family History", "Brief Hospital Course", etc. sections). Then each section was split into sentences using SciSpacy (en core sci md tokenizer).
Pretraining Procedures
The model was trained using code from Google's BERT repository on a GeForce GTX TITAN X 12 GB GPU. Model parameters were initialized with BioBERT (BioBERT-Base v1.0 + PubMed 200K + PMC 270K).
Pretraining Hyperparameters
We used a batch size of 32, a maximum sequence length of 128, and a learning rate of 5 · 10−5 for pre-training our models. The models trained on all MIMIC notes were trained for 150,000 steps. The dup factor for duplicating input data with different masks was set to 5. All other default parameters were used (specifically, masked language model probability = 0.15 and max predictions per sequence = 20).How to use the model
Load the model via the transformers library:
from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")
model = AutoModel.from_pretrained("emilyalsentzer/Bio_ClinicalBERT")More Information
Refer to the original paper, Publicly Available Clinical BERT Embeddings (NAACL Clinical NLP Workshop 2019) for additional details and performance on NLI and NER tasks.
Questions?
Post a Github issue on the clinicalBERT repo or email [email protected] with any questions.
