stanford deidentifier base
简介
Stanford Deidentifier Base 是一款专注于医疗健康领域隐私脱敏的命名实体识别（NER）模型。它能精准识别文本中的患者姓名、日期、地点等敏感个人信息（PHI），帮助开发者在不泄露隐私的前提下处理医疗数据集。该模型基于 Token 分类任务，上手难度低，可直接集成到数据预处理流水线中。对于需要构建符合 HIPAA 等隐私标准、或从事医疗 AI 研发的中国开发者来说，它是一个轻量且高效的预处理工具，能有效替代繁琐的人工脱敏过程。
核心亮点
- 专业医疗隐私脱敏，精准识别 PHI 敏感信息
- 基于 Token 分类，适配多种医疗文本处理场景
- MIT 协议开源，集成简单且部署灵活
- 大幅提升医疗数据预处理效率，确保隐私合规
使用方法
安装依赖
# 安装 Hugging Face transformers
pip install transformers torchSDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("StanfordAIMI/stanford-deidentifier-base")
tokenizer = AutoTokenizer.from_pretrained("StanfordAIMI/stanford-deidentifier-base")Hugging Face 下载
我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。
操作指引：在下载前，请先通过如下命令安装 huggingface_hub：
操作指引
pip install -U huggingface_hub命令行下载
下载完整模型库
下载完整模型库
huggingface-cli download StanfordAIMI/stanford-deidentifier-base下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）
下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）
huggingface-cli download StanfordAIMI/stanford-deidentifier-base config.json --local-dir ./dirSDK 下载
SDK 下载
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('StanfordAIMI/stanford-deidentifier-base')Git 下载
请确保 lfs 已经被正确安装
Git 下载
git lfs install
git clone https://huggingface.co/StanfordAIMI/stanford-deidentifier-base如果您希望跳过 lfs 大文件下载，可以使用如下命令
跳过 LFS
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/StanfordAIMI/stanford-deidentifier-base模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。
PyTorch / Transformers 使用
安装 Transformers
安装 Transformers
pip install -U transformers torch模型加载和推理
模型加载和推理
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('StanfordAIMI/stanford-deidentifier-base')
tokenizer = AutoTokenizer.from_pretrained('StanfordAIMI/stanford-deidentifier-base')完整文档
来源: HuggingFace
---
widget:
- text: "PROCEDURE: Chest xray. COMPARISON: last seen on 1/1/2020 and also record dated of March 1st, 2019. FINDINGS: patchy airspace opacities. IMPRESSION: The results of the chest xray of January 1 2020 are the most concerning ones. The patient was transmitted to another service of UH Medical Center under the responsability of Dr. Perez. We used the system MedClinical data transmitter and sent the data on 2/1/2020, under the ID 5874233. We received the confirmation of Dr Perez. He is reachable at 567-493-1234."
- text: "Dr. Curt Langlotz chose to schedule a meeting on 06/23."
tags:
- token-classification
- sequence-tagger-model
- pytorch
- transformers
- pubmedbert
- uncased
- radiology
- biomedical
- bdf-toolbox
datasets:
- radreports
language:
- en
license: mit
---
Stanford de-identifier was trained on a variety of radiology and biomedical documents with the goal of automatising the de-identification process while reaching satisfactory accuracy for use in production. Manuscript in-proceedings.
These model weights are the recommended ones among all available deidentifier weights.
Associated github repo: https://github.com/MIDRC/Stanford_Penn_Deidentifier
Acknowledgement
This work was supported in part by the Medical Imaging and Data Resource Center (MIDRC), which is funded by the National Institute of Biomedical Imaging and Bioengineering (NIBIB) of the National Institutes of Health under contract 75N92020D00021 and through The Advanced Research Projects Agency for Health (ARPA-H)Citation
bibtex
@article{10.1093/jamia/ocac219,
author = {Chambon, Pierre J and Wu, Christopher and Steinkamp, Jackson M and Adleberg, Jason and Cook, Tessa S and Langlotz, Curtis P},
title = "{Automated deidentification of radiology reports combining transformer and “hide in plain sight” rule-based methods}",
journal = {Journal of the American Medical Informatics Association},
year = {2022},
month = {11},
abstract = "{To develop an automated deidentification pipeline for radiology reports that detect protected health information (PHI) entities and replaces them with realistic surrogates “hiding in plain sight.”In this retrospective study, 999 chest X-ray and CT reports collected between November 2019 and November 2020 were annotated for PHI at the token level and combined with 3001 X-rays and 2193 medical notes previously labeled, forming a large multi-institutional and cross-domain dataset of 6193 documents. Two radiology test sets, from a known and a new institution, as well as i2b2 2006 and 2014 test sets, served as an evaluation set to estimate model performance and to compare it with previously released deidentification tools. Several PHI detection models were developed based on different training datasets, fine-tuning approaches and data augmentation techniques, and a synthetic PHI generation algorithm. These models were compared using metrics such as precision, recall and F1 score, as well as paired samples Wilcoxon tests.Our best PHI detection model achieves 97.9 F1 score on radiology reports from a known institution, 99.6 from a new institution, 99.5 on i2b2 2006, and 98.9 on i2b2 2014. On reports from a known institution, it achieves 99.1 recall of detecting the core of each PHI span.Our model outperforms all deidentifiers it was compared to on all test sets as well as human labelers on i2b2 2014 data. It enables accurate and automatic deidentification of radiology reports.A transformer-based deidentification pipeline can achieve state-of-the-art performance for deidentifying radiology reports and other medical documents.}",
issn = {1527-974X},
doi = {10.1093/jamia/ocac219},
url = {https://doi.org/10.1093/jamia/ocac219},
note = {ocac219},
eprint = {https://academic.oup.com/jamia/advance-article-pdf/doi/10.1093/jamia/ocac219/47220191/ocac219.pdf},
}