biobert large cased v1.1 squad

提供商dmis-lab
分类question-answering
许可证Apache-2.0
下载量16.7K
星标0

简介

BioBERT-Large-Cased-v1.1-SQuAD 是一个专门针对生物医学领域优化的问答模型。它在 BERT-Large 的基础上，利用大规模 PubMed 论文数据进行领域预训练，并针对 SQuAD 问答数据集进行了微调。相比通用模型，它能更精准地理解复杂的医学术语和临床文本，适合用于构建医学知识库问答、电子病历信息提取等专业场景。对于习惯使用 Hugging Face 框架的开发者来说，该模型上手难度较低，可直接作为领域特化插件替换通用 BERT 以提升专业问答的准确率。

核心亮点

  • 深耕生物医学领域，精准识别专业医学术语
  • 基于 SQuAD 微调，擅长从医学文本中提取答案
  • Large 规模参数，比基础版具有更强的语义理解力
  • 兼容 Transformers 库，开发者可快速部署集成

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("dmis-lab/biobert-large-cased-v1.1-squad")
tokenizer = AutoTokenizer.from_pretrained("dmis-lab/biobert-large-cased-v1.1-squad")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download dmis-lab/biobert-large-cased-v1.1-squad

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download dmis-lab/biobert-large-cased-v1.1-squad config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('dmis-lab/biobert-large-cased-v1.1-squad')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/dmis-lab/biobert-large-cased-v1.1-squad

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/dmis-lab/biobert-large-cased-v1.1-squad

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('dmis-lab/biobert-large-cased-v1.1-squad')
tokenizer = AutoTokenizer.from_pretrained('dmis-lab/biobert-large-cased-v1.1-squad')

完整文档

来源: HuggingFace

---
tags:

  • question-answering

  • bert

---

Model Card for biobert-large-cased-v1.1-squad

Model Details

Model Description

More information needed
  • Developed by: DMIS-lab (Data Mining and Information Systems Lab, Korea University)
  • Shared by [Optional]: DMIS-lab (Data Mining and Information Systems Lab, Korea University)
  • Model type: Question Answering
  • Language(s) (NLP): More information needed
  • License: More information needed
  • Resources for more information:
- GitHub Repo - Associated Paper

Uses

Direct Use

This model can be used for the task of question answering.

Downstream Use [Optional]

More information needed.

Out-of-Scope Use

The model should not be used to intentionally create hostile or alienating environments for people.

Bias, Risks, and Limitations

Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

Training Details

Training Data

The model creators note in the associated paper: > We used the BERTBASE model pre-trained on English Wikipedia and BooksCorpus for 1M steps. BioBERT v1.0 (þ PubMed þ PMC) is the version of BioBERT (þ PubMed þ PMC) trained for 470 K steps. When using both the PubMed and PMC corpora, we found that 200K and 270K pre-training steps were optimal for PubMed and PMC, respectively. We also used the ablated versions of BioBERT v1.0, which were pre-trained on only PubMed for 200K steps (BioBERT v1.0 (þ PubMed)) and PMC for 270K steps (BioBERT v1.0 (þ PMC))

Training Procedure

Preprocessing

The model creators note in the associated paper: > We pre-trained BioBERT using Naver Smart Machine Learning (NSML) (Sung et al., 2017), which is utilized for large-scale experiments that need to be run on several GPUs

Speeds, Sizes, Times

The model creators note in the associated paper: > The maximum sequence length was fixed to 512 and the mini-batch size was set to 192, resulting in 98 304 words per iteration.

Evaluation

Testing Data, Factors & Metrics

Testing Data

More information needed

Factors

More information needed

Metrics

More information needed

Results

More information needed

Model Examination

More information needed

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).
  • Hardware Type: More information needed
- Training: Eight NVIDIA V100 (32GB) GPUs [ for training], - Fine-tuning: a single NVIDIA Titan Xp (12GB) GPU to fine-tune BioBERT on each task
  • Hours used: More information needed
  • Cloud Provider: More information needed
  • Compute Region: More information needed
  • Carbon Emitted: More information needed

Technical Specifications [optional]

Model Architecture and Objective

More information needed

Compute Infrastructure

More information needed

Hardware

More information needed

Software

More information needed.

Citation

BibTeX:


bibtex
@misc{mesh-transformer-jax,
 @article{lee2019biobert,
  title={BioBERT: a pre-trained biomedical language representation model for biomedical text mining},
  author={Lee, Jinhyuk and Yoon, Wonjin and Kim, Sungdong and Kim, Donghyeon and Kim, Sunkyu and So, Chan Ho and Kang, Jaewoo},
  journal={arXiv preprint arXiv:1901.08746},
  year={2019}
}





Glossary [optional]



More information needed

More Information [optional]

For help or issues using BioBERT, please submit a GitHub issue. Please contact Jinhyuk Lee(lee.jnhk (at) gmail.com), or Wonjin Yoon (wonjin.info (at) gmail.com) for communication related to BioBERT.

Model Card Authors [optional]

DMIS-lab (Data Mining and Information Systems Lab, Korea University) in collaboration with Ezi Ozoani and the Hugging Face team

Model Card Contact

More information needed

How to Get Started with the Model

Use the code below to get started with the model. <details> <summary> Click to expand </summary>
python
from transformers import AutoTokenizer, AutoModelForQuestionAnswering

tokenizer = AutoTokenizer.from_pretrained("dmis-lab/biobert-large-cased-v1.1-squad")


model = AutoModelForQuestionAnswering.from_pretrained("dmis-lab/biobert-large-cased-v1.1-squad")

</details>

查看官方来源