Name: biobert large cased v1.1 squad
Author: dmis-lab

Overview

BioBERT Large Cased v1.1 SQuAD is a domain-specific transformer model optimized for extractive question answering within the biomedical landscape. Unlike general-purpose BERT models, this version is pre-trained on massive corpora of PubMed abstracts and PMC full-text articles, allowing it to handle complex medical terminology and clinical jargon with higher precision. It is specifically fine-tuned on the SQuAD dataset, making it ideal for building knowledge retrieval systems, medical chatbots, or automated literature review tools. For developers, this means reduced need for extensive domain-specific fine-tuning when deploying QA pipelines for healthcare data, as the model already understands the semantic nuances of biological texts.

Highlights

Domain-optimized for biomedical and clinical text extraction
Fine-tuned on SQuAD for precise extractive QA
Reduced hallucination in medical terminology compared to base BERT
Apache-2.0 license ensures flexible commercial integration
High semantic accuracy for PubMed and PMC data

Usage

Install

# Install Hugging Face transformers
pip install transformers torch

SDK Usage

# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("dmis-lab/biobert-large-cased-v1.1-squad")
tokenizer = AutoTokenizer.from_pretrained("dmis-lab/biobert-large-cased-v1.1-squad")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance

pip install -U huggingface_hub

CLI Download

Download the full repository

huggingface-cli download dmis-lab/biobert-large-cased-v1.1-squad

Download a single file to a local folder (e.g. config.json into ./dir)

huggingface-cli download dmis-lab/biobert-large-cased-v1.1-squad config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('dmis-lab/biobert-large-cased-v1.1-squad')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://huggingface.co/dmis-lab/biobert-large-cased-v1.1-squad

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/dmis-lab/biobert-large-cased-v1.1-squad

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

pip install -U transformers torch

Load the model and run inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('dmis-lab/biobert-large-cased-v1.1-squad')
tokenizer = AutoTokenizer.from_pretrained('dmis-lab/biobert-large-cased-v1.1-squad')

Full Documentation

来源: HuggingFace

---
tags:

question-answering

bert

---

Model Card for biobert-large-cased-v1.1-squad

Model Details

Model Description

More information needed

Developed by: DMIS-lab (Data Mining and Information Systems Lab, Korea University)

Shared by [Optional]: DMIS-lab (Data Mining and Information Systems Lab, Korea University)

Model type: Question Answering

Language(s) (NLP): More information needed

License: More information needed

Parent Model: gpt-neo-2.7B

Resources for more information:

- GitHub Repo - Associated Paper

Uses

Direct Use

This model can be used for the task of question answering.

Downstream Use [Optional]

More information needed.

Out-of-Scope Use

The model should not be used to intentionally create hostile or alienating environments for people.

Bias, Risks, and Limitations

Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.

Recommendations

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.

Training Details

Training Data

The model creators note in the associated paper: > We used the BERTBASE model pre-trained on English Wikipedia and BooksCorpus for 1M steps. BioBERT v1.0 (þ PubMed þ PMC) is the version of BioBERT (þ PubMed þ PMC) trained for 470 K steps. When using both the PubMed and PMC corpora, we found that 200K and 270K pre-training steps were optimal for PubMed and PMC, respectively. We also used the ablated versions of BioBERT v1.0, which were pre-trained on only PubMed for 200K steps (BioBERT v1.0 (þ PubMed)) and PMC for 270K steps (BioBERT v1.0 (þ PMC))

Training Procedure

Preprocessing

The model creators note in the associated paper: > We pre-trained BioBERT using Naver Smart Machine Learning (NSML) (Sung et al., 2017), which is utilized for large-scale experiments that need to be run on several GPUs

Speeds, Sizes, Times

The model creators note in the associated paper: > The maximum sequence length was fixed to 512 and the mini-batch size was set to 192, resulting in 98 304 words per iteration.

Evaluation

Testing Data, Factors & Metrics

Testing Data

More information needed

Factors

More information needed

Metrics

More information needed

Results

More information needed

Model Examination

More information needed

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Hardware Type: More information needed

- Training: Eight NVIDIA V100 (32GB) GPUs [ for training], - Fine-tuning: a single NVIDIA Titan Xp (12GB) GPU to fine-tune BioBERT on each task

Hours used: More information needed

Cloud Provider: More information needed

Compute Region: More information needed

Carbon Emitted: More information needed

Technical Specifications [optional]

Model Architecture and Objective

More information needed

Compute Infrastructure

More information needed

Hardware

More information needed

Software

More information needed.

Citation

BibTeX:

bibtex

@misc{mesh-transformer-jax,
 @article{lee2019biobert,
  title={BioBERT: a pre-trained biomedical language representation model for biomedical text mining},
  author={Lee, Jinhyuk and Yoon, Wonjin and Kim, Sungdong and Kim, Donghyeon and Kim, Sunkyu and So, Chan Ho and Kang, Jaewoo},
  journal={arXiv preprint arXiv:1901.08746},
  year={2019}
}

Glossary [optional]

More information needed

More Information [optional]

For help or issues using BioBERT, please submit a GitHub issue. Please contact Jinhyuk Lee(lee.jnhk (at) gmail.com), or Wonjin Yoon (wonjin.info (at) gmail.com) for communication related to BioBERT.

Model Card Authors [optional]

DMIS-lab (Data Mining and Information Systems Lab, Korea University) in collaboration with Ezi Ozoani and the Hugging Face team

Model Card Contact

More information needed

How to Get Started with the Model

Use the code below to get started with the model. <details> <summary> Click to expand </summary>

python

from transformers import AutoTokenizer, AutoModelForQuestionAnswering
tokenizer = AutoTokenizer.from_pretrained("dmis-lab/biobert-large-cased-v1.1-squad")
model = AutoModelForQuestionAnswering.from_pretrained("dmis-lab/biobert-large-cased-v1.1-squad")

</details>