distilbert base uncased distilled squad

Providerdistilbert
Categoryquestion-answering
Licenseapache-2.0
Downloads83
Stars0

Overview

DistilBERT base uncased distilled SQuAD is a lightweight, transformer-based model optimized for extractive question-answering tasks. By leveraging knowledge distillation from BERT, it retains roughly 97% of its parent's performance while significantly reducing parameter count and inference latency. For developers, this makes it an ideal choice for deploying QA systems on edge devices or in latency-sensitive production environments where a full-sized BERT model would be too resource-intensive. It is pre-trained on the SQuAD dataset, allowing it to identify specific spans of text that answer a given query within a provided context. Integration is straightforward via the Hugging Face Transformers library, offering a drop-in replacement for BERT-base architectures with faster throughput and lower memory overhead.

Highlights

  • Reduced latency and memory footprint via knowledge distillation
  • Optimized for extractive question-answering on SQuAD datasets
  • Maintains high accuracy with significantly faster inference speeds
  • Easy integration through standard Transformers library pipelines
  • Apache-2.0 license ensures flexible commercial deployment

Usage

Install
# Install Hugging Face transformers
pip install transformers torch
SDK Usage
# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("distilbert/distilbert-base-uncased-distilled-squad")
tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased-distilled-squad")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance 
pip install -U huggingface_hub

CLI Download

Download the full repository

Download the full repository 
huggingface-cli download distilbert/distilbert-base-uncased-distilled-squad

Download a single file to a local folder (e.g. config.json into ./dir)

Download a single file to a local folder (e.g. config.json into ./dir) 
huggingface-cli download distilbert/distilbert-base-uncased-distilled-squad config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('distilbert/distilbert-base-uncased-distilled-squad')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://huggingface.co/distilbert/distilbert-base-uncased-distilled-squad

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/distilbert/distilbert-base-uncased-distilled-squad

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

Install Transformers 
pip install -U transformers torch

Load the model and run inference

Load the model and run inference 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('distilbert/distilbert-base-uncased-distilled-squad')
tokenizer = AutoTokenizer.from_pretrained('distilbert/distilbert-base-uncased-distilled-squad')

Model Download

We recommend downloading the model via the ModelScope CLI or SDK.

Guidance：Before downloading, install ModelScope with:

Guidance 
pip install modelscope

CLI Download

Download the full repository

Download the full repository 
modelscope download --model distilbert/distilbert-base-uncased-distilled-squad

Download a single file to a local folder (e.g. README.md into ./dir)

Download a single file to a local folder (e.g. README.md into ./dir) 
modelscope download --model distilbert/distilbert-base-uncased-distilled-squad README.md --local_dir ./dir

See the docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('distilbert/distilbert-base-uncased-distilled-squad')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://www.modelscope.cn/distilbert/distilbert-base-uncased-distilled-squad.git

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/distilbert/distilbert-base-uncased-distilled-squad.git

ModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。

Notebook Quickstart

Install the ModelScope library

Install the ModelScope library 
pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

Load the model and run inference

Load the model and run inference 
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

p = pipeline('text-generation', 'distilbert/distilbert-base-uncased-distilled-squad')

Full Documentation

来源: HuggingFace

---
language: en
datasets:

  • squad

widget:
  • text: "Which name is also used to describe the Amazon rainforest in English?"

context: "The Amazon rainforest (Portuguese: Floresta Amazônica or Amazônia; Spanish: Selva Amazónica, Amazonía or usually Amazonia; French: Forêt amazonienne; Dutch: Amazoneregenwoud), also known in English as Amazonia or the Amazon Jungle, is a moist broadleaf forest that covers most of the Amazon basin of South America. This basin encompasses 7,000,000 square kilometres (2,700,000 sq mi), of which 5,500,000 square kilometres (2,100,000 sq mi) are covered by the rainforest. This region includes territory belonging to nine nations. The majority of the forest is contained within Brazil, with 60% of the rainforest, followed by Peru with 13%, Colombia with 10%, and with minor amounts in Venezuela, Ecuador, Bolivia, Guyana, Suriname and French Guiana. States or departments in four nations contain \"Amazonas\" in their names. The Amazon represents over half of the planet's remaining rainforests, and comprises the largest and most biodiverse tract of tropical rainforest in the world, with an estimated 390 billion individual trees divided into 16,000 species."
  • text: "How many square kilometers of rainforest is covered in the basin?"

context: "The Amazon rainforest (Portuguese: Floresta Amazônica or Amazônia; Spanish: Selva Amazónica, Amazonía or usually Amazonia; French: Forêt amazonienne; Dutch: Amazoneregenwoud), also known in English as Amazonia or the Amazon Jungle, is a moist broadleaf forest that covers most of the Amazon basin of South America. This basin encompasses 7,000,000 square kilometres (2,700,000 sq mi), of which 5,500,000 square kilometres (2,100,000 sq mi) are covered by the rainforest. This region includes territory belonging to nine nations. The majority of the forest is contained within Brazil, with 60% of the rainforest, followed by Peru with 13%, Colombia with 10%, and with minor amounts in Venezuela, Ecuador, Bolivia, Guyana, Suriname and French Guiana. States or departments in four nations contain \"Amazonas\" in their names. The Amazon represents over half of the planet's remaining rainforests, and comprises the largest and most biodiverse tract of tropical rainforest in the world, with an estimated 390 billion individual trees divided into 16,000 species."
license: apache-2.0
---

DistilBERT base uncased distilled SQuAD

Table of Contents

Model Details

Model Description: The DistilBERT model was proposed in the blog post Smaller, faster, cheaper, lighter: Introducing DistilBERT, adistilled version of BERT, and the paper DistilBERT, adistilled version of BERT: smaller, faster, cheaper and lighter. DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than *bert-base-uncased*, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language understanding benchmark.

This model is a fine-tune checkpoint of DistilBERT-base-uncased, fine-tuned using (a second step of) knowledge distillation on SQuAD v1.1.

  • Developed by: Hugging Face
  • Model Type: Transformer-based language model
  • Language(s): English
  • License: Apache 2.0
  • Resources for more information:
- See this repository for more about Distil\* (a class of compressed models including this model) - See Sanh et al. (2019) for more information about knowledge distillation and the training procedure

How to Get Started with the Model

Use the code below to get started with the model.

python
>>> from transformers import pipeline
>>> question_answerer = pipeline("question-answering", model='distilbert-base-uncased-distilled-squad')

>>> context = r"""
... Extractive Question Answering is the task of extracting an answer from a text given a question. An example     of a
... question answering dataset is the SQuAD dataset, which is entirely based on that task. If you would like to fine-tune
... a model on a SQuAD task, you may leverage the examples/pytorch/question-answering/run_squad.py script.
... """


>>> result = question_answerer(question="What is a good example of a question answering dataset?",     context=context)
>>> print(
... f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}"
...)


Answer: 'SQuAD dataset', score: 0.4704, start: 147, end: 160

Here is how to use this model in PyTorch:

python
from transformers import DistilBertTokenizer, DistilBertForQuestionAnswering
import torch
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-uncased-distilled-squad')
model = DistilBertForQuestionAnswering.from_pretrained('distilbert-base-uncased-distilled-squad')

question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"


inputs = tokenizer(question, text, return_tensors="pt")
with torch.no_grad():
    outputs = model(inputs)


answer_start_index = torch.argmax(outputs.start_logits)
answer_end_index = torch.argmax(outputs.end_logits)


predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
tokenizer.decode(predict_answer_tokens)

And in TensorFlow:

python
from transformers import DistilBertTokenizer, TFDistilBertForQuestionAnswering
import tensorflow as tf

tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-uncased-distilled-squad")
model = TFDistilBertForQuestionAnswering.from_pretrained("distilbert-base-uncased-distilled-squad")


question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"


inputs = tokenizer(question, text, return_tensors="tf")
outputs = model(inputs)


answer_start_index = int(tf.math.argmax(outputs.start_logits, axis=-1)[0])
answer_end_index = int(tf.math.argmax(outputs.end_logits, axis=-1)[0])


predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
tokenizer.decode(predict_answer_tokens)

Uses

This model can be used for question answering.

#### Misuse and Out-of-scope Use

The model should not be used to intentionally create hostile or alienating environments for people. In addition, the model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.

Risks, Limitations and Biases

CONTENT WARNING: Readers should be aware that language generated by this model can be disturbing or offensive to some and can propagate historical and current stereotypes.

Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)). Predictions generated by the model can include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. For example:

```python
>>> from transformers import pipeline
>>> question_answerer = pipeline("question-answering", model='distilbert-base-uncased-distilled

View official source
Join our Telegram