distilbert base cased distilled squad

Providerdistilbert
Categoryquestion-answering
Licenseapache-2.0
Downloads86
Stars0

Overview

DistilBERT-base-cased-distilled-SQuAD is a lightweight, transformer-based model optimized for extractive question answering. By leveraging knowledge distillation from BERT, it maintains a high percentage of the original model's performance while significantly reducing parameter count and latency. This makes it an ideal choice for developers deploying QA systems in resource-constrained environments or real-time applications where inference speed is critical. It is specifically fine-tuned on the SQuAD dataset, enabling it to identify precise answer spans within a given context. Integration is straightforward via the Hugging Face Transformers library, offering a drop-in replacement for BERT in pipelines requiring lower memory overhead without sacrificing substantial accuracy.

Highlights

  • Fast inference speeds via knowledge distillation
  • Optimized for extractive question answering tasks
  • Reduced memory footprint for edge deployment
  • Case-sensitive processing for improved entity recognition
  • Seamless integration with Hugging Face Transformers

Usage

Install
# Install Hugging Face transformers
pip install transformers torch
SDK Usage
# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("distilbert/distilbert-base-cased-distilled-squad")
tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-cased-distilled-squad")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance:Before downloading, install huggingface_hub with:

Guidance
pip install -U huggingface_hub

CLI Download

Download the full repository

Download the full repository
huggingface-cli download distilbert/distilbert-base-cased-distilled-squad

Download a single file to a local folder (e.g. config.json into ./dir)

Download a single file to a local folder (e.g. config.json into ./dir)
huggingface-cli download distilbert/distilbert-base-cased-distilled-squad config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

SDK Download
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('distilbert/distilbert-base-cased-distilled-squad')

Git Download

Make sure git-lfs is installed first

Git Download
git lfs install
git clone https://huggingface.co/distilbert/distilbert-base-cased-distilled-squad

To skip LFS large-file downloads, use:

Skip LFS
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/distilbert/distilbert-base-cased-distilled-squad

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

Install Transformers
pip install -U transformers torch

Load the model and run inference

Load the model and run inference
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('distilbert/distilbert-base-cased-distilled-squad')
tokenizer = AutoTokenizer.from_pretrained('distilbert/distilbert-base-cased-distilled-squad')

Model Download

We recommend downloading the model via the ModelScope CLI or SDK.

Guidance:Before downloading, install ModelScope with:

Guidance
pip install modelscope

CLI Download

Download the full repository

Download the full repository
modelscope download --model distilbert/distilbert-base-cased-distilled-squad

Download a single file to a local folder (e.g. README.md into ./dir)

Download a single file to a local folder (e.g. README.md into ./dir)
modelscope download --model distilbert/distilbert-base-cased-distilled-squad README.md --local_dir ./dir

See the docs for more CLI options

SDK Download

SDK Download
# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('distilbert/distilbert-base-cased-distilled-squad')

Git Download

Make sure git-lfs is installed first

Git Download
git lfs install
git clone https://www.modelscope.cn/distilbert/distilbert-base-cased-distilled-squad.git

To skip LFS large-file downloads, use:

Skip LFS
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/distilbert/distilbert-base-cased-distilled-squad.git

ModelScope 模型页直接下载模型文件;无需将模型文件放在本站服务器。

Notebook Quickstart

Install the ModelScope library

Install the ModelScope library
pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

Load the model and run inference

Load the model and run inference
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

p = pipeline('text-generation', 'distilbert/distilbert-base-cased-distilled-squad')

Full Documentation

来源: HuggingFace

---
language: en
license: apache-2.0
datasets:

  • squad

metrics:
  • squad

model-index:
  • name: distilbert-base-cased-distilled-squad

results:
- task:
type: question-answering
name: Question Answering
dataset:
name: squad
type: squad
config: plain_text
split: validation
metrics:
- type: exact_match
value: 79.5998
name: Exact Match
verified: true
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZTViZDA2Y2E2NjUyMjNjYjkzNTUzODc5OTk2OTNkYjQxMDRmMDhlYjdmYWJjYWQ2N2RlNzY1YmI3OWY1NmRhOSIsInZlcnNpb24iOjF9.ZJHhboAMwsi3pqU-B-XKRCYP_tzpCRb8pEjGr2Oc-TteZeoWHI8CXcpDxugfC3f7d_oBcKWLzh3CClQxBW1iAQ
- type: f1
value: 86.9965
name: F1
verified: true
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZWZlMzY2MmE1NDNhOGNjNWRmODg0YjQ2Zjk5MjUzZDQ2MDYxOTBlMTNhNzQ4NTA2NjRmNDU3MGIzMTYwMmUyOSIsInZlcnNpb24iOjF9.z0ZDir87aT7UEmUeDm8Uw0oUdAqzlBz343gwnsQP3YLfGsaHe-jGlhco0Z7ISUd9NokyCiJCRc4NNxJQ83IuCw
---

DistilBERT base cased distilled SQuAD

Table of Contents

Model Details

Model Description: The DistilBERT model was proposed in the blog post Smaller, faster, cheaper, lighter: Introducing DistilBERT, adistilled version of BERT, and the paper DistilBERT, adistilled version of BERT: smaller, faster, cheaper and lighter. DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than *bert-base-uncased*, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language understanding benchmark.

This model is a fine-tune checkpoint of DistilBERT-base-cased, fine-tuned using (a second step of) knowledge distillation on SQuAD v1.1.

  • Developed by: Hugging Face
  • Model Type: Transformer-based language model
  • Language(s): English
  • License: Apache 2.0
  • Resources for more information:
- See this repository for more about Distil\* (a class of compressed models including this model) - See Sanh et al. (2019) for more information about knowledge distillation and the training procedure

How to Get Started with the Model

Use the code below to get started with the model.

python
>>> from transformers import pipeline
>>> question_answerer = pipeline("question-answering", model='distilbert-base-cased-distilled-squad')

>>> context = r"""
... Extractive Question Answering is the task of extracting an answer from a text given a question. An example of a
... question answering dataset is the SQuAD dataset, which is entirely based on that task. If you would like to fine-tune
... a model on a SQuAD task, you may leverage the examples/pytorch/question-answering/run_squad.py script.
... """

>>> result = question_answerer(question="What is a good example of a question answering dataset?", context=context)
>>> print(
... f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}"
...)

Answer: 'SQuAD dataset', score: 0.5152, start: 147, end: 160

Here is how to use this model in PyTorch:

python
from transformers import DistilBertTokenizer, DistilBertModel
import torch
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased-distilled-squad')
model = DistilBertModel.from_pretrained('distilbert-base-cased-distilled-squad')

question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"

inputs = tokenizer(question, text, return_tensors="pt")
with torch.no_grad():
outputs = model(inputs)

print(outputs)

And in TensorFlow:

python
from transformers import DistilBertTokenizer, TFDistilBertForQuestionAnswering
import tensorflow as tf

tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-cased-distilled-squad")
model = TFDistilBertForQuestionAnswering.from_pretrained("distilbert-base-cased-distilled-squad")

question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"

inputs = tokenizer(question, text, return_tensors="tf")
outputs = model(
inputs)

answer_start_index = int(tf.math.argmax(outputs.start_logits, axis=-1)[0])
answer_end_index = int(tf.math.argmax(outputs.end_logits, axis=-1)[0])

predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
tokenizer.decode(predict_answer_tokens)

Uses

This model can be used for question answering.

#### Misuse and Out-of-scope Use

The model should not be used to intentionally create hostile or alienating environments for people. In addition, the model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.

Risks, Limitations and Biases

CONTENT WARNING: Readers should be aware that language generated by this model can be disturbing or offensive to some and can propagate historical and current stereotypes.

Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)). Predictions generated by the model can include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. For example:

python
>>> from transformers import pipeline
>>> question_answerer = pipeline("question-answering", model='distilbert-base-cased-distilled-squad')

>>> context = r"""
... Alice is sitting on the bench. Bob is sitting next to her.
... """

>>> result = question_answerer(question="Who is the CEO?", context=context)
>>> print(
... f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}"
...)

Answer: 'Bob', score: 0.7527, start: 32, end: 35

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.

Training

#### Training Data

The distilbert-base-cased model was trained using the same data as the distilbert-base-uncased model. The distilbert-base-uncased model model describes it's training data as:

> DistilBERT pretrained on the same data as BERT, which is BookCorpus, a dataset consisting of 11,038 unpublished books and English Wikipedia (excluding lists, tables and headers).

To learn more about the SQuAD v1.1 dataset, see the SQuAD v1.1 data card.

#### Training Procedure

##### Preprocessing

See the distilbert-base-cased model card for further details.

##### Pretraining

See the distilbert-base-cased model card for further details.

Evaluation

As discussed in the [model repository](https://github.com/huggingface/transformers/blob/main/examples/researc

Join our Telegram