distilbert base cased distilled squad
Overview
Highlights
- Fast inference speeds via knowledge distillation
- Optimized for extractive question answering tasks
- Reduced memory footprint for edge deployment
- Case-sensitive processing for improved entity recognition
- Seamless integration with Hugging Face Transformers
Usage
# Install Hugging Face transformers
pip install transformers torch
# Load model with transformers
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("distilbert/distilbert-base-cased-distilled-squad")
tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-cased-distilled-squad")
Hugging Face Download
We recommend downloading the model via the Hugging Face CLI or Hub SDK.
Guidance:Before downloading, install huggingface_hub with:
pip install -U huggingface_hub
CLI Download
Download the full repository
huggingface-cli download distilbert/distilbert-base-cased-distilled-squad
Download a single file to a local folder (e.g. config.json into ./dir)
huggingface-cli download distilbert/distilbert-base-cased-distilled-squad config.json --local-dir ./dir
See the official docs for more CLI options
SDK Download
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('distilbert/distilbert-base-cased-distilled-squad')
Git Download
Make sure git-lfs is installed first
git lfs install
git clone https://huggingface.co/distilbert/distilbert-base-cased-distilled-squad
To skip LFS large-file downloads, use:
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/distilbert/distilbert-base-cased-distilled-squad
Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.
PyTorch / Transformers Usage
Install Transformers
pip install -U transformers torch
Load the model and run inference
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('distilbert/distilbert-base-cased-distilled-squad')
tokenizer = AutoTokenizer.from_pretrained('distilbert/distilbert-base-cased-distilled-squad')
Model Download
We recommend downloading the model via the ModelScope CLI or SDK.
Guidance:Before downloading, install ModelScope with:
pip install modelscope
CLI Download
Download the full repository
modelscope download --model distilbert/distilbert-base-cased-distilled-squad
Download a single file to a local folder (e.g. README.md into ./dir)
modelscope download --model distilbert/distilbert-base-cased-distilled-squad README.md --local_dir ./dir
See the docs for more CLI options
SDK Download
# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('distilbert/distilbert-base-cased-distilled-squad')
Git Download
Make sure git-lfs is installed first
git lfs install
git clone https://www.modelscope.cn/distilbert/distilbert-base-cased-distilled-squad.git
To skip LFS large-file downloads, use:
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/distilbert/distilbert-base-cased-distilled-squad.git
ModelScope 模型页直接下载模型文件;无需将模型文件放在本站服务器。
Notebook Quickstart
Install the ModelScope library
pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html
Load the model and run inference
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks
p = pipeline('text-generation', 'distilbert/distilbert-base-cased-distilled-squad')
Full Documentation
---
language: en
license: apache-2.0
datasets:
- squad
metrics:
- squad
model-index:
- name: distilbert-base-cased-distilled-squad
results:
- task:
type: question-answering
name: Question Answering
dataset:
name: squad
type: squad
config: plain_text
split: validation
metrics:
- type: exact_match
value: 79.5998
name: Exact Match
verified: true
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZTViZDA2Y2E2NjUyMjNjYjkzNTUzODc5OTk2OTNkYjQxMDRmMDhlYjdmYWJjYWQ2N2RlNzY1YmI3OWY1NmRhOSIsInZlcnNpb24iOjF9.ZJHhboAMwsi3pqU-B-XKRCYP_tzpCRb8pEjGr2Oc-TteZeoWHI8CXcpDxugfC3f7d_oBcKWLzh3CClQxBW1iAQ
- type: f1
value: 86.9965
name: F1
verified: true
verifyToken: eyJhbGciOiJFZERTQSIsInR5cCI6IkpXVCJ9.eyJoYXNoIjoiZWZlMzY2MmE1NDNhOGNjNWRmODg0YjQ2Zjk5MjUzZDQ2MDYxOTBlMTNhNzQ4NTA2NjRmNDU3MGIzMTYwMmUyOSIsInZlcnNpb24iOjF9.z0ZDir87aT7UEmUeDm8Uw0oUdAqzlBz343gwnsQP3YLfGsaHe-jGlhco0Z7ISUd9NokyCiJCRc4NNxJQ83IuCw
---
DistilBERT base cased distilled SQuAD
Table of Contents
Model Details
Model Description: The DistilBERT model was proposed in the blog post Smaller, faster, cheaper, lighter: Introducing DistilBERT, adistilled version of BERT, and the paper DistilBERT, adistilled version of BERT: smaller, faster, cheaper and lighter. DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than *bert-base-uncased*, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language understanding benchmark.
This model is a fine-tune checkpoint of DistilBERT-base-cased, fine-tuned using (a second step of) knowledge distillation on SQuAD v1.1.
- Developed by: Hugging Face
- Model Type: Transformer-based language model
- Language(s): English
- License: Apache 2.0
- Related Models: DistilBERT-base-cased
- Resources for more information:
How to Get Started with the Model
Use the code below to get started with the model.
>>> from transformers import pipeline
>>> question_answerer = pipeline("question-answering", model='distilbert-base-cased-distilled-squad')
>>> context = r"""
... Extractive Question Answering is the task of extracting an answer from a text given a question. An example of a
... question answering dataset is the SQuAD dataset, which is entirely based on that task. If you would like to fine-tune
... a model on a SQuAD task, you may leverage the examples/pytorch/question-answering/run_squad.py script.
... """
>>> result = question_answerer(question="What is a good example of a question answering dataset?", context=context)
>>> print(
... f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}"
...)
Answer: 'SQuAD dataset', score: 0.5152, start: 147, end: 160
Here is how to use this model in PyTorch:
from transformers import DistilBertTokenizer, DistilBertModel
import torch
tokenizer = DistilBertTokenizer.from_pretrained('distilbert-base-cased-distilled-squad')
model = DistilBertModel.from_pretrained('distilbert-base-cased-distilled-squad')
question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
inputs = tokenizer(question, text, return_tensors="pt")
with torch.no_grad():
outputs = model(inputs)
print(outputs)
And in TensorFlow:
from transformers import DistilBertTokenizer, TFDistilBertForQuestionAnswering
import tensorflow as tf
tokenizer = DistilBertTokenizer.from_pretrained("distilbert-base-cased-distilled-squad")
model = TFDistilBertForQuestionAnswering.from_pretrained("distilbert-base-cased-distilled-squad")
question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
inputs = tokenizer(question, text, return_tensors="tf")
outputs = model(inputs)
answer_start_index = int(tf.math.argmax(outputs.start_logits, axis=-1)[0])
answer_end_index = int(tf.math.argmax(outputs.end_logits, axis=-1)[0])
predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index + 1]
tokenizer.decode(predict_answer_tokens)
Uses
This model can be used for question answering.
#### Misuse and Out-of-scope Use
The model should not be used to intentionally create hostile or alienating environments for people. In addition, the model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.
Risks, Limitations and Biases
CONTENT WARNING: Readers should be aware that language generated by this model can be disturbing or offensive to some and can propagate historical and current stereotypes.
Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)). Predictions generated by the model can include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. For example:
>>> from transformers import pipeline
>>> question_answerer = pipeline("question-answering", model='distilbert-base-cased-distilled-squad')
>>> context = r"""
... Alice is sitting on the bench. Bob is sitting next to her.
... """
>>> result = question_answerer(question="Who is the CEO?", context=context)
>>> print(
... f"Answer: '{result['answer']}', score: {round(result['score'], 4)}, start: {result['start']}, end: {result['end']}"
...)
Answer: 'Bob', score: 0.7527, start: 32, end: 35
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
Training
#### Training Data
The distilbert-base-cased model was trained using the same data as the distilbert-base-uncased model. The distilbert-base-uncased model model describes it's training data as:
> DistilBERT pretrained on the same data as BERT, which is BookCorpus, a dataset consisting of 11,038 unpublished books and English Wikipedia (excluding lists, tables and headers).
To learn more about the SQuAD v1.1 dataset, see the SQuAD v1.1 data card.
#### Training Procedure
##### Preprocessing
See the distilbert-base-cased model card for further details.
##### Pretraining
See the distilbert-base-cased model card for further details.
Evaluation
As discussed in the [model repository](https://github.com/huggingface/transformers/blob/main/examples/researc