deberta v3 base
Overview
Highlights
- Disentangled attention for superior structural text understanding
- RTD pre-training yields higher efficiency than standard MLM
- Optimized for NLU tasks like classification and NER
- Lightweight footprint with high benchmark performance
- Easy integration via Hugging Face Transformers library
Usage
# Install Hugging Face transformers
pip install transformers torch# Load model with transformers
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("microsoft/deberta-v3-base")
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-v3-base")Hugging Face Download
We recommend downloading the model via the Hugging Face CLI or Hub SDK.
Guidance：Before downloading, install huggingface_hub with:
pip install -U huggingface_hubCLI Download
Download the full repository
huggingface-cli download microsoft/deberta-v3-baseDownload a single file to a local folder (e.g. config.json into ./dir)
huggingface-cli download microsoft/deberta-v3-base config.json --local-dir ./dirSee the official docs for more CLI options
SDK Download
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('microsoft/deberta-v3-base')Git Download
Make sure git-lfs is installed first
git lfs install
git clone https://huggingface.co/microsoft/deberta-v3-baseTo skip LFS large-file downloads, use:
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/microsoft/deberta-v3-baseModel files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.
PyTorch / Transformers Usage
Install Transformers
pip install -U transformers torchLoad the model and run inference
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('microsoft/deberta-v3-base')
tokenizer = AutoTokenizer.from_pretrained('microsoft/deberta-v3-base')Model Download
We recommend downloading the model via the ModelScope CLI or SDK.
Guidance：Before downloading, install ModelScope with:
pip install modelscopeCLI Download
Download the full repository
modelscope download --model microsoft/deberta-v3-baseDownload a single file to a local folder (e.g. README.md into ./dir)
modelscope download --model microsoft/deberta-v3-base README.md --local_dir ./dirSee the docs for more CLI options
SDK Download
# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('microsoft/deberta-v3-base')Git Download
Make sure git-lfs is installed first
git lfs install
git clone https://www.modelscope.cn/microsoft/deberta-v3-base.gitTo skip LFS large-file downloads, use:
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/microsoft/deberta-v3-base.gitModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。
Notebook Quickstart
Install the ModelScope library
pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.htmlLoad the model and run inference
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks
p = pipeline('text-generation', 'microsoft/deberta-v3-base')Full Documentation
---
language: en
tags:
- deberta
- deberta-v3
- fill-mask
thumbnail: https://huggingface.co/front/thumbnails/microsoft.png
license: mit
---
DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing
DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data.
In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper.
Please check the official repository for more implementation details and updates.
The DeBERTa V3 base model comes with 12 layers and a hidden size of 768. It has only 86M backbone parameters with a vocabulary containing 128K tokens which introduces 98M parameters in the Embedding layer. This model was trained using the 160GB data as DeBERTa V2.
#### Fine-tuning on NLU tasks
We present the dev results on SQuAD 2.0 and MNLI tasks.
| Model |Vocabulary(K)|Backbone #Params(M)| SQuAD 2.0(F1/EM) | MNLI-m/mm(ACC)|
|-------------------|----------|-------------------|-----------|----------|
| RoBERTa-base |50 |86 | 83.7/80.5 | 87.6/- |
| XLNet-base |32 |92 | -/80.2 | 86.8/- |
| ELECTRA-base |30 |86 | -/80.5 | 88.8/ |
| DeBERTa-base |50 |100 | 86.2/83.1| 88.8/88.5|
| DeBERTa-v3-base |128|86 | 88.4/85.4 | 90.6/90.7|
| DeBERTa-v3-base + SiFT |128|86 | -/- | 91.0/-|
We present the dev results on SQuAD 1.1/2.0 and MNLI tasks.
#### Fine-tuning with HF transformers
#!/bin/bash
cd transformers/examples/pytorch/text-classification/
pip install datasets
export TASK_NAME=mnli
output_dir="ds_results"
num_gpus=8
batch_size=8
python -m torch.distributed.launch --nproc_per_node=${num_gpus} \
run_glue.py \
--model_name_or_path microsoft/deberta-v3-base \
--task_name $TASK_NAME \
--do_train \
--do_eval \
--evaluation_strategy steps \
--max_seq_length 256 \
--warmup_steps 500 \
--per_device_train_batch_size ${batch_size} \
--learning_rate 2e-5 \
--num_train_epochs 3 \
--output_dir $output_dir \
--overwrite_output_dir \
--logging_steps 1000 \
--logging_dir $output_dir
Citation
If you find DeBERTa useful for your work, please cite the following papers:
`` latex``
@misc{he2021debertav3,
title={DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing},
author={Pengcheng He and Jianfeng Gao and Weizhu Chen},
year={2021},
eprint={2111.09543},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
@inproceedings{
he2021deberta,
title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=XPZIaotutsD}
}