Name: deberta xlarge mnli
Author: microsoft

Overview

DeBERTa-v3-xlarge-mnli is a high-performance encoder model optimized for Natural Language Inference (NLI) and general-purpose text classification. Unlike standard BERT models, it utilizes a disentangled attention mechanism that handles relative positions more effectively, significantly boosting its ability to understand nuanced logical relationships between sentence pairs. For developers, this makes it a reliable choice for building automated labeling pipelines, sentiment analysis tools, or zero-shot classification systems where high precision is required. It integrates seamlessly with the Hugging Face Transformers library, offering a strong balance between inference latency and accuracy compared to larger LLMs, making it ideal for deployment in production environments where specialized classification is the primary goal.

Highlights

Superior logical reasoning via disentangled attention mechanism
Optimized for high-accuracy Natural Language Inference tasks
Seamless integration with Hugging Face Transformers library
Efficient alternative to LLMs for specialized text classification
Permissive MIT license for flexible commercial deployment

Usage

Install

# Install Hugging Face transformers
pip install transformers torch

SDK Usage

# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("microsoft/deberta-xlarge-mnli")
tokenizer = AutoTokenizer.from_pretrained("microsoft/deberta-xlarge-mnli")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance

pip install -U huggingface_hub

CLI Download

Download the full repository

huggingface-cli download microsoft/deberta-xlarge-mnli

Download a single file to a local folder (e.g. config.json into ./dir)

huggingface-cli download microsoft/deberta-xlarge-mnli config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('microsoft/deberta-xlarge-mnli')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://huggingface.co/microsoft/deberta-xlarge-mnli

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/microsoft/deberta-xlarge-mnli

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

pip install -U transformers torch

Load the model and run inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('microsoft/deberta-xlarge-mnli')
tokenizer = AutoTokenizer.from_pretrained('microsoft/deberta-xlarge-mnli')

Model Download

We recommend downloading the model via the ModelScope CLI or SDK.

Guidance：Before downloading, install ModelScope with:

Guidance

pip install modelscope

CLI Download

Download the full repository

modelscope download --model microsoft/deberta-xlarge-mnli

Download a single file to a local folder (e.g. README.md into ./dir)

modelscope download --model microsoft/deberta-xlarge-mnli README.md --local_dir ./dir

See the docs for more CLI options

SDK Download

# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('microsoft/deberta-xlarge-mnli')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://www.modelscope.cn/microsoft/deberta-xlarge-mnli.git

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/microsoft/deberta-xlarge-mnli.git

ModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。

Notebook Quickstart

Install the ModelScope library

pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

Load the model and run inference

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

p = pipeline('text-generation', 'microsoft/deberta-xlarge-mnli')

Full Documentation

来源: HuggingFace

---
language: en
tags:

deberta-v1

deberta-mnli

tasks: mnli
thumbnail: https://huggingface.co/front/thumbnails/microsoft.png
license: mit
widget:

text: "[CLS] I love you. [SEP] I like you. [SEP]"

---

DeBERTa: Decoding-enhanced BERT with Disentangled Attention

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. It outperforms BERT and RoBERTa on majority of NLU tasks with 80GB training data.

Please check the official repository for more details and updates.

This the DeBERTa xlarge model(750M) fine-tuned with mnli task.

Fine-tuning on NLU tasks

We present the dev results on SQuAD 1.1/2.0 and several GLUE benchmark tasks.

| Model | SQuAD 1.1 | SQuAD 2.0 | MNLI-m/mm | SST-2 | QNLI | CoLA | RTE | MRPC | QQP |STS-B |
|---------------------------|-----------|-----------|-------------|-------|------|------|--------|-------|-------|------|
| | F1/EM | F1/EM | Acc | Acc | Acc | MCC | Acc |Acc/F1 |Acc/F1 |P/S |
| BERT-Large | 90.9/84.1 | 81.8/79.0 | 86.6/- | 93.2 | 92.3 | 60.6 | 70.4 | 88.0/- | 91.3/- |90.0/- |
| RoBERTa-Large | 94.6/88.9 | 89.4/86.5 | 90.2/- | 96.4 | 93.9 | 68.0 | 86.6 | 90.9/- | 92.2/- |92.4/- |
| XLNet-Large | 95.1/89.7 | 90.6/87.9 | 90.8/- | 97.0 | 94.9 | 69.0 | 85.9 | 90.8/- | 92.3/- |92.5/- |
| DeBERTa-Large1 | 95.5/90.1 | 90.7/88.0 | 91.3/91.1| 96.5|95.3| 69.5| 91.0| 92.6/94.6| 92.3/- |92.8/92.5 |
| DeBERTa-XLarge1 | -/- | -/- | 91.5/91.2| 97.0 | - | - | 93.1 | 92.1/94.3 | - |92.9/92.7|
| DeBERTa-V2-XLarge1|95.8/90.8| 91.4/88.9|91.7/91.6| 97.5| 95.8|71.1|93.9|92.0/94.2|92.3/89.8|92.9/92.9|
|DeBERTa-V2-XXLarge1,2|96.1/91.4|92.2/89.7|91.7/91.9|97.2|96.0|72.0| 93.5| 93.1/94.9|92.7/90.3 |93.2/93.1 |
--------
#### Notes.
- 1 Following RoBERTa, for RTE, MRPC, STS-B, we fine-tune the tasks based on DeBERTa-Large-MNLI, DeBERTa-XLarge-MNLI, DeBERTa-V2-XLarge-MNLI, DeBERTa-V2-XXLarge-MNLI. The results of SST-2/QQP/QNLI/SQuADv2 will also be slightly improved when start from MNLI fine-tuned models, however, we only report the numbers fine-tuned from pretrained base models for those 4 tasks.
- 2 To try the XXLarge model with HF transformers, you need to specify --sharded_ddp

``bash cd transformers/examples/text-classification/ export TASK_NAME=mrpc python -m torch.distributed.launch --nproc_per_node=8 run_glue.py --model_name_or_path microsoft/deberta-v2-xxlarge \\ --task_name $TASK_NAME --do_train --do_eval --max_seq_length 128 --per_device_train_batch_size 4 \\ --learning_rate 3e-6 --num_train_epochs 3 --output_dir /tmp/$TASK_NAME/ --overwrite_output_dir --sharded_ddp --fp16

code

### Citation
If you find DeBERTa useful for your work, please cite the following paper:

 latex
@inproceedings{
he2021deberta,
title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=XPZIaotutsD}
}

``