Overview

The mdeberta-v3-base-squad2 is a multilingual encoder model optimized for extractive question answering. Built on the DeBERTa v3 architecture, it leverages replaced token detection and disentangled attention to outperform standard BERT or RoBERTa models in understanding linguistic nuances across multiple languages. This model is specifically fine-tuned on the SQuAD 2.0 dataset, meaning it can distinguish between answerable questions and those where the provided context contains no valid answer—a critical feature for reducing hallucinations in production RAG pipelines. For developers, it offers a lightweight footprint suitable for CPU-based inference while maintaining high accuracy for reading comprehension tasks across diverse language sets.

Highlights

Multilingual support for diverse global language datasets
Fine-tuned on SQuAD 2.0 for answerability detection
Efficient DeBERTa v3 architecture for lower latency
Ideal for extractive QA and RAG system integration
Permissive MIT license for commercial application

Usage

Install

# Install Hugging Face transformers
pip install transformers torch

SDK Usage

# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("timpal0l/mdeberta-v3-base-squad2")
tokenizer = AutoTokenizer.from_pretrained("timpal0l/mdeberta-v3-base-squad2")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance

pip install -U huggingface_hub

CLI Download

Download the full repository

huggingface-cli download timpal0l/mdeberta-v3-base-squad2

Download a single file to a local folder (e.g. config.json into ./dir)

huggingface-cli download timpal0l/mdeberta-v3-base-squad2 config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('timpal0l/mdeberta-v3-base-squad2')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://huggingface.co/timpal0l/mdeberta-v3-base-squad2

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/timpal0l/mdeberta-v3-base-squad2

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

pip install -U transformers torch

Load the model and run inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('timpal0l/mdeberta-v3-base-squad2')
tokenizer = AutoTokenizer.from_pretrained('timpal0l/mdeberta-v3-base-squad2')

Full Documentation

来源: HuggingFace

---
datasets:

squad_v2

language:

multilingual

af

am

ar

as

az

be

bg

bn

br

bs

ca

cs

cy

da

de

el

en

eo

es

et

eu

fa

fi

fr

fy

ga

gd

gl

gu

ha

he

hi

hr

hu

hy

id

is

it

ja

jv

ka

kk

km

kn

ko

ku

ky

la

lo

lt

lv

mg

mk

ml

mn

mr

ms

my

ne

nl

'no'

om

or

pa

pl

ps

pt

ro

ru

sa

sd

si

sk

sl

so

sq

sr

su

sv

sw

ta

te

th

tl

tr

ug

uk

ur

uz

vi

xh

yi

zh

tags:

deberta

deberta-v3

mdeberta

question-answering

qa

multilingual

thumbnail: https://huggingface.co/front/thumbnails/microsoft.png
license: mit
base_model:

microsoft/mdeberta-v3-base

---

This model can be used for Extractive QA

It has been finetuned for 3 epochs on SQuAD2.0.

Usage

python

from transformers import pipeline
qa_model = pipeline("question-answering", "timpal0l/mdeberta-v3-base-squad2")
question = "Where do I live?"
context = "My name is Tim and I live in Sweden."
qa_model(question = question, context = context)
{'score': 0.975547730922699, 'start': 28, 'end': 36, 'answer': ' Sweden.'}

Evaluation on SQuAD2.0 dev set

bash

{
    "epoch": 3.0,
    "eval_HasAns_exact": 79.65587044534414,
    "eval_HasAns_f1": 85.91387795001529,
    "eval_HasAns_total": 5928,
    "eval_NoAns_exact": 82.10260723296888,
    "eval_NoAns_f1": 82.10260723296888,
    "eval_NoAns_total": 5945,
    "eval_best_exact": 80.8809904826076,
    "eval_best_exact_thresh": 0.0,
    "eval_best_f1": 84.00551406448994,
    "eval_best_f1_thresh": 0.0,
    "eval_exact": 80.8809904826076,
    "eval_f1": 84.00551406449004,
    "eval_samples": 12508,
    "eval_total": 11873,
    "train_loss": 0.7729689576483615,
    "train_runtime": 9118.953,
    "train_samples": 134891,
    "train_samples_per_second": 44.377,
    "train_steps_per_second": 0.925
}

DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. With those two improvements, DeBERTa out perform RoBERTa on a majority of NLU tasks with 80GB training data.

In DeBERTa V3, we further improved the efficiency of DeBERTa using ELECTRA-Style pre-training with Gradient Disentangled Embedding Sharing. Compared to DeBERTa, our V3 version significantly improves the model performance on downstream tasks. You can find more technique details about the new model from our paper.

Please check the official repository for more implementation details and updates.

mDeBERTa is multilingual version of DeBERTa which use the same structure as DeBERTa and was trained with CC100 multilingual data.
The mDeBERTa V3 base model comes with 12 layers and a hidden size of 768. It has 86M backbone parameters with a vocabulary containing 250K tokens which introduces 190M parameters in the Embedding layer. This model was trained using the 2.5T CC100 data as XLM-R.

mdeberta v3 base squad2

Overview

Highlights

Usage

Hugging Face Download

CLI Download

SDK Download

Git Download

PyTorch / Transformers Usage

Full Documentation

This model can be used for Extractive QA

Usage

`{'score': 0.975547730922699, 'start': 28, 'end': 36, 'answer': ' Sweden.'}`

Evaluation on SQuAD2.0 dev set

DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing