distilbart mnli 12 1

Providervalhalla
Categoryzero-shot-classification
LicenseApache-2.0
Downloads82.6K
Stars0

Overview

DistilBART-MNLI-12-1 is a streamlined, encoder-decoder model optimized for zero-shot text classification. By leveraging a distillation of BART trained on the Multi-Genre Natural Language Inference (MNLI) dataset, it allows developers to categorize text into arbitrary labels without requiring task-specific training data. This makes it an ideal choice for rapid prototyping or deploying classification pipelines where labels change frequently. Compared to full-sized BART or RoBERTa models, it offers a significantly lower memory footprint and faster inference latency while maintaining competitive accuracy for NLI-based classification tasks. It integrates easily into standard Hugging Face pipelines, making it a plug-and-play solution for sentiment analysis, topic tagging, and intent recognition.

Highlights

  • Efficient zero-shot classification without task-specific fine-tuning
  • Reduced latency and memory usage via model distillation
  • Seamless integration with Hugging Face Transformers library
  • Apache-2.0 license for flexible commercial deployment
  • Strong performance on MNLI-based natural language inference

Usage

Install
# Install Hugging Face transformers
pip install transformers torch
SDK Usage
# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("valhalla/distilbart-mnli-12-1")
tokenizer = AutoTokenizer.from_pretrained("valhalla/distilbart-mnli-12-1")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance 
pip install -U huggingface_hub

CLI Download

Download the full repository

Download the full repository 
huggingface-cli download valhalla/distilbart-mnli-12-1

Download a single file to a local folder (e.g. config.json into ./dir)

Download a single file to a local folder (e.g. config.json into ./dir) 
huggingface-cli download valhalla/distilbart-mnli-12-1 config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('valhalla/distilbart-mnli-12-1')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://huggingface.co/valhalla/distilbart-mnli-12-1

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/valhalla/distilbart-mnli-12-1

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

Install Transformers 
pip install -U transformers torch

Load the model and run inference

Load the model and run inference 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('valhalla/distilbart-mnli-12-1')
tokenizer = AutoTokenizer.from_pretrained('valhalla/distilbart-mnli-12-1')

Full Documentation

来源: HuggingFace

---
datasets:

  • mnli

tags:
  • distilbart

  • distilbart-mnli

pipeline_tag: zero-shot-classification
---

DistilBart-MNLI

distilbart-mnli is the distilled version of bart-large-mnli created using the No Teacher Distillation technique proposed for BART summarisation by Huggingface, here.

We just copy alternating layers from bart-large-mnli and finetune more on the same data.

| | matched acc | mismatched acc |
| ------------------------------------------------------------------------------------ | ----------- | -------------- |
| bart-large-mnli (baseline, 12-12) | 89.9 | 90.01 |
| distilbart-mnli-12-1 | 87.08 | 87.5 |
| distilbart-mnli-12-3 | 88.1 | 88.19 |
| distilbart-mnli-12-6 | 89.19 | 89.01 |
| distilbart-mnli-12-9 | 89.56 | 89.52 |

This is a very simple and effective technique, as we can see the performance drop is very little.

Detailed performace trade-offs will be posted in this sheet.

Fine-tuning

If you want to train these models yourself, clone the distillbart-mnli repo and follow the steps below

Clone and install transformers from source

bash
git clone https://github.com/huggingface/transformers.git
pip install -qqq -U ./transformers

Download MNLI data

bash
python transformers/utils/download_glue_data.py --data_dir glue_data --tasks MNLI

Create student model

bash
python create_student.py \
  --teacher_model_name_or_path facebook/bart-large-mnli \
  --student_encoder_layers 12 \
  --student_decoder_layers 6 \
  --save_path student-bart-mnli-12-6 \

Start fine-tuning

bash
python run_glue.py args.json

You can find the logs of these trained models in this wandb project.

View official source
Join our Telegram