Qwen3.6 27B NVFP4

提供商unsloth
分类image-text-to-text
许可证apache-2.0
下载量35.4K
星标24

简介

Qwen3.6-27B-NVFP4 是基于阿里通义千问最新系列的量化版本，由 Unsloth 团队优化。该模型主打多模态理解，能够高效处理图文输入。得益于 NVFP4 量化技术，它在大幅降低显存占用的同时，尽可能保留了 27B 参数规模的推理能力，使得中端消费级显卡也能流畅运行。对于开发者而言，它是一个极佳的本地部署选择，既能胜任复杂的视觉分析任务，又无需昂贵的企业级计算资源，上手门槛极低。

核心亮点

  • 支持图文多模态输入，视觉理解能力强
  • NVFP4 量化显著降低显存，部署更轻量
  • 27B 参数规模在性能与速度间取得平衡
  • Apache-2.0 协议，商业使用极其友好

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("unsloth/Qwen3.6-27B-NVFP4")
tokenizer = AutoTokenizer.from_pretrained("unsloth/Qwen3.6-27B-NVFP4")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download unsloth/Qwen3.6-27B-NVFP4

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download unsloth/Qwen3.6-27B-NVFP4 config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('unsloth/Qwen3.6-27B-NVFP4')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/unsloth/Qwen3.6-27B-NVFP4

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/unsloth/Qwen3.6-27B-NVFP4

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('unsloth/Qwen3.6-27B-NVFP4')
tokenizer = AutoTokenizer.from_pretrained('unsloth/Qwen3.6-27B-NVFP4')

模型下载

我们推荐使用命令行或者 ModelScope SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 ModelScope：

操作指引 
pip install modelscope

命令行下载

下载完整模型库

下载完整模型库 
modelscope download --model unsloth/Qwen3.6-27B-NVFP4

下载单个文件到指定本地文件夹（以下载 README.md 到当前路径下 dir 目录为例）

下载单个文件到指定本地文件夹（以下载 README.md 到当前路径下 dir 目录为例） 
modelscope download --model unsloth/Qwen3.6-27B-NVFP4 README.md --local_dir ./dir

更多更丰富的命令行下载选项，可参见具体文档

SDK 下载

SDK 下载 
# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('unsloth/Qwen3.6-27B-NVFP4')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://www.modelscope.cn/unsloth/Qwen3.6-27B-NVFP4.git

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/unsloth/Qwen3.6-27B-NVFP4.git

ModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。

Notebook 快速开发

下载并安装 ModelScope library

下载并安装 ModelScope library 
pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

模型加载和推理

模型加载和推理 
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

p = pipeline('text-generation', 'unsloth/Qwen3.6-27B-NVFP4')

完整文档

来源: HuggingFace

---
library_name: transformers
license: apache-2.0
license_link: https://huggingface.co/Qwen/Qwen3.6-27B/blob/main/LICENSE
pipeline_tag: image-text-to-text
base_model:

  • Qwen/Qwen3.6-27B

tags:
  • unsloth

  • qwen

  • qwen3_5

---

Read our How to Run Qwen3.6 NVFP4 Guide!


<div>
<p style="margin: 0 0 0px 0; margin-top: 0px;">
<em>See <a href="https://unsloth.ai/docs/basics/unsloth-dynamic-v2.0-gguf">Unsloth Dynamic 2.0 GGUFs</a> for our quantization benchmarks.</em>
</p>
<div style="display: flex; gap: 5px; align-items: center; margin-bottom: 0px;">
<a href="https://github.com/unslothai/unsloth/">
<img src="https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png" width="133">
</a>
<a href="https://discord.gg/unsloth">
<img src="https://github.com/unslothai/unsloth/raw/main/images/Discord%20button.png" width="173">
</a>
<a href="https://unsloth.ai/docs/models/qwen3.6">
<img src="https://raw.githubusercontent.com/unslothai/unsloth/refs/heads/main/images/documentation%20green%20button.png" width="143">
</a>
</div>

  • 2.5x faster throughput than other NVFP4 quants.
  • This is an Unsloth NVFP4 quantized checkpoint calibrated on a mixture of our Unsloth dataset + UltraChat dataset.
  • Works on a 24GB VRAM GPU. Benchmarks on 1xB200 128 concurrency.

<img width="800px" src="https://unsloth.ai/docs/~gitbook/image?url=https%3A%2F%2F3215535692-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FxhOjnexMCB3dmuQFQ2Zq%252Fuploads%252F8zM7cg0Xgo2pAlop1iPW%252F01_qwen36_combined_throughput.png%3Falt%3Dmedia%26token%3D23131a96-4c41-42d7-aa16-5bfb21bb44a3&width=768&dpr=3&quality=100&sign=ee3c70f9&sv=2">

NVFP4 Accuracy Benchmarks

For accuracy benchmarks, we conducted MMLU-Pro, AIME 2025, GPQA for FP8, BF16, NVIDIA's NVFP4 and our NVFP4s - we show our faster quants do similarly on all:

| Provider | MMLU-Pro | GPQA | AIME 2025 |
| -------- | -------: | ----: | --------: |
| Unsloth NVFP4 | 86.25 | 86.34 | 93.12 |
| NVIDIA NVFP4 | 85.96 | 86.87 | 93.12 |
| FP8 | 86.11 | 86.87 | 93.75 |
| BF16 | 85.96 | 88.13 | 93.33 |

Read all benchmarks in our NVFP4 blog

vLLM Run Instructions

To install vLLM in a separate venv:

shell
uv venv unsloth-nvfp4-env --python 3.13
source unsloth-nvfp4-env/bin/activate
uv pip install "vllm>=0.25.0" "flashinfer-python>=0.6.13" "nvidia-cutlass-dsl>=4.5.2" \
    --torch-backend=auto

Then to serve the 27B NVFP4 quant:
code
vllm serve unsloth/Qwen3.6-27B-NVFP4 \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'

Also do NOT use the Marlin backend since it's 2x slower - use the native vLLM or cute-DSL / CUTLASS / flashinfer_trtllm backends!
| Model | backend | decode tok/s | thr out tok/s |
|---|---|---|---|
| nvidia 27B | marlin (auto) | 115.6 | 2,403 |
| unsloth 27B | cute-DSL (auto) | 125.9 | 6,863 |
| nvidia 35B-A3B | marlin (auto) | 240.8 | 8,721 |
| unsloth 35B-A3B | cute-DSL + trtllm (auto) | 295.2 | 15,636 |

DGX Spark

You must use the below or you will get 2x slower inference!

shell
export CUTE_DSL_ARCH=sm_121a
vllm serve unsloth/Qwen3.6-27B-NVFP4 --moe-backend flashinfer_b12x

Multi-Token Prediction (MTP)

This checkpoint includes the MTP module, so it can act as its own speculative draft for faster decoding, but somewhat less throughput

shell
vllm serve unsloth/Qwen3.6-27B-NVFP4 \
    --speculative-config '{"method": "mtp", "num_speculative_tokens": 2}'

![Qwen Chat](https://chat.qwen.ai)

> [!Note]
> This repository contains model weights and configuration files for the post-trained model in the Hugging Face Transformers format.
>
> These artifacts are compatible with Hugging Face Transformers, vLLM, SGLang, KTransformers, etc.

Following the February release of the Qwen3.5 series, we're pleased to share the first open-weight variant of Qwen3.6. Built on direct feedback from the community, Qwen3.6 prioritizes stability and real-world utility, offering developers a more intuitive, responsive, and genuinely productive coding experience.

Qwen3.6 Highlights

This release delivers substantial upgrades, particularly in

  • Agentic Coding: the model now handles frontend workflows and repository-level reasoning with greater fluency and precision.
  • Thinking Preservation: we've introduced a new option to retain reasoning context from historical messages, streamlining iterative development and reducing overhead.

!Benchmark Results

For more details, please refer to our blog post Qwen3.6-27B.

Model Overview

  • Type: Causal Language Model with Vision Encoder
  • Training Stage: Pre-training & Post-training
  • Language Model
- Number of Parameters: 27B - Hidden Dimension: 5120 - Token Embedding: 248320 (Padded) - Number of Layers: 64 - Hidden Layout: 16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN)) - Gated DeltaNet: - Number of Linear Attention Heads: 48 for V and 16 for QK - Head Dimension: 128 - Gated Attention: - Number of Attention Heads: 24 for Q and 4 for KV - Head Dimension: 256 - Rotary Position Embedding Dimension: 64 - Feed Forward Network: - Intermediate Dimension: 17408 - LM Output: 248320 (Padded) - MTP: trained with multi-steps
  • Context Length: 262,144 natively and extensible up to 1,010,000 tokens.

Original Qwen3.6 BF16 Reference Benchmarks

Language

<div style="font-family:-apple-system,BlinkMacSystemFont,'Segoe UI',Roboto,sans-serif;max-width:1000px;margin:0 auto;padding:16px 0">
<table style="width:100%;border-collapse:collapse;font-size:13px">
<thead><tr>
<th style="padding:10px 7px;text-align:left;font-weight:600;border-bottom:2px solid #7c3aed;color:#7c3aed"></th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #7c3aed;color:#7c3aed;font-size: 14px;">Qwen3.5-27B</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #7c3aed;color:#7c3aed;font-size: 14px;">Qwen3.5-397B-A17B</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #7c3aed;color:#7c3aed;font-size: 14px;">Gemma4-31B</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #7c3aed;color:#7c3aed;font-size: 14px;">Claude 4.5 Opus</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #7c3aed;color:#7c3aed;font-size: 14px;">Qwen3.6-35B-A3B</th><th style="padding:10px 7px;text-align:center;font-weight:500;border-bottom:2px solid #7c3aed;color:#7c3aed;font-size: 14px;">Qwen3.6-27B</th></tr></thead>
<tbody>
<tr><td colspan="7" style="padding:8px 12px;font-weight:600;color:#7c3aed;border-bottom:1px solid rgba(124, 58, 237, 0.2);background:rgba(124, 58, 237, 0.1)">Coding Agent</td></tr>
<tr>
<td style="padding:7px 7px;padding-left:20px;border-bottom:1px solid rgba(128, 128, 128, 0.15);">SWE-bench Verified</td>
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15)">75.0</td>
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15)">76.2</td>
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15)">52.0</td>
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15)">80.9</td>
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15)">73.4</td>
<td style="padding:7px 7px;text-align:center;border-bottom:1px solid rgba(128, 128, 128, 0.15)">77.2</td>
</tr>
<tr>
<td style="

查看官方来源