简介

Qwen3-VL-8B-Instruct-NVFP4 是阿里通义千问最新视觉语言模型的量化版本。该模型主打多模态理解，能够精准识别图像中的细节、阅读复杂图表并进行逻辑推理。由于采用了 NVFP4 量化技术，它在保持强大视觉分析能力的同时，大幅降低了显存占用并提升了推理速度，使得在消费级显卡上流畅运行 8B 参数规模的视觉模型成为可能。对于需要处理文档数字化、图像问答或自动化视觉分析的开发者来说，这是一个兼顾性能与部署成本的理想选择。

核心亮点

强悍的视觉理解与图表分析能力
NVFP4 量化大幅降低显存占用
推理速度快，适配消费级硬件
Apache-2.0 协议，企业部署友好

使用方法

安装依赖

# 安装 Hugging Face transformers
pip install transformers torch

SDK 使用

# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4")
tokenizer = AutoTokenizer.from_pretrained("JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引

pip install -U huggingface_hub

命令行下载

下载完整模型库

huggingface-cli download JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

huggingface-cli download JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4 config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4')

Git 下载

请确保 lfs 已经被正确安装

Git 下载

git lfs install
git clone https://huggingface.co/JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

pip install -U transformers torch

模型加载和推理

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4')
tokenizer = AutoTokenizer.from_pretrained('JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4')

完整文档

来源: HuggingFace

---
license: apache-2.0
base_model: Qwen/Qwen3-VL-8B-Instruct
tags:

vision-language

multimodal

qwen3-vl

nvfp4

quantization

llmcompressor

smoothquant

pipeline_tag: image-to-text
---

Qwen3VL-8B-Instruct-NVFP4

This is an NVFP4 quantized version of Qwen3-VL-8B-Instruct, a powerful vision-language model for multimodal understanding and generation tasks.

Model Details

Base Model

Base Model: Qwen/Qwen3-VL-8B-Instruct

Architecture: Qwen3VLForConditionalGeneration

Model Type: Vision-Language Model (VLM)

Quantization Details

Quantization Method: NVFP4 with SmoothQuant

Quantization Tool: llmcompressor

Smoothing Strength: 0.8

Calibration Dataset: lmms-lab/flickr30k (512 samples from test split)

Max Sequence Length: 32,768 tokens

Quantization Configuration

Weight Quantization: NVFP4 (4-bit NVIDIA Floating Point)

- Strategy: Channel-wise - Observer: MinMax - Symmetric: True

Activation Quantization: NVFP4

- Strategy: Token-wise - Dynamic scaling: Enabled - Symmetric: True

Excluded Modules

The following modules were excluded from quantization to maintain model quality:

lm_head (language model head)

Visual encoder modules (model.visual.*)

MLP gate projections (.*mlp.gate$)

Model Use

Installation

bash

pip install transformers torch qwen-vl-utils pillow

Basic Usage

python

from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
from PIL import Image
import requests
Load model and processor
model = Qwen3VLForConditionalGeneration.from_pretrained(
    "JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4",
    torch_dtype="auto",
    device_map="auto"
)
processor = AutoProcessor.from_pretrained("JEILDLWLMRA/Qwen3-VL-8B-Instruct-NVFP4")
Prepare inputs
image_url = "http://images.cocodataset.org/train2017/000000231895.jpg"
image = Image.open(requests.get(image_url, stream=True).raw)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image_url},
            {"type": "text", "text": "What does the image show?"},
        ],
    }
]
Process and generate
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(
    text=[prompt],
    images=[image],
    padding=False,
    return_tensors="pt",
).to(model.device)
output = model.generate(inputs, max_new_tokens=100, temperature=0.7)
generated_text = processor.decode(output[0], skip_special_tokens=True)
print(generated_text)

Using with vLLM

For faster inference, you can use this model with vLLM:

python
from vllm import LLM from vllm.multimodal.utils import encode_image_base64 from PIL import Image import base64 from io import BytesIO Initialize vLLM engine llm = LLM( model="JEILDLWLMRA/Qwen3-VL-8B-Instruct-NVFP4", max_model_len=8192, limit_mm_per_prompt={"image": 1, "video": 0}, trust_remote_code=True, ) Prepare image image = Image.open("path/to/image.jpg") buffered = BytesIO() image.save(buffered, format="PNG") img_str = base64.b64encode(buffered.getvalue()).decode() Generate messages = [ { "role": "user", "content": [ {"type": "image", "image": f"data:image/png;base64,{img_str}"}, {"type": "text", "text": "Describe this image."}, ], } ]
outputs = llm.chat(messages, max_tokens=100) print(outputs[0].outputs[0].text)

Performance

Memory Benefits

Reduced Memory Footprint: NVFP4 quantization significantly reduces model size and memory requirements (approximately 80% reduction compared to BF16)

Faster Inference: Lower precision enables faster computation on modern GPUs with NVFP4 support (2.5-3.5x speedup)

Optimized for NVIDIA Hardware: NVFP4 is optimized for NVIDIA GPUs, especially Jetson Thor platform

Quality
This quantized model maintains high quality for vision-language tasks while significantly reducing memory usage. The SmoothQuant technique helps preserve model accuracy during quantization. Typical quality degradation is 2-5% compared to the full-precision model.
Training Details

Quantization Process
1. Calibration: Used 512 samples from the flickr30k test dataset 2. SmoothQuant: Applied with smoothing strength of 0.8 to improve quantization quality 3. Sequential Processing**: Applied quantization sequentially to Qwen3VLTextDecoderLayer modules

Hardware

Quantization was performed on NVIDIA GPUs with CUDA support

Optimized for NVIDIA Jetson Thor platform

Limitations

This is a quantized model, so there may be slight quality degradation (2-5%) compared to the full-precision base model

NVFP4 support requires compatible hardware (e.g., NVIDIA H100, A100, Jetson Thor with appropriate CUDA versions)

Maximum sequence length is limited to 32,768 tokens

Citation

If you use this model, please cite the original Qwen3-VL model:

bibtex

@article{qwen3vl,
  title={Qwen3-VL: A Versatile Vision-Language Model},
  author={Qwen Team},
  journal={arXiv preprint},
  year={2024}
}

License

This model inherits the license from the base model Qwen/Qwen3-VL-8B-Instruct. Please refer to the original model's license for details.

Acknowledgments

Base model: Qwen Team

Quantization tool: llmcompressor by vLLM Project

Calibration dataset: flickr30k

NVFP4 format: NVIDIA

Qwen3 VL 8B Instruct NVFP4

简介

核心亮点

使用方法

Hugging Face 下载

命令行下载

SDK 下载

Git 下载

PyTorch / Transformers 使用

完整文档

Qwen3VL-8B-Instruct-NVFP4

Model Details

Base Model

Quantization Details

Quantization Configuration

Excluded Modules

Model Use

Installation

Basic Usage

Load model and processor

Prepare inputs

Process and generate

Using with vLLM

Initialize vLLM engine

Prepare image

Generate

Performance

Memory Benefits

Quality

Training Details

Quantization Process

Hardware

Limitations

Citation

License

Acknowledgments

扫码进交流群