Overview

Qwen3 VL 8B Instruct NVFP4 is a vision-language model optimized for high-throughput deployment via 4-bit Normal Float (NVFP4) quantization. Designed for developers who need a balance between visual reasoning and inference speed, this model excels at complex image-to-text tasks, including OCR, document parsing, and spatial understanding. By utilizing NVFP4, it significantly reduces VRAM overhead compared to full-precision weights without a substantial hit to accuracy, making it ideal for integration into edge servers or resource-constrained cloud environments. It serves as a drop-in replacement for standard 8B VL models where latency and memory efficiency are critical KPIs for the production pipeline.

Highlights

NVFP4 quantization for reduced VRAM and faster inference
Strong multimodal capabilities in OCR and visual reasoning
Optimized for high-throughput image-to-text production workloads
Apache-2.0 license ensuring flexible commercial integration

Usage

Install

# Install Hugging Face transformers
pip install transformers torch

SDK Usage

# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4")
tokenizer = AutoTokenizer.from_pretrained("JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance

pip install -U huggingface_hub

CLI Download

Download the full repository

huggingface-cli download JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4

Download a single file to a local folder (e.g. config.json into ./dir)

huggingface-cli download JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4 config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://huggingface.co/JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

pip install -U transformers torch

Load the model and run inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4')
tokenizer = AutoTokenizer.from_pretrained('JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4')

Full Documentation

来源: HuggingFace

---
license: apache-2.0
base_model: Qwen/Qwen3-VL-8B-Instruct
tags:

vision-language

multimodal

qwen3-vl

nvfp4

quantization

llmcompressor

smoothquant

pipeline_tag: image-to-text
---

Qwen3VL-8B-Instruct-NVFP4

This is an NVFP4 quantized version of Qwen3-VL-8B-Instruct, a powerful vision-language model for multimodal understanding and generation tasks.

Model Details

Base Model

Base Model: Qwen/Qwen3-VL-8B-Instruct

Architecture: Qwen3VLForConditionalGeneration

Model Type: Vision-Language Model (VLM)

Quantization Details

Quantization Method: NVFP4 with SmoothQuant

Quantization Tool: llmcompressor

Smoothing Strength: 0.8

Calibration Dataset: lmms-lab/flickr30k (512 samples from test split)

Max Sequence Length: 32,768 tokens

Quantization Configuration

Weight Quantization: NVFP4 (4-bit NVIDIA Floating Point)

- Strategy: Channel-wise - Observer: MinMax - Symmetric: True

Activation Quantization: NVFP4

- Strategy: Token-wise - Dynamic scaling: Enabled - Symmetric: True

Excluded Modules

The following modules were excluded from quantization to maintain model quality:

lm_head (language model head)

Visual encoder modules (model.visual.*)

MLP gate projections (.*mlp.gate$)

Model Use

Installation

bash

pip install transformers torch qwen-vl-utils pillow

Basic Usage

python

from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
from PIL import Image
import requests
Load model and processor
model = Qwen3VLForConditionalGeneration.from_pretrained(
    "JEILDLWLRMA/Qwen3-VL-8B-Instruct-NVFP4",
    torch_dtype="auto",
    device_map="auto"
)
processor = AutoProcessor.from_pretrained("JEILDLWLMRA/Qwen3-VL-8B-Instruct-NVFP4")
Prepare inputs
image_url = "http://images.cocodataset.org/train2017/000000231895.jpg"
image = Image.open(requests.get(image_url, stream=True).raw)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": image_url},
            {"type": "text", "text": "What does the image show?"},
        ],
    }
]
Process and generate
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(
    text=[prompt],
    images=[image],
    padding=False,
    return_tensors="pt",
).to(model.device)
output = model.generate(inputs, max_new_tokens=100, temperature=0.7)
generated_text = processor.decode(output[0], skip_special_tokens=True)
print(generated_text)

Using with vLLM

For faster inference, you can use this model with vLLM:

python
from vllm import LLM from vllm.multimodal.utils import encode_image_base64 from PIL import Image import base64 from io import BytesIO Initialize vLLM engine llm = LLM( model="JEILDLWLMRA/Qwen3-VL-8B-Instruct-NVFP4", max_model_len=8192, limit_mm_per_prompt={"image": 1, "video": 0}, trust_remote_code=True, ) Prepare image image = Image.open("path/to/image.jpg") buffered = BytesIO() image.save(buffered, format="PNG") img_str = base64.b64encode(buffered.getvalue()).decode() Generate messages = [ { "role": "user", "content": [ {"type": "image", "image": f"data:image/png;base64,{img_str}"}, {"type": "text", "text": "Describe this image."}, ], } ]
outputs = llm.chat(messages, max_tokens=100) print(outputs[0].outputs[0].text)

Performance

Memory Benefits

Reduced Memory Footprint: NVFP4 quantization significantly reduces model size and memory requirements (approximately 80% reduction compared to BF16)

Faster Inference: Lower precision enables faster computation on modern GPUs with NVFP4 support (2.5-3.5x speedup)

Optimized for NVIDIA Hardware: NVFP4 is optimized for NVIDIA GPUs, especially Jetson Thor platform

Quality
This quantized model maintains high quality for vision-language tasks while significantly reducing memory usage. The SmoothQuant technique helps preserve model accuracy during quantization. Typical quality degradation is 2-5% compared to the full-precision model.
Training Details

Quantization Process
1. Calibration: Used 512 samples from the flickr30k test dataset 2. SmoothQuant: Applied with smoothing strength of 0.8 to improve quantization quality 3. Sequential Processing**: Applied quantization sequentially to Qwen3VLTextDecoderLayer modules

Hardware

Quantization was performed on NVIDIA GPUs with CUDA support

Optimized for NVIDIA Jetson Thor platform

Limitations

This is a quantized model, so there may be slight quality degradation (2-5%) compared to the full-precision base model

NVFP4 support requires compatible hardware (e.g., NVIDIA H100, A100, Jetson Thor with appropriate CUDA versions)

Maximum sequence length is limited to 32,768 tokens

Citation

If you use this model, please cite the original Qwen3-VL model:

bibtex

@article{qwen3vl,
  title={Qwen3-VL: A Versatile Vision-Language Model},
  author={Qwen Team},
  journal={arXiv preprint},
  year={2024}
}

License

This model inherits the license from the base model Qwen/Qwen3-VL-8B-Instruct. Please refer to the original model's license for details.

Acknowledgments

Base model: Qwen Team

Quantization tool: llmcompressor by vLLM Project

Calibration dataset: flickr30k

NVFP4 format: NVIDIA