MiniCPM V 4 5 GGUF

Providersecond-state
Categoryvisual-question-answering
LicenseApache-2.0
Downloads2.2K
Stars0

Overview

MiniCPM-V 2.6 (provided here in GGUF format) is a high-efficiency multimodal model designed for edge deployment and resource-constrained environments. Unlike massive proprietary VLMs, this model balances strong visual understanding with a small parameter footprint, making it ideal for local integration via llama.cpp. It excels at high-resolution image analysis, OCR, and visual question answering without requiring massive GPU clusters. For developers, the GGUF quantization ensures compatibility across diverse hardware, allowing for seamless integration into desktop or mobile applications where low latency and privacy are priorities. It serves as a practical alternative for those needing GPT-4V-like capabilities in a self-hosted, open-weight package.

Highlights

  • Optimized GGUF format for efficient local CPU/GPU inference
  • Strong high-resolution image understanding and precise OCR
  • Low memory footprint ideal for edge device deployment
  • Apache-2.0 license allowing flexible commercial integration
  • Competitive visual reasoning performance relative to model size

Usage

Install
# Install Hugging Face transformers
pip install transformers torch
SDK Usage
# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("second-state/MiniCPM-V-4_5-GGUF")
tokenizer = AutoTokenizer.from_pretrained("second-state/MiniCPM-V-4_5-GGUF")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance 
pip install -U huggingface_hub

CLI Download

Download the full repository

Download the full repository 
huggingface-cli download second-state/MiniCPM-V-4_5-GGUF

Download a single file to a local folder (e.g. config.json into ./dir)

Download a single file to a local folder (e.g. config.json into ./dir) 
huggingface-cli download second-state/MiniCPM-V-4_5-GGUF config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

SDK Download 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('second-state/MiniCPM-V-4_5-GGUF')

Git Download

Make sure git-lfs is installed first

Git Download 
git lfs install
git clone https://huggingface.co/second-state/MiniCPM-V-4_5-GGUF

To skip LFS large-file downloads, use:

Skip LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/second-state/MiniCPM-V-4_5-GGUF

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

Install Transformers 
pip install -U transformers torch

Load the model and run inference

Load the model and run inference 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('second-state/MiniCPM-V-4_5-GGUF')
tokenizer = AutoTokenizer.from_pretrained('second-state/MiniCPM-V-4_5-GGUF')

Full Documentation

来源: HuggingFace

---
base_model: openbmb/MiniCPM-V-4_5
model_creator: openbmb
model_name: MiniCPM-V-4_5
quantized_by: Second State Inc.
pipeline_tag: visual-question-answering
language:

  • en

  • zh

---

<!-- header start -->
<!-- 200823 -->
<div style="width: auto; margin-left: auto; margin-right: auto">
<img src="https://github.com/LlamaEdge/LlamaEdge/raw/dev/assets/logo.svg" style="width: 100%; min-width: 400px; display: block; margin: auto;">
</div>
<hr style="margin-top: 1.0em; margin-bottom: 1.0em;">
<!-- header end -->

MiniCPM-V-4_5-GGUF



Original Model



openbmb/MiniCPM-V-4_5

Run with LlamaEdge



  • LlamaEdge version: coming soon


<!-- - LlamaEdge version: v0.25.1 and above -->

  • Prompt template


- Prompt type: minicpmv

- Prompt string

``text
<|system|>
{system_message}<|end|>
<|user|>
{user_message_1}<|end|>
<|assistant|>
{assistant_message_1}<|end|>
<|user|>
{user_message_2}<|end|>
<|assistant|>
`

The {user_message_n} has the format: {image_base64_encoding_string}\n{user_question}.

  • Context size: 128000


  • Run as LlamaEdge service


`bash
wasmedge --dir .:. \
--nn-preload default:GGML:AUTO:MiniCPM-V-4_5-Q5_K_M.gguf \
llama-api-server.wasm \
--prompt-template minicpmv \
--ctx-size 128000 \
--llava-mmproj MiniCPM-V-4_5-mmproj-f16.gguf \
--model-name minicpmv-26
``

Quantized GGUF Models



| Name | Quant method | Bits | Size | Use case |
| ---- | ---- | ---- | ---- | ----- |
| MiniCPM-V-4_5-Q2_K.gguf | Q2_K | 2 | 3.28 GB| smallest, significant quality loss - not recommended for most purposes |
| MiniCPM-V-4_5-Q3_K_L.gguf | Q3_K_L | 3 | 4.43 GB| small, substantial quality loss |
| MiniCPM-V-4_5-Q3_K_M.gguf | Q3_K_M | 3 | 4.12 GB| very small, high quality loss |
| MiniCPM-V-4_5-Q3_K_S.gguf | Q3_K_S | 3 | 3.77 GB| very small, high quality loss |
| MiniCPM-V-4_5-Q4_0.gguf | Q4_0 | 4 | 4.77 GB| legacy; small, very high quality loss - prefer using Q3_K_M |
| MiniCPM-V-4_5-Q4_K_M.gguf | Q4_K_M | 4 | 5.03 GB| medium, balanced quality - recommended |
| MiniCPM-V-4_5-Q4_K_S.gguf | Q4_K_S | 4 | 4.80 GB| small, greater quality loss |
| MiniCPM-V-4_5-Q5_0.gguf | Q5_0 | 5 | 5.72 GB| legacy; medium, balanced quality - prefer using Q4_K_M |
| MiniCPM-V-4_5-Q5_K_M.gguf | Q5_K_M | 5 | 5.85 GB| large, very low quality loss - recommended |
| MiniCPM-V-4_5-Q5_K_S.gguf | Q5_K_S | 5 | 5.72 GB| large, low quality loss - recommended |
| MiniCPM-V-4_5-Q6_K.gguf | Q6_K | 6 | 6.72 GB| very large, extremely low quality loss |
| MiniCPM-V-4_5-Q8_0.gguf | Q8_0 | 8 | 8.71 GB| very large, extremely low quality loss - not recommended |
| MiniCPM-V-4_5-f16.gguf | f16 | 16 | 16.4 GB| |
| MiniCPM-V-4_5-mmproj-f16.gguf | f16 | 16 | 1.10 GB| |

*Quantized with llama.cpp b6301.*

View official source
Join our Telegram