NuExtract3

---
license: apache-2.0
license_link: https://huggingface.co/numind/NuExtract3/blob/main/LICENSE
library_name: transformers
pipeline_tag: image-to-text
tags:

• image-text-to-text

• transformers

• safetensors

• qwen3_5

• vision-language

• vlm

• document-understanding

• structured-extraction

• information-extraction

• ocr

• document-to-markdown

• markdown

• rag

• reasoning

• multilingual

• conversational

• Qwen/Qwen3.5-4B

<p align="center">
<a href="https://nuextract.ai/">
<img src="header.svg" width="900px"/>
</a>
</p>

<p align="center">
🖥️ <a href="https://nuextract.ai/">API / Platform</a>&nbsp;&nbsp; | &nbsp;&nbsp;
📑 <a href="https://numind.ai/blog">Blog</a>&nbsp;&nbsp; | &nbsp;&nbsp;
🗣️ <a href="https://discord.gg/3tsEtJNCDe">Discord</a>&nbsp;&nbsp; | &nbsp;&nbsp;
🛠️ <a href="https://github.com/numindai/nuextract">GitHub</a>
</p>

NuExtract3 is a unified 4B vision-language reasoning model for document understanding.

It combines strong structured information extraction with high-quality image-to-Markdown conversion, making it suitable for extraction pipelines, OCR, and RAG preprocessing for all types of documents such as scans, receipts, forms, invoices, contracts or tables.

Try it out in the 🤗 space!

Overview

• Structured extraction: input (text/images) + JSON template + instructions --> JSON output

• Markdown conversion: input (text/images) --> Markdown

• Multimodal inputs: text, images, or text + images.

• Multilingual documents.

• Reasoning and non-reasoning inference modes.

• Template generation for structured extraction from natural language or input document.

Benchmark results

Structured Extraction

We benchmarked NuExtract on NuMind's internal structured benchmark, measuring model's performances on ~600 documents of diverse types including invoices, movie posters or floor plans. These documents and their ground-truth cover diverse use-cases testing model visual understanding, OCR, reasoning and ability to handle long input and output contexts.
We plan to open-source this benchmark in the coming weeks, along with a extensive leaderboard including most popular open-weight and closed-sourced APIs and a Python library allowing to easily measure model performances on structured extraction.

<img src="st.svg" width="1000"/>

To measure a pair of predicted and ground-truth JSONs, we represent both as trees which we align based on node names, compute metric scores for aligned leaves and report the average of these scores. string and verbatim-string leaves are evaluated with indel distance (i.e. Levenshtein without replacement), while all others are evaluated with exact-match.
Models were evaluated using vllm, with a temperature of 0.25 and a maximum of 65000 output token (for both thinking and answer), which largely exceeds 22000 which is the number of tokens of the largest ground truth output.

<figure>

|Model name |Average score|Num. failed⁽¹⁾|Avg. num tokens thinking|Avg. num tokens answer|
|--------------------|-------------|-----------|------------------------|----------------------|
|NuExtract3.4_4B-RL |0.651 ± 0.019|27 |2036 |1856 |
|gemma-4-E4B-it |0.538 ± 0.023|31 |3005 |1287 |
|Qwen3.5-9B |0.479 ± 0.030|170 |22409 |1257 |
|Qwen3.5-4B |0.417 ± 0.031|229 |27177 |1201 |
|GLM-4.6V-Flash |0.435 ± 0.026|153 |2989 |1357 |
|Nemotron-3-Nano-Omni|0.387 ± 0.028|204 |25827 |522 |
|Ministral-3-3B |0.240 ± 0.022|344 |27586 |362 |

<figcaption>
<small>
(1) number of model outputs that were not JSON deserializable, either directly or by removing leading and trailing backticks.<br>
95% confidence intervals computed using a nonparametric bootstrap over scores distributions.
</small>
</figcaption>
</figure>

The benchmark include samples containing multiple images resulting in large input context, and some with ground-truth containing large numbers of items to extract resulting in large outputs. We found that the reasoning of small models significantly negatively impact their performances. The reason is that many models ended up falling in repetition loops, hitting the output tokens limit and resulting in failed requests.

Document to Markdown

NuExtract can also convert document images into clean Markdown. Output will be Markdown for text (headers etc), HTML for tables, LaTeX for math and ``<figure data-type="image" data-id="img_n"><img src="/NM-dev/model_card-A/resolve/main/img_n.png" alt="Detail description of the images"/>

Modern, format-agnostic benchmarks for complex document understanding are limited, so we explored a new evaluation approach.
We selected 100 documents with challenging layouts and tables, asked each model to convert them into a structured representation, then used Gemini 3 Flash to compare model outputs against the source document and choose the most accurate result.
The rankings aligned with human votes, suggesting this is a promising method for evaluating document-to-Markdown capabilities. More details will be shared in an upcoming technical report.
Here are some results:

<img src="ocr_preferences.svg" width="1000"/>
Using "Markdown-to-structured"
To add other evaluate references, we used our structured extraction benchmark to evaluate models in a two-step fashion: convert the benchmark inputs to Markdown, then use Qwen3.6 27B to perform the structured extraction task on them. Intuitively, it allows to evaluate how models achieve to keep the input document content and layout: good models will allow the "structured extractor" model to perform better scores.
<img src="md2st.svg" width="1000"/>
Using NuExtract
Structured extraction
Structured extraction takes as inputs:
1. An input document, which can be text, image, or both;
2. A JSON template describing the information to extract;
3. (Optional) Instructions, allowing to specify expected output formats or values, to provide with the instructions chat template kwarg;
4. (Optional) In-Context Learning (ICL) examples.

Input JSON template
NuExtract uses a input JSON template whose structure is identical to the output JSON. Its leaf values are specify the types of the output JSON leaves. For examples:

Supported template types include:

verbatim-string: extract text exactly as it appears in the document;

string: generic string field, allowing abstraction or light paraphrasing;

integer: whole number;

number: integer or decimal number;

date-time: ISO-8601 date, time or date-time;

Other specific types such as data, time, country, currency, email and so on.

For more details, read the complete types specifications and examples
Template constructors:
Arrays, for example ["string"];

Enums, for example ["yes", "no", "maybe"];

Multi-enums (multiple possible values), for example [["A", "B", "C"]].

If the model does not find relevant information for a field, it returns null or [].

Converting JSON schema / Pydantic models to NuExtract template
Our Python SDK (pip install numind`) offers a method to convert JSON schemas to NuExtract templates: