Name: layoutlm document qa
Author: impira

Overview

LayoutLM is a specialized transformer model designed for Document AI, bridging the gap between raw text and visual spatial awareness. Unlike standard LLMs that process text linearly, LayoutLM integrates layout coordinates (bounding boxes) and image features, making it ideal for parsing complex documents like invoices, receipts, and forms. For developers, this means significantly higher accuracy in Question Answering (QA) tasks where the answer depends on the physical position of text on a page. It integrates well into OCR pipelines, acting as the intelligence layer that interprets the structured output of tools like Tesseract or AWS Textract to extract specific data points without needing rigid templates.

Highlights

Combines text, layout, and image embeddings for spatial awareness
Optimized for structured document QA and information extraction
Seamlessly integrates with existing OCR preprocessing pipelines
Open-source MIT license for flexible commercial deployment

Usage

Install

# Install Hugging Face transformers
pip install transformers torch

SDK Usage

# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("impira/layoutlm-document-qa")
tokenizer = AutoTokenizer.from_pretrained("impira/layoutlm-document-qa")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance

pip install -U huggingface_hub

CLI Download

Download the full repository

huggingface-cli download impira/layoutlm-document-qa

Download a single file to a local folder (e.g. config.json into ./dir)

huggingface-cli download impira/layoutlm-document-qa config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('impira/layoutlm-document-qa')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://huggingface.co/impira/layoutlm-document-qa

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/impira/layoutlm-document-qa

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

pip install -U transformers torch

Load the model and run inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('impira/layoutlm-document-qa')
tokenizer = AutoTokenizer.from_pretrained('impira/layoutlm-document-qa')

Full Documentation

来源: HuggingFace

---
language: en
license: mit
pipeline_tag: document-question-answering
tags:
- layoutlm
- document-question-answering
- pdf
widget:

text: "What is the invoice number?"

src: "https://huggingface.co/spaces/impira/docquery/resolve/2359223c1837a7587402bda0f2643382a6eefeab/invoice.png"

text: "What is the purchase amount?"

src: "https://huggingface.co/spaces/impira/docquery/resolve/2359223c1837a7587402bda0f2643382a6eefeab/contract.jpeg"
---

LayoutLM for Visual Question Answering

This is a fine-tuned version of the multi-modal LayoutLM model for the task of question answering on documents. It has been fine-tuned using both the SQuAD2.0 and DocVQA datasets.

Getting started with the model

To run these examples, you must have PIL, pytesseract, and PyTorch installed in addition to transformers.

python

from transformers import pipeline
nlp = pipeline(
    "document-question-answering",
    model="impira/layoutlm-document-qa",
)
nlp(
    "https://templates.invoicehome.com/invoice-template-us-neat-750px.png",
    "What is the invoice number?"
)
{'score': 0.9943977, 'answer': 'us-001', 'start': 15, 'end': 15}

nlp(
    "https://miro.medium.com/max/787/1*iECQRIiOGTmEFLdWkVIH2g.jpeg",
    "What is the purchase amount?"
)
{'score': 0.9912159, 'answer': '$1,000,000,000', 'start': 97, 'end': 97}

nlp(
    "https://www.accountingcoach.com/wp-content/uploads/2013/10/[email protected]",
    "What are the 2020 net sales?"
)
{'score': 0.59147286, 'answer': '$ 3,750', 'start': 19, 'end': 20}

NOTE: This model and pipeline was recently landed in transformers via PR #18407 and PR #18414, so you'll need to use a recent version of transformers, for example:

bash

pip install git+https://github.com/huggingface/transformers.git@2ef774211733f0acf8d3415f9284c49ef219e991

About us

This model was created by the team at Impira.