layoutlm document qa

提供商impira
分类document-question-answering
许可证mit
下载量192.9K
星标0

简介

LayoutLM Document QA 是一款专注于文档理解的 AI 模型，它打破了传统 OCR 仅识别文字的局限，能够同时分析文本内容、字体样式和页面布局（如表格、表单位置）。简单来说，它不仅能“读”到字，还能像人一样通过位置关系理解文档结构，非常适合处理发票、合同、研报等复杂排版文档的问答任务。对于开发者而言，它将视觉信息与语义信息结合，极大提升了从非结构化文档中提取精准信息的效率，上手难度中等，可作为 RAG 流程中增强文档解析能力的底层组件。

核心亮点

  • 结合文本与空间布局，精准解析复杂文档结构
  • 擅长处理发票、表单等强格式文档的问答提取
  • 弥补纯文本 OCR 在语义理解上的空间缺失
  • MIT 协议开源，易于集成至企业级文档分析流

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("impira/layoutlm-document-qa")
tokenizer = AutoTokenizer.from_pretrained("impira/layoutlm-document-qa")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download impira/layoutlm-document-qa

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download impira/layoutlm-document-qa config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('impira/layoutlm-document-qa')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/impira/layoutlm-document-qa

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/impira/layoutlm-document-qa

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('impira/layoutlm-document-qa')
tokenizer = AutoTokenizer.from_pretrained('impira/layoutlm-document-qa')

完整文档

来源: HuggingFace

---
language: en
license: mit
pipeline_tag: document-question-answering
tags:
- layoutlm
- document-question-answering
- pdf
widget:

  • text: "What is the invoice number?"

src: "https://huggingface.co/spaces/impira/docquery/resolve/2359223c1837a7587402bda0f2643382a6eefeab/invoice.png"
  • text: "What is the purchase amount?"

src: "https://huggingface.co/spaces/impira/docquery/resolve/2359223c1837a7587402bda0f2643382a6eefeab/contract.jpeg"
---

LayoutLM for Visual Question Answering

This is a fine-tuned version of the multi-modal LayoutLM model for the task of question answering on documents. It has been fine-tuned using both the SQuAD2.0 and DocVQA datasets.

Getting started with the model

To run these examples, you must have PIL, pytesseract, and PyTorch installed in addition to transformers.

python
from transformers import pipeline

nlp = pipeline(
    "document-question-answering",
    model="impira/layoutlm-document-qa",
)


nlp(
    "https://templates.invoicehome.com/invoice-template-us-neat-750px.png",
    "What is the invoice number?"
)
{'score': 0.9943977, 'answer': 'us-001', 'start': 15, 'end': 15}


nlp(
    "https://miro.medium.com/max/787/1*iECQRIiOGTmEFLdWkVIH2g.jpeg",
    "What is the purchase amount?"
)
{'score': 0.9912159, 'answer': '$1,000,000,000', 'start': 97, 'end': 97}


nlp(
    "https://www.accountingcoach.com/wp-content/uploads/2013/10/[email protected]",
    "What are the 2020 net sales?"
)
{'score': 0.59147286, 'answer': '$ 3,750', 'start': 19, 'end': 20}

NOTE: This model and pipeline was recently landed in transformers via PR #18407 and PR #18414, so you'll need to use a recent version of transformers, for example:

bash
pip install git+https://github.com/huggingface/transformers.git@2ef774211733f0acf8d3415f9284c49ef219e991

About us

This model was created by the team at Impira.

查看官方来源