pix2struct ai2d base

提供商google
分类visual-question-answering
许可证apache-2.0
下载量422
星标0

简介

pix2struct ai2d base 是一款由 Google 开发的视觉问答（VQA）模型，其核心能力在于将视觉图像（尤其是包含结构化信息的图表、文档）精准转化为可理解的文本描述。与通用多模态模型不同，它专注于解析图像中的空间结构和逻辑关系，非常适合处理科学图表、流程图等复杂视觉素材。对于开发者而言，该模型上手难度较低，可作为 OCR 增强或自动化数据提取流水线中的关键组件，在需要将图像信息“结构化”的场景中，比单纯的图像分类或通用描述模型更高效。

核心亮点

  • 擅长解析科学图表与结构化文档
  • 将视觉空间关系精准转化为文本
  • 适用于自动化数据提取与文档分析
  • Apache-2.0 协议，企业级部署友好

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("google/pix2struct-ai2d-base")
tokenizer = AutoTokenizer.from_pretrained("google/pix2struct-ai2d-base")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download google/pix2struct-ai2d-base

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download google/pix2struct-ai2d-base config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('google/pix2struct-ai2d-base')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/google/pix2struct-ai2d-base

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/google/pix2struct-ai2d-base

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('google/pix2struct-ai2d-base')
tokenizer = AutoTokenizer.from_pretrained('google/pix2struct-ai2d-base')

模型下载

我们推荐使用命令行或者 ModelScope SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 ModelScope：

操作指引 
pip install modelscope

命令行下载

下载完整模型库

下载完整模型库 
modelscope download --model google/pix2struct-ai2d-base

下载单个文件到指定本地文件夹（以下载 README.md 到当前路径下 dir 目录为例）

下载单个文件到指定本地文件夹（以下载 README.md 到当前路径下 dir 目录为例） 
modelscope download --model google/pix2struct-ai2d-base README.md --local_dir ./dir

更多更丰富的命令行下载选项，可参见具体文档

SDK 下载

SDK 下载 
# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('google/pix2struct-ai2d-base')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://www.modelscope.cn/google/pix2struct-ai2d-base.git

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/google/pix2struct-ai2d-base.git

ModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。

Notebook 快速开发

下载并安装 ModelScope library

下载并安装 ModelScope library 
pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

模型加载和推理

模型加载和推理 
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

p = pipeline('text-generation', 'google/pix2struct-ai2d-base')

完整文档

来源: HuggingFace

---
language:

  • en

  • fr

  • ro

  • de

  • multilingual

inference: false
pipeline_tag: visual-question-answering
license: apache-2.0
---

Model card for Pix2Struct - Finetuned on AI2D (scientific diagram VQA)

!model_image

Table of Contents

0. TL;DR
1. Using the model
2. Contribution
3. Citation

TL;DR

Pix2Struct is an image encoder - text decoder model that is trained on image-text pairs for various tasks, including image captionning and visual question answering. The full list of available models can be found on the Table 1 of the paper:

!Table 1 - paper

The abstract of the model states that:
> Visually-situated language is ubiquitous—sources range from textbooks with diagrams to web pages with images and tables, to mobile apps with buttons and
forms. Perhaps due to this diversity, previous work has typically relied on domainspecific recipes with limited sharing of the underlying data, model architectures,
and objectives. We present Pix2Struct, a pretrained image-to-text model for
purely visual language understanding, which can be finetuned on tasks containing visually-situated language. Pix2Struct is pretrained by learning to parse
masked screenshots of web pages into simplified HTML. The web, with its richness of visual elements cleanly reflected in the HTML structure, provides a large
source of pretraining data well suited to the diversity of downstream tasks. Intuitively, this objective subsumes common pretraining signals such as OCR, language modeling, image captioning. In addition to the novel pretraining strategy,
we introduce a variable-resolution input representation and a more flexible integration of language and vision inputs, where language prompts such as questions
are rendered directly on top of the input image. For the first time, we show that a
single pretrained model can achieve state-of-the-art results in six out of nine tasks
across four domains: documents, illustrations, user interfaces, and natural images.

Using the model

This model has been fine-tuned on VQA, you need to provide a question in a specific format, ideally in the format of a Choices question answering

Running the model

In full precision, on CPU:

You can run the model in full precision on CPU:

python
import requests
from PIL import Image
from transformers import Pix2StructForConditionalGeneration, Pix2StructProcessor


image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"
image = Image.open(requests.get(image_url, stream=True).raw)


model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-ai2d-base")
processor = Pix2StructProcessor.from_pretrained("google/pix2struct-ai2d-base")


question = "What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud"


inputs = processor(images=image, text=question, return_tensors="pt")


predictions = model.generate(inputs)
print(processor.decode(predictions[0], skip_special_tokens=True))
>>> ash cloud

In full precision, on GPU:

You can run the model in full precision on CPU:

python
import requests
from PIL import Image
from transformers import Pix2StructForConditionalGeneration, Pix2StructProcessor


image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"
image = Image.open(requests.get(image_url, stream=True).raw)


model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-ai2d-base").to("cuda")
processor = Pix2StructProcessor.from_pretrained("google/pix2struct-ai2d-base")


question = "What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud"


inputs = processor(images=image, text=question, return_tensors="pt").to("cuda")


predictions = model.generate(inputs)
print(processor.decode(predictions[0], skip_special_tokens=True))
>>> ash cloud

In half precision, on GPU:

You can run the model in full precision on CPU:

python
import requests
from PIL import Image


import torch
from transformers import Pix2StructForConditionalGeneration, Pix2StructProcessor


image_url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg"
image = Image.open(requests.get(image_url, stream=True).raw)


model = Pix2StructForConditionalGeneration.from_pretrained("google/pix2struct-ai2d-base", torch_dtype=torch.bfloat16).to("cuda")
processor = Pix2StructProcessor.from_pretrained("google/pix2struct-ai2d-base")


question = "What does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud"


inputs = processor(images=image, text=question, return_tensors="pt").to("cuda", torch.bfloat16)


predictions = model.generate(**inputs)
print(processor.decode(predictions[0], skip_special_tokens=True))
>>> ash cloud

Converting from T5x to huggingface

You can use the convert_pix2struct_checkpoint_to_pytorch.py script as follows:

bash
python convert_pix2struct_checkpoint_to_pytorch.py --t5x_checkpoint_path PATH_TO_T5X_CHECKPOINTS --pytorch_dump_path PATH_TO_SAVE --is_vqa

if you are converting a large model, run:
bash
python convert_pix2struct_checkpoint_to_pytorch.py --t5x_checkpoint_path PATH_TO_T5X_CHECKPOINTS --pytorch_dump_path PATH_TO_SAVE --use-large --is_vqa

Once saved, you can push your converted model with the following snippet:
python
from transformers import Pix2StructForConditionalGeneration, Pix2StructProcessor


model = Pix2StructForConditionalGeneration.from_pretrained(PATH_TO_SAVE)
processor = Pix2StructProcessor.from_pretrained(PATH_TO_SAVE)


model.push_to_hub("USERNAME/MODEL_NAME")
processor.push_to_hub("USERNAME/MODEL_NAME")

Contribution

This model was originally contributed by Kenton Lee, Mandar Joshi et al. and added to the Hugging Face ecosystem by Younes Belkada.

Citation

If you want to cite this work, please consider citing the original paper:

code
@misc{https://doi.org/10.48550/arxiv.2210.03347,
  doi = {10.48550/ARXIV.2210.03347},
  
  url = {https://arxiv.org/abs/2210.03347},
  
  author = {Lee, Kenton and Joshi, Mandar and Turc, Iulia and Hu, Hexiang and Liu, Fangyu and Eisenschlos, Julian and Khandelwal, Urvashi and Shaw, Peter and Chang, Ming-Wei and Toutanova, Kristina},
  
  keywords = {Computation and Language (cs.CL), Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
  
  title = {Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding},
  
  publisher = {arXiv},
  
  year = {2022},
  
  copyright = {Creative Commons Attribution 4.0 International}
}

查看官方来源