vilt b32 finetuned vqa

提供商dandelin
分类visual-question-answering
许可证apache-2.0
下载量74.8K
星标0

简介

ViLT-B32-Finetuned-VQA 是一款基于视觉-语言 Transformer 架构的轻量化视觉问答模型。与传统的 VQA 模型不同，它舍弃了复杂的预训练目标检测网络，直接将图像和文本映射到统一的向量空间，从而显著提升了推理速度并降低了计算开销。该模型经过 VQA 数据集微调，能够理解图像内容并回答相关问题。对于开发者而言，它非常适合部署在资源受限的边缘端，或作为多模态流水线中的快速初筛模块，上手门槛低且兼容主流的 Hugging Face 框架。

核心亮点

  • 采用 ViLT 架构，推理速度快且内存占用低
  • 支持图像内容问答，实现视觉与文本的跨模态理解
  • 无需外部目标检测器，端到端处理流程更简洁
  • Apache-2.0 协议，适合商业化集成与快速二次开发

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("dandelin/vilt-b32-finetuned-vqa")
tokenizer = AutoTokenizer.from_pretrained("dandelin/vilt-b32-finetuned-vqa")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download dandelin/vilt-b32-finetuned-vqa

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download dandelin/vilt-b32-finetuned-vqa config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('dandelin/vilt-b32-finetuned-vqa')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/dandelin/vilt-b32-finetuned-vqa

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/dandelin/vilt-b32-finetuned-vqa

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('dandelin/vilt-b32-finetuned-vqa')
tokenizer = AutoTokenizer.from_pretrained('dandelin/vilt-b32-finetuned-vqa')

完整文档

来源: HuggingFace

---
tags:

  • visual-question-answering

license: apache-2.0
widget:
  • text: "What's the animal doing?"

src: "https://huggingface.co/datasets/mishig/sample_images/resolve/main/tiger.jpg"
  • text: "What is on top of the building?"

src: "https://huggingface.co/datasets/mishig/sample_images/resolve/main/palace.jpg"
---

Vision-and-Language Transformer (ViLT), fine-tuned on VQAv2

Vision-and-Language Transformer (ViLT) model fine-tuned on VQAv2. It was introduced in the paper ViLT: Vision-and-Language Transformer
Without Convolution or Region Supervision by Kim et al. and first released in this repository.

Disclaimer: The team releasing ViLT did not write a model card for this model so this model card has been written by the Hugging Face team.

Intended uses & limitations

You can use the raw model for visual question answering.

How to use

Here is how to use this model in PyTorch:

python
from transformers import ViltProcessor, ViltForQuestionAnswering
import requests
from PIL import Image

prepare image + question

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
text = "How many cats are there?"

processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa")
model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa")


prepare inputs

encoding = processor(image, text, return_tensors="pt")

forward pass

outputs = model(**encoding)
logits = outputs.logits
idx = logits.argmax(-1).item()
print("Predicted answer:", model.config.id2label[idx])

Training data

(to do)

Training procedure

Preprocessing

(to do)

Pretraining

(to do)

Evaluation results

(to do)

BibTeX entry and citation info

bibtex
@misc{kim2021vilt,
      title={ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision}, 
      author={Wonjae Kim and Bokyung Son and Ildoo Kim},
      year={2021},
      eprint={2102.03334},
      archivePrefix={arXiv},
      primaryClass={stat.ML}
}
查看官方来源