Name: blip image captioning large
Author: Salesforce

Overview

BLIP (Bootstrapping Language-Image Pre-training) Large is a versatile vision-language model designed for high-fidelity image captioning and visual question answering. Unlike basic image-to-text models, BLIP is trained to bridge the gap between noisy web data and clean synthetic captions, resulting in descriptions that are more contextually accurate and descriptive. For developers, it serves as a robust backbone for automating alt-text generation, indexing visual libraries, or building accessibility tools. It integrates well into Python-based ML pipelines via Hugging Face Transformers, offering a strong balance between inference speed and descriptive quality compared to smaller CLIP-based encoders.

Highlights

High-accuracy image captioning and visual question answering
Optimized for descriptive and contextually relevant text output
Seamless integration with Hugging Face Transformers library
Permissive BSD-3-Clause license for commercial deployment
Superior performance over standard image-to-text baselines

Usage

Install

# Install Hugging Face transformers
pip install transformers torch

SDK Usage

# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("Salesforce/blip-image-captioning-large")
tokenizer = AutoTokenizer.from_pretrained("Salesforce/blip-image-captioning-large")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance

pip install -U huggingface_hub

CLI Download

Download the full repository

huggingface-cli download Salesforce/blip-image-captioning-large

Download a single file to a local folder (e.g. config.json into ./dir)

huggingface-cli download Salesforce/blip-image-captioning-large config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('Salesforce/blip-image-captioning-large')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://huggingface.co/Salesforce/blip-image-captioning-large

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/Salesforce/blip-image-captioning-large

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

pip install -U transformers torch

Load the model and run inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('Salesforce/blip-image-captioning-large')
tokenizer = AutoTokenizer.from_pretrained('Salesforce/blip-image-captioning-large')

Model Download

We recommend downloading the model via the ModelScope CLI or SDK.

Guidance：Before downloading, install ModelScope with:

Guidance

pip install modelscope

CLI Download

Download the full repository

modelscope download --model Salesforce/blip-image-captioning-large

Download a single file to a local folder (e.g. README.md into ./dir)

modelscope download --model Salesforce/blip-image-captioning-large README.md --local_dir ./dir

See the docs for more CLI options

SDK Download

# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('Salesforce/blip-image-captioning-large')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://www.modelscope.cn/Salesforce/blip-image-captioning-large.git

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/Salesforce/blip-image-captioning-large.git

ModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。

Notebook Quickstart

Install the ModelScope library

pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

Load the model and run inference

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

p = pipeline('text-generation', 'Salesforce/blip-image-captioning-large')

Full Documentation

来源: HuggingFace

---
pipeline_tag: image-to-text
tags:

image-captioning

languages:

en

license: bsd-3-clause
---

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

Model card for image captioning pretrained on COCO dataset - base architecture (with ViT large backbone).

TL;DR

Authors from the paper write in the abstract:

*Vision-Language Pre-training (VLP) has advanced the performance for many vision-language tasks. However, most existing pre-trained models only excel in either understanding-based tasks or generation-based tasks. Furthermore, performance improvement has been largely achieved by scaling up the dataset with noisy image-text pairs collected from the web, which is a suboptimal source of supervision. In this paper, we propose BLIP, a new VLP framework which transfers flexibly to both vision-language understanding and generation tasks. BLIP effectively utilizes the noisy web data by bootstrapping the captions, where a captioner generates synthetic captions and a filter removes the noisy ones. We achieve state-of-the-art results on a wide range of vision-language tasks, such as image-text retrieval (+2.7% in average recall@1), image captioning (+2.8% in CIDEr), and VQA (+1.6% in VQA score). BLIP also demonstrates strong generalization ability when directly transferred to videolanguage tasks in a zero-shot manner. Code, models, and datasets are released.*

Usage

You can use this model for conditional and un-conditional image captioning

Using the Pytorch model

#### Running the model on CPU

<details>
<summary> Click to expand </summary>

python

import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large")
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt")
out = model.generate(inputs)
print(processor.decode(out[0], skip_special_tokens=True))

unconditional image captioning
inputs = processor(raw_image, return_tensors="pt")
out = model.generate(inputs)
print(processor.decode(out[0], skip_special_tokens=True))

</details>

#### Running the model on GPU

##### In full precision

<details>
<summary> Click to expand </summary>

python

import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large").to("cuda")
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt").to("cuda")
out = model.generate(inputs)
print(processor.decode(out[0], skip_special_tokens=True))

unconditional image captioning
inputs = processor(raw_image, return_tensors="pt").to("cuda")
out = model.generate(inputs)
print(processor.decode(out[0], skip_special_tokens=True))

</details>

##### In half precision (float16)

<details>
<summary> Click to expand </summary>

python

import torch
import requests
from PIL import Image
from transformers import BlipProcessor, BlipForConditionalGeneration
processor = BlipProcessor.from_pretrained("Salesforce/blip-image-captioning-large")
model = BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-large", torch_dtype=torch.float16).to("cuda")
img_url = 'https://storage.googleapis.com/sfr-vision-language-research/BLIP/demo.jpg' 
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
conditional image captioning
text = "a photography of"
inputs = processor(raw_image, text, return_tensors="pt").to("cuda", torch.float16)
out = model.generate(inputs)
print(processor.decode(out[0], skip_special_tokens=True))
>>> a photography of a woman and her dog

unconditional image captioning
inputs = processor(raw_image, return_tensors="pt").to("cuda", torch.float16)
out = model.generate(inputs)
print(processor.decode(out[0], skip_special_tokens=True))
>>> a woman sitting on the beach with her dog

</details>

Ethical Considerations

This release is for research purposes only in support of an academic paper. Our models, datasets, and code are not specifically designed or evaluated for all downstream purposes. We strongly recommend users evaluate and address potential concerns related to accuracy, safety, and fairness before deploying this model. We encourage users to consider the common limitations of AI, comply with applicable laws, and leverage best practices when selecting use cases, particularly for high-risk scenarios where errors or misuse could significantly impact people’s lives, rights, or safety. For further guidance on use cases, refer to our AUP and AI AUP.

BibTex and citation info

code

@misc{https://doi.org/10.48550/arxiv.2201.12086,
  doi = {10.48550/ARXIV.2201.12086},
  
  url = {https://arxiv.org/abs/2201.12086},
  
  author = {Li, Junnan and Li, Dongxu and Xiong, Caiming and Hoi, Steven},
  
  keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
  
  title = {BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation},
  
  publisher = {arXiv},
  
  year = {2022},
  
  copyright = {Creative Commons Attribution 4.0 International}
}