Name: ImageTextRetrieval
Author: ohgnues

Overview

ImageTextRetrieval is a specialized model designed for cross-modal alignment, allowing developers to perform efficient semantic searches across image and text datasets. Unlike standard classification models, this architecture maps both visual and textual inputs into a shared embedding space. This makes it ideal for building reverse image search engines, automated tagging systems, or content discovery tools where natural language queries must retrieve relevant visual assets. It integrates easily into RAG (Retrieval-Augmented Generation) pipelines by serving as the encoder for vector databases, offering a lightweight alternative to massive multimodal LLMs when the primary goal is retrieval speed and precision rather than generative output.

Highlights

Maps images and text into a shared embedding space
Optimized for high-performance semantic cross-modal search
Apache-2.0 license ensures flexible commercial integration
Ideal for vector database indexing and retrieval pipelines

Usage

Install

# Install Hugging Face transformers
pip install transformers torch

SDK Usage

# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("ohgnues/ImageTextRetrieval")
tokenizer = AutoTokenizer.from_pretrained("ohgnues/ImageTextRetrieval")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance

pip install -U huggingface_hub

CLI Download

Download the full repository

huggingface-cli download ohgnues/ImageTextRetrieval

Download a single file to a local folder (e.g. config.json into ./dir)

huggingface-cli download ohgnues/ImageTextRetrieval config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('ohgnues/ImageTextRetrieval')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://huggingface.co/ohgnues/ImageTextRetrieval

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/ohgnues/ImageTextRetrieval

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

pip install -U transformers torch

Load the model and run inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('ohgnues/ImageTextRetrieval')
tokenizer = AutoTokenizer.from_pretrained('ohgnues/ImageTextRetrieval')

Full Documentation

来源: HuggingFace

Repo

multi-modal-retrieval

This repository contains code for multi modal retrieval

This project involves implementing a multi-modal Bi-encoder using both ResNet and BERT for image and text representations.

Data

Sample Data

The pretraining was conducted using the dataset from Hugging Face's "poloclub/diffusiondb" dataset.

I used 50k randomly sampled images and prompts for my project.

If you want to use a different dataset, follow the steps below

Data Format

Only images and the corresponding text for those images are necessary, and other elements are irrelevant. In this case, the text can serve as prompts or captions for the images.

You specify the names of the columns for images and text in the training command.

bash

python3 train.py --text_column_name text --image_column_name img

Pretrained models

Pretrained models can be downloaded huggingface or Specify the model name "ohgnues/ImageTextRetrieval" in the training command.

bash

python3 train.py --pretrained_model_name_or_path ohgnues/ImageTextRetrieval

The model "ohgnues/ImageTextRetrieval" was trained for 10 epochs using a Tesla P100 GPU.

Usage

Train

bash

python3 train.py --name 2m_random_50k --cache_dir /data/.cache --max_length 100 --num_train_epochs 10

For detailed instructions, please refer to the official Hugging Face documentation or consult the dataclass within the "train.py" script.

Encode

python

def encode(self, model_name: Literal["text", "image"],
            input_ids: Optional[torch.Tensor] = None,
            attention_mask: Optional[torch.Tensor] = None,
            token_type_ids: Optional[torch.Tensor] = None,
            position_ids: Optional[torch.Tensor] = None,
            head_mask: Optional[torch.Tensor] = None,
            inputs_embeds: Optional[torch.Tensor] = None,
            output_attentions: Optional[bool] = None,
            output_hidden_states: Optional[bool] = None,
            return_dict: Optional[bool] = None,
            pixel_values: Tensor = None
            ):
        
        if model_name == "text":
            return self.text_encoder(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
            ).last_hidden_state[:, 0, :]
        
        elif model_name == "image":
            return self.image_encoder(
            pixel_values=pixel_values,
            output_hidden_states=output_hidden_states,
            ).pooler_output[:, :, 0, 0]