Overview

Touch Vision Language Models (TVLMs) extend traditional multimodal architectures by integrating tactile sensing with visual and linguistic data. For developers, this means moving beyond static image-text pairs to models capable of processing physical interaction data. These models are particularly useful for robotics, haptic interface design, and autonomous agents that require a physical understanding of texture, pressure, and material properties to operate in real-world environments. By bridging the gap between seeing and feeling, TVLMs enable more precise manipulation tasks and sensory-grounded reasoning. Integration typically follows standard multimodal pipelines, though developers should account for the specific data formats associated with tactile sensors. Compared to standard VLMs, Touch VLMs provide a higher degree of spatial and physical awareness, reducing reliance on purely visual estimation.

Highlights

Integrates tactile sensing with visual and linguistic processing
Ideal for robotics and haptic interface development
Enables sensory-grounded reasoning for real-world physical interactions
Apache-2.0 license ensures flexible commercial integration
Provides superior physical awareness over standard vision models

Usage

Install

# Install Hugging Face transformers
pip install transformers torch

SDK Usage

# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("mlfu7/Touch-Vision-Language-Models")
tokenizer = AutoTokenizer.from_pretrained("mlfu7/Touch-Vision-Language-Models")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance

pip install -U huggingface_hub

CLI Download

Download the full repository

huggingface-cli download mlfu7/Touch-Vision-Language-Models

Download a single file to a local folder (e.g. config.json into ./dir)

huggingface-cli download mlfu7/Touch-Vision-Language-Models config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('mlfu7/Touch-Vision-Language-Models')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://huggingface.co/mlfu7/Touch-Vision-Language-Models

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/mlfu7/Touch-Vision-Language-Models

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

pip install -U transformers torch

Load the model and run inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('mlfu7/Touch-Vision-Language-Models')
tokenizer = AutoTokenizer.from_pretrained('mlfu7/Touch-Vision-Language-Models')

Full Documentation

来源: HuggingFace

---
license: apache-2.0
---

A Touch, Vision, and Language Dataset for Multimodal Alignment

by <a href="https://max-fu.github.io">Max (Letian) Fu</a>, <a href="https://www.linkedin.com/in/gaurav-datta/">Gaurav Datta*</a>, <a href="https://qingh097.github.io/">Huang Huang*</a>, <a href="https://autolab.berkeley.edu/people">William Chung-Ho Panitch*</a>, <a href="https://www.linkedin.com/in/jaimyn-drake/">Jaimyn Drake*</a>, <a href="https://joeaortiz.github.io/">Joseph Ortiz</a>, <a href="https://www.mustafamukadam.com/">Mustafa Mukadam</a>, <a href="https://scholar.google.com/citations?user=p6DCMrQAAAAJ&hl=en">Mike Lambeta</a>, <a href="https://lasr.org/">Roberto Calandra</a>, <a href="https://goldberg.berkeley.edu">Ken Goldberg</a> at UC Berkeley, Meta AI, TU Dresden, and CeTI (*equal contribution).

[Paper] | [Project Page] | [Checkpoints] | [Dataset] | [Citation]

This repo contains the official checkpoints for *A Touch, Vision, and Language Dataset for Multimodal Alignment*.

The tactile encoders comes in three different sizes: ViT-Tiny, ViT-Small, and ViT-Base, all of which are stored in

bash

ckpt/tvl_enc

TVL-LLaMA, the generative counterparts, are stored in

bash

ckpt/tvl_llama

Inference

For zero-shot classification, we would require OpenCLIP with the following configuration:

bash

CLIP_VISION_MODEL = "ViT-L-14"
CLIP_PRETRAIN_DATA = "datacomp_xl_s13b_b90k"

For TVL-LLaMA, please request access to the pre-trained LLaMA-2 from this form. In particular, we use llama-2-7b as the base model. The weights here contains the trained adapter, the tactile encoder, and the vision encoder for the ease of loading.

For the complete info, please take a look at the GitHub repo to see instructions on pretraining, fine-tuning, and evaluation with these models.

Citation

Please give us a star 🌟 on Github to support us!

Please cite our work if you find our work inspiring or use our code in your work:

code

@article{fu2024tvl,
    title={A Touch, Vision, and Language Dataset for Multimodal Alignment}, 
    author={Letian Fu and Gaurav Datta and Huang Huang and William Chung-Ho Panitch and Jaimyn Drake and Joseph Ortiz and Mustafa Mukadam and Mike Lambeta and Roberto Calandra and Ken Goldberg},
    journal={arXiv preprint arXiv:2402.13232},
    year={2024}
}