Touch Vision Language Models
简介
核心亮点
- 融合触觉与视觉信号，增强物理世界感知力
- 核心应用于具身智能与机器人精细化操作
- 提供多模态特征表示，降低触觉数据处理难度
- 采用 Apache-2.0 协议，利于商业化部署
使用方法
# 安装 Hugging Face transformers
pip install transformers torch# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("mlfu7/Touch-Vision-Language-Models")
tokenizer = AutoTokenizer.from_pretrained("mlfu7/Touch-Vision-Language-Models")Hugging Face 下载
我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。
操作指引：在下载前，请先通过如下命令安装 huggingface_hub：
pip install -U huggingface_hub命令行下载
下载完整模型库
huggingface-cli download mlfu7/Touch-Vision-Language-Models下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）
huggingface-cli download mlfu7/Touch-Vision-Language-Models config.json --local-dir ./dirSDK 下载
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('mlfu7/Touch-Vision-Language-Models')Git 下载
请确保 lfs 已经被正确安装
git lfs install
git clone https://huggingface.co/mlfu7/Touch-Vision-Language-Models如果您希望跳过 lfs 大文件下载，可以使用如下命令
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/mlfu7/Touch-Vision-Language-Models模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。
PyTorch / Transformers 使用
安装 Transformers
pip install -U transformers torch模型加载和推理
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('mlfu7/Touch-Vision-Language-Models')
tokenizer = AutoTokenizer.from_pretrained('mlfu7/Touch-Vision-Language-Models')完整文档
---
license: apache-2.0
---
A Touch, Vision, and Language Dataset for Multimodal Alignment
by <a href="https://max-fu.github.io">Max (Letian) Fu</a>, <a href="https://www.linkedin.com/in/gaurav-datta/">Gaurav Datta*</a>, <a href="https://qingh097.github.io/">Huang Huang*</a>, <a href="https://autolab.berkeley.edu/people">William Chung-Ho Panitch*</a>, <a href="https://www.linkedin.com/in/jaimyn-drake/">Jaimyn Drake*</a>, <a href="https://joeaortiz.github.io/">Joseph Ortiz</a>, <a href="https://www.mustafamukadam.com/">Mustafa Mukadam</a>, <a href="https://scholar.google.com/citations?user=p6DCMrQAAAAJ&hl=en">Mike Lambeta</a>, <a href="https://lasr.org/">Roberto Calandra</a>, <a href="https://goldberg.berkeley.edu">Ken Goldberg</a> at UC Berkeley, Meta AI, TU Dresden, and CeTI (*equal contribution).
[Paper] | [Project Page] | [Checkpoints] | [Dataset] | [Citation]
<p align="center">
<img src="img/splash_figure_alt.png" width="800">
</p>
This repo contains the official checkpoints for *A Touch, Vision, and Language Dataset for Multimodal Alignment*.
The tactile encoders comes in three different sizes: ViT-Tiny, ViT-Small, and ViT-Base, all of which are stored in
ckpt/tvl_encTVL-LLaMA, the generative counterparts, are stored in
ckpt/tvl_llamaInference
For zero-shot classification, we would require OpenCLIP with the following configuration:CLIP_VISION_MODEL = "ViT-L-14"
CLIP_PRETRAIN_DATA = "datacomp_xl_s13b_b90k"For TVL-LLaMA, please request access to the pre-trained LLaMA-2 from this form. In particular, we use llama-2-7b as the base model. The weights here contains the trained adapter, the tactile encoder, and the vision encoder for the ease of loading.
For the complete info, please take a look at the GitHub repo to see instructions on pretraining, fine-tuning, and evaluation with these models.
Citation
Please give us a star 🌟 on Github to support us!Please cite our work if you find our work inspiring or use our code in your work:
@article{fu2024tvl,
title={A Touch, Vision, and Language Dataset for Multimodal Alignment},
author={Letian Fu and Gaurav Datta and Huang Huang and William Chung-Ho Panitch and Jaimyn Drake and Joseph Ortiz and Mustafa Mukadam and Mike Lambeta and Roberto Calandra and Ken Goldberg},
journal={arXiv preprint arXiv:2402.13232},
year={2024}
}