vjepa2 vitl fpc64 256

提供商facebook
分类video-classification
许可证mit
下载量792
星标1

简介

V-JEPA 是 Meta 推出的一款基于自监督学习的视频表征模型。不同之于传统的逐帧分析，它旨在学习视频中的空间-时间预测，能够更高效地捕捉动作逻辑和场景变化。该版本采用 ViT-L 骨干网络，特别强化了特征提取能力，非常适合需要处理复杂视频语义的开发者。它不是一个直接对话的聊天机器人，而是一个强大的‘视频特征提取器’，可作为下游任务（如视频分类、动作识别）的预训练底座，显著降低从零训练模型的算力门槛。

核心亮点

  • 自监督预训练，无需海量标注数据即可理解视频
  • 基于 ViT-L 架构，提供高质量的视频语义表征
  • 适用于视频分类、动作检测等计算机视觉任务
  • MIT 协议开源，方便开发者快速集成到自有管线

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("facebook/vjepa2-vitl-fpc64-256")
tokenizer = AutoTokenizer.from_pretrained("facebook/vjepa2-vitl-fpc64-256")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download facebook/vjepa2-vitl-fpc64-256

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download facebook/vjepa2-vitl-fpc64-256 config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('facebook/vjepa2-vitl-fpc64-256')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/facebook/vjepa2-vitl-fpc64-256

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/facebook/vjepa2-vitl-fpc64-256

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('facebook/vjepa2-vitl-fpc64-256')
tokenizer = AutoTokenizer.from_pretrained('facebook/vjepa2-vitl-fpc64-256')

模型下载

我们推荐使用命令行或者 ModelScope SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 ModelScope：

操作指引 
pip install modelscope

命令行下载

下载完整模型库

下载完整模型库 
modelscope download --model facebook/vjepa2-vitl-fpc64-256

下载单个文件到指定本地文件夹（以下载 README.md 到当前路径下 dir 目录为例）

下载单个文件到指定本地文件夹（以下载 README.md 到当前路径下 dir 目录为例） 
modelscope download --model facebook/vjepa2-vitl-fpc64-256 README.md --local_dir ./dir

更多更丰富的命令行下载选项，可参见具体文档

SDK 下载

SDK 下载 
# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('facebook/vjepa2-vitl-fpc64-256')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://www.modelscope.cn/facebook/vjepa2-vitl-fpc64-256.git

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/facebook/vjepa2-vitl-fpc64-256.git

ModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。

Notebook 快速开发

下载并安装 ModelScope library

下载并安装 ModelScope library 
pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

模型加载和推理

模型加载和推理 
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

p = pipeline('text-generation', 'facebook/vjepa2-vitl-fpc64-256')

完整文档

来源: HuggingFace

---
license: mit
pipeline_tag: video-classification
tags:

  • video

library_name: transformers
---

V-JEPA 2

A frontier video understanding model developed by FAIR, Meta, which extends the pretraining objectives of VJEPA, resulting in state-of-the-art video understanding capabilities, leveraging data and model sizes at scale.
The code is released in this repository.

<img src="https://github.com/user-attachments/assets/914942d8-6a1e-409d-86ff-ff856b7346ab">&nbsp;

Installation

To run V-JEPA 2 model, ensure you have installed the latest transformers:

bash
pip install -U git+https://github.com/huggingface/transformers

Intended Uses

V-JEPA 2 is intended to represent any video (and image) to perform video classification, retrieval, or as a video encoder for VLMs.

python
from transformers import AutoVideoProcessor, AutoModel

hf_repo = "facebook/vjepa2-vitl-fpc64-256"


model = AutoModel.from_pretrained(hf_repo)
processor = AutoVideoProcessor.from_pretrained(hf_repo)

To load a video, sample the number of frames according to the model. For this model, we use 64.

python
import torch
from torchcodec.decoders import VideoDecoder
import numpy as np

video_url = "https://huggingface.co/datasets/nateraw/kinetics-mini/resolve/main/val/archery/-Qz25rXdMjE_000014_000024.mp4"
vr = VideoDecoder(video_url)
frame_idx = np.arange(0, 64) # choosing some frames. here, you can define more complex sampling strategy
video = vr.get_frames_at(indices=frame_idx).data  # T x C x H x W
video = processor(video, return_tensors="pt").to(model.device)
with torch.no_grad():
    video_embeddings = model.get_vision_features(**video)


print(video_embeddings.shape)

To load an image, simply copy the image to the desired number of frames.

python
from transformers.image_utils import load_image

image = load_image("https://huggingface.co/datasets/merve/coco/resolve/main/val2017/000000000285.jpg")
pixel_values = processor(image, return_tensors="pt").to(model.device)["pixel_values_videos"]
pixel_values = pixel_values.repeat(1, 16, 1, 1, 1) # repeating image 16 times


with torch.no_grad():
    image_embeddings = model.get_vision_features(pixel_values)


print(image_embeddings.shape)

For more code examples, please refer to the V-JEPA 2 documentation.

Citation

code
@techreport{assran2025vjepa2,
  title={V-JEPA~2: Self-Supervised Video Models Enable Understanding, Prediction and Planning},
  author={Assran, Mahmoud and Bardes, Adrien and Fan, David and Garrido, Quentin and Howes, Russell and
Komeili, Mojtaba and Muckley, Matthew and Rizvi, Ammar and Roberts, Claire and Sinha, Koustuv and Zholus, Artem and
Arnaud, Sergio and Gejji, Abha and Martin, Ada and Robert Hogan, Francois and Dugas, Daniel and
Bojanowski, Piotr and Khalidov, Vasil and Labatut, Patrick and Massa, Francisco and Szafraniec, Marc and
Krishnakumar, Kapil and Li, Yong and Ma, Xiaodong and Chandar, Sarath and Meier, Franziska and LeCun, Yann and
Rabbat, Michael and Ballas, Nicolas},
  institution={FAIR at Meta},
  year={2025}
}
查看官方来源