Name: VideoLLaMA2.1 7B AV
Author: DAMO-NLP-SG

Overview

VideoLLaMA2.1 7B AV is a multimodal LLM optimized for high-fidelity video understanding and visual question answering. Unlike basic image-to-text models, this architecture is designed to handle temporal dynamics, allowing developers to extract semantic meaning from video sequences rather than just isolated frames. It is particularly effective for automated video indexing, content summarization, and complex visual reasoning tasks. Built on a 7B parameter backbone and released under the Apache-2.0 license, it offers a balanced trade-off between inference latency and reasoning depth, making it suitable for integration into scalable production pipelines via standard LLM frameworks. Compared to previous iterations, it demonstrates improved alignment between visual tokens and linguistic outputs, reducing hallucinations in long-form video analysis.

Highlights

Optimized for temporal reasoning and video question answering.
Apache-2.0 license ensures flexible commercial integration.
Balanced 7B parameter size for efficient deployment.
High-fidelity alignment between visual inputs and text outputs.

Usage

Install

# Install Hugging Face transformers
pip install transformers torch

SDK Usage

# Load model with transformers
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("DAMO-NLP-SG/VideoLLaMA2.1-7B-AV")
tokenizer = AutoTokenizer.from_pretrained("DAMO-NLP-SG/VideoLLaMA2.1-7B-AV")

Hugging Face Download

We recommend downloading the model via the Hugging Face CLI or Hub SDK.

Guidance：Before downloading, install huggingface_hub with:

Guidance

pip install -U huggingface_hub

CLI Download

Download the full repository

huggingface-cli download DAMO-NLP-SG/VideoLLaMA2.1-7B-AV

Download a single file to a local folder (e.g. config.json into ./dir)

huggingface-cli download DAMO-NLP-SG/VideoLLaMA2.1-7B-AV config.json --local-dir ./dir

See the official docs for more CLI options

SDK Download

# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('DAMO-NLP-SG/VideoLLaMA2.1-7B-AV')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2.1-7B-AV

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/DAMO-NLP-SG/VideoLLaMA2.1-7B-AV

Model files are hosted on the Hugging Face Hub — download directly via HF CLI / SDK / Git, not through this site.

PyTorch / Transformers Usage

Install Transformers

pip install -U transformers torch

Load the model and run inference

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('DAMO-NLP-SG/VideoLLaMA2.1-7B-AV')
tokenizer = AutoTokenizer.from_pretrained('DAMO-NLP-SG/VideoLLaMA2.1-7B-AV')

Model Download

We recommend downloading the model via the ModelScope CLI or SDK.

Guidance：Before downloading, install ModelScope with:

Guidance

pip install modelscope

CLI Download

Download the full repository

modelscope download --model DAMO-NLP-SG/VideoLLaMA2.1-7B-AV

Download a single file to a local folder (e.g. README.md into ./dir)

modelscope download --model DAMO-NLP-SG/VideoLLaMA2.1-7B-AV README.md --local_dir ./dir

See the docs for more CLI options

SDK Download

# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('DAMO-NLP-SG/VideoLLaMA2.1-7B-AV')

Git Download

Make sure git-lfs is installed first

Git Download

git lfs install
git clone https://www.modelscope.cn/DAMO-NLP-SG/VideoLLaMA2.1-7B-AV.git

To skip LFS large-file downloads, use:

Skip LFS

GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/DAMO-NLP-SG/VideoLLaMA2.1-7B-AV.git

ModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。

Notebook Quickstart

Install the ModelScope library

pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

Load the model and run inference

from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

p = pipeline('text-generation', 'DAMO-NLP-SG/VideoLLaMA2.1-7B-AV')

Full Documentation

来源: HuggingFace

---
license: apache-2.0
datasets:

lmms-lab/ClothoAQA

Loie/VGGSound

language:

en

metrics:

accuracy

pipeline_tag: visual-question-answering
library_name: transformers
tags:

Audio-visual Question Answering

Audio Question Answering

multimodal large language model

---

<h3 align="center"><a href="https://arxiv.org/abs/2406.07476">VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs</a></h3>
<h5 align="center"> If you like our project, please give us a star ⭐ on <a href="https://github.com/DAMO-NLP-SG/VideoLLaMA2">Github</a> for the latest update. </h2>

📰 News

[2024.10.22] Release checkpoints of VideoLLaMA2.1-7B-AV

[2024.10.15] Release checkpoints of VideoLLaMA2.1-7B-16F-Base and VideoLLaMA2.1-7B-16F

[2024.08.14] Release checkpoints of VideoLLaMA2-72B-Base and VideoLLaMA2-72B

[2024.07.30] Release checkpoints of VideoLLaMA2-8x7B-Base and VideoLLaMA2-8x7B.

[2024.06.25] 🔥🔥 As of Jun 25, our VideoLLaMA2-7B-16F is the Top-1 ~7B-sized VideoLLM on the MLVU Leaderboard.

[2024.06.18] 🔥🔥 As of Jun 18, our VideoLLaMA2-7B-16F is the Top-1 ~7B-sized VideoLLM on the VideoMME Leaderboard.

[2024.06.17] 👋👋 Update technical report with the latest results and the missing references. If you have works closely related to VideoLLaMA 2 but not mentioned in the paper, feel free to let us know.

[2024.06.14] 🔥🔥 Online Demo is available.

[2024.06.03] Release training, evaluation, and serving codes of VideoLLaMA 2.

🌎 Model Zoo

Vision-Only Checkpoints

Audio-Visual Checkpoints

| Model Name | Type | Audio Encoder | Language Decoder | |:-------------------|:--------------:|:----------------|:----------------------:| | VideoLLaMA2.1-7B-AV (This Checkpoint) | Chat | Fine-tuned BEATs_iter3+(AS2M)(cpt2) | VideoLLaMA2.1-7B-16F |

🚀 Main Results

Multi-Choice Video QA & Video Captioning

Open-Ended Video QA

Multi-Choice & Open-Ended Audio QA

Open-Ended Audio-Visual QA

🤖 Inference with VideoLLaMA2-AV

```python import sys sys.path.append('./') from videollama2 import model_init, mm_infer from videollama2.utils import disable_torch_init import argparse

def inference(args):

model_path = args.model_path
model, processor, tokenizer = model_init(model_path)

if args.modal_type == "a":
model.model.vision_tower = None
elif args.modal_type == "v":
model.model.audio_tower = None
elif args.modal_type == "av":
pass
else:
raise NotImplementedError
# Audio-visual Inference
audio_video_path = "assets/00003491.mp4"
preprocess = processor['audio' if args.modal_type == "a" else "video"]
if args.modal_type == "a":
audio_video_tensor = preprocess(audio_video_path)
else:
audio_video_tensor = preprocess(audio_video_path, va=True if args.modal_type == "av" else False)
question = f"Please describe the video with audio information."

# Audio Inference
audio_video_path = "assets/bird-twitter-car.wav"
preprocess = processor['audio' if args.modal_type == "a" else "video"]
if args.modal_type == "a":
audio_video_tensor = preprocess(audio_video_path)
else:
audio_video_tensor = preprocess(audio_video_path, va=True if args.modal_type == "av" else False)
question = f"Please describe the audio."

# Video Inferen