Wan2.2 Fun Reward LoRAs

提供商alibaba-pai
分类text-to-video
许可证apache-2.0
下载量40.4K
星标0

简介

Wan2.2 Fun Reward LoRAs 是由阿里巴巴 PAI 团队推出的针对视频生成模型 Wan2.2 的微调权重集。该项目旨在解决通用视频模型在特定视觉风格或动态效果上难以精准控制的痛点，通过引入奖励模型（Reward Model）的引导进行优化，显著提升了生成视频的视觉美感和动作流畅度。对于开发者而言，它无需重新训练大模型，只需在推理时加载 LoRA 插件即可快速切换风格，极大降低了高质量短视频创作的门槛，是目前增强 Wan2.2 画面表现力的实用工具。

核心亮点

  • 基于奖励模型优化，大幅提升视频视觉质量
  • 即插即用，无需全量微调即可改变视频风格
  • 显著增强动作流畅度，减少生成画面的崩坏
  • 适配 Apache-2.0 协议，方便开发者商业集成

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("alibaba-pai/Wan2.2-Fun-Reward-LoRAs")
tokenizer = AutoTokenizer.from_pretrained("alibaba-pai/Wan2.2-Fun-Reward-LoRAs")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download alibaba-pai/Wan2.2-Fun-Reward-LoRAs

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download alibaba-pai/Wan2.2-Fun-Reward-LoRAs config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('alibaba-pai/Wan2.2-Fun-Reward-LoRAs')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/alibaba-pai/Wan2.2-Fun-Reward-LoRAs

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/alibaba-pai/Wan2.2-Fun-Reward-LoRAs

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('alibaba-pai/Wan2.2-Fun-Reward-LoRAs')
tokenizer = AutoTokenizer.from_pretrained('alibaba-pai/Wan2.2-Fun-Reward-LoRAs')

完整文档

来源: HuggingFace

---
license: apache-2.0
base_model:

  • alibaba-pai/Wan2.2-Fun-A14B-InP

  • Wan-AI/Wan2.2-T2V-A14B

library_name: videox_fun
pipeline_tag: text-to-video
---

Wan2.2-Fun-Reward-LoRAs


Introduction


We explore the Reward Backpropagation technique <sup>1 2</sup> to optimized the generated videos by Wan2.2-Fun for better alignment with human preferences.
We provide the following pre-trained models (i.e. LoRAs) along with the training script. You can use these LoRAs to enhance the corresponding base model as a plug-in or train your own reward LoRA.

For more details, please refer to our GitHub repo.

| Name | Base Model | Reward Model | Hugging Face | Description |
|--|--|--|--|--|
| Wan2.2-Fun-A14B-InP-high-noise-HPS2.1.safetensors | Wan2.2-Fun-A14B-InP (high noise) | HPS v2.1 | 🤗Link | Official HPS v2.1 reward LoRA (rank=128 and network_alpha=64) for Wan2.2-Fun-A14B-InP (high noise). It is trained with a batch size of 8 for 5,000 steps.|
| Wan2.2-Fun-A14B-InP-low-noise-HPS2.1.safetensors | Wan2.2-Fun-A14B-InP (low noise) | MPS | 🤗Link | Official HPS v2.1 reward LoRA (rank=128 and network_alpha=64) for Wan2.2-Fun-A14B-InP (low noise). It is trained with a batch size of 8 for 2,700 steps.|
| Wan2.2-Fun-A14B-InP-high-noise-MPS.safetensors | Wan2.2-Fun-A14B-InP (high noise) | HPS v2.1 | 🤗Link | Official MPS reward LoRA (rank=128 and network_alpha=64) for Wan2.2-Fun-A14B-InP (high noise). It is trained with a batch size of 8 for 5,000 steps.|
| Wan2.2-Fun-A14B-InP-low-noise-MPS.safetensors | Wan2.2-Fun-A14B-InP (low noise) | MPS | 🤗Link | Official MPS reward LoRA (rank=128 and network_alpha=64) for Wan2.2-Fun-A14B-InP (low noise). It is trained with a batch size of 8 for 4,500 steps.|

> [!NOTE]
> We found that, MPS reward LoRA for the low-noise model converges significantly more slowly than on the other models, and may not deliver satisfactory results. Therefore, for the low-noise model, we recommend using HPSv2.1 reward LoRA.

Demo

Wan2.2-Fun-A14B-InP

<table border="0" style="width: 100%; text-align: center; margin-top: 20px;">
<thead>
<tr>
<th style="text-align: center;" width="10%">Prompt</sup></th>
<th style="text-align: center;" width="30%">Wan2.2-Fun-A14B-InP</th>
<th style="text-align: center;" width="30%">Wan2.2-Fun-A14B-InP <br> high + low HPSv2.1 Reward LoRA</th>
<th style="text-align: center;" width="30%">Wan2.2-Fun-A14B-InP <br> high MPS + low HPSv2.1 Reward LoRA</th>
</tr>
</thead>
<tr>
<td>
A panda eats bamboo while a monkey swings from branch to branch
<details>
<summary>Expanded</summary>
<p>In a lush green forest, a panda sits comfortably against a tree, leisurely munching on bamboo stalks. Nearby, a lively monkey swings energetically from branch to branch, its tail curling around the limbs. Sunlight filters through the canopy, casting dappled shadows on the forest floor.</p>
</details>
</td>
<td>
<video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset_Wan2_2/reward_lora/14B_baseline_00000001.mp4" width="100%" controls autoplay loop></video>
</td>
<td>
<video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset_Wan2_2/reward_lora/14B_high_hpsv2.1%2Blow_hps2.1_00000001.mp4" width="100%" controls autoplay loop></video>
</td>
<td>
<video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset_Wan2_2/reward_lora/14B_high_mps%2Blow_hpsv2.1_00000001.mp4" width="100%" controls autoplay loop></video>
</td>
</tr>
<tr>
<td>
A dog runs through a field while a cat climbs a tree
<details>
<summary>Expanded</summary>
<p>In a sunlit, expansive green field surrounded by tall trees, a playful golden retriever sprints energetically across the grass, its fur gleaming in the afternoon sun. Nearby, a nimble tabby cat gracefully climbs a sturdy tree, its claws gripping the bark effortlessly. The sky is clear blue with occasional birds flying.</p>
</details>
</td>
<td>
<video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset_Wan2_2/reward_lora/14B_baseline_00000002.mp4" width="100%" controls autoplay loop></video>
</td>
<td>
<video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset_Wan2_2/reward_lora/14B_high_hpsv2.1%2Blow_hps2.1_00000002.mp4" width="100%" controls autoplay loop></video>
</td>
<td>
<video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset_Wan2_2/reward_lora/14B_high_mps%2Blow_hpsv2.1_00000002.mp4" width="100%" controls autoplay loop></video>
</td>
</tr>
<tr>
<td>
A penguin waddles on the ice, a camel treks by
<details>
<summary>Expanded</summary>
<p>A small penguin waddles slowly across a vast, icy surface under a clear blue sky. The penguin's short, flipper-like wings sway at its sides as it moves. Nearby, a camel treks steadily, its long legs navigating the snowy terrain with ease. The camel's fur is thick, providing warmth in the cold environment.</p>
</details>
</td>
<td>
<video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset_Wan2_2/reward_lora/14B_baseline_00000004.mp4" width="100%" controls autoplay loop></video>
</td>
<td>
<video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset_Wan2_2/reward_lora/14B_high_hpsv2.1%2Blow_hps2.1_00000004.mp4" width="100%" controls autoplay loop></video>
</td>
<td>
<video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset_Wan2_2/reward_lora/14B_high_mps%2Blow_hpsv2.1_00000004.mp4" width="100%" controls autoplay loop></video>
</td>
</tr>
<tr>
<td>
Pig with wings flying above a diamond mountain
<details>
<summary>Expanded</summary>
<p>A whimsical pig, complete with delicate feathered wings, soars gracefully above a shimmering diamond mountain. The pig's pink skin glistens in the sunlight as it flaps its wings. The mountain below sparkles with countless facets, reflecting brilliant rays of light into the clear blue sky.</p>
</details>
</td>
<td>
<video src="https://pai-aigc-photog.oss-cn-hangzhou.aliyuncs.com/wan_fun/asset_Wan2_2/reward_lora/14B_baseline_00000008.mp4" width="100%" controls autoplay loop></video>
</td>
<td>
<v

查看官方来源