Skip to content
Python

feat: support avatar-v1.5

lile47fc778719a178
diff --git a/README.md b/README.md
index c64a48a..4e7563d 100644
--- a/README.md
+++ b/README.md
@@ -13,7 +13,7 @@
<a href='https://huggingface.co/meituan-longcat/LongCat-Video'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue'></a>
</div>
<div align="center" style="line-height: 1;">
- <img src='assets/longcat_video_avatar_title.svg' alt="LongCat-Video-Avatar">
+ <img src='assets/longcat_video_avatar_1.5_title.svg' alt="LongCat-Video-Avatar 1.5">
<a href='https://meigen-ai.github.io/LongCat-Video-Avatar/'><img src='https://img.shields.io/badge/Project-Page-green'></a>
<a href='https://github.com/meituan-longcat/LongCat-Video/blob/main/assets/LongCat-Video-Avatar-Tech-Report.pdf'><img src='https://img.shields.io/badge/Technique-Report-red'></a>
<a href='https://huggingface.co/meituan-longcat/LongCat-Video-Avatar'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue'></a>
@@ -45,6 +45,7 @@ For more detail, please refer to the comprehensive [***LongCat-Video Technical R
</div>
## 🔥 Latest News!!
+- May 21, 2026: 🚀 We release [***LongCat-Video-Avatar-1.5***](https://meigen-ai.github.io/LongCat-Video-Avatar-1.5/), an upgraded open-source framework for audio-driven human video generation. 1.5 replaces Wav2Vec2 with Whisper-Large for smoother lip dynamics, achieves production-ready stability with robust long-video generation, generalizes to stylized domains (anime, animals, complex real-world conditions), and accelerates inference to 8 steps via DMD2 distillation. [ [***code***](https://github.com/meituan-longcat/LongCat-Video) | 🤗 [***weights***](https://huggingface.co/meituan-longcat/LongCat-Video-Avatar-1.5) | [***project page***](https://meigen-ai.github.io/LongCat-Video-Avatar-1.5/) ]
- Dec 16, 2025: 🚀 We are excited to announce the release of [***LongCat-Video-Avatar***](https://meigen-ai.github.io/LongCat-Video-Avatar/), a unified model that delivers expressive and highly dynamic audio-driven character animation, supporting native tasks including *Audio-Text-to-Video*, *Audio-Text-Image-to-Video*, and *Video Continuation* with seamless compatibility for both *single-stream* and *multi-stream* audio inputs. The release includes our [***Technical Report***](https://github.com/meituan-longcat/LongCat-Video), [***inference code***](https://github.com/meituan-longcat/LongCat-Video), 🤗 [***model weights***](https://huggingface.co/meituan-longcat/LongCat-Video-Avatar), and [***project page***](https://meigen-ai.github.io/LongCat-Video-Avatar/).
- Oct 25, 2025: 🚀 We've released LongCat-Video, a foundational video generation model. Tech report and models are available at [***LongCat-Video Technical Report***](https://arxiv.org/abs/2510.22200) and 🤗 [***Huggingface***](https://huggingface.co/meituan-longcat/LongCat-Video) !
@@ -94,13 +95,15 @@ FlashAttention-2 is enabled in the model config by default; you can also change
| Models | Description | Download Link |
| --- | --- | --- |
| LongCat-Video | foundational video generation | 🤗 [Huggingface](https://huggingface.co/meituan-longcat/LongCat-Video) |
-| LongCat-Video-Avatar | single- and multi-character audio-driven video generation | 🤗 [Huggingface](https://huggingface.co/meituan-longcat/LongCat-Video-Avatar) |
+| LongCat-Video-Avatar | single- and multi-character audio-driven video generation (wav2vec2) | 🤗 [Huggingface](https://huggingface.co/meituan-longcat/LongCat-Video-Avatar) |
+| LongCat-Video-Avatar-1.5 | upgraded avatar model with Whisper-large-v3 audio encoder, distillation-based fast inference | 🤗 [Huggingface](https://huggingface.co/meituan-longcat/LongCat-Video-Avatar-1.5) |
Download models using huggingface-cli:
```shell
pip install "huggingface_hub[cli]"
huggingface-cli download meituan-longcat/LongCat-Video --local-dir ./weights/LongCat-Video
huggingface-cli download meituan-longcat/LongCat-Video-Avatar --local-dir ./weights/LongCat-Video-Avatar
+huggingface-cli download meituan-longcat/LongCat-Video-Avatar-1.5 --local-dir ./weights/LongCat-Video-Avatar-1.5
```
### Run Text-to-Video
@@ -154,35 +157,55 @@ torchrun --nproc_per_node=2 run_demo_interactive_video.py --context_parallel_siz
```
### Run LongCat-Video-Avatar
-💡 User tips
+<details>
+<summary>💡 User tips for 1.5</summary>
+
+> - **Lip synchronization accuracy:** Audio CFG works optimally between 3–5. Increase the audio CFG value for better synchronization.
+> - **Prompt Enhancement:** Longer, more descriptive prompts yield better consistency and naturalness than short ones. We recommend including rich details such as character appearance, actions, and scene context (e.g., *"A young woman with long black hair is speaking and smiling, wearing a white blouse, sitting in a bright café"*) for best results.
+> - **Mitigate repeated actions:** Setting the reference image index(--ref_img_index, default to 10) between 0 and 24 ensures better consistency; setting it to 30 helps reduce repeated actions. Additionally, increasing the mask frame range (--mask_frame_range, default to 3) can further help mitigate repeated actions, but excessively large values may introduce artifacts.
+> - **Super resolution:** Our model is compatible with both 480P and 720P, which can be controlled via --resolution.
+> - **Dual-Audio Modes:** Merge mode (set audio_type to para) requires two audio clips of equal length, and the resulting audio is obtained by summing the two clips; Concatenation mode (set audio_type to add) does not require equal-length inputs, and the resulting audio is formed by sequentially concatenating the two clips with silence padding for any gaps, where by default person1 speaks first and person2 speaks afterward.
+> - **Model versions:** `--model_type avatar-v1.0` uses wav2vec2 audio encoder (default); `--model_type avatar-v1.5` uses Whisper-large-v3 audio encoder for better lip sync quality.
+> - **Distillation mode:** Add `--use_distill` to enable distillation sampling (fewer steps, faster inference). This is **required** when using `--model_type avatar-v1.5`.
+> - **INT8 quantization:** Add `--use_int8` to load the INT8 quantized DiT model for reduced VRAM usage. Only supported with `--model_type avatar-v1.5`.
+
+</details>
+
+<details>
+<summary>💡 User tips for 1.0</summary>
+
> - Lip synchronization accuracy:​​ Audio CFG works optimally between 3–5. Increase the audio CFG value for better synchronization.
> - Prompt Enhancement: Include clear verbal-action cues (e.g., talking, speaking) in the prompt to achieve more natural lip movements.
> - Mitigate repeated actions: Setting the reference image index(--ref_img_index, default to 10) between 0 and 24 ensures better consistency, while selecting other ranges (e.g., -10 or 30) helps reduce repeated actions. Additionally, increasing the mask frame range (--mask_frame_range, default to 3) can further help mitigate repeated actions, but excessively large values may introduce artifacts.
> - Super resolution: Our model is compatible with both 480P and 720P, which can be controlled via --resolution.
> - Dual-Audio Modes: Merge mode (set audio_type to para) requires two audio clips of equal length, and the resulting audio is obtained by summing the two clips; Concatenation mode (set audio_type to add) does not require equal-length inputs, and the resulting audio is formed by sequentially concatenating the two clips with silence padding for any gaps, where by default person1 speaks first and person2 speaks afterward.
+</details>
+
+#### LongCat-Video-Avatar-1.5
+
- Single-Audio-to-Video Generation
```shell
# Audio-Text-to-Video
-torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar --stage_1=at2v --input_json=assets/avatar/single_example_1.json
+torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --stage_1=at2v --input_json=assets/avatar/single_example_1.json --use_distill --model_type avatar-v1.5 --use_int8
# Audio-Image-to-Video
-torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar --stage_1=ai2v --input_json=assets/avatar/single_example_1.json
+torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --stage_1=ai2v --input_json=assets/avatar/single_example_1.json --use_distill --model_type avatar-v1.5 --use_int8
# Audio-Text-to-Video and Video-Continuation
-torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar --stage_1=at2v --input_json=assets/avatar/single_example_1.json --num_segments=5 --ref_img_index=10 --mask_frame_range=3
+torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --stage_1=at2v --input_json=assets/avatar/single_example_1.json --num_segments=5 --ref_img_index=10 --mask_frame_range=3 --use_distill --model_type avatar-v1.5 --use_int8
# Audio-Image-to-Video and Video-Continuation
-torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar --stage_1=ai2v --input_json=assets/avatar/single_example_1.json --num_segments=5 --ref_img_index=10 --mask_frame_range=3
+torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --stage_1=ai2v --input_json=assets/avatar/single_example_1.json --num_segments=5 --ref_img_index=10 --mask_frame_range=3 --use_distill --model_type avatar-v1.5 --use_int8
```
- Multi-Audio-to-Video Generation
```shell
# Audio-Image-to-Video
-torchrun --nproc_per_node=2 run_demo_avatar_multi_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar --input_json=assets/avatar/multi_example_1.json
+torchrun --nproc_per_node=2 run_demo_avatar_multi_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --input_json=assets/avatar/multi_example_1.json --use_distill --model_type avatar-v1.5 --use_int8
# Audio-Image-to-Video and Video-Continuation
-torchrun --nproc_per_node=2 run_demo_avatar_multi_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar --input_json=assets/avatar/multi_example_1.json --num_segments=5 --ref_img_index=10 --mask_frame_range=3
+torchrun --nproc_per_node=2 run_demo_avatar_multi_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --input_json=assets/avatar/multi_example_1.json --num_segments=5 --ref_img_index=10 --mask_frame_range=3 --use_distill --model_type avatar-v1.5 --use_int8
```
### Run Streamlit
@@ -262,6 +285,15 @@ We kindly encourage citation of our work if you find it useful.
primaryClass={cs.CV},
url={https://arxiv.org/abs/2510.22200},
}
+@misc{meituanlongcatteam2025longcatvideoavatar15technicalreport,
+ title={LongCat-Video-Avatar 1.5 Technical Report},
+ author={Meituan LongCat Team},
+ year={2026},
+ eprint={},
+ archivePrefix={arXiv},
+ primaryClass={cs.CV},
+ url={},
+}
@misc{meituanlongcatteam2025longcatvideoavatartechnicalreport,
title={LongCat-Video-Avatar Technical Report},
author={Meituan LongCat Team},
diff --git a/assets/longcat_video_avatar_1.5_title.svg b/assets/longcat_video_avatar_1.5_title.svg
new file mode 100644
index 0000000..f60fb24
--- /dev/null
+++ b/assets/longcat_video_avatar_1.5_title.svg
@@ -0,0 +1 @@
+<svg xmlns="http://www.w3.org/2000/svg" width="150" height="20"><rect width="100" height="100" x="50" y="50" fill="none"/><text x="71" y="10" font-family="Arial" font-size="13" fill="black" font-weight="bold" alignment-baseline="middle" text-anchor="middle">LongCat-Video-Avatar 1.5</text></svg>
diff --git a/longcat_video/audio_process/__init__.py b/longcat_video/audio_process/__init__.py
new file mode 100644
index 0000000..f98cf6c
--- /dev/null
+++ b/longcat_video/audio_process/__init__.py
@@ -0,0 +1,18 @@
+from .wav2vec2 import Wav2Vec2ModelWrapper
+from transformers import Wav2Vec2FeatureExtractor, WhisperModel, AutoFeatureExtractor
+
+def get_audio_encoder(checkpoint_path, model_type="avatar-v1.0"):
+ if model_type == "avatar-v1.0":
+ model = Wav2Vec2ModelWrapper(checkpoint_path)
+ model.feature_extractor._freeze_parameters()
+ return model
+ if model_type == "avatar-v1.5":
+ model = WhisperModel.from_pretrained(checkpoint_path).eval()
+ model.requires_grad_(False)
+ return model
+
+def get_audio_feature_extractor(checkpoint_path, model_type="avatar-v1.0"):
+ if model_type == "avatar-v1.0":
+ return Wav2Vec2FeatureExtractor(checkpoint_path, local_files_only=True)
+ if model_type == "avatar-v1.5":
+ return AutoFeatureExtractor.from_pretrained(checkpoint_path)
diff --git a/longcat_video/modules/avatar/attention.py b/longcat_video/modules/avatar/attention.py
index 97ee132..a169a7a 100644
--- a/longcat_video/modules/avatar/attention.py
+++ b/longcat_video/modules/avatar/attention.py
@@ -326,7 +326,7 @@ class SingleStreamAttention(nn.Module):
self.rope_bak = int(self.class_range // 2)
self.rope_1d = RotaryPositionalEmbedding1D(self.head_dim)
- def _process_cross_attn(self, x, cond, frames_num=None, x_ref_attn_map=None):
+ def _process_cross_attn(self, x, cond, frames_num=None, x_ref_attn_map=None, human_num=None):
N_t = frames_num
out_dtype = x.dtype
@@ -351,8 +351,9 @@ class SingleStreamAttention(nn.Module):
human1 = normalize_and_scale(x_ref_attn_map[0], (human1_min_value, human1_max_value), (self.rope_h1[0], self.rope_h1[1]))
human2 = normalize_and_scale(x_ref_attn_map[1], (human2_min_value, human2_max_value), (self.rope_h2[0], self.rope_h2[1]))
- back = torch.full((x_ref_attn_map.size(1),), self.rope_bak, dtype=human1.dtype).to(human1.device)
- max_indices = x_ref_attn_map.argmax(dim=0)
+ background_pos = self.rope_bak if x_ref_attn_map.shape[0] <= 3 else 100
+ back = torch.full((x_ref_attn_map.size(1),), background_pos, dtype=human1.dtype).to(human1.device)
+ max_indices = x_ref_attn_map.argmax(dim=0).clamp(max=2)
normalized_map = torch.stack([human1, human2, back], dim=1)
normalized_pos = normalized_map[range(x_ref_attn_map.size(1)), max_indices]
@@ -373,8 +374,17 @@ class SingleStreamAttention(nn.Module):
# multitalk with rope1d pe
if x_ref_attn_map is not None:
per_frame = torch.zeros(N_a, dtype=encoder_k.dtype).to(encoder_k.device)
- per_frame[:per_frame.size(0)//2] = (self.rope_h1[0] + self.rope_h1[1]) / 2
- per_frame[per_frame.size(0)//2:] = (self.rope_h2[0] + self.rope_h2[1]) / 2
+ human1_pos = (self.rope_h1[0] + self.rope_h1[1]) / 2
+ human2_pos = (self.rope_h2[0] + self.rope_h2[1]) / 2
+ if human_num is not None and human_num > 2:
+ background_pos = self.rope_bak if x_ref_attn_map.shape[0] <= 3 else 100
+ tokens_per_human = per_frame.size(0) // human_num
+ per_frame[:tokens_per_human] = human1_pos
+ per_frame[tokens_per_human:2*tokens_per_human] = human2_pos
+ per_frame[2*tokens_per_human:] = background_pos
+ else:
+ per_frame[:per_frame.size(0)//2] = human1_pos
+ per_frame[per_frame.size(0)//2:] = human2_pos
encoder_pos = torch.concat([per_frame]*N_t, dim=0)
encoder_k = rearrange(encoder_k, "(B N_t) H S C -> B H (N_t S) C", N_t=N_t)
encoder_k = self.rope_1d(encoder_k, encoder_pos)
@@ -434,7 +444,7 @@ class SingleStreamAttention(nn.Module):
B, N, C = x.shape
if (num_cond_latents is None or num_cond_latents == 0):
# text to video
- output = self._process_cross_attn(x, cond, shape[0], x_ref_attn_map)
+ output = self._process_cross_attn(x, cond, shape[0], x_ref_attn_map, human_num=human_num)
return None, output
elif num_cond_latents is not None and num_cond_latents > 0:
# image to video or video continuation
@@ -448,8 +458,10 @@ class SingleStreamAttention(nn.Module):
if human_num is not None and human_num == 2:
# multitalk mode
output_noise = self._process_cross_attn(x_noise, cond, frames_num, x_ref_attn_map)
+ elif human_num is not None and human_num > 2:
+ # multitalk mode
+ output_noise = self._process_cross_attn(x_noise, cond, frames_num, x_ref_attn_map, human_num)
else:
- # singletalk mode
output_noise = self._process_cross_attn(x_noise, cond, frames_num)
output_cond = torch.zeros((B, num_cond_latents_thw, C), dtype=output_noise.dtype, device=output_noise.device)
return output_cond, output_noise
diff --git a/longcat_video/modules/avatar/longcat_video_dit_avatar.py b/longcat_video/modules/avatar/longcat_video_dit_avatar.py
index a30136a..44593e5 100644
--- a/longcat_video/modules/avatar/longcat_video_dit_avatar.py
+++ b/longcat_video/modules/avatar/longcat_video_dit_avatar.py
@@ -220,6 +220,8 @@ class LongCatVideoAvatarTransformer3DModel(
text_tokens_zero_pad: bool = False,
# avatar config
audio_window: int = 5,
+ audio_block: int = 12,
+ audio_channel: int = 768,
intermediate_dim: int = 512,
output_dim: int = 768,
context_tokens: int = 32,
@@ -269,6 +271,8 @@ class LongCatVideoAvatarTransformer3DModel(
self.audio_proj = AudioProjModel(
seq_len=audio_window,
seq_len_vf=audio_window+vae_scale-1,
+ blocks=audio_block,
+ channels=audio_channel,
intermediate_dim=intermediate_dim,
output_dim=output_dim,
context_tokens=context_tokens
diff --git a/longcat_video/modules/lora_utils.py b/longcat_video/modules/lora_utils.py
index 68ce476..ce3e892 100644
--- a/longcat_video/modules/lora_utils.py
+++ b/longcat_video/modules/lora_utils.py
@@ -41,7 +41,8 @@ class LoRAModule(torch.nn.Module):
super().__init__()
self.lora_name = lora_name
- assert org_module.__class__.__name__ == "Linear"
+ assert org_module.__class__.__name__ in ("Linear", "QuantizedLinear"), \
+ f"LoRA target must be Linear or QuantizedLinear, got {org_module.__class__.__name__}"
in_dim = org_module.in_features
out_dim = org_module.out_features
@@ -116,7 +117,7 @@ class LoRANetwork(torch.nn.Module):
except Exception as e:
print(f"Cannot find module: {module_name}, error: {e}")
continue
- if module.__class__.__name__ != "Linear":
+ if module.__class__.__name__ not in ("Linear", "QuantizedLinear"):
continue
# 推断 n_seperate
diff --git a/longcat_video/modules/quantization.py b/longcat_video/modules/quantization.py
new file mode 100644
index 0000000..17c775e
--- /dev/null
+++ b/longcat_video/modules/quantization.py
@@ -0,0 +1,239 @@
+import os
+import json
+import shutil
+from typing import Optional, Set
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from safetensors.torch import save_file, load_file
+
+
+class QuantizedLinear(nn.Module):
+ """INT8 weight-only quantized linear layer with per-channel symmetric quantization."""
+
+ def __init__(self, in_features: int, out_features: int, bias: bool = False):
+ super().__init__()
+ self.in_features = in_features
+ self.out_features = out_features
+ self.register_buffer("weight_int8", torch.zeros(out_features, in_features, dtype=torch.int8))
+ self.register_buffer("weight_scale", torch.zeros(out_features, dtype=torch.float32))
+ if bias:
+ self.register_buffer("bias", torch.zeros(out_features, dtype=torch.bfloat16))
+ else:
+ self.bias = None
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ # Dequantize weight to input dtype for computation
+ compute_dtype = x.dtype
+ weight = self.weight_int8.to(compute_dtype) * self.weight_scale.to(compute_dtype).unsqueeze(1)
+ bias = self.bias.to(compute_dtype) if self.bias is not None else None
+ return F.linear(x, weight, bias)
+
+ @classmethod
+ def from_linear(cls, linear: nn.Linear) -> "QuantizedLinear":
+ """Quantize a standard Linear layer to INT8."""
+ has_bias = linear.bias is not None
+ ql = cls(linear.in_features, linear.out_features, bias=has_bias)
+
+ weight = linear.weight.data.float()
+ # Per-channel symmetric quantization
+ scale = weight.abs().amax(dim=1).clamp(min=1e-8) / 127.0
+ weight_int8 = (weight / scale.unsqueeze(1)).round().clamp(-128, 127).to(torch.int8)
+
+ ql.weight_int8 = weight_int8
+ ql.weight_scale = scale
+ if has_bias:
+ ql.bias = linear.bias.data.to(torch.bfloat16)
+ return ql
+
+ def extra_repr(self) -> str:
+ return f"in_features={self.in_features}, out_features={self.out_features}, bias={self.bias is not None}"
+
+
+# Layers to skip quantization (sensitive to precision)
+DEFAULT_SKIP_PATTERNS = {
+ "final_layer.linear", # Final output projection, precision-sensitive
+}
+
+
+def quantize_model(model: nn.Module, skip_patterns: Optional[Set[str]] = None) -> nn.Module:
+ """Replace all nn.Linear layers in the model with QuantizedLinear (INT8 weight-only).
+
+ Args:
+ model: The model to quantize (modified in-place).
+ skip_patterns: Set of module name patterns to skip. If None, uses DEFAULT_SKIP_PATTERNS.
+
+ Returns:
+ The quantized model.
+ """
+ if skip_patterns is None:
+ skip_patterns = DEFAULT_SKIP_PATTERNS
+
+ modules_to_replace = {}
+ for name, module in model.named_modules():
+ if isinstance(module, nn.Linear):
+ # Check if this module should be skipped
+ should_skip = any(pattern in name for pattern in skip_patterns)
+ if not should_skip:
+ modules_to_replace[name] = module
+
+ for name, linear in modules_to_replace.items():
+ # Navigate to the parent module
+ parts = name.split(".")
+ parent = model
+ for part in parts[:-1]:
+ parent = getattr(parent, part)
+ # Replace with quantized version
+ setattr(parent, parts[-1], QuantizedLinear.from_linear(linear))
+
+ return model
+
+
+def save_quantized_state_dict(model: nn.Module, save_dir: str, config_source_dir: Optional[str] = None):
+ """Save the quantized model's state dict using safetensors format.
+
+ Saves quantized weights in shards to avoid single large files.
+
+ Args:
+ model: The quantized model.
+ save_dir: Directory to save quantized weights.
+ config_source_dir: If provided, copy config.json from this directory.
+ """
+ os.makedirs(save_dir, exist_ok=True)
+
+ state_dict = model.state_dict()
+
+ # Split into shards (~4GB each for manageable files)
+ max_shard_size = 4 * 1024 * 1024 * 1024 # 4GB
+
+ shards = []
+ current_shard = {}
+ current_size = 0
+
+ for key, tensor in state_dict.items():
+ tensor_size = tensor.numel() * tensor.element_size()
+ if current_size + tensor_size > max_shard_size and current_shard:
+ shards.append(current_shard)
+ current_shard = {}
+ current_size = 0
+ current_shard[key] = tensor
+ current_size += tensor_size
+
+ if current_shard:
+ shards.append(current_shard)
+
+ # Save each shard
+ index = {"metadata": {"total_size": sum(t.numel() * t.element_size() for t in state_dict.values())}, "weight_map": {}}
+
+ for i, shard in enumerate(shards):
+ shard_name = f"quantized_model-{i+1:05d}-of-{len(shards):05d}.safetensors"
+ save_file(shard, os.path.join(save_dir, shard_name))
+ for key in shard:
+ index["weight_map"][key] = shard_name
+
+ # Save index
+ index_path = os.path.join(save_dir, "quantized_model.safetensors.index.json")
+ with open(index_path, "w") as f:
+ json.dump(index, f, indent=2)
+
+ # Copy config if source provided
+ if config_source_dir:
+ src_config = os.path.join(config_source_dir, "config.json")
+ if os.path.exists(src_config):
+ shutil.copy2(src_config, os.path.join(save_dir, "config.json"))
+
+ # Save quantization metadata
+ quant_config = {
+ "quantization_method": "int8_per_channel_symmetric",
+ "skip_patterns": list(DEFAULT_SKIP_PATTERNS),
+ "description": "Weight-only INT8 quantization with per-channel symmetric scaling"
+ }
+ with open(os.path.join(save_dir, "quantization_config.json"), "w") as f:
+ json.dump(quant_config, f, indent=2)
+
+ print(f"Saved {len(shards)} shard(s) to {save_dir}")
+
+
+def load_quantized_dit(checkpoint_dir: str, subfolder: str = "base_model_int8", **kwargs):
+ """Load a quantized DiT model.
+
+ Args:
+ checkpoint_dir: Base checkpoint directory.
+ subfolder: Subfolder containing quantized weights (default: 'base_model_int8').
+ **kwargs: Additional kwargs passed to the model constructor (e.g., cp_split_hw).
+
+ Returns:
+ The quantized DiT model ready for inference.
+ """
+ from .avatar.longcat_video_dit_avatar import LongCatVideoAvatarTransformer3DModel
+
+ quantized_dir = os.path.join(checkpoint_dir, subfolder)
+
+ # Load config
+ config_path = os.path.join(quantized_dir, "config.json")
+ with open(config_path, "r") as f:
+ config = json.load(f)
+
+ # Remove non-constructor keys
+ config.pop("_class_name", None)
+ config.pop("architectures", None)
+ config.pop("_diffusers_version", None)
+ config.pop("model_max_length", None)
+
+ # Override with kwargs
+ config.update(kwargs)
+
+ # Instantiate model (empty weights)
+ model = LongCatVideoAvatarTransformer3DModel(**config)
+
+ # Replace Linear layers with QuantizedLinear (empty)
+ skip_patterns = DEFAULT_SKIP_PATTERNS
+ modules_to_replace = {}
+ for name, module in model.named_modules():
+ if isinstance(module, nn.Linear):
+ should_skip = any(pattern in name for pattern in skip_patterns)
+ if not should_skip:
+ ql = QuantizedLinear(module.in_features, module.out_features, bias=module.bias is not None)
+ modules_to_replace[name] = ql
+
+ for name, ql in modules_to_replace.items():
+ parts = name.split(".")
+ parent = model
+ for part in parts[:-1]:
+ parent = getattr(parent, part)
+ setattr(parent, parts[-1], ql)
+
+ # Load quantized state dict
+ index_path = os.path.join(quantized_dir, "quantized_model.safetensors.index.json")
+ if os.path.exists(index_path):
+ with open(index_path, "r") as f:
+ index = json.load(f)
+ # Load from shards
+ shard_files = set(index["weight_map"].values())
+ state_dict = {}
+ for shard_file in sorted(shard_files):
+ shard_path = os.path.join(quantized_dir, shard_file)
+ shard_dict = load_file(shard_path, device="cpu")
+ state_dict.update(shard_dict)
+ else:
+ # Single file fallback
+ files = [f for f in os.listdir(quantized_dir) if f.endswith(".safetensors") and "index" not in f]
+ state_dict = {}
+ for f in sorted(files):
+ shard_dict = load_file(os.path.join(quantized_dir, f), device="cpu")
+ state_dict.update(shard_dict)
+
+ model.load_state_dict(state_dict, strict=True)
+ model.eval()
+
+ # Cast non-quantized parameters (Conv3d, LayerNorm, etc.) to bfloat16
+ # QuantizedLinear buffers (int8, float32 scale) are kept as-is
+ for name, module in model.named_modules():
+ if isinstance(module, QuantizedLinear):
+ continue
+ for param_name, param in module.named_parameters(recurse=False):
+ if param.dtype == torch.float32:
+ param.data = param.data.to(torch.bfloat16)
+
+ return model
diff --git a/longcat_video/pipeline_longcat_video_avatar.py b/longcat_video/pipeline_longcat_video_avatar.py
index ec22c07..5caf1c5 100644
--- a/longcat_video/pipeline_longcat_video_avatar.py
+++ b/longcat_video/pipeline_longcat_video_avatar.py
@@ -27,8 +27,6 @@ import html
# -------- avatar related --------
import scipy.signal as ss
import pyloudnorm as pyln
-from longcat_video.audio_process.wav2vec2 import Wav2Vec2ModelWrapper
-from transformers import Wav2Vec2FeatureExtractor
from diffusers.image_processor import is_valid_image, is_valid_image_imagelist
import warnings
@@ -120,8 +118,9 @@ class LongCatVideoAvatarPipeline:
vae: AutoencoderKLWan,
scheduler: FlowMatchEulerDiscreteScheduler,
dit: LongCatVideoAvatarTransformer3DModel,
- audio_encoder: Wav2Vec2ModelWrapper,
- wav2vec_feature_extractor: Wav2Vec2FeatureExtractor
+ audio_encoder = None,
+ audio_feature_extractor = None,
+ model_type="avatar-v1.0",
):
self.vae = vae
self.text_encoder = text_encoder
@@ -138,8 +137,14 @@ class LongCatVideoAvatarPipeline:
self._num_timesteps = 1000
self._num_distill_sample_steps = 50
- self.audio_encoder=audio_encoder
- self.wav2vec_feature_extractor = wav2vec_feature_extractor
+ self.audio_encoder = audio_encoder
+ self.audio_feature_extractor = audio_feature_extractor
+
+ self.model_type = model_type
+ self.use_vcond = True
+ if self.model_type == "avatar-v1.5":
+ self.use_vcond = False
+ self._num_distill_sample_steps = 8
def _get_t5_prompt_embeds(
self,
@@ -373,14 +378,20 @@ class LongCatVideoAvatarPipeline:
return self._attention_kwargs
def get_timesteps_sigmas(self, sampling_steps: int, use_distill: bool=False):
- if use_distill:
- distill_indices = torch.arange(1, self.num_distill_sample_steps + 1, dtype=torch.float32)
- distill_indices = (distill_indices * (self.num_timesteps // self.num_distill_sample_steps)).round().long()
-
- inference_indices = np.linspace(0, self.num_distill_sample_steps, num=sampling_steps, endpoint=False)
- inference_indices = np.floor(inference_indices).astype(np.int64)
+ if use_distill:
+ if self.model_type == "avatar-v1.5":
+ distill_indices = torch.arange(1, self.num_distill_sample_steps + 1, dtype=torch.float32)
+ distill_indices = (distill_indices * (self.num_timesteps // self.num_distill_sample_steps)).round().long()
+ distill_indices = self.num_timesteps - distill_indices
+ sigmas = torch.flip(torch.linspace(0, 1, self.num_timesteps), [0])
+ sigmas = torch.flip(sigmas[distill_indices], [0]).float()
+ else:
+ distill_indices = torch.arange(1, self.num_distill_sample_steps + 1, dtype=torch.float32)
+ distill_indices = (distill_indices * (self.num_timesteps // self.num_distill_sample_steps)).round().long()
+ inference_indices = np.linspace(0, self.num_distill_sample_steps, num=sampling_steps, endpoint=False)
+ inference_indices = np.floor(inference_indices).astype(np.int64)
- sigmas = torch.flip(distill_indices, [0])[inference_indices].float() / self.num_timesteps
+ sigmas = torch.flip(distill_indices, [0])[inference_indices].float() / self.num_timesteps
else:
sigmas = torch.linspace(1, 0.001, sampling_steps)
sigmas = sigmas.to(torch.float32)
@@ -524,8 +535,84 @@ class LongCatVideoAvatarPipeline:
else:
raise NotImplementedError(f"Not supported resize_mode {resize_mode}")
+ def get_audio_embedding(self, speech_array, fps=32, device='cpu', sample_rate=16000, model_type="avatar-v1.0"):
+ if model_type == "avatar-v1.0":
+ return self.get_audio_embedding_wav2vec2(speech_array, fps, device, sample_rate)
+ if model_type == "avatar-v1.5":
+ return self.get_audio_embedding_whisper(speech_array, fps, device, sample_rate)
+
@torch.no_grad()
- def get_audio_embedding(self, speech_array, fps=32, device='cpu', sample_rate=16000):
+ def get_audio_embedding_whisper(self, speech_array, fps=25, device='cpu', sample_rate=16000):
+ """使用 Whisper encoder 提取音频特征。
+ Args:
+ speech_array: 原始音频波形 (numpy array, 单声道, sample_rate=16000)
+ fps: 目标视频帧率
+ device: 推理设备
+ sample_rate: 音频采样率
+ Returns:
+ audio_emb: [T, 5, D],T = int(audio_duration * fps)
+ """
+ def linear_interpolation_fps(features, input_fps, output_fps, output_len=None):
+ """将音频特征从 input_fps 插值到 output_fps。
+ Args:
+ features: [B, T, D]
+ input_fps: 源帧率
+ output_fps: 目标帧率
+ output_len: 若指定则直接用该长度,否则按帧率比换算
+ Returns:
+ [B, output_len, D]
+ """
+ features = features.transpose(1, 2) # [B, D, T]
+ if output_len is None:
+ output_len = int(features.shape[2] / float(input_fps) * output_fps)
+ output_features = F.interpolate(features, size=output_len, align_corners=True, mode='linear')
+ return output_features.transpose(1, 2)
+ # ---- 常量 ----
+ MEL_CHUNK = 750 * 640 # feature extractor 滑窗大小(样本数)
+ ENC_CHUNK = 3000 # encoder 滑窗大小(mel 帧数)
+ ENC_FPS = 50 # encoder 输出帧率
+
+ # ---- 时长 / 帧数 ----
+ audio_duration = len(speech_array) / sample_rate
+ video_length = int(audio_duration * fps)
+
+ # ---- 音频预处理 ----
+ speech_array = self._loudness_norm(speech_array, sample_rate)
+
+ # ---- Whisper feature extractor:wav → mel spectrogram ----
+ mel_chunks = []
+ for i in range(0, len(speech_array), MEL_CHUNK):
+ mel = self.audio_feature_extractor(
+ speech_array[i: i + MEL_CHUNK],
+ sampling_rate=sample_rate,
+ return_tensors="pt",
+ ).input_features # [1, mel_bins, T_mel]
+ mel_chunks.append(mel)
+ audio_features = torch.cat(mel_chunks, dim=-1) # [1, mel_bins, T_mel_total]
+ audio_features = audio_features.to(self.audio_encoder.dtype)
+
+ # ---- Whisper encoder:mel → hidden states ----
+ enc_chunks = []
+ for i in range(0, audio_features.shape[-1], ENC_CHUNK):
+ chunk_hs = self.audio_encoder.encoder(
+ audio_features[:, :, i: i + ENC_CHUNK].to(device),
+ output_hidden_states=True,
+ ).hidden_states # tuple: (n_layers+1,) x [1, T_enc, D]
+ enc_chunks.append(torch.stack(chunk_hs, dim=2)) # [1, T_enc, n_layers, D]
+ audio_prompts = torch.cat(enc_chunks, dim=1) # [1, T_enc_total, n_layers, D]
+ audio_prompts = audio_prompts[:, :video_length * 2] # 截取有效帧
+
+ # ---- 按层分组 + 插值到目标帧数 ----
+ feat0 = linear_interpolation_fps(audio_prompts[:, :, 0: 8].mean(dim=2), ENC_FPS, fps, video_length)
+ feat1 = linear_interpolation_fps(audio_prompts[:, :, 8:16].mean(dim=2), ENC_FPS, fps, video_length)
+ feat2 = linear_interpolation_fps(audio_prompts[:, :, 16:24].mean(dim=2), ENC_FPS, fps, video_length)
+ feat3 = linear_interpolation_fps(audio_prompts[:, :, 24:32].mean(dim=2), ENC_FPS, fps, video_length)
+ feat4 = linear_interpolation_fps(audio_prompts[:, :, 32], ENC_FPS, fps, video_length)
+ audio_emb = torch.stack([feat0, feat1, feat2, feat3, feat4], dim=2)[0] # [T, 5, D]
+ return audio_emb
+
+ @torch.no_grad()
+ def get_audio_embedding_wav2vec2(self, speech_array, fps=32, device='cpu', sample_rate=16000):
audio_duration = len(speech_array) / sample_rate
video_length = audio_duration * fps
@@ -537,7 +624,7 @@ class LongCatVideoAvatarPipeline:
# wav2vec_feature_extractor
audio_feature = np.squeeze(
- self.wav2vec_feature_extractor(speech_array, sampling_rate=sample_rate).input_values
+ self.audio_feature_extractor(speech_array, sampling_rate=sample_rate).input_values
)
audio_feature = torch.from_numpy(audio_feature).float().to(device=device)
audio_feature = audio_feature.unsqueeze(0)
@@ -711,7 +798,7 @@ class LongCatVideoAvatarPipeline:
)
if context_parallel_util.get_cp_size() > 1:
context_parallel_util.cp_broadcast(latents)
-
+
# 6. Denoising loop
if context_parallel_util.get_cp_size() > 1:
torch.distributed.barrier(group=context_parallel_util.get_cp_group())
@@ -923,6 +1010,7 @@ class LongCatVideoAvatarPipeline:
context_parallel_util.cp_broadcast(negative_prompt_attention_mask)
audio_cond_embs = torch.cat([audio_emb] * num_videos_per_prompt, dim=0)
+ audio_num = audio_cond_embs.shape[0]
if self.do_classifier_free_guidance:
prompt_embeds = torch.cat([negative_prompt_embeds, prompt_embeds], dim=0)
prompt_attention_mask = torch.cat([negative_prompt_attention_mask, prompt_attention_mask], dim=0)
@@ -979,14 +1067,14 @@ class LongCatVideoAvatarPipeline:
timestep[:, :1] = 0
if self.do_classifier_free_guidance and ref_target_masks is not None:
- # Multitalk mode
+ # Multitalk mode with CFG
noise_pred_uncond_text = self.dit(
hidden_states=latent_model_input[:1],
timestep=timestep[:1],
encoder_hidden_states=prompt_embeds[:1],
encoder_attention_mask=prompt_attention_mask[:1],
num_cond_latents=1,
- audio_embs=audio_cond_embs[:2],
+ audio_embs=audio_cond_embs[:audio_num],
ref_target_masks=ref_target_masks
)
noise_pred_cond = self.dit(
@@ -995,19 +1083,21 @@ class LongCatVideoAvatarPipeline:
encoder_hidden_states=prompt_embeds[1:],
encoder_attention_mask=prompt_attention_mask[1:],
num_cond_latents=1,
- audio_embs=audio_cond_embs[2:],
+ audio_embs=audio_cond_embs[audio_num:2*audio_num],
ref_target_masks=ref_target_masks
)
noise_pred = torch.cat([noise_pred_uncond_text, noise_pred_cond])
else:
# Singletalk mode
+ # Multitalk mode without CFG (e.g. use_distill with scale=1.0)
noise_pred = self.dit(
hidden_states=latent_model_input,
timestep=timestep,
encoder_hidden_states=prompt_embeds,
encoder_attention_mask=prompt_attention_mask,
num_cond_latents=1,
- audio_embs=audio_cond_embs
+ audio_embs=audio_cond_embs,
+ ref_target_masks=ref_target_masks
)
if self.do_classifier_free_guidance:
@@ -1222,6 +1312,7 @@ class LongCatVideoAvatarPipeline:
context_parallel_util.cp_broadcast(negative_prompt_attention_mask)
audio_cond_embs = torch.cat([audio_emb] * num_videos_per_prompt, dim=0)
+ audio_num = audio_cond_embs.shape[0]
if self.do_classifier_free_guidance:
prompt_embeds = torch.cat([negative_prompt_embeds, prompt_embeds], dim=0)
prompt_attention_mask = torch.cat([negative_prompt_attention_mask, prompt_attention_mask], dim=0)
@@ -1267,10 +1358,13 @@ class LongCatVideoAvatarPipeline:
latents=latents,
need_encode=False
)
- if context_parallel_util.get_cp_size() > 1:
- context_parallel_util.cp_broadcast(latents)
num_cond_latents = 1 + (num_cond_frames - 1) // self.vae_scale_factor_temporal
+ if not self.use_vcond:
+ latents[:, :, :num_cond_latents] = cond_videos_latents
+
+ if context_parallel_util.get_cp_size() > 1:
+ context_parallel_util.cp_broadcast(latents)
# 6. Prepare ref_target_masks from source size to latent size
if ref_target_masks is not None:
@@ -1311,7 +1405,7 @@ class LongCatVideoAvatarPipeline:
timestep[:, :num_cond_latents] = 0
if self.do_classifier_free_guidance and ref_target_masks is not None:
- # Multitalk mode
+ # Multitalk mode with CFG
noise_pred_uncond_text = self.dit(
hidden_states=latent_model_input[:1],
timestep=timestep[:1],
@@ -1319,7 +1413,7 @@ class LongCatVideoAvatarPipeline:
encoder_attention_mask=prompt_attention_mask[:1],
num_cond_latents=num_cond_latents,
kv_cache_dict=kv_cache_dict,
- audio_embs=audio_cond_embs[:2],
+ audio_embs=audio_cond_embs[:audio_num],
num_ref_latents=num_ref_latents,
ref_img_index=ref_img_index,
mask_frame_range=mask_frame_range,
@@ -1332,7 +1426,7 @@ class LongCatVideoAvatarPipeline:
encoder_attention_mask=prompt_attention_mask[1:],
num_cond_latents=num_cond_latents,
kv_cache_dict=kv_cache_dict,
- audio_embs=audio_cond_embs[2:],
+ audio_embs=audio_cond_embs[audio_num:2*audio_num],
num_ref_latents=num_ref_latents,
ref_img_index=ref_img_index,
mask_frame_range=mask_frame_range,
@@ -1341,6 +1435,7 @@ class LongCatVideoAvatarPipeline:
noise_pred = torch.cat([noise_pred_uncond_text, noise_pred_cond])
else:
# Singletalk mode
+ # Multitalk mode without CFG (e.g. use_distill with scale=1.0)
noise_pred = self.dit(
hidden_states=latent_model_input,
timestep=timestep,
@@ -1351,7 +1446,8 @@ class LongCatVideoAvatarPipeline:
audio_embs=audio_cond_embs,
num_ref_latents=num_ref_latents,
ref_img_index=ref_img_index,
- mask_frame_range=mask_frame_range
+ mask_frame_range=mask_frame_range,
+ ref_target_masks=ref_target_masks
)
if self.do_classifier_free_guidance:
diff --git a/run_demo_avatar_multi_audio_to_video.py b/run_demo_avatar_multi_audio_to_video.py
index 2cf14d1..f867be5 100644
--- a/run_demo_avatar_multi_audio_to_video.py
+++ b/run_demo_avatar_multi_audio_to_video.py
@@ -19,14 +19,14 @@ from longcat_video.pipeline_longcat_video_avatar import LongCatVideoAvatarPipeli
from longcat_video.modules.scheduling_flow_match_euler_discrete import FlowMatchEulerDiscreteScheduler
from longcat_video.modules.autoencoder_kl_wan import AutoencoderKLWan
from longcat_video.modules.avatar.longcat_video_dit_avatar import LongCatVideoAvatarTransformer3DModel
+from longcat_video.modules.quantization import load_quantized_dit
from longcat_video.context_parallel import context_parallel_util
# -------- avatar related --------
import librosa
import soundfile as sf
-from longcat_video.audio_process.wav2vec2 import Wav2Vec2ModelWrapper
+from longcat_video.audio_process import get_audio_encoder, get_audio_feature_extractor
from longcat_video.audio_process.torch_utils import save_video_ffmpeg
-from transformers import Wav2Vec2FeatureExtractor
from audio_separator.separator import Separator
@@ -78,7 +78,7 @@ def audio_prepare_multi(left_temp_vocal_path, right_temp_vocal_path, generate_du
right_speech_array_ext = np.concatenate([np.zeros_like(left_speech_array), right_speech_array])
merge_raw_speech = np.concatenate([left_raw_speech_array, np.zeros_like(right_raw_speech_array)]) + \
np.concatenate([np.zeros_like(left_raw_speech_array), right_raw_speech_array])
- elif audio_type == 'para':
+ elif audio_type == 'para':
left_speech_array_ext = left_speech_array
right_speech_array_ext = right_speech_array
merge_raw_speech = left_raw_speech_array + right_raw_speech_array
@@ -109,12 +109,23 @@ def generate(args):
resolution = args.resolution
num_segments = max(1, args.num_segments)
output_dir = args.output_dir
+ model_type = args.model_type
+ use_distill = args.use_distill
+ use_int8 = args.use_int8
+
+ if use_distill and model_type == "avatar-v1.5":
+ num_inference_steps = 8
+ text_guidance_scale = 1.0
+ audio_guidance_scale = 1.0
# set up default inference params
save_fps = 16
+ audio_stride = 2
+ if model_type == "avatar-v1.5":
+ save_fps = 25
+ audio_stride = 1
num_frames = 93
num_cond_frames = 13
- audio_stride = 2
if resolution == '480p':
height, width = 480, 832
@@ -130,11 +141,14 @@ def generate(args):
right_raw_speech_path = input_data['cond_audio'].get('person2', None)
assert left_raw_speech_path is not None or right_raw_speech_path is not None, f"At least one speech is required."
left_person_bbox, right_person_bbox = None, None
+ use_background_silent_audio = False
if 'bbox' in input_data:
# bbox: [left_y_min, left_x_min, left_y_max, left_x_max]
# x and y coordinates correspond to the width and height dimensions, respectively
left_person_bbox = input_data['bbox'].get('person1', None)
right_person_bbox = input_data['bbox'].get('person2', None)
+ other_person_bbox = input_data['bbox'].get('others', None)
+ use_background_silent_audio = other_person_bbox is not None and len(other_person_bbox) > 0
audio_type = 'para'
if 'audio_type' in input_data:
audio_type = input_data.get('audio_type', 'para')
@@ -158,15 +172,35 @@ def generate(args):
tokenizer = AutoTokenizer.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="tokenizer", torch_dtype=torch.bfloat16)
text_encoder = UMT5EncoderModel.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="text_encoder", torch_dtype=torch.bfloat16)
vae = AutoencoderKLWan.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="vae", torch_dtype=torch.bfloat16)
- scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="scheduler", torch_dtype=torch.bfloat16)
- dit = LongCatVideoAvatarTransformer3DModel.from_pretrained(checkpoint_dir, subfolder="avatar_multi", cp_split_hw=cp_split_hw, torch_dtype=torch.bfloat16)
+ if model_type == "avatar-v1.0":
+ scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="scheduler", torch_dtype=torch.bfloat16)
+ elif model_type == "avatar-v1.5":
+ scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(checkpoint_dir, subfolder="scheduler", torch_dtype=torch.bfloat16)
+ else:
+ raise ValueError(f"Unsupported model_type: {model_type}. Expected 'avatar-v1.0' or 'avatar-v1.5'.")
+
+ if model_type == "avatar-v1.0":
+ dit = LongCatVideoAvatarTransformer3DModel.from_pretrained(checkpoint_dir, subfolder="avatar_multi", cp_split_hw=cp_split_hw, torch_dtype=torch.bfloat16)
+ elif model_type == "avatar-v1.5":
+ if use_int8:
+ print("[INFO] Loading INT8 quantized DiT model...")
+ dit = load_quantized_dit(checkpoint_dir, subfolder="base_model_int8", cp_split_hw=cp_split_hw)
+ else:
+ dit = LongCatVideoAvatarTransformer3DModel.from_pretrained(checkpoint_dir, subfolder="base_model", cp_split_hw=cp_split_hw, torch_dtype=torch.bfloat16)
+ if use_distill:
+ distill_checkpoint_path = os.path.join(checkpoint_dir, 'lora', f'dmd_lora.safetensors')
+ if os.path.exists(distill_checkpoint_path):
+ dit.load_lora(distill_checkpoint_path, "dmd", multiplier=1.0, lora_network_dim=128, lora_network_alpha=64)
+ dit.enable_loras(["dmd"])
# initialize audio models
- wav2vec_path = os.path.join(checkpoint_dir, 'chinese-wav2vec2-base')
- audio_encoder = Wav2Vec2ModelWrapper(wav2vec_path).to(local_rank)
- audio_encoder.feature_extractor._freeze_parameters()
+ if model_type == "avatar-v1.0":
+ audio_model_checkpoint_path = os.path.join(checkpoint_dir, 'chinese-wav2vec2-base')
+ elif model_type == "avatar-v1.5":
+ audio_model_checkpoint_path = os.path.join(checkpoint_dir, 'whisper-large-v3')
+ audio_encoder = get_audio_encoder(audio_model_checkpoint_path, model_type).to(local_rank)
+ audio_feature_extractor = get_audio_feature_extractor(audio_model_checkpoint_path, model_type)
- wav2vec_feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(wav2vec_path, local_files_only=True)
vocal_separator_path = os.path.join(checkpoint_dir, 'vocal_separator/Kim_Vocal_2.onnx')
audio_output_dir_temp = f"./audio_temp_file"
os.makedirs(audio_output_dir_temp, exist_ok=True)
@@ -189,7 +223,8 @@ def generate(args):
scheduler = scheduler,
dit = dit,
audio_encoder=audio_encoder,
- wav2vec_feature_extractor=wav2vec_feature_extractor
+ audio_feature_extractor=audio_feature_extractor,
+ model_type=model_type
)
pipe.to(local_rank)
@@ -202,8 +237,10 @@ def generate(args):
if cp_rank == 0:
# extract vocal
sr = 16000
- left_temp_vocal_path = extract_vocal_from_speech(left_raw_speech_path, f"/tmp/temp_speech_{generate_random_uid()}_{global_rank}_vocal.wav", vocal_separator, audio_output_dir_temp)
- right_temp_vocal_path = extract_vocal_from_speech(right_raw_speech_path, f"/tmp/temp_speech_{generate_random_uid()}_{global_rank}_vocal.wav", vocal_separator, audio_output_dir_temp)
+ left_temp_vocal_path = os.path.join(audio_output_dir_temp, f"{generate_random_uid()}_left_temp_vocal.wav")
+ right_temp_vocal_path = os.path.join(audio_output_dir_temp, f"{generate_random_uid()}_right_temp_vocal.wav")
+ left_temp_vocal_path = extract_vocal_from_speech(left_raw_speech_path, left_temp_vocal_path, vocal_separator, audio_output_dir_temp)
+ right_temp_vocal_path = extract_vocal_from_speech(right_raw_speech_path, right_temp_vocal_path, vocal_separator, audio_output_dir_temp)
# prepare each vocal and synthesize the sum audio
generate_duration = num_frames / save_fps + (num_segments-1) * (num_frames-num_cond_frames) / save_fps
@@ -212,12 +249,15 @@ def generate(args):
merge_speech_path = f"/tmp/temp_speech_{generate_random_uid()}_{global_rank}_merge.wav"
sf.write(merge_speech_path, merge_speech, 16000)
- left_full_audio_emb = pipe.get_audio_embedding(left_speech_array_ext, fps=save_fps*audio_stride, device=local_rank, sample_rate=sr)
- right_full_audio_emb = pipe.get_audio_embedding(right_speech_array_ext, fps=save_fps*audio_stride, device=local_rank, sample_rate=sr)
+ left_full_audio_emb = pipe.get_audio_embedding(left_speech_array_ext, fps=save_fps*audio_stride, device=local_rank, sample_rate=sr, model_type=model_type)
+ right_full_audio_emb = pipe.get_audio_embedding(right_speech_array_ext, fps=save_fps*audio_stride, device=local_rank, sample_rate=sr, model_type=model_type)
if torch.isnan(left_full_audio_emb).any() or torch.isnan(right_full_audio_emb).any():
raise ValueError(f"broken audio embedding with nan values")
-
+ if use_background_silent_audio:
+ back_full_audio_emb = pipe.get_audio_embedding(np.zeros_like(left_speech_array_ext), fps=save_fps*audio_stride, device=local_rank, sample_rate=sr, model_type=model_type)
assert left_full_audio_emb.shape == right_full_audio_emb.shape, f"Inconsistent audio embedding shape."
+ if use_background_silent_audio:
+ assert left_full_audio_emb.shape == back_full_audio_emb.shape, f"Inconsistent audio embedding shape between speaker and background."
if context_parallel_util.get_cp_size() > 1:
full_audio_emb_shape_list = list(left_full_audio_emb.size())
@@ -225,6 +265,8 @@ def generate(args):
context_parallel_util.cp_broadcast(full_audio_emb_tensor_shape_list)
context_parallel_util.cp_broadcast(left_full_audio_emb)
context_parallel_util.cp_broadcast(right_full_audio_emb)
+ if use_background_silent_audio:
+ context_parallel_util.cp_broadcast(back_full_audio_emb)
if left_temp_vocal_path is not None and os.path.exists(left_temp_vocal_path):
os.remove(left_temp_vocal_path)
@@ -239,6 +281,9 @@ def generate(args):
context_parallel_util.cp_broadcast(left_full_audio_emb)
right_full_audio_emb = torch.zeros(*full_audio_emb_shape_list, dtype=torch.float32, device=local_rank)
context_parallel_util.cp_broadcast(right_full_audio_emb)
+ if use_background_silent_audio:
+ back_full_audio_emb = torch.zeros(*full_audio_emb_shape_list, dtype=torch.float32, device=local_rank)
+ context_parallel_util.cp_broadcast(back_full_audio_emb)
indices = torch.arange(2 * 2 + 1) - 2
@@ -250,7 +295,10 @@ def generate(args):
center_indices = torch.clamp(center_indices, min=0, max=left_full_audio_emb.shape[0]-1)
left_audio_emb = left_full_audio_emb[center_indices][None,...].to(local_rank)
right_audio_emb = right_full_audio_emb[center_indices][None,...].to(local_rank)
- audio_embs = torch.cat([left_audio_emb, right_audio_emb])
+ audio_embs = [left_audio_emb, right_audio_emb]
+ if use_background_silent_audio:
+ audio_embs.append(back_full_audio_emb[center_indices][None,...].to(local_rank))
+ audio_embs = torch.cat(audio_embs)
# ==============================
@@ -282,7 +330,14 @@ def generate(args):
background_mask += human_mask1
background_mask += human_mask2
background_mask = torch.where(background_mask > 0, torch.tensor(0), torch.tensor(1))
- ref_target_masks = torch.stack([human_mask1, human_mask2, background_mask], dim=0).to(local_rank)
+ total_mask = [human_mask1, human_mask2, background_mask]
+ if use_background_silent_audio:
+ for i in range(len(other_person_bbox)//4):
+ other_person_bbox_i = other_person_bbox[i*4:(i+1)*4]
+ other_person_mask = torch.zeros([src_height, src_width])
+ other_person_mask[other_person_bbox_i[0]:other_person_bbox_i[2], other_person_bbox_i[1]:other_person_bbox_i[3]] = 1
+ total_mask.append(other_person_mask)
+ ref_target_masks = torch.stack(total_mask, dim=0).to(local_rank)
# generate video
output_tuple = pipe.generate_ai2v(
@@ -297,7 +352,8 @@ def generate(args):
output_type='both',
generator=generator,
audio_emb=audio_embs,
- ref_target_masks=ref_target_masks
+ ref_target_masks=ref_target_masks,
+ use_distill=use_distill,
)
output, latent = output_tuple
output = output[0]
@@ -336,7 +392,10 @@ def generate(args):
center_indices = torch.clamp(center_indices, min=0, max=left_full_audio_emb.shape[0]-1)
left_audio_emb = left_full_audio_emb[center_indices][None,...].to(local_rank)
right_audio_emb = right_full_audio_emb[center_indices][None,...].to(local_rank)
- audio_embs = torch.cat([left_audio_emb, right_audio_emb])
+ audio_embs = [left_audio_emb, right_audio_emb]
+ if use_background_silent_audio:
+ audio_embs.append(back_full_audio_emb[center_indices][None,...].to(local_rank))
+ audio_embs = torch.cat(audio_embs)
output_tuple = pipe.generate_avc(
video=current_video,
@@ -354,12 +413,13 @@ def generate(args):
output_type='both',
use_kv_cache=True,
offload_kv_cache=False,
- enhance_hf=True,
+ enhance_hf=True if not use_distill else False,
audio_emb=audio_embs,
ref_latent=ref_latent,
ref_img_index=ref_img_index,
mask_frame_range=mask_frame_range,
- ref_target_masks=ref_target_masks
+ ref_target_masks=ref_target_masks,
+ use_distill=use_distill,
)
output, latent = output_tuple
@@ -441,6 +501,20 @@ def _parse_args():
type=str,
default="./weights/LongCat-Video-Avatar",
)
+ parser.add_argument(
+ "--model_type",
+ type=str,
+ default="avatar-v1.0",
+ )
+ parser.add_argument(
+ "--use_distill",
+ action='store_true',
+ )
+ parser.add_argument(
+ "--use_int8",
+ action='store_true',
+ help="Load INT8 quantized DiT model for reduced VRAM usage"
+ )
args = parser.parse_args()
diff --git a/run_demo_avatar_single_audio_to_video.py b/run_demo_avatar_single_audio_to_video.py
index aec2f5f..5fc822a 100644
--- a/run_demo_avatar_single_audio_to_video.py
+++ b/run_demo_avatar_single_audio_to_video.py
@@ -19,13 +19,13 @@ from longcat_video.pipeline_longcat_video_avatar import LongCatVideoAvatarPipeli
from longcat_video.modules.scheduling_flow_match_euler_discrete import FlowMatchEulerDiscreteScheduler
from longcat_video.modules.autoencoder_kl_wan import AutoencoderKLWan
from longcat_video.modules.avatar.longcat_video_dit_avatar import LongCatVideoAvatarTransformer3DModel
+from longcat_video.modules.quantization import load_quantized_dit
from longcat_video.context_parallel import context_parallel_util
# -------- avatar related --------
import librosa
-from longcat_video.audio_process.wav2vec2 import Wav2Vec2ModelWrapper
+from longcat_video.audio_process import get_audio_encoder, get_audio_feature_extractor
from longcat_video.audio_process.torch_utils import save_video_ffmpeg
-from transformers import Wav2Vec2FeatureExtractor
from audio_separator.separator import Separator
@@ -64,12 +64,23 @@ def generate(args):
resolution = args.resolution
num_segments = max(1, args.num_segments)
output_dir = args.output_dir
+ model_type = args.model_type
+ use_distill = args.use_distill
+ use_int8 = args.use_int8
+
+ if use_distill and model_type == "avatar-v1.5":
+ num_inference_steps = 8
+ text_guidance_scale = 1.0
+ audio_guidance_scale = 1.0
# set up default inference params
save_fps = 16
+ audio_stride = 2
+ if model_type == "avatar-v1.5":
+ save_fps = 25
+ audio_stride = 1
num_frames = 93
num_cond_frames = 13
- audio_stride = 2
if resolution == '480p':
height, width = 480, 832
@@ -102,15 +113,37 @@ def generate(args):
tokenizer = AutoTokenizer.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="tokenizer", torch_dtype=torch.bfloat16)
text_encoder = UMT5EncoderModel.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="text_encoder", torch_dtype=torch.bfloat16)
vae = AutoencoderKLWan.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="vae", torch_dtype=torch.bfloat16)
- scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="scheduler", torch_dtype=torch.bfloat16)
- dit = LongCatVideoAvatarTransformer3DModel.from_pretrained(checkpoint_dir, subfolder="avatar_single", cp_split_hw=cp_split_hw, torch_dtype=torch.bfloat16)
+ if model_type == "avatar-v1.0":
+ scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="scheduler", torch_dtype=torch.bfloat16)
+ elif model_type == "avatar-v1.5":
+ scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(checkpoint_dir, subfolder="scheduler", torch_dtype=torch.bfloat16)
+ else:
+ raise ValueError(f"Unsupported model_type: {model_type}. Expected 'avatar-v1.0' or 'avatar-v1.5'.")
+
+ if model_type == "avatar-v1.0":
+ dit = LongCatVideoAvatarTransformer3DModel.from_pretrained(checkpoint_dir, subfolder="avatar_single", cp_split_hw=cp_split_hw, torch_dtype=torch.bfloat16)
+ elif model_type == "avatar-v1.5":
+ if use_int8:
+ print("[INFO] Loading INT8 quantized DiT model...")
+ dit = load_quantized_dit(checkpoint_dir, subfolder="base_model_int8", cp_split_hw=cp_split_hw)
+ else:
+ dit = LongCatVideoAvatarTransformer3DModel.from_pretrained(checkpoint_dir, subfolder="base_model", cp_split_hw=cp_split_hw, torch_dtype=torch.bfloat16)
+ if use_distill:
+ distill_checkpoint_path = os.path.join(checkpoint_dir, 'lora', f'dmd_lora.safetensors')
+ if os.path.exists(distill_checkpoint_path):
+ dit.load_lora(distill_checkpoint_path, "dmd", multiplier=1.0, lora_network_dim=128, lora_network_alpha=64)
+ dit.enable_loras(["dmd"])
+ else:
+ raise ValueError(f"Unsupported model_type: {model_type}. Expected 'avatar-v1.0' or 'avatar-v1.5'.")
# initialize audio models
- wav2vec_path = os.path.join(checkpoint_dir, 'chinese-wav2vec2-base')
- audio_encoder = Wav2Vec2ModelWrapper(wav2vec_path).to(local_rank)
- audio_encoder.feature_extractor._freeze_parameters()
+ if model_type == "avatar-v1.0":
+ audio_model_checkpoint_path = os.path.join(checkpoint_dir, 'chinese-wav2vec2-base')
+ elif model_type == "avatar-v1.5":
+ audio_model_checkpoint_path = os.path.join(checkpoint_dir, 'whisper-large-v3')
+ audio_encoder = get_audio_encoder(audio_model_checkpoint_path, model_type).to(local_rank)
+ audio_feature_extractor = get_audio_feature_extractor(audio_model_checkpoint_path, model_type)
- wav2vec_feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(wav2vec_path, local_files_only=True)
vocal_separator_path = os.path.join(checkpoint_dir, 'vocal_separator/Kim_Vocal_2.onnx')
audio_output_dir_temp = f"./audio_temp_file"
os.makedirs(audio_output_dir_temp, exist_ok=True)
@@ -133,7 +166,8 @@ def generate(args):
scheduler = scheduler,
dit = dit,
audio_encoder=audio_encoder,
- wav2vec_feature_extractor=wav2vec_feature_extractor
+ audio_feature_extractor=audio_feature_extractor,
+ model_type=model_type
)
pipe.to(local_rank)
@@ -157,7 +191,7 @@ def generate(args):
speech_array = np.append(speech_array, [0.]*added_sample_nums)
# audio embedding
- full_audio_emb = pipe.get_audio_embedding(speech_array, fps=save_fps*audio_stride, device=local_rank, sample_rate=sr)
+ full_audio_emb = pipe.get_audio_embedding(speech_array, fps=save_fps*audio_stride, device=local_rank, sample_rate=sr, model_type=model_type)
if torch.isnan(full_audio_emb).any():
raise ValueError(f"broken audio embedding with nan values")
@@ -205,7 +239,8 @@ def generate(args):
audio_guidance_scale=audio_guidance_scale,
generator=generator,
output_type='both',
- audio_emb=audio_emb
+ audio_emb=audio_emb,
+ use_distill=use_distill,
)
output, latent = output_tuple
output = output[0]
@@ -235,7 +270,8 @@ def generate(args):
audio_guidance_scale=audio_guidance_scale,
output_type='both',
generator=generator,
- audio_emb=audio_emb
+ audio_emb=audio_emb,
+ use_distill=use_distill,
)
output, latent = output_tuple
output = output[0]
@@ -292,11 +328,12 @@ def generate(args):
output_type='both',
use_kv_cache=True,
offload_kv_cache=False,
- enhance_hf=True,
+ enhance_hf=True if not use_distill else False,
audio_emb=audio_emb,
ref_latent=ref_latent,
ref_img_index=ref_img_index,
- mask_frame_range=mask_frame_range
+ mask_frame_range=mask_frame_range,
+ use_distill=use_distill,
)
output, latent = output_tuple
@@ -379,6 +416,20 @@ def _parse_args():
type=str,
default="./weights/LongCat-Video-Avatar",
)
+ parser.add_argument(
+ "--model_type",
+ type=str,
+ default="avatar-v1.0",
+ )
+ parser.add_argument(
+ "--use_distill",
+ action='store_true',
+ )
+ parser.add_argument(
+ "--use_int8",
+ action='store_true',
+ help="Load INT8 quantized DiT model for reduced VRAM usage"
+ )
args = parser.parse_args()