Python
feat: support avatar-v1.5
lile47fc778719a178
diff --git a/README.md b/README.mdindex c64a48a..4e7563d 100644--- a/README.md+++ b/README.md@@ -13,7 +13,7 @@<a href='https://huggingface.co/meituan-longcat/LongCat-Video'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue'></a></div><div align="center" style="line-height: 1;">- <img src='assets/longcat_video_avatar_title.svg' alt="LongCat-Video-Avatar">+ <img src='assets/longcat_video_avatar_1.5_title.svg' alt="LongCat-Video-Avatar 1.5"><a href='https://meigen-ai.github.io/LongCat-Video-Avatar/'><img src='https://img.shields.io/badge/Project-Page-green'></a><a href='https://github.com/meituan-longcat/LongCat-Video/blob/main/assets/LongCat-Video-Avatar-Tech-Report.pdf'><img src='https://img.shields.io/badge/Technique-Report-red'></a><a href='https://huggingface.co/meituan-longcat/LongCat-Video-Avatar'><img src='https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue'></a>@@ -45,6 +45,7 @@ For more detail, please refer to the comprehensive [***LongCat-Video Technical R</div>## 🔥 Latest News!!+- May 21, 2026: 🚀 We release [***LongCat-Video-Avatar-1.5***](https://meigen-ai.github.io/LongCat-Video-Avatar-1.5/), an upgraded open-source framework for audio-driven human video generation. 1.5 replaces Wav2Vec2 with Whisper-Large for smoother lip dynamics, achieves production-ready stability with robust long-video generation, generalizes to stylized domains (anime, animals, complex real-world conditions), and accelerates inference to 8 steps via DMD2 distillation. [ [***code***](https://github.com/meituan-longcat/LongCat-Video) | 🤗 [***weights***](https://huggingface.co/meituan-longcat/LongCat-Video-Avatar-1.5) | [***project page***](https://meigen-ai.github.io/LongCat-Video-Avatar-1.5/) ]- Dec 16, 2025: 🚀 We are excited to announce the release of [***LongCat-Video-Avatar***](https://meigen-ai.github.io/LongCat-Video-Avatar/), a unified model that delivers expressive and highly dynamic audio-driven character animation, supporting native tasks including *Audio-Text-to-Video*, *Audio-Text-Image-to-Video*, and *Video Continuation* with seamless compatibility for both *single-stream* and *multi-stream* audio inputs. The release includes our [***Technical Report***](https://github.com/meituan-longcat/LongCat-Video), [***inference code***](https://github.com/meituan-longcat/LongCat-Video), 🤗 [***model weights***](https://huggingface.co/meituan-longcat/LongCat-Video-Avatar), and [***project page***](https://meigen-ai.github.io/LongCat-Video-Avatar/).- Oct 25, 2025: 🚀 We've released LongCat-Video, a foundational video generation model. Tech report and models are available at [***LongCat-Video Technical Report***](https://arxiv.org/abs/2510.22200) and 🤗 [***Huggingface***](https://huggingface.co/meituan-longcat/LongCat-Video) !@@ -94,13 +95,15 @@ FlashAttention-2 is enabled in the model config by default; you can also change| Models | Description | Download Link || --- | --- | --- || LongCat-Video | foundational video generation | 🤗 [Huggingface](https://huggingface.co/meituan-longcat/LongCat-Video) |-| LongCat-Video-Avatar | single- and multi-character audio-driven video generation | 🤗 [Huggingface](https://huggingface.co/meituan-longcat/LongCat-Video-Avatar) |+| LongCat-Video-Avatar | single- and multi-character audio-driven video generation (wav2vec2) | 🤗 [Huggingface](https://huggingface.co/meituan-longcat/LongCat-Video-Avatar) |+| LongCat-Video-Avatar-1.5 | upgraded avatar model with Whisper-large-v3 audio encoder, distillation-based fast inference | 🤗 [Huggingface](https://huggingface.co/meituan-longcat/LongCat-Video-Avatar-1.5) |Download models using huggingface-cli:```shellpip install "huggingface_hub[cli]"huggingface-cli download meituan-longcat/LongCat-Video --local-dir ./weights/LongCat-Videohuggingface-cli download meituan-longcat/LongCat-Video-Avatar --local-dir ./weights/LongCat-Video-Avatar+huggingface-cli download meituan-longcat/LongCat-Video-Avatar-1.5 --local-dir ./weights/LongCat-Video-Avatar-1.5```### Run Text-to-Video@@ -154,35 +157,55 @@ torchrun --nproc_per_node=2 run_demo_interactive_video.py --context_parallel_siz```### Run LongCat-Video-Avatar-💡 User tips+<details>+<summary>💡 User tips for 1.5</summary>++> - **Lip synchronization accuracy:** Audio CFG works optimally between 3–5. Increase the audio CFG value for better synchronization.+> - **Prompt Enhancement:** Longer, more descriptive prompts yield better consistency and naturalness than short ones. We recommend including rich details such as character appearance, actions, and scene context (e.g., *"A young woman with long black hair is speaking and smiling, wearing a white blouse, sitting in a bright café"*) for best results.+> - **Mitigate repeated actions:** Setting the reference image index(--ref_img_index, default to 10) between 0 and 24 ensures better consistency; setting it to 30 helps reduce repeated actions. Additionally, increasing the mask frame range (--mask_frame_range, default to 3) can further help mitigate repeated actions, but excessively large values may introduce artifacts.+> - **Super resolution:** Our model is compatible with both 480P and 720P, which can be controlled via --resolution.+> - **Dual-Audio Modes:** Merge mode (set audio_type to para) requires two audio clips of equal length, and the resulting audio is obtained by summing the two clips; Concatenation mode (set audio_type to add) does not require equal-length inputs, and the resulting audio is formed by sequentially concatenating the two clips with silence padding for any gaps, where by default person1 speaks first and person2 speaks afterward.+> - **Model versions:** `--model_type avatar-v1.0` uses wav2vec2 audio encoder (default); `--model_type avatar-v1.5` uses Whisper-large-v3 audio encoder for better lip sync quality.+> - **Distillation mode:** Add `--use_distill` to enable distillation sampling (fewer steps, faster inference). This is **required** when using `--model_type avatar-v1.5`.+> - **INT8 quantization:** Add `--use_int8` to load the INT8 quantized DiT model for reduced VRAM usage. Only supported with `--model_type avatar-v1.5`.++</details>++<details>+<summary>💡 User tips for 1.0</summary>+> - Lip synchronization accuracy: Audio CFG works optimally between 3–5. Increase the audio CFG value for better synchronization.> - Prompt Enhancement: Include clear verbal-action cues (e.g., talking, speaking) in the prompt to achieve more natural lip movements.> - Mitigate repeated actions: Setting the reference image index(--ref_img_index, default to 10) between 0 and 24 ensures better consistency, while selecting other ranges (e.g., -10 or 30) helps reduce repeated actions. Additionally, increasing the mask frame range (--mask_frame_range, default to 3) can further help mitigate repeated actions, but excessively large values may introduce artifacts.> - Super resolution: Our model is compatible with both 480P and 720P, which can be controlled via --resolution.> - Dual-Audio Modes: Merge mode (set audio_type to para) requires two audio clips of equal length, and the resulting audio is obtained by summing the two clips; Concatenation mode (set audio_type to add) does not require equal-length inputs, and the resulting audio is formed by sequentially concatenating the two clips with silence padding for any gaps, where by default person1 speaks first and person2 speaks afterward.+</details>++#### LongCat-Video-Avatar-1.5+- Single-Audio-to-Video Generation```shell# Audio-Text-to-Video-torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar --stage_1=at2v --input_json=assets/avatar/single_example_1.json+torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --stage_1=at2v --input_json=assets/avatar/single_example_1.json --use_distill --model_type avatar-v1.5 --use_int8# Audio-Image-to-Video-torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar --stage_1=ai2v --input_json=assets/avatar/single_example_1.json+torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --stage_1=ai2v --input_json=assets/avatar/single_example_1.json --use_distill --model_type avatar-v1.5 --use_int8# Audio-Text-to-Video and Video-Continuation-torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar --stage_1=at2v --input_json=assets/avatar/single_example_1.json --num_segments=5 --ref_img_index=10 --mask_frame_range=3+torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --stage_1=at2v --input_json=assets/avatar/single_example_1.json --num_segments=5 --ref_img_index=10 --mask_frame_range=3 --use_distill --model_type avatar-v1.5 --use_int8# Audio-Image-to-Video and Video-Continuation-torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar --stage_1=ai2v --input_json=assets/avatar/single_example_1.json --num_segments=5 --ref_img_index=10 --mask_frame_range=3+torchrun --nproc_per_node=2 run_demo_avatar_single_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --stage_1=ai2v --input_json=assets/avatar/single_example_1.json --num_segments=5 --ref_img_index=10 --mask_frame_range=3 --use_distill --model_type avatar-v1.5 --use_int8```- Multi-Audio-to-Video Generation```shell# Audio-Image-to-Video-torchrun --nproc_per_node=2 run_demo_avatar_multi_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar --input_json=assets/avatar/multi_example_1.json+torchrun --nproc_per_node=2 run_demo_avatar_multi_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --input_json=assets/avatar/multi_example_1.json --use_distill --model_type avatar-v1.5 --use_int8# Audio-Image-to-Video and Video-Continuation-torchrun --nproc_per_node=2 run_demo_avatar_multi_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar --input_json=assets/avatar/multi_example_1.json --num_segments=5 --ref_img_index=10 --mask_frame_range=3+torchrun --nproc_per_node=2 run_demo_avatar_multi_audio_to_video.py --context_parallel_size=2 --checkpoint_dir=./weights/LongCat-Video-Avatar-1.5 --input_json=assets/avatar/multi_example_1.json --num_segments=5 --ref_img_index=10 --mask_frame_range=3 --use_distill --model_type avatar-v1.5 --use_int8```### Run Streamlit@@ -262,6 +285,15 @@ We kindly encourage citation of our work if you find it useful.primaryClass={cs.CV},url={https://arxiv.org/abs/2510.22200},}+@misc{meituanlongcatteam2025longcatvideoavatar15technicalreport,+ title={LongCat-Video-Avatar 1.5 Technical Report},+ author={Meituan LongCat Team},+ year={2026},+ eprint={},+ archivePrefix={arXiv},+ primaryClass={cs.CV},+ url={},+}@misc{meituanlongcatteam2025longcatvideoavatartechnicalreport,title={LongCat-Video-Avatar Technical Report},author={Meituan LongCat Team},diff --git a/assets/longcat_video_avatar_1.5_title.svg b/assets/longcat_video_avatar_1.5_title.svgnew file mode 100644index 0000000..f60fb24--- /dev/null+++ b/assets/longcat_video_avatar_1.5_title.svg@@ -0,0 +1 @@+<svg xmlns="http://www.w3.org/2000/svg" width="150" height="20"><rect width="100" height="100" x="50" y="50" fill="none"/><text x="71" y="10" font-family="Arial" font-size="13" fill="black" font-weight="bold" alignment-baseline="middle" text-anchor="middle">LongCat-Video-Avatar 1.5</text></svg>diff --git a/longcat_video/audio_process/__init__.py b/longcat_video/audio_process/__init__.pynew file mode 100644index 0000000..f98cf6c--- /dev/null+++ b/longcat_video/audio_process/__init__.py@@ -0,0 +1,18 @@+from .wav2vec2 import Wav2Vec2ModelWrapper+from transformers import Wav2Vec2FeatureExtractor, WhisperModel, AutoFeatureExtractor++def get_audio_encoder(checkpoint_path, model_type="avatar-v1.0"):+ if model_type == "avatar-v1.0":+ model = Wav2Vec2ModelWrapper(checkpoint_path)+ model.feature_extractor._freeze_parameters()+ return model+ if model_type == "avatar-v1.5":+ model = WhisperModel.from_pretrained(checkpoint_path).eval()+ model.requires_grad_(False)+ return model++def get_audio_feature_extractor(checkpoint_path, model_type="avatar-v1.0"):+ if model_type == "avatar-v1.0":+ return Wav2Vec2FeatureExtractor(checkpoint_path, local_files_only=True)+ if model_type == "avatar-v1.5":+ return AutoFeatureExtractor.from_pretrained(checkpoint_path)diff --git a/longcat_video/modules/avatar/attention.py b/longcat_video/modules/avatar/attention.pyindex 97ee132..a169a7a 100644--- a/longcat_video/modules/avatar/attention.py+++ b/longcat_video/modules/avatar/attention.py@@ -326,7 +326,7 @@ class SingleStreamAttention(nn.Module):self.rope_bak = int(self.class_range // 2)self.rope_1d = RotaryPositionalEmbedding1D(self.head_dim)- def _process_cross_attn(self, x, cond, frames_num=None, x_ref_attn_map=None):+ def _process_cross_attn(self, x, cond, frames_num=None, x_ref_attn_map=None, human_num=None):N_t = frames_numout_dtype = x.dtype@@ -351,8 +351,9 @@ class SingleStreamAttention(nn.Module):human1 = normalize_and_scale(x_ref_attn_map[0], (human1_min_value, human1_max_value), (self.rope_h1[0], self.rope_h1[1]))human2 = normalize_and_scale(x_ref_attn_map[1], (human2_min_value, human2_max_value), (self.rope_h2[0], self.rope_h2[1]))- back = torch.full((x_ref_attn_map.size(1),), self.rope_bak, dtype=human1.dtype).to(human1.device)- max_indices = x_ref_attn_map.argmax(dim=0)+ background_pos = self.rope_bak if x_ref_attn_map.shape[0] <= 3 else 100+ back = torch.full((x_ref_attn_map.size(1),), background_pos, dtype=human1.dtype).to(human1.device)+ max_indices = x_ref_attn_map.argmax(dim=0).clamp(max=2)normalized_map = torch.stack([human1, human2, back], dim=1)normalized_pos = normalized_map[range(x_ref_attn_map.size(1)), max_indices]@@ -373,8 +374,17 @@ class SingleStreamAttention(nn.Module):# multitalk with rope1d peif x_ref_attn_map is not None:per_frame = torch.zeros(N_a, dtype=encoder_k.dtype).to(encoder_k.device)- per_frame[:per_frame.size(0)//2] = (self.rope_h1[0] + self.rope_h1[1]) / 2- per_frame[per_frame.size(0)//2:] = (self.rope_h2[0] + self.rope_h2[1]) / 2+ human1_pos = (self.rope_h1[0] + self.rope_h1[1]) / 2+ human2_pos = (self.rope_h2[0] + self.rope_h2[1]) / 2+ if human_num is not None and human_num > 2:+ background_pos = self.rope_bak if x_ref_attn_map.shape[0] <= 3 else 100+ tokens_per_human = per_frame.size(0) // human_num+ per_frame[:tokens_per_human] = human1_pos+ per_frame[tokens_per_human:2*tokens_per_human] = human2_pos+ per_frame[2*tokens_per_human:] = background_pos+ else:+ per_frame[:per_frame.size(0)//2] = human1_pos+ per_frame[per_frame.size(0)//2:] = human2_posencoder_pos = torch.concat([per_frame]*N_t, dim=0)encoder_k = rearrange(encoder_k, "(B N_t) H S C -> B H (N_t S) C", N_t=N_t)encoder_k = self.rope_1d(encoder_k, encoder_pos)@@ -434,7 +444,7 @@ class SingleStreamAttention(nn.Module):B, N, C = x.shapeif (num_cond_latents is None or num_cond_latents == 0):# text to video- output = self._process_cross_attn(x, cond, shape[0], x_ref_attn_map)+ output = self._process_cross_attn(x, cond, shape[0], x_ref_attn_map, human_num=human_num)return None, outputelif num_cond_latents is not None and num_cond_latents > 0:# image to video or video continuation@@ -448,8 +458,10 @@ class SingleStreamAttention(nn.Module):if human_num is not None and human_num == 2:# multitalk modeoutput_noise = self._process_cross_attn(x_noise, cond, frames_num, x_ref_attn_map)+ elif human_num is not None and human_num > 2:+ # multitalk mode+ output_noise = self._process_cross_attn(x_noise, cond, frames_num, x_ref_attn_map, human_num)else:- # singletalk modeoutput_noise = self._process_cross_attn(x_noise, cond, frames_num)output_cond = torch.zeros((B, num_cond_latents_thw, C), dtype=output_noise.dtype, device=output_noise.device)return output_cond, output_noisediff --git a/longcat_video/modules/avatar/longcat_video_dit_avatar.py b/longcat_video/modules/avatar/longcat_video_dit_avatar.pyindex a30136a..44593e5 100644--- a/longcat_video/modules/avatar/longcat_video_dit_avatar.py+++ b/longcat_video/modules/avatar/longcat_video_dit_avatar.py@@ -220,6 +220,8 @@ class LongCatVideoAvatarTransformer3DModel(text_tokens_zero_pad: bool = False,# avatar configaudio_window: int = 5,+ audio_block: int = 12,+ audio_channel: int = 768,intermediate_dim: int = 512,output_dim: int = 768,context_tokens: int = 32,@@ -269,6 +271,8 @@ class LongCatVideoAvatarTransformer3DModel(self.audio_proj = AudioProjModel(seq_len=audio_window,seq_len_vf=audio_window+vae_scale-1,+ blocks=audio_block,+ channels=audio_channel,intermediate_dim=intermediate_dim,output_dim=output_dim,context_tokens=context_tokensdiff --git a/longcat_video/modules/lora_utils.py b/longcat_video/modules/lora_utils.pyindex 68ce476..ce3e892 100644--- a/longcat_video/modules/lora_utils.py+++ b/longcat_video/modules/lora_utils.py@@ -41,7 +41,8 @@ class LoRAModule(torch.nn.Module):super().__init__()self.lora_name = lora_name- assert org_module.__class__.__name__ == "Linear"+ assert org_module.__class__.__name__ in ("Linear", "QuantizedLinear"), \+ f"LoRA target must be Linear or QuantizedLinear, got {org_module.__class__.__name__}"in_dim = org_module.in_featuresout_dim = org_module.out_features@@ -116,7 +117,7 @@ class LoRANetwork(torch.nn.Module):except Exception as e:print(f"Cannot find module: {module_name}, error: {e}")continue- if module.__class__.__name__ != "Linear":+ if module.__class__.__name__ not in ("Linear", "QuantizedLinear"):continue# 推断 n_seperatediff --git a/longcat_video/modules/quantization.py b/longcat_video/modules/quantization.pynew file mode 100644index 0000000..17c775e--- /dev/null+++ b/longcat_video/modules/quantization.py@@ -0,0 +1,239 @@+import os+import json+import shutil+from typing import Optional, Set++import torch+import torch.nn as nn+import torch.nn.functional as F+from safetensors.torch import save_file, load_file+++class QuantizedLinear(nn.Module):+ """INT8 weight-only quantized linear layer with per-channel symmetric quantization."""++ def __init__(self, in_features: int, out_features: int, bias: bool = False):+ super().__init__()+ self.in_features = in_features+ self.out_features = out_features+ self.register_buffer("weight_int8", torch.zeros(out_features, in_features, dtype=torch.int8))+ self.register_buffer("weight_scale", torch.zeros(out_features, dtype=torch.float32))+ if bias:+ self.register_buffer("bias", torch.zeros(out_features, dtype=torch.bfloat16))+ else:+ self.bias = None++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ # Dequantize weight to input dtype for computation+ compute_dtype = x.dtype+ weight = self.weight_int8.to(compute_dtype) * self.weight_scale.to(compute_dtype).unsqueeze(1)+ bias = self.bias.to(compute_dtype) if self.bias is not None else None+ return F.linear(x, weight, bias)++ @classmethod+ def from_linear(cls, linear: nn.Linear) -> "QuantizedLinear":+ """Quantize a standard Linear layer to INT8."""+ has_bias = linear.bias is not None+ ql = cls(linear.in_features, linear.out_features, bias=has_bias)++ weight = linear.weight.data.float()+ # Per-channel symmetric quantization+ scale = weight.abs().amax(dim=1).clamp(min=1e-8) / 127.0+ weight_int8 = (weight / scale.unsqueeze(1)).round().clamp(-128, 127).to(torch.int8)++ ql.weight_int8 = weight_int8+ ql.weight_scale = scale+ if has_bias:+ ql.bias = linear.bias.data.to(torch.bfloat16)+ return ql++ def extra_repr(self) -> str:+ return f"in_features={self.in_features}, out_features={self.out_features}, bias={self.bias is not None}"+++# Layers to skip quantization (sensitive to precision)+DEFAULT_SKIP_PATTERNS = {+ "final_layer.linear", # Final output projection, precision-sensitive+}+++def quantize_model(model: nn.Module, skip_patterns: Optional[Set[str]] = None) -> nn.Module:+ """Replace all nn.Linear layers in the model with QuantizedLinear (INT8 weight-only).++ Args:+ model: The model to quantize (modified in-place).+ skip_patterns: Set of module name patterns to skip. If None, uses DEFAULT_SKIP_PATTERNS.++ Returns:+ The quantized model.+ """+ if skip_patterns is None:+ skip_patterns = DEFAULT_SKIP_PATTERNS++ modules_to_replace = {}+ for name, module in model.named_modules():+ if isinstance(module, nn.Linear):+ # Check if this module should be skipped+ should_skip = any(pattern in name for pattern in skip_patterns)+ if not should_skip:+ modules_to_replace[name] = module++ for name, linear in modules_to_replace.items():+ # Navigate to the parent module+ parts = name.split(".")+ parent = model+ for part in parts[:-1]:+ parent = getattr(parent, part)+ # Replace with quantized version+ setattr(parent, parts[-1], QuantizedLinear.from_linear(linear))++ return model+++def save_quantized_state_dict(model: nn.Module, save_dir: str, config_source_dir: Optional[str] = None):+ """Save the quantized model's state dict using safetensors format.++ Saves quantized weights in shards to avoid single large files.++ Args:+ model: The quantized model.+ save_dir: Directory to save quantized weights.+ config_source_dir: If provided, copy config.json from this directory.+ """+ os.makedirs(save_dir, exist_ok=True)++ state_dict = model.state_dict()++ # Split into shards (~4GB each for manageable files)+ max_shard_size = 4 * 1024 * 1024 * 1024 # 4GB++ shards = []+ current_shard = {}+ current_size = 0++ for key, tensor in state_dict.items():+ tensor_size = tensor.numel() * tensor.element_size()+ if current_size + tensor_size > max_shard_size and current_shard:+ shards.append(current_shard)+ current_shard = {}+ current_size = 0+ current_shard[key] = tensor+ current_size += tensor_size++ if current_shard:+ shards.append(current_shard)++ # Save each shard+ index = {"metadata": {"total_size": sum(t.numel() * t.element_size() for t in state_dict.values())}, "weight_map": {}}++ for i, shard in enumerate(shards):+ shard_name = f"quantized_model-{i+1:05d}-of-{len(shards):05d}.safetensors"+ save_file(shard, os.path.join(save_dir, shard_name))+ for key in shard:+ index["weight_map"][key] = shard_name++ # Save index+ index_path = os.path.join(save_dir, "quantized_model.safetensors.index.json")+ with open(index_path, "w") as f:+ json.dump(index, f, indent=2)++ # Copy config if source provided+ if config_source_dir:+ src_config = os.path.join(config_source_dir, "config.json")+ if os.path.exists(src_config):+ shutil.copy2(src_config, os.path.join(save_dir, "config.json"))++ # Save quantization metadata+ quant_config = {+ "quantization_method": "int8_per_channel_symmetric",+ "skip_patterns": list(DEFAULT_SKIP_PATTERNS),+ "description": "Weight-only INT8 quantization with per-channel symmetric scaling"+ }+ with open(os.path.join(save_dir, "quantization_config.json"), "w") as f:+ json.dump(quant_config, f, indent=2)++ print(f"Saved {len(shards)} shard(s) to {save_dir}")+++def load_quantized_dit(checkpoint_dir: str, subfolder: str = "base_model_int8", **kwargs):+ """Load a quantized DiT model.++ Args:+ checkpoint_dir: Base checkpoint directory.+ subfolder: Subfolder containing quantized weights (default: 'base_model_int8').+ **kwargs: Additional kwargs passed to the model constructor (e.g., cp_split_hw).++ Returns:+ The quantized DiT model ready for inference.+ """+ from .avatar.longcat_video_dit_avatar import LongCatVideoAvatarTransformer3DModel++ quantized_dir = os.path.join(checkpoint_dir, subfolder)++ # Load config+ config_path = os.path.join(quantized_dir, "config.json")+ with open(config_path, "r") as f:+ config = json.load(f)++ # Remove non-constructor keys+ config.pop("_class_name", None)+ config.pop("architectures", None)+ config.pop("_diffusers_version", None)+ config.pop("model_max_length", None)++ # Override with kwargs+ config.update(kwargs)++ # Instantiate model (empty weights)+ model = LongCatVideoAvatarTransformer3DModel(**config)++ # Replace Linear layers with QuantizedLinear (empty)+ skip_patterns = DEFAULT_SKIP_PATTERNS+ modules_to_replace = {}+ for name, module in model.named_modules():+ if isinstance(module, nn.Linear):+ should_skip = any(pattern in name for pattern in skip_patterns)+ if not should_skip:+ ql = QuantizedLinear(module.in_features, module.out_features, bias=module.bias is not None)+ modules_to_replace[name] = ql++ for name, ql in modules_to_replace.items():+ parts = name.split(".")+ parent = model+ for part in parts[:-1]:+ parent = getattr(parent, part)+ setattr(parent, parts[-1], ql)++ # Load quantized state dict+ index_path = os.path.join(quantized_dir, "quantized_model.safetensors.index.json")+ if os.path.exists(index_path):+ with open(index_path, "r") as f:+ index = json.load(f)+ # Load from shards+ shard_files = set(index["weight_map"].values())+ state_dict = {}+ for shard_file in sorted(shard_files):+ shard_path = os.path.join(quantized_dir, shard_file)+ shard_dict = load_file(shard_path, device="cpu")+ state_dict.update(shard_dict)+ else:+ # Single file fallback+ files = [f for f in os.listdir(quantized_dir) if f.endswith(".safetensors") and "index" not in f]+ state_dict = {}+ for f in sorted(files):+ shard_dict = load_file(os.path.join(quantized_dir, f), device="cpu")+ state_dict.update(shard_dict)++ model.load_state_dict(state_dict, strict=True)+ model.eval()++ # Cast non-quantized parameters (Conv3d, LayerNorm, etc.) to bfloat16+ # QuantizedLinear buffers (int8, float32 scale) are kept as-is+ for name, module in model.named_modules():+ if isinstance(module, QuantizedLinear):+ continue+ for param_name, param in module.named_parameters(recurse=False):+ if param.dtype == torch.float32:+ param.data = param.data.to(torch.bfloat16)++ return modeldiff --git a/longcat_video/pipeline_longcat_video_avatar.py b/longcat_video/pipeline_longcat_video_avatar.pyindex ec22c07..5caf1c5 100644--- a/longcat_video/pipeline_longcat_video_avatar.py+++ b/longcat_video/pipeline_longcat_video_avatar.py@@ -27,8 +27,6 @@ import html# -------- avatar related --------import scipy.signal as ssimport pyloudnorm as pyln-from longcat_video.audio_process.wav2vec2 import Wav2Vec2ModelWrapper-from transformers import Wav2Vec2FeatureExtractorfrom diffusers.image_processor import is_valid_image, is_valid_image_imagelistimport warnings@@ -120,8 +118,9 @@ class LongCatVideoAvatarPipeline:vae: AutoencoderKLWan,scheduler: FlowMatchEulerDiscreteScheduler,dit: LongCatVideoAvatarTransformer3DModel,- audio_encoder: Wav2Vec2ModelWrapper,- wav2vec_feature_extractor: Wav2Vec2FeatureExtractor+ audio_encoder = None,+ audio_feature_extractor = None,+ model_type="avatar-v1.0",):self.vae = vaeself.text_encoder = text_encoder@@ -138,8 +137,14 @@ class LongCatVideoAvatarPipeline:self._num_timesteps = 1000self._num_distill_sample_steps = 50- self.audio_encoder=audio_encoder- self.wav2vec_feature_extractor = wav2vec_feature_extractor+ self.audio_encoder = audio_encoder+ self.audio_feature_extractor = audio_feature_extractor++ self.model_type = model_type+ self.use_vcond = True+ if self.model_type == "avatar-v1.5":+ self.use_vcond = False+ self._num_distill_sample_steps = 8def _get_t5_prompt_embeds(self,@@ -373,14 +378,20 @@ class LongCatVideoAvatarPipeline:return self._attention_kwargsdef get_timesteps_sigmas(self, sampling_steps: int, use_distill: bool=False):- if use_distill:- distill_indices = torch.arange(1, self.num_distill_sample_steps + 1, dtype=torch.float32)- distill_indices = (distill_indices * (self.num_timesteps // self.num_distill_sample_steps)).round().long()-- inference_indices = np.linspace(0, self.num_distill_sample_steps, num=sampling_steps, endpoint=False)- inference_indices = np.floor(inference_indices).astype(np.int64)+ if use_distill:+ if self.model_type == "avatar-v1.5":+ distill_indices = torch.arange(1, self.num_distill_sample_steps + 1, dtype=torch.float32)+ distill_indices = (distill_indices * (self.num_timesteps // self.num_distill_sample_steps)).round().long()+ distill_indices = self.num_timesteps - distill_indices+ sigmas = torch.flip(torch.linspace(0, 1, self.num_timesteps), [0])+ sigmas = torch.flip(sigmas[distill_indices], [0]).float()+ else:+ distill_indices = torch.arange(1, self.num_distill_sample_steps + 1, dtype=torch.float32)+ distill_indices = (distill_indices * (self.num_timesteps // self.num_distill_sample_steps)).round().long()+ inference_indices = np.linspace(0, self.num_distill_sample_steps, num=sampling_steps, endpoint=False)+ inference_indices = np.floor(inference_indices).astype(np.int64)- sigmas = torch.flip(distill_indices, [0])[inference_indices].float() / self.num_timesteps+ sigmas = torch.flip(distill_indices, [0])[inference_indices].float() / self.num_timestepselse:sigmas = torch.linspace(1, 0.001, sampling_steps)sigmas = sigmas.to(torch.float32)@@ -524,8 +535,84 @@ class LongCatVideoAvatarPipeline:else:raise NotImplementedError(f"Not supported resize_mode {resize_mode}")+ def get_audio_embedding(self, speech_array, fps=32, device='cpu', sample_rate=16000, model_type="avatar-v1.0"):+ if model_type == "avatar-v1.0":+ return self.get_audio_embedding_wav2vec2(speech_array, fps, device, sample_rate)+ if model_type == "avatar-v1.5":+ return self.get_audio_embedding_whisper(speech_array, fps, device, sample_rate)+@torch.no_grad()- def get_audio_embedding(self, speech_array, fps=32, device='cpu', sample_rate=16000):+ def get_audio_embedding_whisper(self, speech_array, fps=25, device='cpu', sample_rate=16000):+ """使用 Whisper encoder 提取音频特征。+ Args:+ speech_array: 原始音频波形 (numpy array, 单声道, sample_rate=16000)+ fps: 目标视频帧率+ device: 推理设备+ sample_rate: 音频采样率+ Returns:+ audio_emb: [T, 5, D],T = int(audio_duration * fps)+ """+ def linear_interpolation_fps(features, input_fps, output_fps, output_len=None):+ """将音频特征从 input_fps 插值到 output_fps。+ Args:+ features: [B, T, D]+ input_fps: 源帧率+ output_fps: 目标帧率+ output_len: 若指定则直接用该长度,否则按帧率比换算+ Returns:+ [B, output_len, D]+ """+ features = features.transpose(1, 2) # [B, D, T]+ if output_len is None:+ output_len = int(features.shape[2] / float(input_fps) * output_fps)+ output_features = F.interpolate(features, size=output_len, align_corners=True, mode='linear')+ return output_features.transpose(1, 2)+ # ---- 常量 ----+ MEL_CHUNK = 750 * 640 # feature extractor 滑窗大小(样本数)+ ENC_CHUNK = 3000 # encoder 滑窗大小(mel 帧数)+ ENC_FPS = 50 # encoder 输出帧率++ # ---- 时长 / 帧数 ----+ audio_duration = len(speech_array) / sample_rate+ video_length = int(audio_duration * fps)++ # ---- 音频预处理 ----+ speech_array = self._loudness_norm(speech_array, sample_rate)++ # ---- Whisper feature extractor:wav → mel spectrogram ----+ mel_chunks = []+ for i in range(0, len(speech_array), MEL_CHUNK):+ mel = self.audio_feature_extractor(+ speech_array[i: i + MEL_CHUNK],+ sampling_rate=sample_rate,+ return_tensors="pt",+ ).input_features # [1, mel_bins, T_mel]+ mel_chunks.append(mel)+ audio_features = torch.cat(mel_chunks, dim=-1) # [1, mel_bins, T_mel_total]+ audio_features = audio_features.to(self.audio_encoder.dtype)++ # ---- Whisper encoder:mel → hidden states ----+ enc_chunks = []+ for i in range(0, audio_features.shape[-1], ENC_CHUNK):+ chunk_hs = self.audio_encoder.encoder(+ audio_features[:, :, i: i + ENC_CHUNK].to(device),+ output_hidden_states=True,+ ).hidden_states # tuple: (n_layers+1,) x [1, T_enc, D]+ enc_chunks.append(torch.stack(chunk_hs, dim=2)) # [1, T_enc, n_layers, D]+ audio_prompts = torch.cat(enc_chunks, dim=1) # [1, T_enc_total, n_layers, D]+ audio_prompts = audio_prompts[:, :video_length * 2] # 截取有效帧++ # ---- 按层分组 + 插值到目标帧数 ----+ feat0 = linear_interpolation_fps(audio_prompts[:, :, 0: 8].mean(dim=2), ENC_FPS, fps, video_length)+ feat1 = linear_interpolation_fps(audio_prompts[:, :, 8:16].mean(dim=2), ENC_FPS, fps, video_length)+ feat2 = linear_interpolation_fps(audio_prompts[:, :, 16:24].mean(dim=2), ENC_FPS, fps, video_length)+ feat3 = linear_interpolation_fps(audio_prompts[:, :, 24:32].mean(dim=2), ENC_FPS, fps, video_length)+ feat4 = linear_interpolation_fps(audio_prompts[:, :, 32], ENC_FPS, fps, video_length)+ audio_emb = torch.stack([feat0, feat1, feat2, feat3, feat4], dim=2)[0] # [T, 5, D]+ return audio_emb++ @torch.no_grad()+ def get_audio_embedding_wav2vec2(self, speech_array, fps=32, device='cpu', sample_rate=16000):audio_duration = len(speech_array) / sample_ratevideo_length = audio_duration * fps@@ -537,7 +624,7 @@ class LongCatVideoAvatarPipeline:# wav2vec_feature_extractoraudio_feature = np.squeeze(- self.wav2vec_feature_extractor(speech_array, sampling_rate=sample_rate).input_values+ self.audio_feature_extractor(speech_array, sampling_rate=sample_rate).input_values)audio_feature = torch.from_numpy(audio_feature).float().to(device=device)audio_feature = audio_feature.unsqueeze(0)@@ -711,7 +798,7 @@ class LongCatVideoAvatarPipeline:)if context_parallel_util.get_cp_size() > 1:context_parallel_util.cp_broadcast(latents)-+# 6. Denoising loopif context_parallel_util.get_cp_size() > 1:torch.distributed.barrier(group=context_parallel_util.get_cp_group())@@ -923,6 +1010,7 @@ class LongCatVideoAvatarPipeline:context_parallel_util.cp_broadcast(negative_prompt_attention_mask)audio_cond_embs = torch.cat([audio_emb] * num_videos_per_prompt, dim=0)+ audio_num = audio_cond_embs.shape[0]if self.do_classifier_free_guidance:prompt_embeds = torch.cat([negative_prompt_embeds, prompt_embeds], dim=0)prompt_attention_mask = torch.cat([negative_prompt_attention_mask, prompt_attention_mask], dim=0)@@ -979,14 +1067,14 @@ class LongCatVideoAvatarPipeline:timestep[:, :1] = 0if self.do_classifier_free_guidance and ref_target_masks is not None:- # Multitalk mode+ # Multitalk mode with CFGnoise_pred_uncond_text = self.dit(hidden_states=latent_model_input[:1],timestep=timestep[:1],encoder_hidden_states=prompt_embeds[:1],encoder_attention_mask=prompt_attention_mask[:1],num_cond_latents=1,- audio_embs=audio_cond_embs[:2],+ audio_embs=audio_cond_embs[:audio_num],ref_target_masks=ref_target_masks)noise_pred_cond = self.dit(@@ -995,19 +1083,21 @@ class LongCatVideoAvatarPipeline:encoder_hidden_states=prompt_embeds[1:],encoder_attention_mask=prompt_attention_mask[1:],num_cond_latents=1,- audio_embs=audio_cond_embs[2:],+ audio_embs=audio_cond_embs[audio_num:2*audio_num],ref_target_masks=ref_target_masks)noise_pred = torch.cat([noise_pred_uncond_text, noise_pred_cond])else:# Singletalk mode+ # Multitalk mode without CFG (e.g. use_distill with scale=1.0)noise_pred = self.dit(hidden_states=latent_model_input,timestep=timestep,encoder_hidden_states=prompt_embeds,encoder_attention_mask=prompt_attention_mask,num_cond_latents=1,- audio_embs=audio_cond_embs+ audio_embs=audio_cond_embs,+ ref_target_masks=ref_target_masks)if self.do_classifier_free_guidance:@@ -1222,6 +1312,7 @@ class LongCatVideoAvatarPipeline:context_parallel_util.cp_broadcast(negative_prompt_attention_mask)audio_cond_embs = torch.cat([audio_emb] * num_videos_per_prompt, dim=0)+ audio_num = audio_cond_embs.shape[0]if self.do_classifier_free_guidance:prompt_embeds = torch.cat([negative_prompt_embeds, prompt_embeds], dim=0)prompt_attention_mask = torch.cat([negative_prompt_attention_mask, prompt_attention_mask], dim=0)@@ -1267,10 +1358,13 @@ class LongCatVideoAvatarPipeline:latents=latents,need_encode=False)- if context_parallel_util.get_cp_size() > 1:- context_parallel_util.cp_broadcast(latents)num_cond_latents = 1 + (num_cond_frames - 1) // self.vae_scale_factor_temporal+ if not self.use_vcond:+ latents[:, :, :num_cond_latents] = cond_videos_latents++ if context_parallel_util.get_cp_size() > 1:+ context_parallel_util.cp_broadcast(latents)# 6. Prepare ref_target_masks from source size to latent sizeif ref_target_masks is not None:@@ -1311,7 +1405,7 @@ class LongCatVideoAvatarPipeline:timestep[:, :num_cond_latents] = 0if self.do_classifier_free_guidance and ref_target_masks is not None:- # Multitalk mode+ # Multitalk mode with CFGnoise_pred_uncond_text = self.dit(hidden_states=latent_model_input[:1],timestep=timestep[:1],@@ -1319,7 +1413,7 @@ class LongCatVideoAvatarPipeline:encoder_attention_mask=prompt_attention_mask[:1],num_cond_latents=num_cond_latents,kv_cache_dict=kv_cache_dict,- audio_embs=audio_cond_embs[:2],+ audio_embs=audio_cond_embs[:audio_num],num_ref_latents=num_ref_latents,ref_img_index=ref_img_index,mask_frame_range=mask_frame_range,@@ -1332,7 +1426,7 @@ class LongCatVideoAvatarPipeline:encoder_attention_mask=prompt_attention_mask[1:],num_cond_latents=num_cond_latents,kv_cache_dict=kv_cache_dict,- audio_embs=audio_cond_embs[2:],+ audio_embs=audio_cond_embs[audio_num:2*audio_num],num_ref_latents=num_ref_latents,ref_img_index=ref_img_index,mask_frame_range=mask_frame_range,@@ -1341,6 +1435,7 @@ class LongCatVideoAvatarPipeline:noise_pred = torch.cat([noise_pred_uncond_text, noise_pred_cond])else:# Singletalk mode+ # Multitalk mode without CFG (e.g. use_distill with scale=1.0)noise_pred = self.dit(hidden_states=latent_model_input,timestep=timestep,@@ -1351,7 +1446,8 @@ class LongCatVideoAvatarPipeline:audio_embs=audio_cond_embs,num_ref_latents=num_ref_latents,ref_img_index=ref_img_index,- mask_frame_range=mask_frame_range+ mask_frame_range=mask_frame_range,+ ref_target_masks=ref_target_masks)if self.do_classifier_free_guidance:diff --git a/run_demo_avatar_multi_audio_to_video.py b/run_demo_avatar_multi_audio_to_video.pyindex 2cf14d1..f867be5 100644--- a/run_demo_avatar_multi_audio_to_video.py+++ b/run_demo_avatar_multi_audio_to_video.py@@ -19,14 +19,14 @@ from longcat_video.pipeline_longcat_video_avatar import LongCatVideoAvatarPipelifrom longcat_video.modules.scheduling_flow_match_euler_discrete import FlowMatchEulerDiscreteSchedulerfrom longcat_video.modules.autoencoder_kl_wan import AutoencoderKLWanfrom longcat_video.modules.avatar.longcat_video_dit_avatar import LongCatVideoAvatarTransformer3DModel+from longcat_video.modules.quantization import load_quantized_ditfrom longcat_video.context_parallel import context_parallel_util# -------- avatar related --------import librosaimport soundfile as sf-from longcat_video.audio_process.wav2vec2 import Wav2Vec2ModelWrapper+from longcat_video.audio_process import get_audio_encoder, get_audio_feature_extractorfrom longcat_video.audio_process.torch_utils import save_video_ffmpeg-from transformers import Wav2Vec2FeatureExtractorfrom audio_separator.separator import Separator@@ -78,7 +78,7 @@ def audio_prepare_multi(left_temp_vocal_path, right_temp_vocal_path, generate_duright_speech_array_ext = np.concatenate([np.zeros_like(left_speech_array), right_speech_array])merge_raw_speech = np.concatenate([left_raw_speech_array, np.zeros_like(right_raw_speech_array)]) + \np.concatenate([np.zeros_like(left_raw_speech_array), right_raw_speech_array])- elif audio_type == 'para':+ elif audio_type == 'para':left_speech_array_ext = left_speech_arrayright_speech_array_ext = right_speech_arraymerge_raw_speech = left_raw_speech_array + right_raw_speech_array@@ -109,12 +109,23 @@ def generate(args):resolution = args.resolutionnum_segments = max(1, args.num_segments)output_dir = args.output_dir+ model_type = args.model_type+ use_distill = args.use_distill+ use_int8 = args.use_int8++ if use_distill and model_type == "avatar-v1.5":+ num_inference_steps = 8+ text_guidance_scale = 1.0+ audio_guidance_scale = 1.0# set up default inference paramssave_fps = 16+ audio_stride = 2+ if model_type == "avatar-v1.5":+ save_fps = 25+ audio_stride = 1num_frames = 93num_cond_frames = 13- audio_stride = 2if resolution == '480p':height, width = 480, 832@@ -130,11 +141,14 @@ def generate(args):right_raw_speech_path = input_data['cond_audio'].get('person2', None)assert left_raw_speech_path is not None or right_raw_speech_path is not None, f"At least one speech is required."left_person_bbox, right_person_bbox = None, None+ use_background_silent_audio = Falseif 'bbox' in input_data:# bbox: [left_y_min, left_x_min, left_y_max, left_x_max]# x and y coordinates correspond to the width and height dimensions, respectivelyleft_person_bbox = input_data['bbox'].get('person1', None)right_person_bbox = input_data['bbox'].get('person2', None)+ other_person_bbox = input_data['bbox'].get('others', None)+ use_background_silent_audio = other_person_bbox is not None and len(other_person_bbox) > 0audio_type = 'para'if 'audio_type' in input_data:audio_type = input_data.get('audio_type', 'para')@@ -158,15 +172,35 @@ def generate(args):tokenizer = AutoTokenizer.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="tokenizer", torch_dtype=torch.bfloat16)text_encoder = UMT5EncoderModel.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="text_encoder", torch_dtype=torch.bfloat16)vae = AutoencoderKLWan.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="vae", torch_dtype=torch.bfloat16)- scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="scheduler", torch_dtype=torch.bfloat16)- dit = LongCatVideoAvatarTransformer3DModel.from_pretrained(checkpoint_dir, subfolder="avatar_multi", cp_split_hw=cp_split_hw, torch_dtype=torch.bfloat16)+ if model_type == "avatar-v1.0":+ scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="scheduler", torch_dtype=torch.bfloat16)+ elif model_type == "avatar-v1.5":+ scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(checkpoint_dir, subfolder="scheduler", torch_dtype=torch.bfloat16)+ else:+ raise ValueError(f"Unsupported model_type: {model_type}. Expected 'avatar-v1.0' or 'avatar-v1.5'.")++ if model_type == "avatar-v1.0":+ dit = LongCatVideoAvatarTransformer3DModel.from_pretrained(checkpoint_dir, subfolder="avatar_multi", cp_split_hw=cp_split_hw, torch_dtype=torch.bfloat16)+ elif model_type == "avatar-v1.5":+ if use_int8:+ print("[INFO] Loading INT8 quantized DiT model...")+ dit = load_quantized_dit(checkpoint_dir, subfolder="base_model_int8", cp_split_hw=cp_split_hw)+ else:+ dit = LongCatVideoAvatarTransformer3DModel.from_pretrained(checkpoint_dir, subfolder="base_model", cp_split_hw=cp_split_hw, torch_dtype=torch.bfloat16)+ if use_distill:+ distill_checkpoint_path = os.path.join(checkpoint_dir, 'lora', f'dmd_lora.safetensors')+ if os.path.exists(distill_checkpoint_path):+ dit.load_lora(distill_checkpoint_path, "dmd", multiplier=1.0, lora_network_dim=128, lora_network_alpha=64)+ dit.enable_loras(["dmd"])# initialize audio models- wav2vec_path = os.path.join(checkpoint_dir, 'chinese-wav2vec2-base')- audio_encoder = Wav2Vec2ModelWrapper(wav2vec_path).to(local_rank)- audio_encoder.feature_extractor._freeze_parameters()+ if model_type == "avatar-v1.0":+ audio_model_checkpoint_path = os.path.join(checkpoint_dir, 'chinese-wav2vec2-base')+ elif model_type == "avatar-v1.5":+ audio_model_checkpoint_path = os.path.join(checkpoint_dir, 'whisper-large-v3')+ audio_encoder = get_audio_encoder(audio_model_checkpoint_path, model_type).to(local_rank)+ audio_feature_extractor = get_audio_feature_extractor(audio_model_checkpoint_path, model_type)- wav2vec_feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(wav2vec_path, local_files_only=True)vocal_separator_path = os.path.join(checkpoint_dir, 'vocal_separator/Kim_Vocal_2.onnx')audio_output_dir_temp = f"./audio_temp_file"os.makedirs(audio_output_dir_temp, exist_ok=True)@@ -189,7 +223,8 @@ def generate(args):scheduler = scheduler,dit = dit,audio_encoder=audio_encoder,- wav2vec_feature_extractor=wav2vec_feature_extractor+ audio_feature_extractor=audio_feature_extractor,+ model_type=model_type)pipe.to(local_rank)@@ -202,8 +237,10 @@ def generate(args):if cp_rank == 0:# extract vocalsr = 16000- left_temp_vocal_path = extract_vocal_from_speech(left_raw_speech_path, f"/tmp/temp_speech_{generate_random_uid()}_{global_rank}_vocal.wav", vocal_separator, audio_output_dir_temp)- right_temp_vocal_path = extract_vocal_from_speech(right_raw_speech_path, f"/tmp/temp_speech_{generate_random_uid()}_{global_rank}_vocal.wav", vocal_separator, audio_output_dir_temp)+ left_temp_vocal_path = os.path.join(audio_output_dir_temp, f"{generate_random_uid()}_left_temp_vocal.wav")+ right_temp_vocal_path = os.path.join(audio_output_dir_temp, f"{generate_random_uid()}_right_temp_vocal.wav")+ left_temp_vocal_path = extract_vocal_from_speech(left_raw_speech_path, left_temp_vocal_path, vocal_separator, audio_output_dir_temp)+ right_temp_vocal_path = extract_vocal_from_speech(right_raw_speech_path, right_temp_vocal_path, vocal_separator, audio_output_dir_temp)# prepare each vocal and synthesize the sum audiogenerate_duration = num_frames / save_fps + (num_segments-1) * (num_frames-num_cond_frames) / save_fps@@ -212,12 +249,15 @@ def generate(args):merge_speech_path = f"/tmp/temp_speech_{generate_random_uid()}_{global_rank}_merge.wav"sf.write(merge_speech_path, merge_speech, 16000)- left_full_audio_emb = pipe.get_audio_embedding(left_speech_array_ext, fps=save_fps*audio_stride, device=local_rank, sample_rate=sr)- right_full_audio_emb = pipe.get_audio_embedding(right_speech_array_ext, fps=save_fps*audio_stride, device=local_rank, sample_rate=sr)+ left_full_audio_emb = pipe.get_audio_embedding(left_speech_array_ext, fps=save_fps*audio_stride, device=local_rank, sample_rate=sr, model_type=model_type)+ right_full_audio_emb = pipe.get_audio_embedding(right_speech_array_ext, fps=save_fps*audio_stride, device=local_rank, sample_rate=sr, model_type=model_type)if torch.isnan(left_full_audio_emb).any() or torch.isnan(right_full_audio_emb).any():raise ValueError(f"broken audio embedding with nan values")-+ if use_background_silent_audio:+ back_full_audio_emb = pipe.get_audio_embedding(np.zeros_like(left_speech_array_ext), fps=save_fps*audio_stride, device=local_rank, sample_rate=sr, model_type=model_type)assert left_full_audio_emb.shape == right_full_audio_emb.shape, f"Inconsistent audio embedding shape."+ if use_background_silent_audio:+ assert left_full_audio_emb.shape == back_full_audio_emb.shape, f"Inconsistent audio embedding shape between speaker and background."if context_parallel_util.get_cp_size() > 1:full_audio_emb_shape_list = list(left_full_audio_emb.size())@@ -225,6 +265,8 @@ def generate(args):context_parallel_util.cp_broadcast(full_audio_emb_tensor_shape_list)context_parallel_util.cp_broadcast(left_full_audio_emb)context_parallel_util.cp_broadcast(right_full_audio_emb)+ if use_background_silent_audio:+ context_parallel_util.cp_broadcast(back_full_audio_emb)if left_temp_vocal_path is not None and os.path.exists(left_temp_vocal_path):os.remove(left_temp_vocal_path)@@ -239,6 +281,9 @@ def generate(args):context_parallel_util.cp_broadcast(left_full_audio_emb)right_full_audio_emb = torch.zeros(*full_audio_emb_shape_list, dtype=torch.float32, device=local_rank)context_parallel_util.cp_broadcast(right_full_audio_emb)+ if use_background_silent_audio:+ back_full_audio_emb = torch.zeros(*full_audio_emb_shape_list, dtype=torch.float32, device=local_rank)+ context_parallel_util.cp_broadcast(back_full_audio_emb)indices = torch.arange(2 * 2 + 1) - 2@@ -250,7 +295,10 @@ def generate(args):center_indices = torch.clamp(center_indices, min=0, max=left_full_audio_emb.shape[0]-1)left_audio_emb = left_full_audio_emb[center_indices][None,...].to(local_rank)right_audio_emb = right_full_audio_emb[center_indices][None,...].to(local_rank)- audio_embs = torch.cat([left_audio_emb, right_audio_emb])+ audio_embs = [left_audio_emb, right_audio_emb]+ if use_background_silent_audio:+ audio_embs.append(back_full_audio_emb[center_indices][None,...].to(local_rank))+ audio_embs = torch.cat(audio_embs)# ==============================@@ -282,7 +330,14 @@ def generate(args):background_mask += human_mask1background_mask += human_mask2background_mask = torch.where(background_mask > 0, torch.tensor(0), torch.tensor(1))- ref_target_masks = torch.stack([human_mask1, human_mask2, background_mask], dim=0).to(local_rank)+ total_mask = [human_mask1, human_mask2, background_mask]+ if use_background_silent_audio:+ for i in range(len(other_person_bbox)//4):+ other_person_bbox_i = other_person_bbox[i*4:(i+1)*4]+ other_person_mask = torch.zeros([src_height, src_width])+ other_person_mask[other_person_bbox_i[0]:other_person_bbox_i[2], other_person_bbox_i[1]:other_person_bbox_i[3]] = 1+ total_mask.append(other_person_mask)+ ref_target_masks = torch.stack(total_mask, dim=0).to(local_rank)# generate videooutput_tuple = pipe.generate_ai2v(@@ -297,7 +352,8 @@ def generate(args):output_type='both',generator=generator,audio_emb=audio_embs,- ref_target_masks=ref_target_masks+ ref_target_masks=ref_target_masks,+ use_distill=use_distill,)output, latent = output_tupleoutput = output[0]@@ -336,7 +392,10 @@ def generate(args):center_indices = torch.clamp(center_indices, min=0, max=left_full_audio_emb.shape[0]-1)left_audio_emb = left_full_audio_emb[center_indices][None,...].to(local_rank)right_audio_emb = right_full_audio_emb[center_indices][None,...].to(local_rank)- audio_embs = torch.cat([left_audio_emb, right_audio_emb])+ audio_embs = [left_audio_emb, right_audio_emb]+ if use_background_silent_audio:+ audio_embs.append(back_full_audio_emb[center_indices][None,...].to(local_rank))+ audio_embs = torch.cat(audio_embs)output_tuple = pipe.generate_avc(video=current_video,@@ -354,12 +413,13 @@ def generate(args):output_type='both',use_kv_cache=True,offload_kv_cache=False,- enhance_hf=True,+ enhance_hf=True if not use_distill else False,audio_emb=audio_embs,ref_latent=ref_latent,ref_img_index=ref_img_index,mask_frame_range=mask_frame_range,- ref_target_masks=ref_target_masks+ ref_target_masks=ref_target_masks,+ use_distill=use_distill,)output, latent = output_tuple@@ -441,6 +501,20 @@ def _parse_args():type=str,default="./weights/LongCat-Video-Avatar",)+ parser.add_argument(+ "--model_type",+ type=str,+ default="avatar-v1.0",+ )+ parser.add_argument(+ "--use_distill",+ action='store_true',+ )+ parser.add_argument(+ "--use_int8",+ action='store_true',+ help="Load INT8 quantized DiT model for reduced VRAM usage"+ )args = parser.parse_args()diff --git a/run_demo_avatar_single_audio_to_video.py b/run_demo_avatar_single_audio_to_video.pyindex aec2f5f..5fc822a 100644--- a/run_demo_avatar_single_audio_to_video.py+++ b/run_demo_avatar_single_audio_to_video.py@@ -19,13 +19,13 @@ from longcat_video.pipeline_longcat_video_avatar import LongCatVideoAvatarPipelifrom longcat_video.modules.scheduling_flow_match_euler_discrete import FlowMatchEulerDiscreteSchedulerfrom longcat_video.modules.autoencoder_kl_wan import AutoencoderKLWanfrom longcat_video.modules.avatar.longcat_video_dit_avatar import LongCatVideoAvatarTransformer3DModel+from longcat_video.modules.quantization import load_quantized_ditfrom longcat_video.context_parallel import context_parallel_util# -------- avatar related --------import librosa-from longcat_video.audio_process.wav2vec2 import Wav2Vec2ModelWrapper+from longcat_video.audio_process import get_audio_encoder, get_audio_feature_extractorfrom longcat_video.audio_process.torch_utils import save_video_ffmpeg-from transformers import Wav2Vec2FeatureExtractorfrom audio_separator.separator import Separator@@ -64,12 +64,23 @@ def generate(args):resolution = args.resolutionnum_segments = max(1, args.num_segments)output_dir = args.output_dir+ model_type = args.model_type+ use_distill = args.use_distill+ use_int8 = args.use_int8++ if use_distill and model_type == "avatar-v1.5":+ num_inference_steps = 8+ text_guidance_scale = 1.0+ audio_guidance_scale = 1.0# set up default inference paramssave_fps = 16+ audio_stride = 2+ if model_type == "avatar-v1.5":+ save_fps = 25+ audio_stride = 1num_frames = 93num_cond_frames = 13- audio_stride = 2if resolution == '480p':height, width = 480, 832@@ -102,15 +113,37 @@ def generate(args):tokenizer = AutoTokenizer.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="tokenizer", torch_dtype=torch.bfloat16)text_encoder = UMT5EncoderModel.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="text_encoder", torch_dtype=torch.bfloat16)vae = AutoencoderKLWan.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="vae", torch_dtype=torch.bfloat16)- scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="scheduler", torch_dtype=torch.bfloat16)- dit = LongCatVideoAvatarTransformer3DModel.from_pretrained(checkpoint_dir, subfolder="avatar_single", cp_split_hw=cp_split_hw, torch_dtype=torch.bfloat16)+ if model_type == "avatar-v1.0":+ scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(os.path.join(checkpoint_dir, '..', 'LongCat-Video'), subfolder="scheduler", torch_dtype=torch.bfloat16)+ elif model_type == "avatar-v1.5":+ scheduler = FlowMatchEulerDiscreteScheduler.from_pretrained(checkpoint_dir, subfolder="scheduler", torch_dtype=torch.bfloat16)+ else:+ raise ValueError(f"Unsupported model_type: {model_type}. Expected 'avatar-v1.0' or 'avatar-v1.5'.")++ if model_type == "avatar-v1.0":+ dit = LongCatVideoAvatarTransformer3DModel.from_pretrained(checkpoint_dir, subfolder="avatar_single", cp_split_hw=cp_split_hw, torch_dtype=torch.bfloat16)+ elif model_type == "avatar-v1.5":+ if use_int8:+ print("[INFO] Loading INT8 quantized DiT model...")+ dit = load_quantized_dit(checkpoint_dir, subfolder="base_model_int8", cp_split_hw=cp_split_hw)+ else:+ dit = LongCatVideoAvatarTransformer3DModel.from_pretrained(checkpoint_dir, subfolder="base_model", cp_split_hw=cp_split_hw, torch_dtype=torch.bfloat16)+ if use_distill:+ distill_checkpoint_path = os.path.join(checkpoint_dir, 'lora', f'dmd_lora.safetensors')+ if os.path.exists(distill_checkpoint_path):+ dit.load_lora(distill_checkpoint_path, "dmd", multiplier=1.0, lora_network_dim=128, lora_network_alpha=64)+ dit.enable_loras(["dmd"])+ else:+ raise ValueError(f"Unsupported model_type: {model_type}. Expected 'avatar-v1.0' or 'avatar-v1.5'.")# initialize audio models- wav2vec_path = os.path.join(checkpoint_dir, 'chinese-wav2vec2-base')- audio_encoder = Wav2Vec2ModelWrapper(wav2vec_path).to(local_rank)- audio_encoder.feature_extractor._freeze_parameters()+ if model_type == "avatar-v1.0":+ audio_model_checkpoint_path = os.path.join(checkpoint_dir, 'chinese-wav2vec2-base')+ elif model_type == "avatar-v1.5":+ audio_model_checkpoint_path = os.path.join(checkpoint_dir, 'whisper-large-v3')+ audio_encoder = get_audio_encoder(audio_model_checkpoint_path, model_type).to(local_rank)+ audio_feature_extractor = get_audio_feature_extractor(audio_model_checkpoint_path, model_type)- wav2vec_feature_extractor = Wav2Vec2FeatureExtractor.from_pretrained(wav2vec_path, local_files_only=True)vocal_separator_path = os.path.join(checkpoint_dir, 'vocal_separator/Kim_Vocal_2.onnx')audio_output_dir_temp = f"./audio_temp_file"os.makedirs(audio_output_dir_temp, exist_ok=True)@@ -133,7 +166,8 @@ def generate(args):scheduler = scheduler,dit = dit,audio_encoder=audio_encoder,- wav2vec_feature_extractor=wav2vec_feature_extractor+ audio_feature_extractor=audio_feature_extractor,+ model_type=model_type)pipe.to(local_rank)@@ -157,7 +191,7 @@ def generate(args):speech_array = np.append(speech_array, [0.]*added_sample_nums)# audio embedding- full_audio_emb = pipe.get_audio_embedding(speech_array, fps=save_fps*audio_stride, device=local_rank, sample_rate=sr)+ full_audio_emb = pipe.get_audio_embedding(speech_array, fps=save_fps*audio_stride, device=local_rank, sample_rate=sr, model_type=model_type)if torch.isnan(full_audio_emb).any():raise ValueError(f"broken audio embedding with nan values")@@ -205,7 +239,8 @@ def generate(args):audio_guidance_scale=audio_guidance_scale,generator=generator,output_type='both',- audio_emb=audio_emb+ audio_emb=audio_emb,+ use_distill=use_distill,)output, latent = output_tupleoutput = output[0]@@ -235,7 +270,8 @@ def generate(args):audio_guidance_scale=audio_guidance_scale,output_type='both',generator=generator,- audio_emb=audio_emb+ audio_emb=audio_emb,+ use_distill=use_distill,)output, latent = output_tupleoutput = output[0]@@ -292,11 +328,12 @@ def generate(args):output_type='both',use_kv_cache=True,offload_kv_cache=False,- enhance_hf=True,+ enhance_hf=True if not use_distill else False,audio_emb=audio_emb,ref_latent=ref_latent,ref_img_index=ref_img_index,- mask_frame_range=mask_frame_range+ mask_frame_range=mask_frame_range,+ use_distill=use_distill,)output, latent = output_tuple@@ -379,6 +416,20 @@ def _parse_args():type=str,default="./weights/LongCat-Video-Avatar",)+ parser.add_argument(+ "--model_type",+ type=str,+ default="avatar-v1.0",+ )+ parser.add_argument(+ "--use_distill",+ action='store_true',+ )+ parser.add_argument(+ "--use_int8",+ action='store_true',+ help="Load INT8 quantized DiT model for reduced VRAM usage"+ )args = parser.parse_args()