Python
added ltx2.3 ic lora support, svi2 pro end frame, matanyone2
DeepBeepMeep46f340d4e12a
diff --git a/README.md b/README.mdindex cc8ef77..7218bb3 100644--- a/README.md+++ b/README.md@@ -33,16 +33,27 @@ WanGP supports the Wan (and derived models) but also Hunyuan Video, Flux, Qwen,## 🔥 Latest Updates :-### March 6th 2026: WanGP v10.98, Expecting an Update ?+### March 7th 2026: WanGP v10.981, Expecting an Update ?- **LTX-2 2.3**: 0 day delivery of LTX 2 latest version with better *audio*, *image 2 video* and *greater details*. This model is bigger (22B versus 19B), but with WanGP VRAM usage will be still ridiculously low. Try it at 720p or 1080p, this is where it will shine the most !+*Control Video Support* (*Ic lora Union Control*) will let you transfer *Human Motion*, *Edges*, ... in your new video.++For expert users, *Dev* finetune offers extra new configurable settings (*modality guidance*, *audio guidance*, *STG pertubation/skip self attention *, *guidance rescaling*). LTX team suggests: Cfg=3, Audio cfg=7, Modality Cfg=3, Rescale=0.7, STG Perturbation Skip Attention on all steps.++I recommend to stick to the *Distilled* finetune for higher resolutions (see sample video below) as it seems to have been distilled from a higher quality model (pro model?).+- **Kiwi Edit**: a great model that lets you edit video and / or inject objects in a video. It exists in 3 flavours depending on what you want to do+- **SVI PRO2 End Frames**: this should allow in theory to generate very long shots by splitting one shot into sub shots (sliding windows) by inserting key frames (the *End Frames*). This is an alternative to the *Infinitalk* references frames method (see my old release notes). I am waiting for your feedback to know which method is the best one.+- **Upgraded Models Selector** with *already Downloaded indicator*: Next to each model or finetune, you will find a colored square: *Blue* = fully downloaded & available, *Yellow* = partially downloaded & *Black* = not downloaded at all. Please note that the square color will depend on your current choices of requested model quantization.- **Upgraded Models Manager**: colors squares have also been added so that you can see in glance what has already been downloaded. New filter for a quick model lookout. List of missing files per finetune.+- **Matanyone 2**: everyone favorite Mask extractor has been been updated and is now more precise++*update 10.981*: LTX2.3 Ic Lora Support & expert settings, Matanyone 2, SVI Pro end frames### February 19th 2026: WanGP v10.951, Breaking the Sound Barrier, Mach 2Here comes the (last ?) missing bit in WanGP of the *Text To Speech* offering: **emotions**diff --git a/defaults/hunyuan_t2v_accvideo.json b/defaults/hunyuan_t2v_accvideo.jsonindex 2da984a..3c56dd7 100644--- a/defaults/hunyuan_t2v_accvideo.json+++ b/defaults/hunyuan_t2v_accvideo.json@@ -22,8 +22,8 @@"temporal_upsampling": "","spatial_upsampling": "","RIFLEx_setting": 0,- "slg_start_perc": 10,- "slg_end_perc": 90,+ "perturbation_start_perc": 10,+ "perturbation_end_perc": 90,"prompt_enhancer": "","activated_loras": []diff --git a/defaults/hunyuan_t2v_fast.json b/defaults/hunyuan_t2v_fast.jsonindex 4019e24..54bb889 100644--- a/defaults/hunyuan_t2v_fast.json+++ b/defaults/hunyuan_t2v_fast.json@@ -24,8 +24,8 @@"temporal_upsampling": "","spatial_upsampling": "","RIFLEx_setting": 0,- "slg_start_perc": 10,- "slg_end_perc": 90,+ "perturbation_start_perc": 10,+ "perturbation_end_perc": 90,"prompt_enhancer": "","activated_loras": []diff --git a/defaults/ltx2_19B.json b/defaults/ltx2_19B.jsonindex 1d909ff..949ebf8 100644--- a/defaults/ltx2_19B.json+++ b/defaults/ltx2_19B.json@@ -20,5 +20,14 @@"prompt": "A warm sunny backyard. The camera starts in a tight cinematic close-up of a woman and a man in their 30s, facing each other with serious expressions. The woman, emotional and dramatic, says softly, \"That's it... Dad's lost it. And we've lost Dad.\"The man exhales, slightly annoyed: \"Stop being so dramatic, Jess.\"A beat. He glances aside, then mutters defensively, \"He's just having fun.\"The camera slowly pans right, revealing the grandfather in the garden wearing enormous butterfly wings, waving his arms in the air like he's trying to take off.He shouts, \"Wheeeew!\" as he flaps his wings with full commitment.The woman covers her face, on the verge of tears. The tone is deadpan, absurd, and quietly tragic.","num_inference_steps": 40,"video_length": 241,- "guidance_scale": 4.0+ "guidance_scale": 3.0,+ "audio_guidance_scale": 7.0,+ "alt_guidance_scale": 3.0,+ "alt_scale": 0.7,+ "perturbation_switch": 2,+ "perturbation_layers": [29],+ "perturbation_start_perc": 0,+ "perturbation_end_perc": 100,+ "apg_switch": 0,+ "cfg_star_switch": 0}diff --git a/defaults/ltx2_22B.json b/defaults/ltx2_22B.jsonindex 9594dca..71a6187 100644--- a/defaults/ltx2_22B.json+++ b/defaults/ltx2_22B.json@@ -2,11 +2,14 @@"model": {"name": "LTX-2 2.3 Dev 22B","architecture": "ltx2_22B",- "description": "LTX-2.3 dev checkpoint (22B) for video+audio generation with start/end keyframes and sliding-window continuation.",+ "description": "LTX-2.3 generates video up to 20s with an audio soundtrack. Supports start/end keyframes and sliding-window continuation. You can force the audio soundtrack by providing an Audio Prompt..","URLs": ["https://huggingface.co/DeepBeepMeep/LTX-2/resolve/main/ltx-2.3-22b-dev_diffusion_model.safetensors","https://huggingface.co/DeepBeepMeep/LTX-2/resolve/main/ltx-2.3-22b-dev_diffusion_model_quanto_int8.safetensors"],+ "preload_URLs": [+ "https://huggingface.co/DeepBeepMeep/LTX-2/resolve/main/ltx-2.3-22b-ic-lora-union-control-ref0.5.safetensors"+ ],"loras": ["https://huggingface.co/DeepBeepMeep/LTX-2/resolve/main/ltx-2.3-22b-distilled-lora-384.safetensors"],@@ -18,5 +21,14 @@"num_inference_steps": 30,"video_length": 241,"resolution": "1280x720",- "guidance_scale": 3.0+ "guidance_scale": 3.0,+ "audio_guidance_scale": 7.0,+ "alt_guidance_scale": 3.0,+ "alt_scale": 0.7,+ "perturbation_switch": 2,+ "perturbation_layers": [28],+ "perturbation_start_perc": 0,+ "perturbation_end_perc": 100,+ "apg_switch": 0,+ "cfg_star_switch": 0}diff --git a/defaults/t2v_fusionix.json b/defaults/t2v_fusionix.jsonindex 6ecdf0c..907fa1d 100644--- a/defaults/t2v_fusionix.json+++ b/defaults/t2v_fusionix.json@@ -28,9 +28,9 @@"temporal_upsampling": "","spatial_upsampling": "","RIFLEx_setting": 0,- "slg_switch": 0,- "slg_start_perc": 10,- "slg_end_perc": 90,+ "perturbation_switch": 0,+ "perturbation_start_perc": 10,+ "perturbation_end_perc": 90,"cfg_star_switch": 0,"cfg_zero_step": -1,"prompt_enhancer": "",diff --git a/defaults/vace_14B_fusionix.json b/defaults/vace_14B_fusionix.jsonindex 95639f4..34f062f 100644--- a/defaults/vace_14B_fusionix.json+++ b/defaults/vace_14B_fusionix.json@@ -26,9 +26,9 @@"temporal_upsampling": "","spatial_upsampling": "","RIFLEx_setting": 0,- "slg_switch": 0,- "slg_start_perc": 10,- "slg_end_perc": 90,+ "perturbation_switch": 0,+ "perturbation_start_perc": 10,+ "perturbation_end_perc": 90,"cfg_star_switch": 0,"cfg_zero_step": -1,"prompt_enhancer": "",diff --git a/defaults/vace_14B_sf.json b/defaults/vace_14B_sf.jsonindex 7dc495d..f52e2d8 100644--- a/defaults/vace_14B_sf.json+++ b/defaults/vace_14B_sf.json@@ -31,9 +31,9 @@"temporal_upsampling": "","spatial_upsampling": "","RIFLEx_setting": 0,- "slg_switch": 0,- "slg_start_perc": 10,- "slg_end_perc": 90,+ "perturbation_switch": 0,+ "perturbation_start_perc": 10,+ "perturbation_end_perc": 90,"cfg_star_switch": 0,"cfg_zero_step": -1,"prompt_enhancer": "",diff --git a/defaults/vace_multitalk_14B.json b/defaults/vace_multitalk_14B.jsonindex c35a048..599ba2e 100644--- a/defaults/vace_multitalk_14B.json+++ b/defaults/vace_multitalk_14B.json@@ -31,9 +31,9 @@"temporal_upsampling": "","spatial_upsampling": "","RIFLEx_setting": 0,- "slg_switch": 0,- "slg_start_perc": 10,- "slg_end_perc": 90,+ "perturbation_switch": 0,+ "perturbation_start_perc": 10,+ "perturbation_end_perc": 90,"cfg_star_switch": 0,"cfg_zero_step": -1,"prompt_enhancer": "",diff --git a/models/_settings.json b/models/_settings.jsonindex 2a55778..7d0cd73 100644--- a/models/_settings.json+++ b/models/_settings.json@@ -38,6 +38,7 @@"audio_scale": 1,"embedded_guidance_scale": 6.0,"alt_guidance_scale": 1.0,+ "alt_scale": 0.0,"temperature": 0.8,"top_p": 0.9,"top_k": 50,@@ -63,10 +64,10 @@"MMAudio_setting": 0,"MMAudio_prompt": "","MMAudio_neg_prompt": "",- "slg_switch": 0,- "slg_layers": [9],- "slg_start_perc": 10,- "slg_end_perc": 90,+ "perturbation_switch": 0,+ "perturbation_layers": [9],+ "perturbation_start_perc": 10,+ "perturbation_end_perc": 90,"apg_switch": 0,"cfg_star_switch": 0,"cfg_zero_step": -1,diff --git a/models/ltx2/ltx2.py b/models/ltx2/ltx2.pyindex 71278c5..5dc4a54 100644--- a/models/ltx2/ltx2.py+++ b/models/ltx2/ltx2.py@@ -1,6 +1,7 @@import jsonimport mathimport os+import reimport typesfrom typing import Callable, Iterator@@ -298,7 +299,9 @@ def _attach_lora_preprocessor(transformer: torch.nn.Module) -> None:return None, ""new_sd = {}+ dropped_keys = []for key, value in sd.items():+ original_key = keyif key.startswith("model."):key = key[len("model.") :]if key.startswith("diffusion_model."):@@ -312,14 +315,23 @@ def _attach_lora_preprocessor(transformer: torch.nn.Module) -> None:module_name, suffix = split_lora_key(key)if not module_name:+ dropped_keys.append(original_key)continueif module_name not in module_names:prefixed_name = f"velocity_model.{module_name}"if prefixed_name in module_names:module_name = prefixed_nameelse:+ dropped_keys.append(original_key)continuenew_sd[f"{module_name}{suffix}"] = value+ if dropped_keys:+ sample = ", ".join(dropped_keys[:8])+ if len(dropped_keys) > 8:+ sample += ", ..."+ raise ValueError(+ f"LTX2 LoRA preprocessing dropped {len(dropped_keys)} unmatched keys for model '{model_type}': {sample}"+ )return new_sdtransformer.preprocess_loras = types.MethodType(preprocess_loras, transformer)@@ -394,6 +406,29 @@ def _build_tiling_config(tile_size: int | tuple | list | None, fps: float | Nonereturn TilingConfig(spatial_config=spatial_config, temporal_config=temporal_config)+def _infer_ic_lora_downscale_factor(loras_selected) -> int | None:+ factors = []+ for lora_path in loras_selected or []:+ name = os.path.basename(str(lora_path)).lower()+ if "ic-lora" not in name:+ continue+ match = re.search(r"-ref([0-9]+(?:\.[0-9]+)?)", name)+ if not match:+ factors.append(1)+ continue+ ref_ratio = float(match.group(1))+ if ref_ratio <= 0:+ factors.append(1)+ continue+ factors.append(max(1, int(round(1.0 / ref_ratio))))+ if not factors:+ return None+ unique_factors = sorted(set(factors))+ if len(unique_factors) > 1:+ raise ValueError(f"Conflicting IC-LoRA reference downscale factors in selected LoRAs: {unique_factors}")+ return unique_factors[0]++def _collect_video_chunks(video: Iterator[torch.Tensor] | torch.Tensor,interrupt_check: Callable[[], bool] | None = None,@@ -434,6 +469,7 @@ class LTX2:self.device = torch.device("cuda")self.dtype = dtypeself.VAE_dtype = VAE_dtype+ self.base_model_type = base_model_typeself.model_def = model_defself._interrupt = Falseself.vae = _LTX2VAEHelper()@@ -514,7 +550,7 @@ class LTX2:config = _load_config_from_checkpoint(path, fallback_config_path=fallback_config_path)return config or base_config- def _load_component(model, path, sd_ops=None, postprocess=None):+ def _load_component(model, path, sd_ops=None, postprocess=None, ignore_unused_weights=False):if postprocess is None and sd_ops is not None:postprocess = _make_sd_postprocess(sd_ops)mmgp_offload.load_model_data(@@ -524,6 +560,7 @@ class LTX2:default_dtype=self.dtype,writable_tensors=False,ignore_missing_keys=False,+ ignore_unused_weights=ignore_unused_weights,)model.eval().requires_grad_(False)return model@@ -541,7 +578,7 @@ class LTX2:video_encoder = VideoEncoderConfigurator.from_config(video_config)video_decoder = VideoDecoderConfigurator.from_config(video_config)video_vae = _VAEContainer(video_encoder, video_decoder)- video_vae = _load_component(video_vae, video_vae_path, postprocess=_make_vae_postprocess("vae."))+ video_vae = _load_component(video_vae, video_vae_path, postprocess=_make_vae_postprocess("vae."), ignore_unused_weights=True)video_encoder = video_vae.encodervideo_decoder = video_vae.decoder@@ -642,21 +679,30 @@ class LTX2:trans = self.modelreturn trans, None- def get_loras_transformer(self, get_model_recursive_prop, model_type, video_prompt_type, **kwargs):- map = {+ def get_loras_transformer(self, get_model_recursive_prop, model_type, video_prompt_type, base_model_type=None, **kwargs):+ control_map = {"P": "pose","D": "depth","E": "canny",}loras = []video_prompt_type = video_prompt_type or ""+ if (get_model_recursive_prop(model_type, "ltx2_pipeline") or "two_stage") != "distilled":+ return [], []preload_urls = get_model_recursive_prop(model_type, "preload_URLs")- for letter, signature in map.items():- if letter in video_prompt_type:+ if (base_model_type or self.base_model_type) == "ltx2_22B":+ if any(letter in video_prompt_type for letter in control_map):for file_name in preload_urls:- if signature in file_name:+ if "union-control" in os.path.basename(file_name):loras.append(fl.locate_file(os.path.basename(file_name)))break+ else:+ for letter, signature in control_map.items():+ if letter in video_prompt_type:+ for file_name in preload_urls:+ if signature in file_name:+ loras.append(fl.locate_file(os.path.basename(file_name)))+ breakloras_mult = [1.0] * len(loras)return loras, loras_mult@@ -669,6 +715,36 @@ class LTX2:sampling_steps: int = 40,guide_scale: float = 4.0,alt_guide_scale: float = 1.0,+ input_video=None,+ prefix_frames_count: int = 0,+ input_frames=None,+ input_frames2=None,+ input_masks=None,+ input_masks2=None,+ masking_strength: float | None = None,+ input_video_strength: float | None = None,+ return_latent_slice=None,+ video_prompt_type: str = "",+ denoising_strength: float | None = None,+ cfg_star_switch: int = 0,+ apg_switch: int = 0,+ perturbation_switch: int = 0,+ perturbation_layers: list[int] | None = None,+ perturbation_start: float = 0.0,+ perturbation_end: float = 1.0,+ audio_cfg_scale: float | None = None,+ alt_scale: float = 0.0,+ self_refiner_setting: int = 0,+ self_refiner_plan: str = "",+ self_refiner_f_uncertainty: float = 0.1,+ self_refiner_certain_percentage: float = 0.999,+ loras_slists=None,+ loras_selected=None,+ text_connectors=None,+ input_waveform=None,+ input_waveform_sample_rate=None,+ audio_scale: float | None = None,+ masking_source: dict | None = None,frame_num: int = 121,height: int = 1024,width: int = 1536,@@ -684,27 +760,8 @@ class LTX2:image_start = _coerce_image_list(image_start)image_end = _coerce_image_list(image_end)- input_video = kwargs.get("input_video")- prefix_frames_count = int(kwargs.get("prefix_frames_count") or 0)- input_frames = kwargs.get("input_frames")- input_frames2 = kwargs.get("input_frames2")- input_masks = kwargs.get("input_masks")- input_masks2 = kwargs.get("input_masks2")- masking_strength = kwargs.get("masking_strength")- input_video_strength = kwargs.get("input_video_strength")- return_latent_slice = kwargs.get("return_latent_slice")- video_prompt_type = kwargs.get("video_prompt_type") or ""- denoising_strength = kwargs.get("denoising_strength")- cfg_star_switch = kwargs.get("cfg_star_switch", 0)- apg_switch = kwargs.get("apg_switch", 0)- slg_switch = kwargs.get("slg_switch", 0)- slg_layers = kwargs.get("slg_layers")- slg_start = kwargs.get("slg_start", 0.0)- slg_end = kwargs.get("slg_end", 1.0)- self_refiner_setting = kwargs.get("self_refiner_setting", 0)- self_refiner_plan = kwargs.get("self_refiner_plan", "")- self_refiner_f_uncertainty = kwargs.get("self_refiner_f_uncertainty", 0.1)- self_refiner_certain_percentage = kwargs.get("self_refiner_certain_percentage", 0.999)+ prefix_frames_count = int(prefix_frames_count or 0)+ video_prompt_type = video_prompt_type or ""self_refiner_max_plans = int(self.model_def.get("self_refiner_max_plans", 1))def _get_frame_dim(video_tensor: torch.Tensor) -> int | None:@@ -760,6 +817,10 @@ class LTX2:if "G" not in video_prompt_type:denoising_strength = 1.0masking_strength = 0.0+ ic_lora_downscale_factor = None+ if self.model_def.get("ltx2_pipeline", "two_stage") == "distilled":+ ic_lora_downscale_factor = _infer_ic_lora_downscale_factor(loras_selected)+ video_conditioning_downscale_factor = ic_lora_downscale_factor or 1video_conditioning = Nonemasking_source = None@@ -862,14 +923,10 @@ class LTX2:tiling_config = _build_tiling_config(VAE_tile_size, fps)interrupt_check = lambda: self._interrupt- loras_slists = kwargs.get("loras_slists")- text_connectors = getattr(self, "_text_connectors", None)+ text_connectors = text_connectors or getattr(self, "_text_connectors", None)audio_conditionings = None- input_waveform = kwargs.get("input_waveform")- input_waveform_sample_rate = kwargs.get("input_waveform_sample_rate")if input_waveform is not None:- audio_scale = kwargs.get("audio_scale")if audio_scale is None:audio_scale = 1.0audio_strength = max(0.0, min(1.0, float(audio_scale)))@@ -983,17 +1040,20 @@ class LTX2:frame_rate=float(fps),num_inference_steps=int(sampling_steps),cfg_guidance_scale=float(guide_scale),+ audio_cfg_guidance_scale=float(guide_scale if audio_cfg_scale is None else audio_cfg_scale),cfg_star_switch=cfg_star_switch,apg_switch=apg_switch,- slg_switch=slg_switch,- slg_layers=slg_layers,- slg_start=slg_start,- slg_end=slg_end,+ perturbation_switch=perturbation_switch,+ perturbation_layers=perturbation_layers,+ perturbation_start=perturbation_start,+ perturbation_end=perturbation_end,alt_guidance_scale=float(alt_guide_scale),+ alt_scale=float(alt_scale),images=images,guiding_images=guiding_images or None,images_stage2=images_stage2 if stage2_override else None,video_conditioning=video_conditioning,+ video_conditioning_downscale_factor=video_conditioning_downscale_factor,latent_conditioning_stage2=latent_conditioning_stage2,tiling_config=tiling_config,enhance_prompt=False,@@ -1022,6 +1082,7 @@ class LTX2:images=images,alt_guidance_scale=float(alt_guide_scale),video_conditioning=video_conditioning,+ video_conditioning_downscale_factor=video_conditioning_downscale_factor,latent_conditioning_stage2=latent_conditioning_stage2,tiling_config=tiling_config,enhance_prompt=False,diff --git a/models/ltx2/ltx2_handler.py b/models/ltx2/ltx2_handler.pyindex 423eba3..9aece38 100644--- a/models/ltx2/ltx2_handler.py+++ b/models/ltx2/ltx2_handler.py@@ -30,6 +30,7 @@ _ARCH_SPECS = {"spatial_upscaler": "ltx-2.3-spatial-upscaler-x2-1.0.safetensors","temporal_upscaler": "ltx-2.3-temporal-upscaler-x2-1.0.safetensors","distilled_lora": "ltx-2.3-22b-distilled-lora-384.safetensors",+ "union_control_lora": "ltx-2.3-22b-ic-lora-union-control-ref0.5.safetensors","video_vae": "ltx-2.3-22b_vae.safetensors","audio_vae": "ltx-2.3-22b_audio_vae.safetensors","vocoder": "ltx-2.3-22b_vocoder.safetensors",@@ -45,10 +46,27 @@ def _get_arch_spec(base_model_type: str | None) -> dict:return _ARCH_SPECS.get(base_model_type or "", _ARCH_SPECS["ltx2_19B"])-def _default_slg_layers(base_model_type: str | None) -> list[int]:+def _default_perturbation_layers(base_model_type: str | None) -> list[int]:return [28] if base_model_type == "ltx2_22B" else [29]+def _default_dev_settings(base_model_type: str | None) -> dict:+ return {+ "num_inference_steps": 30 if base_model_type == "ltx2_22B" else 40,+ "guidance_scale": 3.0,+ # "audio_guidance_scale": 7.0,+ # "alt_guidance_scale": 3.0,+ # "alt_scale": 0.7,+ # "perturbation_switch": 2,+ "perturbation_layers": _default_perturbation_layers(base_model_type),+ "perturbation_start_perc": 0,+ "perturbation_end_perc": 100,+ "apg_switch": 0,+ "cfg_star_switch": 0,+ "guidance_phases": 2,+ }++def _get_embeddings_connector_filename(model_def, base_model_type):spec = _get_arch_spec(base_model_type)shared_connector = spec.get("embeddings_connector")@@ -105,15 +123,14 @@ class family_handler:def query_model_def(base_model_type, model_def):spec = _get_arch_spec(base_model_type)pipeline_kind = model_def.get("ltx2_pipeline", "two_stage")- is_ltx23 = base_model_type == "ltx2_22B"distilled = pipeline_kind == "distilled"- audio_prompt_selection = ["", "A"] if is_ltx23 else ["", "A", "K"]+ audio_prompt_selection = ["", "A", "K"] if distilled else ["", "A"]audio_prompt_labels = {"": "Generate Video & Soundtrack based on Text Prompt","A": "Generate Video based on Soundtrack and Text Prompt",}- if not is_ltx23:+ if distilled:audio_prompt_labels["K"] = "Generate Video based on Control Video + its Audio Track and Text Prompt"extra_model_def = {@@ -141,7 +158,7 @@ class family_handler:},"audio_guide_window_slicing": True,"output_audio_is_input_audio": True,- "custom_denoising_strength": True,+ "custom_denoising_strength": distilled,"profiles_dir": [spec["profiles_dir"]],"ltx2_spatial_upscaler_file": spec["spatial_upscaler"],"self_refiner": True,@@ -150,7 +167,7 @@ class family_handler:extra_model_def["extra_control_frames"] = 1extra_model_def["dont_cat_preguide"] = Trueextra_model_def["input_video_strength"] = "Image / Source Video Strength (you may try values lower value than 1 to get more motion)"- if not is_ltx23:+ if distilled:extra_model_def["guide_preprocessing"] = {"selection": ["", "PVG", "DVG", "EVG", "VG"],"labels": {@@ -183,10 +200,18 @@ class family_handler:else:extra_model_def.update({+ "audio_guidance": True,"adaptive_projected_guidance": True,"cfg_star": True,- "skip_layer_guidance": True,+ "perturbation": True,"alt_guidance": "Modality Guidance",+ "alt_scale": "Guidance Rescale",+ "perturbation_choices": [+ ("Off", 0),+ ("Skip Layer Guidance", 1),+ ("Skip Self Attention", 2),+ ],+ "perturbation_layers_max": 48,})extra_model_def["guidance_max_phases"] = 2@@ -337,7 +362,7 @@ class family_handler:@staticmethoddef fix_settings(base_model_type, settings_version, model_def, ui_defaults):- default_slg_layers = _default_slg_layers(base_model_type)+ default_perturbation_layers = _default_perturbation_layers(base_model_type)pipeline_kind = model_def.get("ltx2_pipeline", "two_stage")if pipeline_kind != "distilled" and ui_defaults.get("guidance_phases", 0) < 2:ui_defaults["guidance_phases"] = 2@@ -354,7 +379,7 @@ class family_handler:ui_defaults.update({"alt_guidance_scale": 1.0,- "slg_layers": default_slg_layers,+ "perturbation_layers": default_perturbation_layers,})@@ -365,6 +390,15 @@ class family_handler:})+ if settings_version < 2.55 and pipeline_kind != "distilled":+ ui_defaults.update({+ "audio_guidance_scale": 1.0,+ "alt_guidance_scale": 1.0,+ "alt_scale": 0.0,+ })++ # _default_dev_settings(base_model_type)+if settings_version < 2.52:plan = ui_defaults.get("self_refiner_plan")if isinstance(plan, list):@@ -373,7 +407,7 @@ class family_handler:@staticmethoddef update_default_settings(base_model_type, model_def, ui_defaults):- default_slg_layers = _default_slg_layers(base_model_type)+ default_perturbation_layers = _default_perturbation_layers(base_model_type)ui_defaults.update({"sliding_window_size": 481,@@ -381,14 +415,12 @@ class family_handler:"denoising_strength": 1.0,"masking_strength": 0,"audio_prompt_type": "",- "alt_guidance_scale": 1.0,- "slg_layers": default_slg_layers,+ "perturbation_layers": default_perturbation_layers,})ui_defaults.setdefault("audio_scale", 1.0)- ui_defaults.setdefault("alt_guidance_scale", 1.0)pipeline_kind = model_def.get("ltx2_pipeline", "two_stage")if pipeline_kind != "distilled":- ui_defaults.setdefault("guidance_phases", 2)+ ui_defaults.update(_default_dev_settings(base_model_type))else:ui_defaults.setdefault("guidance_phases", 1)diff --git a/models/ltx2/ltx_core/conditioning/__init__.py b/models/ltx2/ltx_core/conditioning/__init__.pyindex 72f5ce1..d509b7f 100644--- a/models/ltx2/ltx_core/conditioning/__init__.py+++ b/models/ltx2/ltx_core/conditioning/__init__.py@@ -2,7 +2,7 @@from .exceptions import ConditioningErrorfrom .item import ConditioningItem-from .types import AudioConditionByLatent, VideoConditionByKeyframeIndex, VideoConditionByLatentIndex+from .types import AudioConditionByLatent, VideoConditionByKeyframeIndex, VideoConditionByLatentIndex, VideoConditionByReferenceLatent__all__ = ["ConditioningError",@@ -10,4 +10,5 @@ __all__ = ["AudioConditionByLatent","VideoConditionByKeyframeIndex","VideoConditionByLatentIndex",+ "VideoConditionByReferenceLatent",]diff --git a/models/ltx2/ltx_core/conditioning/types/__init__.py b/models/ltx2/ltx_core/conditioning/types/__init__.pyindex 9ee69b9..5bdacc1 100644--- a/models/ltx2/ltx_core/conditioning/types/__init__.py+++ b/models/ltx2/ltx_core/conditioning/types/__init__.py@@ -2,9 +2,11 @@from .keyframe_cond import VideoConditionByKeyframeIndexfrom .latent_cond import AudioConditionByLatent, VideoConditionByLatentIndex+from .reference_video_cond import VideoConditionByReferenceLatent__all__ = ["VideoConditionByKeyframeIndex","VideoConditionByLatentIndex",+ "VideoConditionByReferenceLatent","AudioConditionByLatent",]diff --git a/models/ltx2/ltx_pipelines/distilled.py b/models/ltx2/ltx_pipelines/distilled.pyindex 75fb0fd..f077667 100644--- a/models/ltx2/ltx_pipelines/distilled.py+++ b/models/ltx2/ltx_pipelines/distilled.py@@ -36,6 +36,7 @@ from .utils.helpers import (prepare_mask_injection,simple_denoising_func,video_conditionings_by_keyframe,+ video_conditionings_by_reference_latent,)from .utils.media_io import encode_videofrom .utils.types import PipelineComponents@@ -163,6 +164,7 @@ class DistilledPipeline:images: list[tuple[str, int, float]],alt_guidance_scale: float = 1.0,video_conditioning: list[tuple[str, float]] | None = None,+ video_conditioning_downscale_factor: int = 1,latent_conditioning_stage2: torch.Tensor | None = None,tiling_config: TilingConfig | None = None,enhance_prompt: bool = False,@@ -313,16 +315,29 @@ class DistilledPipeline:tiling_config=tiling_config,)if video_conditioning:- stage_1_conditionings += video_conditionings_by_keyframe(- video_conditioning=video_conditioning,- height=stage_1_output_shape.height,- width=stage_1_output_shape.width,- num_frames=num_frames,- video_encoder=video_encoder,- dtype=dtype,- device=self.device,- tiling_config=tiling_config,- )+ if int(video_conditioning_downscale_factor or 1) > 1:+ stage_1_conditionings += video_conditionings_by_reference_latent(+ video_conditioning=video_conditioning,+ height=stage_1_output_shape.height,+ width=stage_1_output_shape.width,+ num_frames=num_frames,+ video_encoder=video_encoder,+ dtype=dtype,+ device=self.device,+ downscale_factor=video_conditioning_downscale_factor,+ tiling_config=tiling_config,+ )+ else:+ stage_1_conditionings += video_conditionings_by_keyframe(+ video_conditioning=video_conditioning,+ height=stage_1_output_shape.height,+ width=stage_1_output_shape.width,+ num_frames=num_frames,+ video_encoder=video_encoder,+ dtype=dtype,+ device=self.device,+ tiling_config=tiling_config,+ )mask_context = prepare_mask_injection(masking_source=masking_source,diff --git a/models/ltx2/ltx_pipelines/keyframe_interpolation.py b/models/ltx2/ltx_pipelines/keyframe_interpolation.pyindex f0118ea..a387e25 100644--- a/models/ltx2/ltx_pipelines/keyframe_interpolation.py+++ b/models/ltx2/ltx_pipelines/keyframe_interpolation.py@@ -94,13 +94,15 @@ class KeyframeInterpolationPipeline:num_inference_steps: int,cfg_guidance_scale: float,images: list[tuple[str, int, float]],+ audio_cfg_guidance_scale: float | None = None,cfg_star_switch: int = 0,apg_switch: int = 0,- slg_switch: int = 0,- slg_layers: list[int] | None = None,- slg_start: float = 0.0,- slg_end: float = 1.0,+ perturbation_switch: int = 0,+ perturbation_layers: list[int] | None = None,+ perturbation_start: float = 0.0,+ perturbation_end: float = 1.0,alt_guidance_scale: float = 1.0,+ alt_scale: float = 0.0,tiling_config: TilingConfig | None = None,enhance_prompt: bool = False,audio_conditionings: list | None = None,@@ -117,12 +119,14 @@ class KeyframeInterpolationPipeline:mask_generator = torch.Generator(device=self.device).manual_seed(int(seed) + 1)noiser = GaussianNoiser(generator=generator)stepper = EulerDiffusionStep()+ audio_cfg_guidance_scale = cfg_guidance_scale if audio_cfg_guidance_scale is None else audio_cfg_guidance_scale+ guider_cls = CFGGuiderif apg_switch:- cfg_guider = LtxAPGGuider(cfg_guidance_scale)+ guider_cls = LtxAPGGuiderelif cfg_star_switch:- cfg_guider = CFGStarRescalingGuider(cfg_guidance_scale)- else:- cfg_guider = CFGGuider(cfg_guidance_scale)+ guider_cls = CFGStarRescalingGuider+ video_cfg_guider = guider_cls(cfg_guidance_scale)+ audio_cfg_guider = guider_cls(audio_cfg_guidance_scale)dtype = torch.bfloat16text_encoder = self.stage_1_model_ledger.text_encoder()@@ -185,17 +189,19 @@ class KeyframeInterpolationPipeline:audio_state=audio_state,stepper=stepper,denoise_fn=guider_denoising_func(- cfg_guider,+ video_cfg_guider,+ audio_cfg_guider,v_context_p,v_context_n,a_context_p,a_context_n,transformer=transformer, # noqa: F821alt_guidance_scale=alt_guidance_scale,- slg_switch=slg_switch,- slg_layers=slg_layers,- slg_start=slg_start,- slg_end=slg_end,+ alt_scale=alt_scale,+ perturbation_switch=perturbation_switch,+ perturbation_layers=perturbation_layers,+ perturbation_start=perturbation_start,+ perturbation_end=perturbation_end,),mask_context=mask_context,interrupt_check=interrupt_check,diff --git a/models/ltx2/ltx_pipelines/ti2vid_one_stage.py b/models/ltx2/ltx_pipelines/ti2vid_one_stage.pyindex f54734e..7acd635 100644--- a/models/ltx2/ltx_pipelines/ti2vid_one_stage.py+++ b/models/ltx2/ltx_pipelines/ti2vid_one_stage.py@@ -82,13 +82,15 @@ class TI2VidOneStagePipeline:num_inference_steps: int,cfg_guidance_scale: float,images: list[tuple[str, int, float]],+ audio_cfg_guidance_scale: float | None = None,cfg_star_switch: int = 0,apg_switch: int = 0,- slg_switch: int = 0,- slg_layers: list[int] | None = None,- slg_start: float = 0.0,- slg_end: float = 1.0,+ perturbation_switch: int = 0,+ perturbation_layers: list[int] | None = None,+ perturbation_start: float = 0.0,+ perturbation_end: float = 1.0,alt_guidance_scale: float = 1.0,+ alt_scale: float = 0.0,enhance_prompt: bool = False,audio_conditionings: list | None = None,callback: Callable[..., None] | None = None,@@ -128,12 +130,14 @@ class TI2VidOneStagePipeline:self_refiner_certain_percentage,channel_dim=-1,)+ audio_cfg_guidance_scale = cfg_guidance_scale if audio_cfg_guidance_scale is None else audio_cfg_guidance_scale+ guider_cls = CFGGuiderif apg_switch:- cfg_guider = LtxAPGGuider(cfg_guidance_scale)+ guider_cls = LtxAPGGuiderelif cfg_star_switch:- cfg_guider = CFGStarRescalingGuider(cfg_guidance_scale)- else:- cfg_guider = CFGGuider(cfg_guidance_scale)+ guider_cls = CFGStarRescalingGuider+ video_cfg_guider = guider_cls(cfg_guidance_scale)+ audio_cfg_guider = guider_cls(audio_cfg_guidance_scale)dtype = torch.bfloat16text_encoder = self.model_ledger.text_encoder()@@ -196,17 +200,19 @@ class TI2VidOneStagePipeline:audio_state=audio_state,stepper=stepper,denoise_fn=guider_denoising_func(- cfg_guider,+ video_cfg_guider,+ audio_cfg_guider,v_context_p,v_context_n,a_context_p,a_context_n,transformer=transformer, # noqa: F821alt_guidance_scale=alt_guidance_scale,- slg_switch=slg_switch,- slg_layers=slg_layers,- slg_start=slg_start,- slg_end=slg_end,+ alt_scale=alt_scale,+ perturbation_switch=perturbation_switch,+ perturbation_layers=perturbation_layers,+ perturbation_start=perturbation_start,+ perturbation_end=perturbation_end,),mask_context=mask_context,interrupt_check=interrupt_check,diff --git a/models/ltx2/ltx_pipelines/ti2vid_two_stages.py b/models/ltx2/ltx_pipelines/ti2vid_two_stages.pyindex 81e7f6e..67810f6 100644--- a/models/ltx2/ltx_pipelines/ti2vid_two_stages.py+++ b/models/ltx2/ltx_pipelines/ti2vid_two_stages.py@@ -37,6 +37,7 @@ from .utils.helpers import (prepare_mask_injection,simple_denoising_func,video_conditionings_by_keyframe,+ video_conditionings_by_reference_latent,)from .utils.media_io import encode_videofrom .utils.types import PipelineComponents@@ -129,16 +130,19 @@ class TI2VidTwoStagesPipeline:num_inference_steps: int,cfg_guidance_scale: float,images: list[tuple[str, int, float]],+ audio_cfg_guidance_scale: float | None = None,cfg_star_switch: int = 0,apg_switch: int = 0,- slg_switch: int = 0,- slg_layers: list[int] | None = None,- slg_start: float = 0.0,- slg_end: float = 1.0,+ perturbation_switch: int = 0,+ perturbation_layers: list[int] | None = None,+ perturbation_start: float = 0.0,+ perturbation_end: float = 1.0,alt_guidance_scale: float = 1.0,+ alt_scale: float = 0.0,guiding_images: list[tuple[str, int, float]] | None = None,images_stage2: list[tuple[str, int, float]] | None = None,video_conditioning: list[tuple[str, float]] | None = None,+ video_conditioning_downscale_factor: int = 1,latent_conditioning_stage2: torch.Tensor | None = None,tiling_config: TilingConfig | None = None,enhance_prompt: bool = False,@@ -199,12 +203,14 @@ class TI2VidTwoStagesPipeline:self_refiner_certain_percentage,channel_dim=-1,)+ audio_cfg_guidance_scale = cfg_guidance_scale if audio_cfg_guidance_scale is None else audio_cfg_guidance_scale+ guider_cls = CFGGuiderif apg_switch:- cfg_guider = LtxAPGGuider(cfg_guidance_scale)+ guider_cls = LtxAPGGuiderelif cfg_star_switch:- cfg_guider = CFGStarRescalingGuider(cfg_guidance_scale)- else:- cfg_guider = CFGGuider(cfg_guidance_scale)+ guider_cls = CFGStarRescalingGuider+ video_cfg_guider = guider_cls(cfg_guidance_scale)+ audio_cfg_guider = guider_cls(audio_cfg_guidance_scale)dtype = torch.bfloat16text_encoder = self._get_stage_model(1, "text_encoder")@@ -271,17 +277,19 @@ class TI2VidTwoStagesPipeline:audio_state=audio_state,stepper=stepper,denoise_fn=guider_denoising_func(- cfg_guider,+ video_cfg_guider,+ audio_cfg_guider,v_context_p,v_context_n,a_context_p,a_context_n,transformer=transformer, # noqa: F821alt_guidance_scale=alt_guidance_scale,- slg_switch=slg_switch,- slg_layers=slg_layers,- slg_start=slg_start,- slg_end=slg_end,+ alt_scale=alt_scale,+ perturbation_switch=perturbation_switch,+ perturbation_layers=perturbation_layers,+ perturbation_start=perturbation_start,+ perturbation_end=perturbation_end,),mask_context=mask_context,interrupt_check=interrupt_check,@@ -321,16 +329,29 @@ class TI2VidTwoStagesPipeline:tiling_config=tiling_config,)if video_conditioning:- stage_1_conditionings += video_conditionings_by_keyframe(- video_conditioning=video_conditioning,- height=stage_1_output_shape.height,- width=stage_1_output_shape.width,- num_frames=num_frames,- video_encoder=video_encoder,- dtype=dtype,- device=self.device,- tiling_config=tiling_config,- )+ if int(video_conditioning_downscale_factor or 1) > 1:+ stage_1_conditionings += video_conditionings_by_reference_latent(+ video_conditioning=video_conditioning,+ height=stage_1_output_shape.height,+ width=stage_1_output_shape.width,+ num_frames=num_frames,+ video_encoder=video_encoder,+ dtype=dtype,+ device=self.device,+ downscale_factor=video_conditioning_downscale_factor,+ tiling_config=tiling_config,+ )+ else:+ stage_1_conditionings += video_conditionings_by_keyframe(+ video_conditioning=video_conditioning,+ height=stage_1_output_shape.height,+ width=stage_1_output_shape.width,+ num_frames=num_frames,+ video_encoder=video_encoder,+ dtype=dtype,+ device=self.device,+ tiling_config=tiling_config,+ )mask_context = prepare_mask_injection(masking_source=masking_source,masking_strength=masking_strength,diff --git a/models/ltx2/ltx_pipelines/utils/helpers.py b/models/ltx2/ltx_pipelines/utils/helpers.pyindex 8c62fd8..d66f4f5 100644--- a/models/ltx2/ltx_pipelines/utils/helpers.py+++ b/models/ltx2/ltx_pipelines/utils/helpers.py@@ -17,6 +17,7 @@ from ...ltx_core.conditioning import (ConditioningItem,VideoConditionByKeyframeIndex,VideoConditionByLatentIndex,+ VideoConditionByReferenceLatent,)from ...ltx_core.guidance.perturbations import (BatchedPerturbationConfig,@@ -163,6 +164,52 @@ def video_conditionings_by_keyframe(return conditionings+def video_conditionings_by_reference_latent(+ video_conditioning: list[tuple],+ height: int,+ width: int,+ num_frames: int,+ video_encoder: VideoEncoder,+ dtype: torch.dtype,+ device: torch.device,+ downscale_factor: int = 1,+ tiling_config: TilingConfig | None = None,+) -> list[ConditioningItem]:+ scale = max(1, int(downscale_factor))+ if scale > 1 and (height % scale != 0 or width % scale != 0):+ raise ValueError(f"Output dimensions ({height}x{width}) must be divisible by reference downscale factor {scale}.")++ ref_height = height // scale+ ref_width = width // scale+ conditionings = []+ for entry in video_conditioning:+ if len(entry) == 2:+ video_path, strength = entry+ frame_idx = 0+ elif len(entry) == 3:+ video_path, frame_idx, strength = entry+ else:+ raise ValueError("Video conditioning entries must be (video, strength) or (video, frame_idx, strength).")+ video = load_video_conditioning(+ video_path=video_path,+ height=ref_height,+ width=ref_width,+ frame_cap=num_frames,+ dtype=dtype,+ device=device,+ )+ encoded_video = vae_encode_video(video, video_encoder, tiling_config)+ conditionings.append(+ VideoConditionByReferenceLatent(+ latent=encoded_video,+ frame_idx=frame_idx,+ strength=strength,+ downscale_factor=scale,+ )+ )+ return conditionings++def latent_conditionings_by_latent_sequence(latents: torch.Tensor,strength: float = 1.0,@@ -837,26 +884,47 @@ def _cross_attn_perturbations(batch_size: int) -> BatchedPerturbationConfig:return BatchedPerturbationConfig(perts)-def _normalize_slg_layers(slg_layers) -> list[int] | None:- if slg_layers is None:+PERTURBATION_perturbation = 1+PERTURBATION_SKIP_SELF_ATTENTION = 2+++def _normalize_perturbation_layers(perturbation_layers) -> list[int] | None:+ if perturbation_layers is None:return None- if isinstance(slg_layers, (list, tuple)):- return [int(layer) for layer in slg_layers if str(layer).strip() != ""]- if isinstance(slg_layers, (int, float)):- return [int(slg_layers)]- if isinstance(slg_layers, str):- return [int(layer.strip()) for layer in slg_layers.split(",") if layer.strip()]+ if isinstance(perturbation_layers, (list, tuple)):+ return [int(layer) for layer in perturbation_layers if str(layer).strip() != ""]+ if isinstance(perturbation_layers, (int, float)):+ return [int(perturbation_layers)]+ if isinstance(perturbation_layers, str):+ return [int(layer.strip()) for layer in perturbation_layers.split(",") if layer.strip()]return None-def _slg_perturbations(batch_size: int, slg_layers: list[int]) -> BatchedPerturbationConfig:+def _legacy_perturbation_layer_configs(+ batch_size: int, perturbation_layers: list[int] | None+) -> BatchedPerturbationConfig:+ perts = [+ PerturbationConfig(+ [+ Perturbation(PerturbationType.SKIP_VIDEO_SELF_ATTN, perturbation_layers),+ Perturbation(PerturbationType.SKIP_AUDIO_SELF_ATTN, perturbation_layers),+ Perturbation(PerturbationType.SKIP_A2V_CROSS_ATTN, perturbation_layers),+ Perturbation(PerturbationType.SKIP_V2A_CROSS_ATTN, perturbation_layers),+ ]+ )+ for _ in range(batch_size)+ ]+ return BatchedPerturbationConfig(perts)+++def _self_attn_perturbation_configs(+ batch_size: int, perturbation_layers: list[int] | None+) -> BatchedPerturbationConfig:perts = [PerturbationConfig([- Perturbation(PerturbationType.SKIP_VIDEO_SELF_ATTN, slg_layers),- Perturbation(PerturbationType.SKIP_AUDIO_SELF_ATTN, slg_layers),- Perturbation(PerturbationType.SKIP_A2V_CROSS_ATTN, slg_layers),- Perturbation(PerturbationType.SKIP_V2A_CROSS_ATTN, slg_layers),+ Perturbation(PerturbationType.SKIP_VIDEO_SELF_ATTN, perturbation_layers),+ Perturbation(PerturbationType.SKIP_AUDIO_SELF_ATTN, perturbation_layers),])for _ in range(batch_size)@@ -864,6 +932,27 @@ def _slg_perturbations(batch_size: int, slg_layers: list[int]) -> BatchedPerturbreturn BatchedPerturbationConfig(perts)+def _perturbation_active(+ step_index: int,+ sigmas: torch.Tensor,+ perturbation_start: float,+ perturbation_end: float,+) -> bool:+ total_steps = max(len(sigmas) - 1, 1)+ start_step = int(perturbation_start * total_steps)+ end_step = int(perturbation_end * total_steps)+ return start_step <= step_index < end_step+++def _rescale_prediction(cond: torch.Tensor | None, pred: torch.Tensor | None, rescale_scale: float) -> torch.Tensor | None:+ if cond is None or pred is None or math.isclose(rescale_scale, 0.0):+ return pred+ pred_std = pred.std().clamp_min(1e-6)+ factor = cond.std() / pred_std+ factor = rescale_scale * factor + (1.0 - rescale_scale)+ return pred * factor++def timesteps_from_mask(denoise_mask: torch.Tensor, sigma: float | torch.Tensor, positions: torch.Tensor | None = None) -> tuple[torch.Tensor, torch.Tensor | None]:@@ -965,18 +1054,23 @@ def simple_denoising_func(def guider_denoising_func(- guider: GuiderProtocol,+ video_guider: GuiderProtocol,+ audio_guider: GuiderProtocol,v_context_p: torch.Tensor,v_context_n: torch.Tensor,a_context_p: torch.Tensor,a_context_n: torch.Tensor,transformer: X0Model,alt_guidance_scale: float = 1.0,- slg_switch: int = 0,- slg_layers: list[int] | None = None,- slg_start: float = 0.0,- slg_end: float = 1.0,+ alt_scale: float = 0.0,+ perturbation_switch: int = 0,+ perturbation_layers: list[int] | None = None,+ perturbation_start: float = 0.0,+ perturbation_end: float = 1.0,) -> DenoisingFunc:+ perturb_all_layers = perturbation_layers is None+ perturbation_layers_norm = _normalize_perturbation_layers(perturbation_layers)+def guider_denoising_step(video_state: LatentState, audio_state: LatentState, sigmas: torch.Tensor, step_index: int) -> tuple[torch.Tensor, torch.Tensor]:@@ -986,38 +1080,48 @@ def guider_denoising_func(if transformer is not None:offload.set_step_no_for_lora(transformer, step_index)- use_cfg = guider.enabled()+ use_video_cfg = video_guider.enabled()+ use_audio_cfg = audio_guider.enabled()+ use_cfg = use_video_cfg or use_audio_cfguse_alt = not math.isclose(alt_guidance_scale, 1.0)- if use_cfg or use_alt:+ use_perturbation = _perturbation_active(step_index, sigmas, perturbation_start, perturbation_end)+ has_perturbation_layers = perturb_all_layers or bool(perturbation_layers_norm)+ use_legacy_perturbation = (+ perturbation_switch == PERTURBATION_perturbation and use_cfg and use_perturbation and has_perturbation_layers+ )+ use_stg = (+ perturbation_switch == PERTURBATION_SKIP_SELF_ATTENTION and use_perturbation and has_perturbation_layers+ )+ selected_layers = None if perturb_all_layers else perturbation_layers_norm+ if use_cfg or use_alt or use_stg:+ batch_size = _get_batch_size(video_state, audio_state)video_list = [pos_video]audio_list = [pos_audio]perturbations: list[BatchedPerturbationConfig | None] = [None]neg_index = None+ stg_index = Nonealt_index = None+if use_cfg:- neg_video = modality_from_latent_state(video_state, v_context_n, sigma)- neg_audio = modality_from_latent_state(audio_state, a_context_n, sigma)+ neg_video_context = v_context_n if use_video_cfg else v_context_p+ neg_audio_context = a_context_n if use_audio_cfg else a_context_pneg_index = len(video_list)- video_list.append(neg_video)- audio_list.append(neg_audio)- batch_size = _get_batch_size(video_state, audio_state)- slg_layers_norm = _normalize_slg_layers(slg_layers)- use_slg = False- if slg_switch and slg_layers_norm:- total_steps = max(len(sigmas) - 1, 1)- start_step = int(slg_start * total_steps)- end_step = int(slg_end * total_steps)- use_slg = start_step <= step_index < end_step+ video_list.append(modality_from_latent_state(video_state, neg_video_context, sigma))+ audio_list.append(modality_from_latent_state(audio_state, neg_audio_context, sigma))perturbations.append(- _slg_perturbations(batch_size, slg_layers_norm) if use_slg else None+ _legacy_perturbation_layer_configs(batch_size, selected_layers) if use_legacy_perturbation else None)++ if use_stg:+ stg_index = len(video_list)+ video_list.append(modality_from_latent_state(video_state, v_context_p, sigma))+ audio_list.append(modality_from_latent_state(audio_state, a_context_p, sigma))+ perturbations.append(_self_attn_perturbation_configs(batch_size, selected_layers))+if use_alt:- alt_video = modality_from_latent_state(video_state, v_context_p, sigma)- alt_audio = modality_from_latent_state(audio_state, a_context_p, sigma)alt_index = len(video_list)- video_list.append(alt_video)- audio_list.append(alt_audio)- batch_size = _get_batch_size(video_state, audio_state)+ video_list.append(modality_from_latent_state(video_state, v_context_p, sigma))+ audio_list.append(modality_from_latent_state(audio_state, a_context_p, sigma))perturbations.append(_cross_attn_perturbations(batch_size))denoised_video_list, denoised_audio_list = transformer(@@ -1029,18 +1133,28 @@ def guider_denoising_func(return None, Nonepos_denoised_video = denoised_video_list[0]pos_denoised_audio = denoised_audio_list[0]- neg_denoised_video = denoised_video_list[neg_index] if neg_index is not None else None- neg_denoised_audio = denoised_audio_list[neg_index] if neg_index is not None else Noneif pos_denoised_video is None and pos_denoised_audio is None:return None, Nonedenoised_video = pos_denoised_videodenoised_audio = pos_denoised_audio+if use_cfg and neg_index is not None:- if denoised_video is not None and neg_denoised_video is not None:- denoised_video = denoised_video + guider.delta(denoised_video, neg_denoised_video)- if denoised_audio is not None and neg_denoised_audio is not None:- denoised_audio = denoised_audio + guider.delta(denoised_audio, neg_denoised_audio)+ neg_denoised_video = denoised_video_list[neg_index]+ neg_denoised_audio = denoised_audio_list[neg_index]+ if denoised_video is not None and neg_denoised_video is not None and use_video_cfg:+ denoised_video = denoised_video + video_guider.delta(pos_denoised_video, neg_denoised_video)+ if denoised_audio is not None and neg_denoised_audio is not None and use_audio_cfg:+ denoised_audio = denoised_audio + audio_guider.delta(pos_denoised_audio, neg_denoised_audio)++ if use_stg and stg_index is not None:+ stg_denoised_video = denoised_video_list[stg_index]+ stg_denoised_audio = denoised_audio_list[stg_index]+ if denoised_video is not None and stg_denoised_video is not None:+ denoised_video = denoised_video + (pos_denoised_video - stg_denoised_video)+ if denoised_audio is not None and stg_denoised_audio is not None:+ denoised_audio = denoised_audio + (pos_denoised_audio - stg_denoised_audio)+if use_alt and alt_index is not None:alt_denoised_video = denoised_video_list[alt_index]alt_denoised_audio = denoised_audio_list[alt_index]@@ -1052,11 +1166,15 @@ def guider_denoising_func(denoised_audio = denoised_audio + (alt_guidance_scale - 1.0) * (pos_denoised_audio - alt_denoised_audio)- neg_video = neg_audio = neg_denoised_video = neg_denoised_audio = Noneelse:denoised_video, denoised_audio = transformer(video=pos_video, audio=pos_audio, perturbations=None)if denoised_video is None and denoised_audio is None:return None, None+ pos_denoised_video = denoised_video+ pos_denoised_audio = denoised_audio++ denoised_video = _rescale_prediction(pos_denoised_video, denoised_video, alt_scale)+ denoised_audio = _rescale_prediction(pos_denoised_audio, denoised_audio, alt_scale)pos_video = pos_audio = Nonereturn denoised_video, denoised_audiodiff --git a/models/wan/any2video.py b/models/wan/any2video.pyindex 0692f74..65e11d0 100644--- a/models/wan/any2video.py+++ b/models/wan/any2video.py@@ -443,9 +443,9 @@ class WanAny2V:enable_RIFLEx = None,VAE_tile_size = 0,joint_pass = False,- slg_layers = None,- slg_start = 0.0,- slg_end = 1.0,+ perturbation_layers = None,+ perturbation_start = 0.0,+ perturbation_end = 1.0,cfg_star_switch = True,cfg_zero_step = 5,audio_scale=None,@@ -664,14 +664,14 @@ class WanAny2V:img_end_frame = image_end.unsqueeze(1).to(self.device)clip_image_start, clip_image_end = image_start, image_end- if any_end_frame:+ remaining_frames = frame_num - control_pre_frames_count+ if any_end_frame and not svi_pro:enc= torch.concat([control_video,torch.zeros( (3, frame_num-control_pre_frames_count-1, height, width), device=self.device, dtype= self.VAE_dtype),img_end_frame,], dim=1).to(self.device)else:- remaining_frames = frame_num - control_pre_frames_countif svi_pro or svi_mode and svi_ref_pad_num != 0:use_extended_overlapped_latents = Falseif input_ref_images is None or len(input_ref_images)==0:@@ -694,6 +694,8 @@ class WanAny2V:lat_y = torch.concat([image_ref_latents, pad_latents], dim=1).to(self.device)else:lat_y = torch.concat([image_ref_latents, overlapped_latents.squeeze(0), pad_latents], dim=1).to(self.device)+ if any_end_frame:+ lat_y[:, -1:] = self.vae.encode([img_end_frame], VAE_tile_size)[0][:, -1:]image_ref_latents = Noneelse:svi_ref_pad_num = remaining_frames if svi_ref_pad_num == -1 else min(svi_ref_pad_num, remaining_frames)@@ -711,7 +713,7 @@ class WanAny2V:msk = torch.ones(1, frame_num + ref_images_count * 4, lat_h, lat_w, device=self.device)if any_end_frame:- msk[:, control_pre_frames_count: -1] = 0+ msk[:, (1 if svi_mode else control_pre_frames_count):-1] = 0if add_frames_for_end_image:msk = torch.concat([ torch.repeat_interleave(msk[:, 0:1], repeats=4, dim=1), msk[:, 1:-1], torch.repeat_interleave(msk[:, -1:], repeats=4, dim=1) ], dim=1)else:@@ -1191,7 +1193,7 @@ class WanAny2V:timestep[:source_latents.shape[2]] = 0kwargs.update({"t": timestep, "current_step_no": i, "real_step_no": start_step_no + i })- kwargs["slg_layers"] = slg_layers if int(slg_start * sampling_steps) <= i < int(slg_end * sampling_steps) else None+ kwargs["perturbation_layers"] = perturbation_layers if int(perturbation_start * sampling_steps) <= i < int(perturbation_end * sampling_steps) else Noneif denoising_strength < 1 and i <= injection_denoising_step:sigma = t / 1000diff --git a/models/wan/df_handler.py b/models/wan/df_handler.pyindex 084f232..279b5f7 100644--- a/models/wan/df_handler.py+++ b/models/wan/df_handler.py@@ -30,7 +30,7 @@ class family_handler():extra_model_def["frames_steps"] = 20extra_model_def["latent_size"] = 4extra_model_def["sliding_window"] = True- extra_model_def["skip_layer_guidance"] = True+ extra_model_def["perturbation"] = Trueextra_model_def["tea_cache"] = Trueextra_model_def["guidance_max_phases"] = 1extra_model_def["flow_shift"] = Truediff --git a/models/wan/diffusion_forcing.py b/models/wan/diffusion_forcing.pyindex 08dcf88..9b7ac0b 100644--- a/models/wan/diffusion_forcing.py+++ b/models/wan/diffusion_forcing.py@@ -223,9 +223,9 @@ class DTT2V:fps: int = 24,VAE_tile_size = 0,joint_pass = False,- slg_layers = None,- slg_start = 0.0,- slg_end = 1.0,+ perturbation_layers = None,+ perturbation_start = 0.0,+ perturbation_end = 1.0,callback = None,loras_slists = None,**bbargs@@ -351,7 +351,7 @@ class DTT2V:kwrags.update(i2v_extra_kwrags)for i, timestep_i in enumerate(tqdm(step_matrix)):- kwrags["slg_layers"] = slg_layers if int(slg_start * updated_num_steps) <= i < int(slg_end * updated_num_steps) else None+ kwrags["perturbation_layers"] = perturbation_layers if int(perturbation_start * updated_num_steps) <= i < int(perturbation_end * updated_num_steps) else Noneoffload.set_step_no_for_lora(self.model, i)update_mask_i = step_update_mask[i]diff --git a/models/wan/modules/model.py b/models/wan/modules/model.pyindex 52adf0f..eae1f5b 100644--- a/models/wan/modules/model.py+++ b/models/wan/modules/model.py@@ -1446,7 +1446,7 @@ class WanModel(ModelMixin, ConfigMixin):real_step_no = 0,x_id= 0,max_steps = 0,- slg_layers=None,+ perturbation_layers=None,callback = None,cam_emb: torch.Tensor = None,fps = None,@@ -1798,7 +1798,7 @@ class WanModel(ModelMixin, ConfigMixin):if not standin_cache_enabled: get_cache("standin").clear()standin_x = block(standin_x, context = None, grid_sizes = None, e= standin_e0, freqs = standin_freqs, standin_phase = 1)- if slg_layers is not None and block_idx in slg_layers:+ if perturbation_layers is not None and block_idx in perturbation_layers:if x_id != 0 or not x_should_calc[0]:continuex_list[0] = block(x_list[0], context = context_list[0], audio_scale= audio_scale_list[0], e= e0, **kwargs)diff --git a/models/wan/ovi/modules/fusion.py b/models/wan/ovi/modules/fusion.pyindex 561ee80..0549a49 100644--- a/models/wan/ovi/modules/fusion.py+++ b/models/wan/ovi/modules/fusion.py@@ -294,7 +294,7 @@ class FusionModel(nn.Module):clip_fea_audio=None,y=None,first_frame_is_clean=False,- computed_slg_layers=None,+ computed_perturbation_layers=None,callback = None,pipeline= None,x_id_list = 0,@@ -333,7 +333,7 @@ class FusionModel(nn.Module):for x_id, one_vid, one_vid_e, one_audio, one_audio_e, one_kwargs in zip(x_id_list, vid_list, vid_e_list, audio_list, audio_e_list, kwargs_list):if pipeline._interrupt:return None, None- if x_id == 1 and computed_slg_layers is not None and i in computed_slg_layers:+ if x_id == 1 and computed_perturbation_layers is not None and i in computed_perturbation_layers:continue# one_vid[...], one_audio[...] = self.single_fusion_block_forward(a, b = self.single_fusion_block_forward(diff --git a/models/wan/ovi_fusion_engine.py b/models/wan/ovi_fusion_engine.pyindex 437c4f1..b8bd334 100644--- a/models/wan/ovi_fusion_engine.py+++ b/models/wan/ovi_fusion_engine.py@@ -142,9 +142,9 @@ class OviFusionEngine:shift=5.0,guide_scale=5.0,audio_cfg_scalecale=4.0,- slg_layers=[11],- slg_start = 0.0,- slg_end = 1.0,+ perturbation_layers=[11],+ perturbation_start = 0.0,+ perturbation_end = 1.0,n_prompt="",audio_negative_prompt="",loras_slists = None,@@ -160,13 +160,13 @@ class OviFusionEngine:if len(audio_negative_prompt) == 0:audio_negative_prompt= "robotic, muffled, echo, distorted" # Artifacts to avoid in audio- slg_layer = None- if isinstance(slg_layers, (list, tuple)) and slg_layers:- slg_layer = int(slg_layers[0])- elif isinstance(slg_layers, (int, float)):- slg_layer = int(slg_layers)- if slg_layer is None:- slg_layer = 11+ perturbation_layer = None+ if isinstance(perturbation_layers, (list, tuple)) and perturbation_layers:+ perturbation_layer = int(perturbation_layers[0])+ elif isinstance(perturbation_layers, (int, float)):+ perturbation_layer = int(perturbation_layers)+ if perturbation_layer is None:+ perturbation_layer = 11video_frame_height_width=(height, width)@@ -262,7 +262,7 @@ class OviFusionEngine:offload.set_step_no_for_lora(self.model.video_model, i)if is_i2v:video_noise[:, :1] = latents_images- computed_slg_layers = slg_layers if int(slg_start * sampling_steps) <= i < int(slg_end * sampling_steps) else None+ computed_perturbation_layers = perturbation_layers if int(perturbation_start * sampling_steps) <= i < int(perturbation_end * sampling_steps) else Noneany_guidance = not (guide_scale == 1 and audio_cfg_scalecale ==1)if any_guidance and not joint_pass:@@ -279,7 +279,7 @@ class OviFusionEngine:audio_context= [text_embeddings_audio_neg],vid_context =[text_embeddings_video_neg],x_id_list =[1],- computed_slg_layers = computed_slg_layers,+ computed_perturbation_layers = computed_perturbation_layers,**kwargs)if pred_vid_neg is None:@@ -288,7 +288,7 @@ class OviFusionEngine:vid, audio = self.model(audio_context= [text_embeddings_audio_pos, text_embeddings_audio_neg],vid_context= [text_embeddings_video_pos, text_embeddings_video_neg],- computed_slg_layers = computed_slg_layers,+ computed_perturbation_layers = computed_perturbation_layers,x_id_list =[0,1],**kwargs)diff --git a/models/wan/ovi_handler.py b/models/wan/ovi_handler.pyindex 039b911..337599b 100644--- a/models/wan/ovi_handler.py+++ b/models/wan/ovi_handler.py@@ -48,7 +48,7 @@ class family_handler:"sliding_window": False,"multiple_submodels": False,"guidance_max_phases": 1,- "skip_layer_guidance": True,+ "perturbation": True,"returns_audio": True,"sample_solvers": [("unipc", "unipc"),@@ -143,10 +143,10 @@ class family_handler:"guidance_scale": 4.0,"audio_guidance_scale": 3.0,"num_inference_steps": 50,- "slg_switch": 1,+ "perturbation_switch": 1,"sliding_window_size": 121,"video_length": 121,- "slg_layers" : [11]+ "perturbation_layers" : [11]})diff --git a/models/wan/wan_handler.py b/models/wan/wan_handler.pyindex 27e9c21..fe1c494 100644--- a/models/wan/wan_handler.py+++ b/models/wan/wan_handler.py@@ -284,7 +284,7 @@ class family_handler():"sliding_window" : base_model_type in ["multitalk", "infinitetalk", "t2v", "t2v_2_2", "fantasy", "animate", "lynx"] or test_class_i2v(base_model_type) or test_wan_5B(base_model_type) or vace_class, #"ti2v_2_2","multiple_submodels" : multiple_submodels,"guidance_max_phases" : 3,- "skip_layer_guidance" : True,+ "perturbation" : True,"flow_shift": True,"cfg_zero" : True,"cfg_star" : True,@@ -673,8 +673,10 @@ class family_handler():image_prompt_types_allowed = "TSVL"elif base_model_type in ["lucy_edit", "kiwi_edit"]:image_prompt_types_allowed = "TVL"- elif multitalk or base_model_type in ["fantasy", "steadydancer", "scail"] or svi2pro:+ elif multitalk or base_model_type in ["fantasy", "steadydancer", "scail"]:image_prompt_types_allowed = "SVL"+ elif svi2pro:+ image_prompt_types_allowed = "SEVL"elif i2v:image_prompt_types_allowed = "SEVL"else:diff --git a/preprocessing/matanyone/app.py b/preprocessing/matanyone/app.pyindex adcf26a..4c621b3 100644--- a/preprocessing/matanyone/app.py+++ b/preprocessing/matanyone/app.py@@ -44,7 +44,7 @@ GPU_process_was_running = Falsedef acquire_GPU(state):global GPU_process_was_runningGPU_process_was_running = any_GPU_process_running(state, "matanyone")- acquire_GPU_ressources(state, "matanyone", "MatAnyone", gr= gr)+ acquire_GPU_ressources(state, "matanyone", "MatAnyone", gr= gr)def release_GPU(state):release_GPU_ressources(state, "matanyone")if GPU_process_was_running:@@ -837,7 +837,7 @@ def display(tabs, tab_state, state, refresh_form_trigger, server_config, get_cur# download assets- gr.Markdown("<B>Mast Edition is provided by MatAnyone, VRAM optimizations & Extended Masks by DeepBeepMeep</B>")+ gr.Markdown("<B>Mask Edition is provided by MatAnyone v2, VRAM optimizations & Extended Masks by DeepBeepMeep</B>")gr.Markdown("If you have some trouble creating the perfect mask, be aware of these tips:")gr.Markdown("- Using the Matanyone Settings you can also define Negative Point Prompts to remove parts of the current selection.")gr.Markdown("- Sometime it is very hard to fit everything you want in a single mask, it may be much easier to combine multiple independent sub Masks before producing the Matting : each sub Mask is created by selecting an area of an image and by clicking the Add Mask button. Sub masks can then be enabled / disabled in the Matanyone settings.")diff --git a/preprocessing/matanyone/utils/model_signature.py b/preprocessing/matanyone/utils/model_signature.pynew file mode 100644index 0000000..4ba5565--- /dev/null+++ b/preprocessing/matanyone/utils/model_signature.py@@ -0,0 +1,20 @@+import os+++MATANYONE2_SIGNATURE_OFFSET = 61576+MATANYONE2_SIGNATURE_BYTES = bytes([155, 244, 100, 61, 100, 179, 194, 60, 92, 239, 124, 186, 254, 127, 168, 61])+++def delete_if_not_matanyone2_model(model_path):+ if model_path is None or not os.path.isfile(model_path):+ return False++ with open(model_path, "rb") as reader:+ reader.seek(MATANYONE2_SIGNATURE_OFFSET)+ actual = reader.read(len(MATANYONE2_SIGNATURE_BYTES))++ if actual == MATANYONE2_SIGNATURE_BYTES:+ return False++ os.remove(model_path)+ return Truediff --git a/profiles/qwen/Lightning Qwen 2512 - 4 Steps.json b/profiles/qwen/Lightning Qwen 2512 - 4 Steps.jsonnew file mode 100644index 0000000..7a164f0--- /dev/null+++ b/profiles/qwen/Lightning Qwen 2512 - 4 Steps.json@@ -0,0 +1,8 @@+{+ "activated_loras": [+ "https://huggingface.co/DeepBeepMeep/Qwen_image/resolve/main/loras_accelerators/Qwen-Image-2512-Lightning-4steps-V1.0-bf16.safetensors"+ ],+ "loras_multipliers": "1",+ "num_inference_steps": 4,+ "guidance_scale": 1+}diff --git a/profiles/qwen/Lightning Qwen 2512 - 8 Steps.json b/profiles/qwen/Lightning Qwen 2512 - 8 Steps.jsonnew file mode 100644index 0000000..b170674--- /dev/null+++ b/profiles/qwen/Lightning Qwen 2512 - 8 Steps.json@@ -0,0 +1,8 @@+{+ "activated_loras": [+ "https://huggingface.co/DeepBeepMeep/Qwen_image/resolve/main/loras_accelerators/Qwen-Image-2512-Lightning-8steps-V1.0-bf16.safetensors"+ ],+ "loras_multipliers": "1",+ "num_inference_steps": 8,+ "guidance_scale": 1+}diff --git a/wgp.py b/wgp.pyindex cee3b0e..4363440 100644--- a/wgp.py+++ b/wgp.py@@ -95,8 +95,8 @@ AUTOSAVE_TEMPLATE_PATH = AUTOSAVE_FILENAMECONFIG_FILENAME = "wgp_config.json"PROMPT_VARS_MAX = 10target_mmgp_version = "3.7.6"-WanGP_version = "10.98"-settings_version = 2.53+WanGP_version = "10.981"+settings_version = 2.55max_source_video_frames = 3000prompt_enhancer_image_caption_model, prompt_enhancer_image_caption_processor, prompt_enhancer_llm_model, prompt_enhancer_llm_tokenizer = None, None, None, Noneimage_names_list = ["image_start", "image_end", "image_refs"]@@ -2440,6 +2440,11 @@ _normalize_profile_defaults(server_config)_normalize_output_paths(server_config)lm_decoder_engine = server_config.get("lm_decoder_engine", "")+from preprocessing.matanyone.utils.model_signature import delete_if_not_matanyone2_model+mask_model_path = fl.locate_file("mask/model.safetensors", error_if_none=False)+if delete_if_not_matanyone2_model(mask_model_path):+ print("Removing old version of 'mask/model.safetensors'. MatAnyone2 will be downloaded next time you use it.")+# Deprecated modelsfor path in ["wan2.1_Vace_1.3B_preview_bf16.safetensors", "sky_reels2_diffusion_forcing_1.3B_bf16.safetensors","sky_reels2_diffusion_forcing_720p_14B_bf16.safetensors","sky_reels2_diffusion_forcing_720p_14B_quanto_int8.safetensors", "sky_reels2_diffusion_forcing_720p_14B_quanto_fp16_int8.safetensors", "wan2.1_image2video_480p_14B_bf16.safetensors", "wan2.1_image2video_480p_14B_quanto_int8.safetensors",@@ -2751,6 +2756,21 @@ def fix_settings(model_type, ui_defaults, min_settings_version = 0):if "lset_name" in ui_defaults: del ui_defaults["lset_name"]+ if settings_version < 2.54:+ renamed_settings = {+ "slg_switch": "perturbation_switch",+ "slg_layers": "perturbation_layers",+ "slg_start_perc": "perturbation_start_perc",+ "slg_end_perc": "perturbation_end_perc",+ }+ for old_name, new_name in renamed_settings.items():+ if old_name in ui_defaults:+ ui_defaults.setdefault(new_name, ui_defaults[old_name])+ del ui_defaults[old_name]++ if settings_version < 2.55:+ ui_defaults.setdefault("alt_scale", 0.0)+audio_prompt_type = ui_defaults.get("audio_prompt_type", None)if settings_version < 2.2:if not base_model_type in ["vace_1.3B","vace_14B", "sky_df_1.3B", "sky_df_14B", "ltxv_13B"]:@@ -4349,6 +4369,7 @@ def select_video(state, current_gallery_tab, input_file_list, file_selected, audvideo_guidance3_scale = configs.get("guidance3_scale", None)video_audio_guidance_scale = configs.get("audio_guidance_scale", None)video_alt_guidance_scale = configs.get("alt_guidance_scale", None)+ video_alt_scale = configs.get("alt_scale", None)video_temperature = configs.get("temperature", None)video_top_p = configs.get("top_p", None)video_top_k = configs.get("top_k", None)@@ -4500,6 +4521,10 @@ def select_video(state, current_gallery_tab, input_file_list, file_selected, audif alt_guidance_type is not None and video_alt_guidance_scale is not None:values += [video_alt_guidance_scale]labels += [alt_guidance_type]+ alt_scale_type = model_def.get("alt_scale", None)+ if alt_scale_type is not None and video_alt_scale is not None:+ values += [video_alt_scale]+ labels += [alt_scale_type]if model_def.get("flow_shift", False):values += [video_flow_shift]labels += ["Shift Scale"]@@ -5693,6 +5718,7 @@ def generate_video(guidance_phases,model_switch_phase,alt_guidance_scale,+ alt_scale,audio_guidance_scale,audio_scale,flow_shift,@@ -5753,10 +5779,10 @@ def generate_video(NAG_scale,NAG_tau,NAG_alpha,- slg_switch,- slg_layers,- slg_start_perc,- slg_end_perc,+ perturbation_switch,+ perturbation_layers,+ perturbation_start_perc,+ perturbation_end_perc,apg_switch,cfg_star_switch,cfg_zero_step,@@ -5882,8 +5908,8 @@ def generate_video(width, height = int(width) // block_size * block_size, int(height) // block_size * block_sizedefault_image_size = (height, width)- if slg_switch == 0:- slg_layers = None+ if perturbation_switch == 0:+ perturbation_layers = Noneoffload.shared_state["_attention"] = attndevice_mem_capacity = torch.cuda.get_device_properties(0).total_memory / 1048576@@ -6587,9 +6613,10 @@ def generate_video(enable_RIFLEx = enable_RIFLEx,VAE_tile_size = VAE_tile_size,joint_pass = joint_pass,- slg_layers = slg_layers,- slg_start = slg_start_perc/100,- slg_end = slg_end_perc/100,+ perturbation_switch = perturbation_switch,+ perturbation_layers = perturbation_layers,+ perturbation_start = perturbation_start_perc/100,+ perturbation_end = perturbation_end_perc/100,apg_switch = apg_switch,cfg_star_switch = cfg_star_switch,cfg_zero_step = cfg_zero_step,@@ -6605,6 +6632,7 @@ def generate_video(audio_context_lens= audio_context_lens,context_scale = context_scale,control_scale_alt = control_net_weight_alt,+ alt_scale = alt_scale,motion_amplitude = motion_amplitude,model_mode = model_mode,causal_block_size = 5,@@ -6647,7 +6675,8 @@ def generate_video(pause_seconds=pause_seconds,top_p=top_p,top_k=top_k,- set_progress_status=set_progress_status,+ set_progress_status=set_progress_status,+ loras_selected=loras_selected,)except Exception as e:if len(control_audio_tracks) > 0 or len(source_audio_tracks) > 0:@@ -8036,6 +8065,9 @@ def prepare_inputs_dict(target, inputs, model_type = None, model_filename = Noneif model_def.get("alt_guidance", None) is None:pop += ["alt_guidance_scale"]+ if model_def.get("alt_scale", None) is None:+ pop += ["alt_scale"]+if not (model_def.get("tea_cache", False) or model_def.get("mag_cache", False)) :pop += ["skip_steps_cache_type", "skip_steps_multiplier", "skip_steps_start_step_perc"]@@ -8059,8 +8091,8 @@ def prepare_inputs_dict(target, inputs, model_type = None, model_filename = Noneif model_def.get("no_negative_prompt", False) :pop += ["negative_prompt" ]- if not model_def.get("skip_layer_guidance", False):- pop += ["slg_switch", "slg_layers", "slg_start_perc", "slg_end_perc"]+ if not model_def.get("perturbation", False):+ pop += ["perturbation_switch", "perturbation_layers", "perturbation_start_perc", "perturbation_end_perc"]if not model_def.get("cfg_zero", False):pop += [ "cfg_zero_step" ]@@ -8646,6 +8678,7 @@ def save_inputs(guidance_phases,model_switch_phase,alt_guidance_scale,+ alt_scale,audio_guidance_scale,audio_scale,flow_shift,@@ -8706,10 +8739,10 @@ def save_inputs(NAG_scale,NAG_tau,NAG_alpha,- slg_switch,- slg_layers,- slg_start_perc,- slg_end_perc,+ perturbation_switch,+ perturbation_layers,+ perturbation_start_perc,+ perturbation_end_perc,apg_switch,cfg_star_switch,cfg_zero_step,@@ -10247,11 +10280,14 @@ def generate_video_tab(update_form = False, state_dict = None, ui_defaults = Nonany_audio_guidance = model_def.get("audio_guidance", False)any_embedded_guidance = model_def.get("embedded_guidance", False)alt_guidance_type = model_def.get("alt_guidance", None)- any_alt_guidance = alt_guidance_type is not None- with gr.Row(visible =any_embedded_guidance or any_audio_guidance or any_alt_guidance) as embedded_guidance_row:+ any_alt_guidance = alt_guidance_type is not None+ alt_scale_type = model_def.get("alt_scale", None)+ any_alt_scale = alt_scale_type is not None+ with gr.Row(visible =any_embedded_guidance or any_audio_guidance or any_alt_guidance or any_alt_scale) as embedded_guidance_row:audio_guidance_scale = gr.Slider(1.0, 20.0, value=ui_get("audio_guidance_scale"), step=0.5, label="Audio Guidance", visible= any_audio_guidance, show_reset_button= False )embedded_guidance_scale = gr.Slider(1.0, 20.0, value=ui_get("embedded_guidance_scale"), step=0.5, label="Embedded Guidance Scale", visible=any_embedded_guidance, show_reset_button= False )alt_guidance_scale = gr.Slider(1.0, 20.0, value=ui_get("alt_guidance_scale"), step=0.5, label= alt_guidance_type if any_alt_guidance else "" , visible=any_alt_guidance, show_reset_button= False )+ alt_scale = gr.Slider(0.0, 1.0, value=ui_get("alt_scale"), step=0.05, label= alt_scale_type if any_alt_scale else "" , visible=any_alt_scale, show_reset_button= False )with gr.Row(visible=model_def.get("pause_between_sentences", False)) as pause_row:pause_seconds = gr.Slider(minimum=0.0, maximum=2.0, value=ui_get("pause_seconds"), step=0.05, label="Pause between Multi Speakers sentences (seconds)", show_reset_button=False,)@@ -10430,39 +10466,38 @@ def generate_video_tab(update_form = False, state_dict = None, ui_defaults = Nongr.Markdown("<B>Add Custom Soundtrack to Video</B>")audio_source = gr.Audio(value= ui_defaults.get("audio_source", None), type="filepath", label="Soundtrack", show_download_button= True)- any_skip_layer_guidance = model_def.get("skip_layer_guidance", False)+ any_perturbation = model_def.get("perturbation", False)any_cfg_zero = model_def.get("cfg_zero", False)any_cfg_star = model_def.get("cfg_star", False)any_apg = model_def.get("adaptive_projected_guidance", False)any_motion_amplitude = model_def.get("motion_amplitude", False) and not image_outputsany_pnp = True # Enable PnP for all supported models (or restriction logic here)- with gr.Tab("Quality", visible = (vace and image_outputs or any_skip_layer_guidance or any_cfg_zero or any_cfg_star or any_apg or any_motion_amplitude or any_pnp) and not audio_only ) as quality_tab:- with gr.Column(visible = any_skip_layer_guidance ) as skip_layer_guidance_row:- gr.Markdown("<B>Skip Layer Guidance (improves video quality, requires guidance > 1)</B>")+ with gr.Tab("Quality", visible = (vace and image_outputs or any_perturbation or any_cfg_zero or any_cfg_star or any_apg or any_motion_amplitude or any_pnp) and not audio_only ) as quality_tab:+ with gr.Column(visible = any_perturbation ) as perturbation_row:+ gr.Markdown("<B>Perturbation (improves video quality, requires guidance > 1)</B>")+ perturbation_choices = model_def.get("perturbation_choices", [("OFF", 0), ("Skip Layer Guidance", 1)])+ perturbation_layers_max = model_def.get("perturbation_layers_max", 40)with gr.Row():- slg_switch = gr.Dropdown(- choices=[- ("OFF", 0),- ("ON", 1),- ],- value=ui_get("slg_switch"),+ perturbation_switch = gr.Dropdown(+ choices=perturbation_choices,+ value=ui_get("perturbation_switch"),visible=True,scale = 1,- label="Skip Layer guidance"+ label="Perturbation")- slg_layers = gr.Dropdown(+ perturbation_layers = gr.Dropdown(choices=[- (str(i), i ) for i in range(40)+ (str(i), i ) for i in range(perturbation_layers_max)],- value=ui_get("slg_layers"),+ value=ui_get("perturbation_layers"),multiselect= True,- label="Skip Layers",+ label="Perturbation Layers",scale= 3)with gr.Row():- slg_start_perc = gr.Slider(0, 100, value=ui_get("slg_start_perc"), step=1, label="Denoising Steps % start", show_reset_button= False)- slg_end_perc = gr.Slider(0, 100, value=ui_get("slg_end_perc"), step=1, label="Denoising Steps % end", show_reset_button= False)+ perturbation_start_perc = gr.Slider(0, 100, value=ui_get("perturbation_start_perc"), step=1, label="Denoising Steps % start", show_reset_button= False)+ perturbation_end_perc = gr.Slider(0, 100, value=ui_get("perturbation_end_perc"), step=1, label="Denoising Steps % end", show_reset_button= False)with gr.Column(visible= any_apg ) as apg_col:gr.Markdown("<B>Correct Progressive Color Saturation during long Video Generations")@@ -10806,7 +10841,7 @@ def generate_video_tab(update_form = False, state_dict = None, ui_defaults = Nonextra_inputs = prompt_vars + [wizard_prompt, wizard_variables_var, wizard_prompt_activated_var, video_prompt_column, image_prompt_column, image_prompt_type_group, image_prompt_type_radio, image_prompt_type_endcheckbox,prompt_column_advanced, prompt_column_wizard_vars, prompt_column_wizard, alt_prompt_row, lset_name, save_lset_prompt_drop, advanced_row, speed_tab, audio_tab, mmaudio_col, quality_tab,- sliding_window_tab, misc_tab, prompt_enhancer_row, inference_steps_row, skip_layer_guidance_row, audio_guide_row, custom_guide_row, RIFLEx_setting_col,+ sliding_window_tab, misc_tab, prompt_enhancer_row, inference_steps_row, perturbation_row, audio_guide_row, custom_guide_row, RIFLEx_setting_col,video_prompt_type_video_guide, video_prompt_type_video_guide_alt, video_prompt_type_video_mask, video_prompt_type_image_refs, video_prompt_type_video_custom_dropbox, video_prompt_type_video_custom_checkbox,apg_col, audio_prompt_type_sources, audio_prompt_type_remux, audio_prompt_type_remux_row, force_fps_col,video_guide_outpainting_col,video_guide_outpainting_top, video_guide_outpainting_bottom, video_guide_outpainting_left, video_guide_outpainting_right,