Skip to content

armgit / LTX-2

Public Imported from GitHub
0

Official Python inference and LoRA trainer package for the LTX-2 audio–video generative model.

Rename LipDub pipeline to Dub-It.

Alex10c99798cd40
diff --git a/README.md b/README.md
index b5d2dcc..cb66d69 100644
--- a/README.md
+++ b/README.md
@@ -63,7 +63,7 @@ For pipelines beyond the quickstart, download the relevant models from the [LTX-
**Temporal Upscaler** - Supported by the model and will be required for future pipeline implementations
* [`ltx-2.3-temporal-upscaler-x2-1.0.safetensors`](https://huggingface.co/Lightricks/LTX-2.3/blob/main/ltx-2.3-temporal-upscaler-x2-1.0.safetensors) - [Download](https://huggingface.co/Lightricks/LTX-2.3/resolve/main/ltx-2.3-temporal-upscaler-x2-1.0.safetensors)
-**Distilled LoRA** - Required for current two-stage pipeline implementations in this repository (except DistilledPipeline, ICLoraPipeline, and LipDubPipeline)
+**Distilled LoRA** - Required for current two-stage pipeline implementations in this repository (except DistilledPipeline, ICLoraPipeline, and DubItPipeline)
* [`ltx-2.3-22b-distilled-lora-384-1.1.safetensors`](https://huggingface.co/Lightricks/LTX-2.3/blob/main/ltx-2.3-22b-distilled-lora-384-1.1.safetensors) - [Download](https://huggingface.co/Lightricks/LTX-2.3/resolve/main/ltx-2.3-22b-distilled-lora-384-1.1.safetensors)
**Gemma Text Encoder** (download all assets from the repository)
@@ -82,7 +82,7 @@ For pipelines beyond the quickstart, download the relevant models from the [LTX-
* [`LTX-2-19b-LoRA-Camera-Control-Jib-Up`](https://huggingface.co/Lightricks/LTX-2-19b-LoRA-Camera-Control-Jib-Up) - [Download](https://huggingface.co/Lightricks/LTX-2-19b-LoRA-Camera-Control-Jib-Up/resolve/main/ltx-2-19b-lora-camera-control-jib-up.safetensors)
* [`LTX-2-19b-LoRA-Camera-Control-Static`](https://huggingface.co/Lightricks/LTX-2-19b-LoRA-Camera-Control-Static) - [Download](https://huggingface.co/Lightricks/LTX-2-19b-LoRA-Camera-Control-Static/resolve/main/ltx-2-19b-lora-camera-control-static.safetensors)
* [`LTX-2.3-22b-IC-LoRA-HDR`](https://huggingface.co/Lightricks/LTX-2.3-22b-IC-LoRA-HDR) - HDR IC-LoRA and pre-computed text embeddings for `HDRICLoraPipeline`
- * [`LTX-2.3-22b-IC-LoRA-LipDub`](https://huggingface.co/Lightricks/LTX-2.3-22b-IC-LoRA-LipDub) - [Download](https://huggingface.co/Lightricks/LTX-2.3-22b-IC-LoRA-LipDub/resolve/main/ltx-2.3-22b-ic-lora-lipdub-0.9.safetensors)
+ * [`LTX-2.3-22b-IC-LoRA-DubIt`](https://huggingface.co/Lightricks/LTX-2.3-22b-IC-LoRA-DubIt) - [Download](https://huggingface.co/Lightricks/LTX-2.3-22b-IC-LoRA-DubIt/resolve/main/ltx-2.3-22b-ic-lora-dubit-0.9.safetensors)
### Available Pipelines
@@ -95,7 +95,7 @@ For pipelines beyond the quickstart, download the relevant models from the [LTX-
* **[A2VidPipelineTwoStage](packages/ltx-pipelines/src/ltx_pipelines/a2vid_two_stage.py)** - Audio-to-video generation conditioned on an input audio file
* **[RetakePipeline](packages/ltx-pipelines/src/ltx_pipelines/retake.py)** - Regenerate a specific time region of an existing video
* **[HDRICLoraPipeline](packages/ltx-pipelines/src/ltx_pipelines/hdr_ic_lora.py)** - Video-to-video with HDR output (linear float frames via LogC3 inverse decode, suitable for EXR export and tonemapping)
-* **[LipDubPipeline](packages/ltx-pipelines/src/ltx_pipelines/lipdub.py)** - Lip dubbing, rephrasing, matching speaker identity (distilled model, single IC-LoRA, Two stages).
+* **[DubItPipeline](packages/ltx-pipelines/src/ltx_pipelines/dubit.py)** - Dub-It: rephrasing while matching speaker identity and lip movements (distilled model, single IC-LoRA, two stages).
### ⚡ Optimization Tips
diff --git a/packages/ltx-pipelines/CLAUDE.md b/packages/ltx-pipelines/CLAUDE.md
index 2aa573d..6b415b8 100644
--- a/packages/ltx-pipelines/CLAUDE.md
+++ b/packages/ltx-pipelines/CLAUDE.md
@@ -24,7 +24,7 @@ Inference pipelines for LTX-2 audio-video generation. Depends on `ltx-core` for
| `KeyframeInterpolationPipeline` | `keyframe_interpolation.py` | 2 | Full + distilled LoRA | Euler | Keyframe interpolation |
| `DistilledPipeline` | `distilled.py` | 2 | Distilled only | Euler | Fastest inference |
| `ICLoraPipeline` | `ic_lora.py` | 2 | Distilled only | Euler | Video-to-video with IC-LoRA control |
-| `LipDubPipeline` | `lipdub.py` | 2 | Distilled only | Euler | Lip dubbing with IC-LoRA + audio ref conditioning |
+| `DubItPipeline` | `dubit.py` | 2 | Distilled only | Euler | Dub-It with IC-LoRA + audio ref conditioning |
| `RetakePipeline` | `retake.py` | 1 | Full or distilled | Euler | Video region regeneration |
## Guidance
@@ -78,7 +78,7 @@ Guided denoisers batch all guidance passes into a **single transformer call**: s
- **HQ**: Res2s second-order sampler for **both** stages, latent-dependent sigma schedule, distilled LoRA on both stages with separate strengths.
- **A2Vid**: Audio frozen in both stages (`frozen=True, noise_scale=0.0`). Returns original audio (not VAE-decoded); no `AudioDecoder`.
- **IC-LoRA**: `VideoConditionByReferenceLatent`, `reference_downscale_factor` from LoRA metadata, `skip_stage_2`, attention mask downsampling. Stage 2 is LoRA-free and uses `combined_image_conditionings` (no IC-LoRA conditioning).
-- **LipDub**: Standalone pipeline; IC reference **video** helpers in `iclora_utils.py`, LipDub-only **audio** patchify/negative positions in `lipdub.py`. Appends frozen audio-reference tokens via `AudioConditionByReferenceLatent` (ltx-core), matching video token order (`[target | ref]`) while keeping reference RoPE positions negative (training-compatible). Single IC-LoRA on both stages; full IC-LoRA video conditioning at stage 1 and 2; stage-2 audio is frozen with S1 latent as initial state and uses S1-derived ref. Final audio decoded from stage 1 latent. The LipDub CLI does not expose `--conditioning-attention-mask`; use `ic_lora.py` if you need spatial IC attention masking.
+- **Dub-It**: Standalone pipeline; IC reference **video** helpers in `iclora_utils.py`, Dub-It-only **audio** patchify/negative positions in `dubit.py`. Appends frozen audio-reference tokens via `AudioConditionByReferenceLatent` (ltx-core), matching video token order (`[target | ref]`) while keeping reference RoPE positions negative (training-compatible). Single IC-LoRA on both stages; full IC-LoRA video conditioning at stage 1 and 2; stage-2 audio is frozen with S1 latent as initial state and uses S1-derived ref. Final audio decoded from stage 1 latent. The Dub-It CLI does not expose `--conditioning-attention-mask`; use `ic_lora.py` if you need spatial IC attention masking.
- **Keyframe**: Uses `image_conditionings_by_adding_guiding_latent` in both stages (all frames as keyframe guidance, no replacement) -- unlike TI2Vid which uses `combined_image_conditionings` (frame_idx=0 replaces, others guide).
- **Retake**: `TemporalRegionMask` for selective time-window regeneration. `regenerate_video`/`regenerate_audio` flags. Conditional distilled/full behavior.
- **Distilled**: Single `self.stage` reused for both stages (not `stage_1`/`stage_2`).
diff --git a/packages/ltx-pipelines/docs/installation.md b/packages/ltx-pipelines/docs/installation.md
index 7b7d8fc..232b00e 100644
--- a/packages/ltx-pipelines/docs/installation.md
+++ b/packages/ltx-pipelines/docs/installation.md
@@ -47,7 +47,7 @@ python -m ltx_pipelines.ti2vid_two_stages --help
- `ltx_pipelines.a2vid_two_stage` - Audio-to-video generation conditioned on an input audio. ([docs](pipelines.md#7-a2vidpipelinetwostage), [source](../src/ltx_pipelines/a2vid_two_stage.py))
- `ltx_pipelines.retake` - Regenerate a time region of an existing video. ([docs](pipelines.md#8-retakepipeline), [source](../src/ltx_pipelines/retake.py))
- `ltx_pipelines.hdr_ic_lora` - Video-to-video with HDR output (linear float via LogC3 inverse decode). ([docs](pipelines.md#9-hdriclorapipeline), [source](../src/ltx_pipelines/hdr_ic_lora.py))
-- `ltx_pipelines.lipdub` - Lip dubbing / re-voicing with IC-LoRA and audio reference conditioning. ([docs](pipelines.md#10-lipdubpipeline), [source](../src/ltx_pipelines/lipdub.py))
+- `ltx_pipelines.dubit` - Dub-It / re-voicing with IC-LoRA and audio reference conditioning. ([docs](pipelines.md#10-dubitpipeline), [source](../src/ltx_pipelines/dubit.py))
Use `--help` with any pipeline module to see all available options and parameters.
diff --git a/packages/ltx-pipelines/docs/pipeline-selection.md b/packages/ltx-pipelines/docs/pipeline-selection.md
index 52a7204..d6edefd 100644
--- a/packages/ltx-pipelines/docs/pipeline-selection.md
+++ b/packages/ltx-pipelines/docs/pipeline-selection.md
@@ -42,7 +42,7 @@ Do you need to condition on existing images/videos?
| [**A2VidPipelineTwoStage**](pipelines.md#7-a2vidpipelinetwostage) | 2 | ✅ | ✅ | Audio + Image | Audio-driven video generation |
| [**RetakePipeline**](pipelines.md#8-retakepipeline) | 1 | ✅ | ❌ | Source Video | Regenerating a time region of a video |
| [**HDRICLoraPipeline**](pipelines.md#9-hdriclorapipeline) | 2 | ❌ | ✅ | Video | HDR video-to-video (linear float output for EXR) |
-| [**LipDubPipeline**](pipelines.md#10-lipdubpipeline) | 2 | ✅ | ✅ | Video + Audio | Lip dubbing with audio ref conditioning |
+| [**DubItPipeline**](pipelines.md#10-dubitpipeline) | 2 | ✅ | ✅ | Video + Audio | Dub-It with audio ref conditioning |
| [**T2AOneStagePipeline**](pipelines.md#11-t2aonestagepipeline) | 1 | Audio only | ❌ | None (text) | Text-to-audio (audio-only output, no video) |
See [Available Pipelines](pipelines.md) for a full description of each.
diff --git a/packages/ltx-pipelines/docs/pipelines.md b/packages/ltx-pipelines/docs/pipelines.md
index b5b0f4b..d9882bf 100644
--- a/packages/ltx-pipelines/docs/pipelines.md
+++ b/packages/ltx-pipelines/docs/pipelines.md
@@ -124,15 +124,15 @@ Two-stage video-to-video on the distilled model with an HDR IC-LoRA. Decoded lat
---
-## 10. LipDubPipeline
+## 10. DubItPipeline
-**Best for:** Lip dubbing, rephrasing while keeping the same speaker identity and matching lip movements to new audio.
+**Best for:** Dub-It — rephrasing while keeping the same speaker identity and matching lip movements to new audio.
-**Source**: [`src/ltx_pipelines/lipdub.py`](../src/ltx_pipelines/lipdub.py)
+**Source**: [`src/ltx_pipelines/dubit.py`](../src/ltx_pipelines/dubit.py)
-Uses IC-LoRA on a **distilled** checkpoint with a **single** lip-dub IC-LoRA applied in **both** stages. The reference clip provides video and audio reference tokens whose VAE latents are appended to the target audio sequence as frozen reference tokens. The frame count and frame rate are derived from the reference video (frame count is silently snapped to the nearest `8k+1`), so the CLI does not accept `--num-frames` or `--frame-rate`. Required: `--reference-video`. Optional: `--reference-strength`. LoRA: [`Lightricks/LTX-2.3-22b-IC-LoRA-LipDub`](https://huggingface.co/Lightricks/LTX-2.3-22b-IC-LoRA-LipDub).
+Uses IC-LoRA on a **distilled** checkpoint with a **single** Dub-It IC-LoRA applied in **both** stages. The reference clip provides video and audio reference tokens whose VAE latents are appended to the target audio sequence as frozen reference tokens. The frame count and frame rate are derived from the reference video (frame count is silently snapped to the nearest `8k+1`), so the CLI does not accept `--num-frames` or `--frame-rate`. Required: `--reference-video`. Optional: `--reference-strength`. LoRA: [`Lightricks/LTX-2.3-22b-IC-LoRA-DubIt`](https://huggingface.co/Lightricks/LTX-2.3-22b-IC-LoRA-DubIt).
-**Note:** Requires a distilled model checkpoint and one lip-dub IC-LoRA (`--lora` exactly once).
+**Note:** Requires a distilled model checkpoint and one Dub-It IC-LoRA (`--lora` exactly once).
**Use when:** Dubbing, rephrasing with matched lips and speaker identity.
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/__init__.py b/packages/ltx-pipelines/src/ltx_pipelines/__init__.py
index 926f4b1..993237c 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/__init__.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/__init__.py
@@ -5,8 +5,8 @@ This package provides ready-to-use pipelines for video generation:
- T2AOneStagePipeline: Text-to-audio in a single stage (audio-only output)
- TI2VidTwoStagesPipeline: Two-stage generation with upsampling
- DistilledPipeline: Fast distilled two-stage generation
+- DubItPipeline: Dub-It with IC-LoRA and audio conditioning
- ICLoraPipeline: Image/video conditioning with distilled LoRA
-- LipDubPipeline: Lip dubbing with IC-LoRA and audio conditioning
- KeyframeInterpolationPipeline: Keyframe-based video interpolation
- RetakePipeline: Regenerate a time region (retake) of an existing video
For more detailed components and utilities, import from specific submodules
@@ -23,9 +23,9 @@ from typing import TYPE_CHECKING
if TYPE_CHECKING:
from ltx_pipelines.a2vid_two_stage import A2VidPipelineTwoStage
from ltx_pipelines.distilled import DistilledPipeline
+ from ltx_pipelines.dubit import DubItPipeline
from ltx_pipelines.ic_lora import ICLoraPipeline
from ltx_pipelines.keyframe_interpolation import KeyframeInterpolationPipeline
- from ltx_pipelines.lipdub import LipDubPipeline
from ltx_pipelines.retake import RetakePipeline
from ltx_pipelines.t2a_one_stage import T2AOneStagePipeline
from ltx_pipelines.ti2vid_one_stage import TI2VidOneStagePipeline
@@ -35,9 +35,9 @@ if TYPE_CHECKING:
_EXPORTS = {
"A2VidPipelineTwoStage": "ltx_pipelines.a2vid_two_stage",
"DistilledPipeline": "ltx_pipelines.distilled",
+ "DubItPipeline": "ltx_pipelines.dubit",
"ICLoraPipeline": "ltx_pipelines.ic_lora",
"KeyframeInterpolationPipeline": "ltx_pipelines.keyframe_interpolation",
- "LipDubPipeline": "ltx_pipelines.lipdub",
"RetakePipeline": "ltx_pipelines.retake",
"T2AOneStagePipeline": "ltx_pipelines.t2a_one_stage",
"TI2VidOneStagePipeline": "ltx_pipelines.ti2vid_one_stage",
@@ -47,9 +47,9 @@ _EXPORTS = {
__all__ = [
"A2VidPipelineTwoStage",
"DistilledPipeline",
+ "DubItPipeline",
"ICLoraPipeline",
"KeyframeInterpolationPipeline",
- "LipDubPipeline",
"RetakePipeline",
"T2AOneStagePipeline",
"TI2VidOneStagePipeline",
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/lipdub.py b/packages/ltx-pipelines/src/ltx_pipelines/dubit.py
similarity index 96%
rename from packages/ltx-pipelines/src/ltx_pipelines/lipdub.py
rename to packages/ltx-pipelines/src/ltx_pipelines/dubit.py
index 2c5a63b..37a7ef1 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/lipdub.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/dubit.py
@@ -1,4 +1,4 @@
-"""Two-stage lip-dubbing pipeline with IC-LoRA and appended audio reference conditioning."""
+"""Two-stage Dub-It pipeline with IC-LoRA and appended audio reference conditioning."""
from __future__ import annotations
@@ -24,7 +24,7 @@ from ltx_pipelines.iclora_utils import (
from ltx_pipelines.utils.allocator_trim_strategy import AllocatorTrimStrategy
from ltx_pipelines.utils.args import (
ImageConditioningInput,
- lipdub_arg_parser,
+ dubit_arg_parser,
resolve_cli_params,
)
from ltx_pipelines.utils.blocks import (
@@ -49,8 +49,8 @@ def _snap_frames_to_8k1(frames: int) -> int:
return ((frames - 1) // time_scale) * time_scale + 1
-class LipDubPipeline:
- """Two-stage lip-dubbing with IC-LoRA video reference and appended audio reference tokens."""
+class DubItPipeline:
+ """Two-stage Dub-It with IC-LoRA video reference and appended audio reference tokens."""
def __init__(
self,
@@ -226,7 +226,7 @@ class LipDubPipeline:
)
def build_audio_ref_conditioning(audio_latent: torch.Tensor) -> AudioConditionByReferenceLatent:
- ref_patch, ref_pos = patchify_lipdub_audio_reference_latent(
+ ref_patch, ref_pos = patchify_dubit_audio_reference_latent(
audio_latent,
negative_positions=True,
device=self.device,
@@ -294,7 +294,7 @@ class LipDubPipeline:
return decoded_video, decoded_audio
-def patchify_lipdub_audio_reference_latent(
+def patchify_dubit_audio_reference_latent(
vae_latents: torch.Tensor,
*,
negative_positions: bool,
@@ -320,13 +320,13 @@ def patchify_lipdub_audio_reference_latent(
def main() -> None:
logging.basicConfig(level=logging.INFO)
params = resolve_cli_params(distilled=True)
- parser = lipdub_arg_parser(params=params)
+ parser = dubit_arg_parser(params=params)
args = parser.parse_args()
if not args.lora or len(args.lora) != 1:
- raise ValueError("LipDub requires exactly one --lora (the lip-dub IC-LoRA).")
+ raise ValueError("Dub-It requires exactly one --lora (the Dub-It IC-LoRA).")
- pipeline = LipDubPipeline(
+ pipeline = DubItPipeline(
distilled_checkpoint_path=args.distilled_checkpoint_path,
spatial_upsampler_path=args.spatial_upsampler_path,
gemma_root=args.gemma_root,
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/iclora_utils.py b/packages/ltx-pipelines/src/ltx_pipelines/iclora_utils.py
index b03268a..c1669c8 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/iclora_utils.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/iclora_utils.py
@@ -1,5 +1,5 @@
"""Shared IC-LoRA helpers: LoRA metadata, mask downsampling, reference-video conditioning.
-Used by ``ic_lora`` and ``lipdub`` (video reference path only). LipDub audio helpers live in ``lipdub.py``.
+Used by ``ic_lora`` and ``dubit`` (video reference path only). Dub-It audio helpers live in ``dubit.py``.
"""
from __future__ import annotations
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/args.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/args.py
index 405da47..d86fdc0 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/args.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/args.py
@@ -486,10 +486,10 @@ def video_editing_arg_parser(
return parser
-def lipdub_arg_parser(
+def dubit_arg_parser(
params: PipelineParams = LTX_2_3_PARAMS,
) -> argparse.ArgumentParser:
- """Argument parser for the lip-dub pipeline.
+ """Argument parser for the Dub-It pipeline.
Frame count and frame rate are derived from the reference video at runtime (the frame count
is silently snapped down to the nearest 8k+1), so this parser intentionally omits
--num-frames, --frame-rate, and --image. Distilled checkpoint only.