Skip to content

armgit / LTX-2

Public Imported from GitHub
0

Official Python inference and LoRA trainer package for the LTX-2 audio–video generative model.

Automated PR - 2026-10-18

github-actions[bot]e98f6d927d83
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 2c2c98d..5512c2c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,36 @@
# Changelog
+## 1.5.0 - 2026-10-18
+
+### Added
+
+- `ImageConditioningInput` supports explicit latent-replacement and keyframe-guide bindings. On video CLIs, pass `--image PATH FRAME_IDX STRENGTH replace|guide [CRF]`; omit the binding keyword for the prior automatic binding (frame 0 replaces, later frames guide). Replacement frame indices are expressed in pixel space and must be `0` or a causal latent boundary (`1 + 8 * k`).
+- `TI2VidTwoStagesPipeline` supports keyframe interpolation through `ImageConditioningInput.guide()` bindings, or
+ `--image … guide` on the CLI. Set `refine_audio=True` or pass `--refine-audio` to preserve the former dedicated
+ pipeline's stage-2 audio refinement; the default preserves stage-1 audio.
+- `ImageConditioner.load_conditionings(images, color_space=None)` resolves each image conditioning's CRF (like `resolve_crf`), decodes each distinct still once (SDR, or EXR with `color_space`) and returns `LoadedImageConditioning`s: the decoded still plus the input's `binding` and `strength`.
+- `MGPUController(cpu_threads_per_rank=8)` caps each rank's intra-op CPU threads (`torch.set_num_threads`) so the ranks' small host tensor ops do not stall on many-core hosts. Pass `None` to keep torch's default.
+- `LTX_MGPU_DECODE_RECEIVER_TIMEOUT_SECONDS` sets how long a multi-GPU video decode waits for a rank that stopped streaming before it gives up (default 60).
+
+### Changed
+
+- Multi-GPU pipelines encode prompts faster with the default Accelerate-sharded Gemma: the other ranks now wait on the host until the source rank's Gemma has finished on every GPU, instead of spinning in an NCCL receive on a GPU Gemma is still using. `BroadcastCoordinator` moved from `ltx_pipelines.multigpu._broadcast` to `ltx_core.multigpu.broadcast`, and `BroadcastGemmaWrapper` (and so `AccelerateGemmaWrapper`) takes an optional `coordinator`. `BroadcastCoordinator.wait_for_signal_change` now consumes exactly one signal per call (a worker that polls late no longer skips a signal) and takes an optional `timeout`.
+- `ImageConditioningInput` now takes an `ImageReplace` or `ImageGuide` `binding` instead of a `frame_idx`. Replace existing `ImageConditioningInput(path, frame_idx, strength, crf)` calls with `ImageConditioningInput.at(path, frame_idx, strength, crf)` to preserve the previous frame-0 replacement and later-frame guidance behavior; use `.replace(path, frame_idx, strength, crf)` or `.guide(path, frame_idx, strength, crf)` to select a binding explicitly. Both bindings use pixel-frame indices.
+- `image_conditionings()` and `image_conditionings_for_chunk()` take `LoadedImageConditioning`s (from `ImageConditioner.load_conditionings`) instead of `ImageConditioningInput`s, and no longer take `color_space`.
+- Performance improvements, in collaboration with doubleAI (https://doubleai.com): faster transformer passes (especially under multi-GPU sequence parallelism and with perturbation guidance) and LoRA switches, denoising samplers without per-step CPU synchronization, a DFR spatial epilogue (`--spatial-upscalings 2`) on fewer tiles (at most 1024 px per side, overlapping by 64 px, instead of a 2x2 first step and 4x4 at a 320 px overlap) that multiplies its long video projections in fp8 on GPUs with fp8 GEMMs (compute capability 8.9+), carry keyframes rebuilt on the rank that decodes them, a multi-GPU video decode that streams to MP4 encoding as it decodes, faster diffusion VAE decoding with lower reserved GPU memory, a faster-compiling `--diffvae-optimization blackwell_dsl` kernel, lower GPU memory during MP4 encoding, faster image conditioning, and faster stage starts with a warm model registry. Output is not bit-identical to previous releases, mainly because of the DFR spatial epilogue's new tiles and fp8 GEMMs; diffusion VAE decodes also differ by rounding, and under multi-GPU sequence parallelism audio cross-attention no longer attends to the video sequence's padding tokens.
+
+### Removed
+
+- `ImageConditioningInput.frame_idx`, `.assert_in_clip()`, `.map_frame_index()` and `.to_item()`; use the same members on the `LoadedImageConditioning` returned by `ImageConditioner.load_conditionings`, or `input.binding.frame_idx`.
+- `KeyframeInterpolationPipeline` and the `ltx_pipelines.keyframe_interpolation` CLI; use `TI2VidTwoStagesPipeline` with `ImageConditioningInput.guide()` and `refine_audio=True`, or pass `--image … guide` with `--refine-audio`.
+
+### Fixed
+
+- DiffVAE 1-tile decode no longer produces gray frame tails from frame 128 on long 1080p clips under torch < 2.13. CUDA RMSNorm wrapped flattened offsets at 2^32 elements; last-dim norms are now T-chunked.
+- Compiling a transformer that uses blockwise FP8/FP6 quantization with `seq_dim_dynamic` no longer pins the sequence length. Previously each new length recompiled the blocks, and a batch size divisible by 4 failed with a `ConstraintViolationError`.
+- A warm multi-GPU DFR job (a second or later request in one process) could fail with "Cannot fit a DiffVAE decode tile". Cold and warm runs and every rank now plan the same decode tiles, and ranks with unequal GPUs no longer hang. `--diffvae-optimization blackwell_dsl` keyframe decodes plan smaller tiles.
+- Diffusion VAE decode with `--diffvae-optimization blackwell_dsl` could hang indefinitely on datacenter Blackwell: the fused stage-5 attention kernel re-armed its per-tile barriers while some warps were still waiting on them.
+
## 1.4.2 - 2026-10-01
### Added
@@ -31,13 +62,21 @@
- `--exr-colorspace` on `ltx_pipelines.hdr_ic_lora` (`acescg` / `acescct` / `srgb_linear`; default `acescg`). Only the EXR sidecar encoding changes; the HLG master stays BT.2020/HLG.
- `--transformer-path` on `ltx_pipelines.hdr_ic_lora`, which runs the pipeline from a split checkpoint pack instead of a distilled monolith. A split transformer carries no VAE weights, so `--video-vae-path` has to accompany it.
- `EXRColorSpace` / `VideoInput` in `ltx_pipelines` and an sRGB EOTF in `ltx_core.hdr`, which convert an SDR source into ACEScct.
+- `TokenTimesteps` in `ltx_core`, which holds either dense per-token timesteps or a compact distinct-value table plus per-token indices. `TokenTimesteps.per_token()` expands either representation.
+- `DenoiseMask` in `ltx_core.types`, which holds a per-token denoise mask and the table of its distinct values. Build the mask on the host; `DenoiseMask.compact(device)` tabulates its values and moves mask and table to the device before denoising.
+
+### Fixed
+
+- `FlashAttention4` accepts current `flash-attn-4` releases that return the output tensor directly while remaining compatible with older tuple-returning releases.
### Changed
- The `ltx_kernels_inductor` compile backend batches the fp8-cast policy's per-forward weight and bias upcasts into a few grouped kernels instead of one per tensor. Outputs are bit-identical.
+- The `ltx_kernels_inductor` and `ltx_kernels_eager` compile backends run unmasked attention on whichever of cuDNN and flash attention is faster for each shape, which moves short-query attention (audio queries over video keys) off cuDNN's non-split fallback. The first call per attention shape times both kernels, so warmup takes slightly longer. This also applies when an `SDPA_*` attention backend is pinned.
- Chunk planning owns generated-keyframe placement and appends slots after pipeline conditionings. Generated slots remain transformer conditioning across every stage; `decode_with_keyframes` now controls only whether the final slots anchor video decode.
- `ChunkConfig.overlap_blend_frames` defaults to the full carry overlap (crossfade over all carried frames). Pass `0` for a hard cut at chunk seams.
- `ChunkConfig.overlap_blend_frames` linearly crossfades a configurable suffix of each decoded overlap instead of cutting directly between windows, buffering only that suffix during streaming.
+- Under sequence parallelism, the `ltx_kernels_inductor` compile backend has the q/k RMS-norm + RoPE kernel and the v projection write each head straight into the all2all exchange buffer of the rank that owns it, instead of producing a local tensor for a separate exchange kernel. Outputs are bit-identical.
- DFR spatial epilogues run their first denoising step on a 2x2 tile grid before retiling any remaining steps to 4x4. A one-step schedule stays on the 2x2 grid.
- `create_initial_audio_latent` now takes a video latent and required `video_scale_factors`. Joint first stages use `create_initial_av_latents`, which also requires `video_scale_factors`. `create_initial_video_latent` requires `scale_factors` (no default). Pass the same video VAE factors used to size the video latent. Audio-only callers use `AudioLatentShape.from_duration`.
- Distilled and DFR sample with Euler ancestral (`eta=1.0`, `s_noise=1.0`) on LTX-2.5+ checkpoints -- distilled on both stages, DFR on stage 1, stage 2 and the spatial epilogue -- so generated output for those checkpoints differs from the previous release. Older generations sample with deterministic Euler. DFR's temporal densify tiles are unchanged at `eta=0.5`.
@@ -52,6 +91,9 @@
- `--distilled-checkpoint-path` on `ltx_pipelines.hdr_ic_lora` is no longer required, since `--transformer-path` can supply the transformer instead. One of the two is still needed.
- DFR seam keyframes on `ltx_pipelines.hdr_ic_lora` are on by default. `--with-keyframes` is replaced by `--no-keyframes`. A 2.4 transformer fails the capability check unless `--no-keyframes` is passed; a 2.5 distilled monolith still needs `--video-vae-path` to a keyframe-trained VAE or the keyframe path is a silent no-op.
- `ltx-core` requires `colour-science`, which supplies the primaries, YUV matrix and HLG transfer definitions that were previously hand-written constants.
+- The transformer projects AdaLN modulation once per distinct timestep value and reads it per token through `Modality.timesteps.indices` when a pipeline supplies a compact timestep table. Outputs are not bit-identical to previous releases. The saving is proportional to the token count while attention grows with its square, so its share of the forward shrinks as resolution rises: about 2-3% on one GPU at 16k tokens (544x960x241), and less per rank under sequence parallelism.
+- `Modality.timesteps` takes a `TokenTimesteps` instead of a tensor; wrap per-token timesteps as `TokenTimesteps(timesteps)`.
+- `LatentState.denoise_mask` takes a `DenoiseMask` instead of a tensor; wrap the mask as `DenoiseMask(mask)` with `mask` on the host, and call `compact(device)` on it before a denoising loop reads the state.
### Removed
diff --git a/LICENSE-2_x b/LICENSE-2_x
index 62bc2ee..010294d 100644
--- a/LICENSE-2_x
+++ b/LICENSE-2_x
@@ -1,5 +1,5 @@
- LTX-2.x Community License Agreement
- License date: August 11, 2026
+ LTX-2.x Community License Agreement
+ License date: August 11, 2026
By downloading, using, accessing or distributing any portion or
@@ -473,10 +473,10 @@
be unaffected thereby and remain valid as if such provision had
not been set forth herein.
- END OF TERMS AND CONDITIONS
+ END OF TERMS AND CONDITIONS
- Attachment A
- Use Restrictions
+ Attachment A
+ Use Restrictions
When using the Outputs, LTX-2.x and any Derivatives thereof, you
agree to comply with the Acceptable Use Policy
diff --git a/README.md b/README.md
index b69cbd7..15cbfee 100644
--- a/README.md
+++ b/README.md
@@ -139,11 +139,10 @@ See **[LTX-2.3 models](MODELS-LTX-2.3.md)** for the full list.
* **[DistilledPipeline](packages/ltx-pipelines/src/ltx_pipelines/distilled.py)** - Fastest text/image-to-video (starting point)
* **[DFRPipeline](packages/ltx-pipelines/src/ltx_pipelines/dfr_pipeline.py)** - Production-quality text/image-to-video (slower, more VRAM): same distilled transformer, generated keyframes, spatial detailing, optional 2x/4x fps. How to run: [DFR in Quick Start](#dfr-production-quality) and [Running DFR](packages/ltx-pipelines/docs/pipelines.md#running-dfr)
-* **[TI2VidTwoStagesPipeline](packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py)** - Guided two-stage text/image-to-video with CFG/STG and 2x upsampling
+* **[TI2VidTwoStagesPipeline](packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py)** - Guided two-stage text/image-to-video and keyframe interpolation with CFG/STG and 2x upsampling
* **[TI2VidTwoStagesHQPipeline](packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq.py)** - Same guided two-stage flow with the res_2s sampler (fewer steps)
* **[TI2VidOneStagePipeline](packages/ltx-pipelines/src/ltx_pipelines/ti2vid_one_stage.py)** - Single-stage generation for quick prototyping
* **[ICLoraPipeline](packages/ltx-pipelines/src/ltx_pipelines/ic_lora.py)** - Video-to-video and image-to-video transformations (uses distilled model.)
-* **[KeyframeInterpolationPipeline](packages/ltx-pipelines/src/ltx_pipelines/keyframe_interpolation.py)** - Interpolate between keyframe images
* **[A2VidPipelineTwoStage](packages/ltx-pipelines/src/ltx_pipelines/a2vid_two_stage.py)** - Audio-to-video generation conditioned on an input audio file
* **[RetakePipeline](packages/ltx-pipelines/src/ltx_pipelines/retake.py)** - Regenerate a specific time region of an existing video
* **[HDRICLoraPipeline](packages/ltx-pipelines/src/ltx_pipelines/hdr_ic_lora.py)** - Video-to-video SDR→HDR with an HDR IC-LoRA, writing a BT.2020/HLG master plus an ACEScct EXR sequence for EXR export and grading
diff --git a/packages/ltx-core/README.md b/packages/ltx-core/README.md
index 3d290b0..dbb24ac 100644
--- a/packages/ltx-core/README.md
+++ b/packages/ltx-core/README.md
@@ -439,10 +439,9 @@ Here's how all the components work together conceptually ([`src/ltx_core/compone
7. **VAE Decoding**: Decode latents to pixel space (with optional upsampling for two-stage)
- [`DFRPipeline`](../ltx-pipelines/src/ltx_pipelines/dfr_pipeline.py) - Production-quality text/image-to-video
-- [`TI2VidTwoStagesPipeline`](../ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py) - Guided two-stage text-to-video
+- [`TI2VidTwoStagesPipeline`](../ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py) - Guided two-stage text/image-to-video and keyframe interpolation
- [`ICLoraPipeline`](../ltx-pipelines/src/ltx_pipelines/ic_lora.py) - Video-to-video with IC-LoRA control
- [`DistilledPipeline`](../ltx-pipelines/src/ltx_pipelines/distilled.py) - Fast inference with distilled model
-- [`KeyframeInterpolationPipeline`](../ltx-pipelines/src/ltx_pipelines/keyframe_interpolation.py) - Keyframe-based interpolation
See the [ltx-pipelines README](../ltx-pipelines/README.md) for usage examples.
diff --git a/packages/ltx-core/pyproject.toml b/packages/ltx-core/pyproject.toml
index f2c3756..42f3314 100644
--- a/packages/ltx-core/pyproject.toml
+++ b/packages/ltx-core/pyproject.toml
@@ -1,6 +1,6 @@
[project]
name = "ltx-core"
-version = "1.4.2"
+version = "1.5.0"
description = "Core implementation of Lightricks' LTX-2 model"
readme = "README.md"
requires-python = ">=3.10"
diff --git a/packages/ltx-core/src/ltx_core/block_streaming/builder.py b/packages/ltx-core/src/ltx_core/block_streaming/builder.py
index cb74a27..f6282d7 100644
--- a/packages/ltx-core/src/ltx_core/block_streaming/builder.py
+++ b/packages/ltx-core/src/ltx_core/block_streaming/builder.py
@@ -28,7 +28,7 @@ from ltx_core.block_streaming.utils import (
from ltx_core.block_streaming.wrapper import BlockStreamingWrapper
from ltx_core.devices import synchronize_device
from ltx_core.loader.fuse_loras import FuseRule, bf16_fuse_rule, fuse_lora_weights
-from ltx_core.loader.helpers import create_meta_model, load_state_dict, read_model_config, read_model_metadata
+from ltx_core.loader.helpers import LazyModelMetadata, create_meta_model, load_state_dict
from ltx_core.loader.module_ops import ModuleOps
from ltx_core.loader.primitives import (
LoraPathStrengthAndSDOps,
@@ -107,6 +107,7 @@ class StreamingModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType])
self._blocks_attr = blocks_attr
self._blocks_prefix = blocks_prefix
self._cpu_slots_count = cpu_slots_count
+ self._metadata = LazyModelMetadata(model_path, self._model_loader)
@property
def model_class_configurator(self) -> type[ModelConfigurator[ModelType]]:
@@ -161,6 +162,10 @@ class StreamingModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType])
# Block weights live in fixed GPU/CPU slots that are reused across builds.
return True
+ @property
+ def builds_per_shape(self) -> bool:
+ return False
+
def with_sd_ops(self, sd_ops: SDOps | None) -> Self:
clone = copy.copy(self)
clone._model_sd_ops = sd_ops
@@ -192,11 +197,11 @@ class StreamingModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType])
def model_config(self) -> dict:
"""Read model configuration from the checkpoint metadata."""
- return read_model_config(self.model_path, self.model_loader)
+ return self.model_metadata().get("config", {})
def model_metadata(self) -> dict:
"""Read the full checkpoint metadata (including sibling keys beyond ``config``)."""
- return read_model_metadata(self.model_path, self.model_loader)
+ return self._metadata.get()
def meta_model(self, metadata: dict, module_ops: tuple[ModuleOps, ...]) -> ModelType:
"""Create a model on the meta device and apply module operations."""
diff --git a/packages/ltx-core/src/ltx_core/color/yuv.py b/packages/ltx-core/src/ltx_core/color/yuv.py
index 81b5d9f..5f8deb4 100644
--- a/packages/ltx-core/src/ltx_core/color/yuv.py
+++ b/packages/ltx-core/src/ltx_core/color/yuv.py
@@ -8,10 +8,13 @@ I420 / planar packing stay here (not provided by colour).
from __future__ import annotations
import enum
+import functools
import colour
import torch
+from ltx_core.devices import upload_from_host
+
class ColorSpace(enum.Enum):
"""YUV color space standard (FFmpeg ``AVCOL_SPC_*`` + colour-science weights)."""
@@ -82,6 +85,13 @@ def _rgb_to_ycbcr_matrix(weights_name: str) -> torch.Tensor:
# Built once; ``ColorSpace.rgb_to_ycbcr_matrix`` reads from here.
_RGB_TO_YCBCR = {cs: _rgb_to_ycbcr_matrix(cs._weights_name) for cs in ColorSpace}
+
[email protected]_cache(maxsize=16)
+def _device_rgb_to_ycbcr_matrix(color_space: ColorSpace, device: torch.device, dtype: torch.dtype) -> torch.Tensor:
+ """``color_space``'s matrix on ``device`` in ``dtype``, uploaded once without waiting for the device."""
+ return upload_from_host(color_space.rgb_to_ycbcr_matrix.to(dtype=dtype), device)
+
+
# cuBLAS GEMM dims are int32. PyTorch folds ``(F, H*W, 3) @ (3, 3)`` into one
# ``mm`` of size ``F*H*W``, reported as ``n`` in column-major. ``n`` must be
# strictly less than this value (see ``at::cuda::blas::gemm``).
@@ -121,7 +131,7 @@ def rgb_to_yuv(image: torch.Tensor, color_space: ColorSpace) -> torch.Tensor:
are concatenated. (``colour.RGB_to_YCbCr`` is NumPy-only and not used here.)
"""
_require_rgb_chw(image)
- mat = color_space.rgb_to_ycbcr_matrix.to(device=image.device, dtype=image.dtype)
+ mat = _device_rgb_to_ycbcr_matrix(color_space, image.device, image.dtype)
orig_shape = image.shape
height, width = orig_shape[-2], orig_shape[-1]
frames = image.reshape(-1, 3, height, width)
diff --git a/packages/ltx-core/src/ltx_core/components/diffusion_steps.py b/packages/ltx-core/src/ltx_core/components/diffusion_steps.py
index 7d74ea3..5097365 100644
--- a/packages/ltx-core/src/ltx_core/components/diffusion_steps.py
+++ b/packages/ltx-core/src/ltx_core/components/diffusion_steps.py
@@ -150,8 +150,7 @@ class Res2sDiffusionStep(DiffusionStepProtocol):
sigma_up = torch.nan_to_num(sigma_up if sigma_up is not None else torch.zeros_like(sigma_next), 0.0)
# Replace NaNs in sigma_down with corresponding sigma_next elements (float32)
- nan_mask = torch.isnan(sigma_down)
- sigma_down[nan_mask] = sigma_next[nan_mask].to(sigma_down.dtype)
+ sigma_down = torch.where(torch.isnan(sigma_down), sigma_next.to(sigma_down.dtype), sigma_down)
alpha_ratio = torch.nan_to_num(alpha_ratio, 1.0)
return alpha_ratio, sigma_down, sigma_up
@@ -180,8 +179,7 @@ class Res2sDiffusionStep(DiffusionStepProtocol):
sigma_next = sigmas[step_index + 1]
alpha_ratio, sigma_down, sigma_up = self.get_sde_coeff(sigma_next, sigma_up=sigma_next * eta)
output_dtype = denoised_sample.dtype
- if torch.any(sigma_up == 0) or torch.any(sigma_next == 0):
- return denoised_sample
+ return_denoised = torch.any(sigma_up == 0) | torch.any(sigma_next == 0)
# Extract epsilon prediction
eps_next = (sample - denoised_sample) / (sigma - sigma_next)
@@ -189,7 +187,7 @@ class Res2sDiffusionStep(DiffusionStepProtocol):
# Mix deterministic and stochastic components
x_noised = alpha_ratio * (denoised_next + sigma_down * eps_next) + sigma_up * noise
- return x_noised.to(output_dtype)
+ return torch.where(return_denoised, denoised_sample, x_noised.to(output_dtype))
class EulerCfgPpDiffusionStep(DiffusionStepProtocol):
diff --git a/packages/ltx-core/src/ltx_core/components/noisers.py b/packages/ltx-core/src/ltx_core/components/noisers.py
index 39122b0..7df5a73 100644
--- a/packages/ltx-core/src/ltx_core/components/noisers.py
+++ b/packages/ltx-core/src/ltx_core/components/noisers.py
@@ -30,7 +30,7 @@ class GaussianNoiser(Noiser):
def __call__(self, latent_state: LatentState, noise_scale: float = 1.0) -> LatentState:
noise = self._sample_noise(latent_state)
latent = torch.lerp(latent_state.latent.float(), noise.float(), noise_scale)
- latent = torch.lerp(latent_state.clean_latent.float(), latent, latent_state.denoise_mask)
+ latent = torch.lerp(latent_state.clean_latent.float(), latent, latent_state.denoise_mask.tensor)
return replace(
latent_state,
latent=latent.to(latent_state.latent.dtype),
diff --git a/packages/ltx-core/src/ltx_core/components/patchifiers.py b/packages/ltx-core/src/ltx_core/components/patchifiers.py
index 77d6521..1c71116 100644
--- a/packages/ltx-core/src/ltx_core/components/patchifiers.py
+++ b/packages/ltx-core/src/ltx_core/components/patchifiers.py
@@ -105,18 +105,12 @@ class VideoLatentPatchifier(Patchifier):
# Shape becomes (3, grid_f, grid_h, grid_w)
patch_starts = torch.stack(grid_coords, dim=0)
- # Create a tensor containing the size of a single patch:
- # (frame_patch_size, height_patch_size, width_patch_size).
- # Reshape to (3, 1, 1, 1) to enable broadcasting when adding to the start coordinates.
- patch_size_delta = torch.tensor(
- self._patch_size,
- device=patch_starts.device,
- dtype=patch_starts.dtype,
- ).view(3, 1, 1, 1)
-
- # Calculate end coordinates: start + patch_size
+ # Calculate end coordinates: start + patch_size, per axis with Python-int patch sizes so
+ # no host-built tensor is copied to the device.
# Shape becomes (3, grid_f, grid_h, grid_w)
- patch_ends = patch_starts + patch_size_delta
+ patch_ends = torch.stack(
+ [start + size for start, size in zip(grid_coords, self._patch_size, strict=True)], dim=0
+ )
# Stack start and end coordinates together along the last dimension
# Shape becomes (3, grid_f, grid_h, grid_w, 2), where the last dimension is [start, end]
@@ -150,18 +144,17 @@ def get_pixel_coords(
causal_fix: When True, rewrites the temporal axis of the first frame so causal VAEs
that treat frame zero differently still yield non-negative timestamps.
"""
- # Broadcast the VAE scale factors so they align with the `(batch, axis, patch, bound)` layout.
- # Axis 1 of `latent_coords` is ordered (frame/time, height, width) — match that explicitly by
- # pulling fields from the NamedTuple rather than relying on tuple iteration order.
- broadcast_shape = [1] * latent_coords.ndim
- broadcast_shape[1] = -1 # axis dimension corresponds to (frame/time, height, width)
- scale_tensor = torch.tensor(
- [scale_factors.time, scale_factors.height, scale_factors.width],
- device=latent_coords.device,
- ).view(*broadcast_shape)
-
- # Apply per-axis scaling to convert latent bounds into pixel-space coordinates.
- pixel_coords = latent_coords * scale_tensor
+ # Axis 1 of `latent_coords` is ordered (frame/time, height, width) -- match that explicitly by
+ # pulling fields from the NamedTuple rather than relying on tuple iteration order. The factors
+ # stay Python ints: a host-built factor tensor would be a blocking copy to the device.
+ pixel_coords = torch.stack(
+ [
+ latent_coords[:, 0] * scale_factors.time,
+ latent_coords[:, 1] * scale_factors.height,
+ latent_coords[:, 2] * scale_factors.width,
+ ],
+ dim=1,
+ )
if causal_fix:
# VAE temporal stride for the very first frame is 1 instead of `scale_factors.time`.
diff --git a/packages/ltx-core/src/ltx_core/conditioning/mask_utils.py b/packages/ltx-core/src/ltx_core/conditioning/mask_utils.py
index 3a4fc54..d068775 100644
--- a/packages/ltx-core/src/ltx_core/conditioning/mask_utils.py
+++ b/packages/ltx-core/src/ltx_core/conditioning/mask_utils.py
@@ -97,7 +97,7 @@ def extend_keyframes_mask(
if existing is None and not marked:
return None
if existing is None:
- existing = torch.zeros_like(latent_state.denoise_mask)
+ existing = torch.zeros_like(latent_state.denoise_mask.tensor, device=latent_state.latent.device)
new = torch.ones if marked else torch.zeros
pad = new(
(existing.shape[0], num_new_tokens, existing.shape[2]),
diff --git a/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_cond.py b/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_cond.py
index 2c8c18a..b23a5d0 100644
--- a/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_cond.py
+++ b/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_cond.py
@@ -4,7 +4,7 @@ from ltx_core.components.patchifiers import get_pixel_coords
from ltx_core.conditioning.item import ConditioningItem
from ltx_core.conditioning.mask_utils import extend_keyframes_mask, update_attention_mask
from ltx_core.tools import VideoLatentTools
-from ltx_core.types import LatentState, VideoLatentShape
+from ltx_core.types import DenoiseMask, LatentState, VideoLatentShape
class VideoConditionByKeyframeIndex(ConditioningItem):
@@ -61,7 +61,7 @@ class VideoConditionByKeyframeIndex(ConditioningItem):
denoise_mask = torch.full(
size=(*tokens.shape[:2], 1),
fill_value=1.0 - self.strength,
- device=self.keyframes.device,
+ device="cpu",
dtype=self.keyframes.dtype,
)
@@ -77,7 +77,7 @@ class VideoConditionByKeyframeIndex(ConditioningItem):
return LatentState(
latent=torch.cat([latent_state.latent, torch.zeros_like(tokens)], dim=1),
- denoise_mask=torch.cat([latent_state.denoise_mask, denoise_mask], dim=1),
+ denoise_mask=DenoiseMask(torch.cat([latent_state.denoise_mask.tensor, denoise_mask], dim=1)),
positions=torch.cat([latent_state.positions, positions], dim=2),
clean_latent=torch.cat([latent_state.clean_latent, tokens], dim=1),
attention_mask=new_attention_mask,
diff --git a/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_slots.py b/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_slots.py
index 99f4904..ee0b535 100644
--- a/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_slots.py
+++ b/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_slots.py
@@ -21,7 +21,7 @@ from ltx_core.components.patchifiers import get_pixel_coords
from ltx_core.conditioning.item import ConditioningItem
from ltx_core.conditioning.mask_utils import extend_keyframes_mask, update_attention_mask
from ltx_core.tools import VideoLatentTools
-from ltx_core.types import GeneratedKeyframeLayout, LatentState
+from ltx_core.types import DenoiseMask, GeneratedKeyframeLayout, LatentState
class VideoGeneratedKeyframeSlots(ConditioningItem):
@@ -116,7 +116,7 @@ class VideoGeneratedKeyframeSlots(ConditioningItem):
dim=1,
)
# denoise_mask 1 => the noiser lerps slot latent toward noise (clean_latent is ignored).
- denoise_mask = torch.ones(batch_size, num_new_tokens, 1, device=device, dtype=latent_state.denoise_mask.dtype)
+ denoise_mask = torch.ones(batch_size, num_new_tokens, 1, dtype=latent_state.denoise_mask.tensor.dtype)
keyframes_mask = extend_keyframes_mask(latent_state, num_new_tokens, marked=True)
@@ -135,7 +135,7 @@ class VideoGeneratedKeyframeSlots(ConditioningItem):
return LatentState(
latent=torch.cat([latent_state.latent, slot_tokens], dim=1),
- denoise_mask=torch.cat([latent_state.denoise_mask, denoise_mask], dim=1),
+ denoise_mask=DenoiseMask(torch.cat([latent_state.denoise_mask.tensor, denoise_mask], dim=1)),
positions=torch.cat([latent_state.positions, positions], dim=2),
clean_latent=torch.cat([latent_state.clean_latent, torch.zeros_like(slot_tokens)], dim=1),
attention_mask=new_attention_mask,
diff --git a/packages/ltx-core/src/ltx_core/conditioning/types/latent_cond.py b/packages/ltx-core/src/ltx_core/conditioning/types/latent_cond.py
index 381f899..4cfbd14 100644
--- a/packages/ltx-core/src/ltx_core/conditioning/types/latent_cond.py
+++ b/packages/ltx-core/src/ltx_core/conditioning/types/latent_cond.py
@@ -83,5 +83,5 @@ def _apply_condition_by_latent_index(
latent_state = latent_state.clone()
latent_state.clean_latent[:, start_token:stop_token] = tokens
- latent_state.denoise_mask[:, start_token:stop_token] = 1.0 - strength
+ latent_state.denoise_mask.tensor[:, start_token:stop_token] = 1.0 - strength
return latent_state
diff --git a/packages/ltx-core/src/ltx_core/conditioning/types/mask_cond.py b/packages/ltx-core/src/ltx_core/conditioning/types/mask_cond.py
index 0464198..153e2ef 100644
--- a/packages/ltx-core/src/ltx_core/conditioning/types/mask_cond.py
+++ b/packages/ltx-core/src/ltx_core/conditioning/types/mask_cond.py
@@ -6,7 +6,7 @@ import torch
from ltx_core.conditioning.item import ConditioningItem
from ltx_core.tools import LatentTools
-from ltx_core.types import LatentState
+from ltx_core.types import DenoiseMask, LatentState
class VideoConditionByMask(ConditioningItem):
@@ -45,5 +45,5 @@ class VideoConditionByMask(ConditioningItem):
return replace(
latent_state,
clean_latent=latent_state.clean_latent * inv + tokens * m,
- denoise_mask=latent_state.denoise_mask * inv + (1.0 - self.strength) * m,
+ denoise_mask=DenoiseMask(latent_state.denoise_mask.tensor * inv.cpu() + (1.0 - self.strength) * m.cpu()),
)
diff --git a/packages/ltx-core/src/ltx_core/conditioning/types/noise_mask_cond.py b/packages/ltx-core/src/ltx_core/conditioning/types/noise_mask_cond.py
index 186ac4e..fbcb3b6 100644
--- a/packages/ltx-core/src/ltx_core/conditioning/types/noise_mask_cond.py
+++ b/packages/ltx-core/src/ltx_core/conditioning/types/noise_mask_cond.py
@@ -22,7 +22,7 @@ class TemporalRegionMask(ConditioningItem):
def apply_to(self, latent_state: LatentState, latent_tools: LatentTools) -> LatentState:
coords = latent_tools.patchifier.get_patch_grid_bounds(
- latent_tools.target_shape, device=latent_state.denoise_mask.device
+ latent_tools.target_shape, device=latent_state.denoise_mask.tensor.device
)
if isinstance(latent_tools.target_shape, AudioLatentShape):
# Audio: patchifier get_patch_grid_bounds returns seconds
@@ -38,8 +38,8 @@ class TemporalRegionMask(ConditioningItem):
t_start, t_end = t_boundaries.unbind(dim=-1) # [B, N]
in_region = (t_end > self.start_time) & (t_start < self.end_time)
state = latent_state.clone()
- mask_val = in_region.to(state.denoise_mask.dtype)
- if state.denoise_mask.dim() == 3:
+ mask_val = in_region.to(state.denoise_mask.tensor.dtype)
+ if state.denoise_mask.tensor.dim() == 3:
mask_val = mask_val.unsqueeze(-1)
- state.denoise_mask.copy_(mask_val)
+ state.denoise_mask.tensor.copy_(mask_val)
return state
diff --git a/packages/ltx-core/src/ltx_core/conditioning/types/reference_audio_cond.py b/packages/ltx-core/src/ltx_core/conditioning/types/reference_audio_cond.py
index 3427025..66292a2 100644
--- a/packages/ltx-core/src/ltx_core/conditioning/types/reference_audio_cond.py
+++ b/packages/ltx-core/src/ltx_core/conditioning/types/reference_audio_cond.py
@@ -6,7 +6,7 @@ import torch
from ltx_core.conditioning.mask_utils import extend_keyframes_mask, update_attention_mask
from ltx_core.tools import LatentTools
-from ltx_core.types import LatentState
+from ltx_core.types import DenoiseMask, LatentState
class AudioConditionByReferenceLatent:
@@ -36,7 +36,7 @@ class AudioConditionByReferenceLatent:
denoise_mask = torch.full(
size=(*tokens.shape[:2], 1),
fill_value=1.0 - self.strength,
- device=tokens.device,
+ device="cpu",
dtype=tokens.dtype,
)
@@ -52,7 +52,7 @@ class AudioConditionByReferenceLatent:
return LatentState(
latent=torch.cat([latent_state.latent, torch.zeros_like(tokens)], dim=1),
- denoise_mask=torch.cat([latent_state.denoise_mask, denoise_mask], dim=1),
+ denoise_mask=DenoiseMask(torch.cat([latent_state.denoise_mask.tensor, denoise_mask], dim=1)),
positions=torch.cat([latent_state.positions, self.positions], dim=2),
clean_latent=torch.cat([latent_state.clean_latent, tokens], dim=1),
attention_mask=new_attention_mask,
diff --git a/packages/ltx-core/src/ltx_core/conditioning/types/reference_video_cond.py b/packages/ltx-core/src/ltx_core/conditioning/types/reference_video_cond.py
index adf4775..eee5754 100644
--- a/packages/ltx-core/src/ltx_core/conditioning/types/reference_video_cond.py
+++ b/packages/ltx-core/src/ltx_core/conditioning/types/reference_video_cond.py
@@ -6,7 +6,7 @@ from ltx_core.components.patchifiers import get_pixel_coords
from ltx_core.conditioning.item import ConditioningItem
from ltx_core.conditioning.mask_utils import extend_keyframes_mask, update_attention_mask
from ltx_core.tools import VideoLatentTools
-from ltx_core.types import LatentState, VideoLatentShape
+from ltx_core.types import DenoiseMask, LatentState, VideoLatentShape
class VideoConditionByReferenceLatent(ConditioningItem):
@@ -87,7 +87,7 @@ class VideoConditionByReferenceLatent(ConditioningItem):
denoise_mask = torch.full(
size=(*tokens.shape[:2], 1),
fill_value=1.0 - self.strength,
- device=self.latent.device,
+ device="cpu",
dtype=self.latent.dtype,
)
@@ -103,7 +103,7 @@ class VideoConditionByReferenceLatent(ConditioningItem):
return LatentState(
latent=torch.cat([latent_state.latent, torch.zeros_like(tokens)], dim=1),
- denoise_mask=torch.cat([latent_state.denoise_mask, denoise_mask], dim=1),
+ denoise_mask=DenoiseMask(torch.cat([latent_state.denoise_mask.tensor, denoise_mask], dim=1)),
positions=torch.cat([latent_state.positions, positions], dim=2),
clean_latent=torch.cat([latent_state.clean_latent, tokens], dim=1),
attention_mask=new_attention_mask,
diff --git a/packages/ltx-core/src/ltx_core/devices.py b/packages/ltx-core/src/ltx_core/devices.py
index f33271b..33e4049 100644
--- a/packages/ltx-core/src/ltx_core/devices.py
+++ b/packages/ltx-core/src/ltx_core/devices.py
@@ -72,6 +72,16 @@ def allow_async_transfer(device: torch.device) -> bool:
return device.type == "cuda"
+def upload_from_host(host: torch.Tensor, device: torch.device) -> torch.Tensor:
+ """Copy a host tensor to *device*, without blocking the host where the backend allows it.
+ The transfer reads a pinned copy, so ``host`` may be modified as soon as this returns. A tensor
+ that is not on the host is moved with ``.to``.
+ """
+ if not allow_async_transfer(device) or host.device.type != "cpu":
+ return host.to(device)
+ return host.pin_memory().to(device, non_blocking=True)
+
+
def synchronize_device(device: DeviceSpec = None) -> None:
"""Synchronize CUDA or MPS work if the selected backend supports it."""
resolved = resolve_device(device)
@@ -110,6 +120,27 @@ def cuda_activation_budget_bytes(device: DeviceSpec = None) -> int:
return min(int(free_raw), under_fraction)
+def cuda_allocatable_bytes(device: DeviceSpec = None) -> int:
+ """Bytes a new CUDA allocation in this process can still get, counting the allocator's cache.
+ Like :func:`cuda_activation_budget_bytes`, but reserved blocks that hold no tensor count as
+ available: the caching allocator serves new allocations from them and releases them to the
+ driver before an allocation fails. ``fraction * device_total - memory_allocated``, capped by
+ raw free memory plus those cached blocks.
+ """
+ if not torch.cuda.is_available():
+ return 0
+ resolved = resolve_device(device)
+ if resolved.type != "cuda":
+ return 0
+ idx = resolved.index if resolved.index is not None else torch.cuda.current_device()
+ free_raw, _total_raw = torch.cuda.mem_get_info(idx)
+ props_total = torch.cuda.get_device_properties(idx).total_memory
+ fraction = torch.cuda.get_per_process_memory_fraction(idx)
+ allocated = int(torch.cuda.memory_allocated(idx))
+ cached = max(0, int(torch.cuda.memory_reserved(idx)) - allocated)
+ return max(0, min(int(free_raw) + cached, int(fraction * props_total) - allocated))
+
+
def mps_activation_budget_bytes(device: DeviceSpec = None) -> int:
"""Bytes still available for new MPS allocations in this process.
``recommended_max_memory`` is Metal's recommended working-set size, which is
diff --git a/packages/ltx-core/src/ltx_core/guidance/perturbations.py b/packages/ltx-core/src/ltx_core/guidance/perturbations.py
index 394f96e..dd9c8bd 100644
--- a/packages/ltx-core/src/ltx_core/guidance/perturbations.py
+++ b/packages/ltx-core/src/ltx_core/guidance/perturbations.py
@@ -4,6 +4,8 @@ from enum import IntEnum
import torch
from torch._prims_common import DeviceLikeType
+from ltx_core.types import HostDeviceTensorPair
+
class PerturbationType(IntEnum):
"""Types of attention perturbations for STG (Spatio-Temporal Guidance).
@@ -52,20 +54,16 @@ class PerturbationConfig:
class BatchedPerturbationConfig:
"""Per-block attention keep-masks for a batch, built once from a list of per-sample configs.
- Construction materializes ``_block_masks`` -- a ``(len(PerturbationType), num_blocks, B)`` tensor
- (1 = keep, 0 = perturbed) whose dim-0 row index is the ``PerturbationType`` value -- from the
- perturbation structure. The per-sample config list is NOT retained: every consumer reads the
- tensor (``mask`` indexes it; ``any_in_batch`` / ``all_in_batch`` read the host mirror).
+ Construction materializes ``_block_masks`` -- paired device and host
+ ``(len(PerturbationType), num_blocks, B)`` tensors (1 = keep, 0 = perturbed) whose dim-0 row
+ index is the ``PerturbationType`` value -- from the perturbation structure. The per-sample
+ config list is NOT retained: every consumer reads the tensor pair.
The host build (reading the Python structure) happens here, in ``__init__``, so it MUST be run
eagerly OUTSIDE any ``torch.compile`` / CUDA-graph-capture region. The compiled block then reads
- perturbation purely as the runtime ``_block_masks`` tensor and never recompiles per config.
+ perturbation purely as the runtime device tensor and never recompiles per config.
"""
- _block_masks: torch.Tensor # keep-mask on the compute device, indexed [PerturbationType, block, sample]
- # Host mirror so any_in_batch / all_in_batch stay sync-free and graph-break-free. Present for
- # configs that may hit the eager skip shortcuts; None for compiled-only configs built via
- # ``from_masks`` (the compiled processor reads only ``_block_masks``).
- _block_masks_cpu: torch.Tensor | None
+ _block_masks: HostDeviceTensorPair
def __init__(
self,
@@ -81,39 +79,46 @@ class BatchedPerturbationConfig:
]
for direction in range(len(PerturbationType))
]
- self._block_masks_cpu = torch.tensor(keep, dtype=dtype, device="cpu")
- self._block_masks = self._block_masks_cpu if device is None else self._block_masks_cpu.to(device)
+ host = torch.tensor(keep, dtype=dtype, device="cpu")
+ if device is not None and torch.device(device).type == "cuda":
+ # CUDA non-blocking copies require pinned source memory. The host mirror is read-only
+ # after construction.
+ host = host.pin_memory()
+ device_mask = host.to(device, non_blocking=True)
+ else:
+ device_mask = host if device is None else host.to(device)
+ self._block_masks = HostDeviceTensorPair(host_tensor=host, device_tensor=device_mask)
@classmethod
- def from_masks(
- cls, block_masks: torch.Tensor, block_masks_cpu: torch.Tensor | None = None
- ) -> "BatchedPerturbationConfig":
+ def from_masks(cls, block_masks: HostDeviceTensorPair) -> "BatchedPerturbationConfig":
"""Construct from prebuilt mask tensors (e.g. a batch-dim slice), bypassing the host build.
- ``block_masks_cpu`` is only consumed by ``any_in_batch`` / ``all_in_batch`` (the eager
- processor's skip shortcuts); pass it when the result may take that path. The compiled
- processor reads only ``_block_masks``, so callers on that path may omit the mirror.
+ The host tensor is consumed by ``any_in_batch`` / ``all_in_batch`` while the compiled
+ processor reads only the device tensor.
"""
obj = cls.__new__(cls)
obj._block_masks = block_masks
- obj._block_masks_cpu = block_masks_cpu
return obj
@property
def block_masks(self) -> torch.Tensor:
"""Keep-mask on the compute device (1 = keep, 0 = perturbed), indexed [type, block, sample]."""
- return self._block_masks
+ return self._block_masks.device_tensor
@property
- def block_masks_cpu(self) -> torch.Tensor | None:
- """Host mirror of the keep-mask, or None for compiled-only configs (see ``from_masks``)."""
- return self._block_masks_cpu
+ def block_masks_cpu(self) -> torch.Tensor:
+ """Host mirror of the keep-mask."""
+ return self._block_masks.host_tensor
def batch_slice(self, start: int, end: int) -> "BatchedPerturbationConfig":
"""A view over samples ``[start:end]`` of the batch, by slicing the mask tensors.
Slicing (never rebuilding) keeps the host mask build outside any compiled / capture region.
"""
- cpu_mask = self._block_masks_cpu[:, :, start:end] if self._block_masks_cpu is not None else None
- return BatchedPerturbationConfig.from_masks(self._block_masks[:, :, start:end], cpu_mask)
+ return BatchedPerturbationConfig.from_masks(
+ HostDeviceTensorPair(
+ host_tensor=self._block_masks.host_tensor[:, :, start:end],
+ device_tensor=self._block_masks.device_tensor[:, :, start:end],
+ )
+ )
def mask(self, perturbation_type: PerturbationType, block: int) -> torch.Tensor:
"""This block's ``(B, 1, 1)`` keep-mask for one perturbation type, as an OWNED tensor.
@@ -121,15 +126,13 @@ class BatchedPerturbationConfig:
(e.g. self- and cross-attention) don't alias the same storage -- aliased graph inputs are
fragile under ``torch.compile``.
"""
- return self._block_masks[perturbation_type, block].reshape(-1, 1, 1).clone()
+ return self._block_masks.device_tensor[perturbation_type, block].reshape(-1, 1, 1).clone()
def any_in_batch(self, perturbation_type: PerturbationType, block: int) -> bool:
- assert self._block_masks_cpu is not None, "host mirror required by the skip-shortcut processor path"
- return bool((self._block_masks_cpu[perturbation_type, block] == 0).any())
+ return bool((self._block_masks.host_tensor[perturbation_type, block] == 0).any())
def all_in_batch(self, perturbation_type: PerturbationType, block: int) -> bool:
- assert self._block_masks_cpu is not None, "host mirror required by the skip-shortcut processor path"
- return bool((self._block_masks_cpu[perturbation_type, block] == 0).all())
+ return bool((self._block_masks.host_tensor[perturbation_type, block] == 0).all())
@staticmethod
def empty(
diff --git a/packages/ltx-core/src/ltx_core/loader/helpers.py b/packages/ltx-core/src/ltx_core/loader/helpers.py
index af2ea04..716d130 100644
--- a/packages/ltx-core/src/ltx_core/loader/helpers.py
+++ b/packages/ltx-core/src/ltx_core/loader/helpers.py
@@ -2,11 +2,14 @@
from __future__ import annotations
+import copy
+from collections.abc import Iterator, Mapping
from os import PathLike, fspath
from typing import TypeVar
import torch
from torch import nn
+from torch.nn.parameter import is_lazy
from ltx_core.loader.module_ops import ModuleOps
from ltx_core.loader.primitives import StateDict, StateDictLoader
@@ -18,6 +21,100 @@ _M = TypeVar("_M", bound=nn.Module)
_PathArg = str | PathLike[str]
+def _customizes_loading(module: nn.Module) -> bool:
+ cls = type(module)
+ return (
+ cls._load_from_state_dict is not nn.Module._load_from_state_dict
+ or cls.set_extra_state is not nn.Module.set_extra_state
+ or bool(module._load_state_dict_pre_hooks)
+ or bool(module._load_state_dict_post_hooks)
+ )
+
+
+def assign_state_dict(module: nn.Module, state_dict: Mapping[str, torch.Tensor]) -> None:
+ """Same result as ``module.load_state_dict(state_dict, strict=False, assign=True)``, in one pass.
+ Each parameter / persistent buffer looks its key up directly, where ``nn.Module.load_state_dict``
+ re-filters the whole state dict for every child module. Keys without a matching tensor are left
+ untouched and unmatched entries are ignored (``strict=False``); a shape mismatch raises.
+ Falls back to ``load_state_dict`` when any submodule customizes loading (an overridden
+ ``_load_from_state_dict`` or ``set_extra_state``, load hooks) or swap-tensors loading is on,
+ because those need the stock per-module protocol.
+ """
+ modules = list(module.named_modules(remove_duplicate=False))
+ if torch.__future__.get_swap_module_params_on_conversion() or any(_customizes_loading(m) for _, m in modules):
+ module.load_state_dict(state_dict, strict=False, assign=True)
+ return
+
+ error_msgs: list[str] = []
+ for prefix, submodule in modules:
+ key_prefix = f"{prefix}." if prefix else ""
+ for name, param in submodule._parameters.items():
+ tensor = None if param is None else state_dict.get(key_prefix + name)
+ if tensor is None:
+ continue
+ tensor = _matching_shape(tensor, param, key_prefix + name, error_msgs)
+ if tensor is None:
+ continue
+ if isinstance(tensor, nn.Parameter):
+ tensor.requires_grad_(param.requires_grad)
+ else:
+ tensor = nn.Parameter(tensor, requires_grad=param.requires_grad)
+ submodule._parameters[name] = tensor
+ for name, buffer in submodule._buffers.items():
+ if buffer is None or name in submodule._non_persistent_buffers_set:
+ continue
+ tensor = state_dict.get(key_prefix + name)
+ if tensor is None:
+ continue
+ tensor = _matching_shape(tensor, buffer, key_prefix + name, error_msgs)
+ if tensor is not None:
+ submodule._buffers[name] = tensor
+ if error_msgs:
+ raise RuntimeError(f"Error(s) in loading state_dict for {type(module).__name__}:\n\t" + "\n\t".join(error_msgs))
+
+
+def _matching_shape(
+ tensor: torch.Tensor, current: torch.Tensor, key: str, error_msgs: list[str]
+) -> torch.Tensor | None:
+ """``tensor`` reshaped the way ``load_state_dict`` accepts it for ``current``, or ``None`` on a mismatch.
+ A lazy ``current`` (``UninitializedParameter`` / ``UninitializedBuffer``) has no shape yet and takes
+ ``tensor`` as is, as in ``load_state_dict``.
+ """
+ if is_lazy(current):
+ return tensor
+ if current.ndim == 0 and tensor.shape == (1,):
+ tensor = tensor[0]
+ if tensor.shape != current.shape:
+ error_msgs.append(
+ f"size mismatch for {key}: copying a param with shape {tensor.shape} from checkpoint, "
+ f"the shape in current model is {current.shape}."
+ )
+ return None
+ return tensor
+
+
+def module_tensors(model: nn.Module) -> Iterator[torch.Tensor]:
+ """Every parameter and buffer of ``model``, read straight off each module.
+ Unlike ``named_parameters`` / ``named_buffers`` there is no de-duplication, so a tensor
+ shared between modules is yielded once per owner.
+ """
+ for module in model.modules():
+ yield from (t for t in module._parameters.values() if t is not None)
+ yield from (t for t in module._buffers.values() if t is not None)
+
+
+def to_device(model: _M, device: torch.device) -> _M:
+ """``model.to(device)``, skipped when every parameter and buffer already lives on ``device``.
+ ``nn.Module.to`` rebuilds every parameter even when nothing moves, which costs a full walk of
+ a large model per call.
+ """
+ if device.type == "cuda" and device.index is None:
+ device = torch.device("cuda", torch.cuda.current_device())
+ if all(t.device == device for t in module_tensors(model)):
+ return model
+ return model.to(device)
+
+
def as_path_list(paths: _PathArg | tuple[_PathArg, ...] | list[_PathArg]) -> list[str]:
"""Normalize checkpoint path args to the ``list[str]`` shape registries key on."""
if isinstance(paths, (str, PathLike)):
@@ -51,6 +148,20 @@ def read_model_metadata(
return loader.metadata(first)
+class LazyModelMetadata:
+ """A checkpoint's parsed metadata, read on first access and shared by every builder copy holding it."""
+
+ def __init__(self, model_path: str | tuple[str, ...], loader: StateDictLoader) -> None:
+ self._model_path = model_path
+ self._loader = loader
+ self._metadata: dict | None = None
+
+ def get(self) -> dict:
+ if self._metadata is None:
+ self._metadata = read_model_metadata(self._model_path, self._loader)
+ return copy.deepcopy(self._metadata)
+
+
def read_model_config(
model_path: str | tuple[str, ...],
loader: StateDictLoader,
diff --git a/packages/ltx-core/src/ltx_core/loader/primitives.py b/packages/ltx-core/src/ltx_core/loader/primitives.py
index 8550bf3..9379c99 100644
--- a/packages/ltx-core/src/ltx_core/loader/primitives.py
+++ b/packages/ltx-core/src/ltx_core/loader/primitives.py
@@ -97,6 +97,11 @@ class ModelBuilderProtocol(BuilderProtocol[BuiltType], Protocol[BuiltType]):
"""Whether rebuilds rebind the same GPU weight storages (safe for CUDA-graph capture)."""
...
+ @property
+ def builds_per_shape(self) -> bool:
+ """Whether a built model serves only the latent shape it was built for (tiled data parallel)."""
+ ...
+
@property
def model_sd_ops(self) -> SDOps | None: ...
diff --git a/packages/ltx-core/src/ltx_core/loader/single_gpu_model_builder.py b/packages/ltx-core/src/ltx_core/loader/single_gpu_model_builder.py
index 58b577b..0d1d9bc 100644
--- a/packages/ltx-core/src/ltx_core/loader/single_gpu_model_builder.py
+++ b/packages/ltx-core/src/ltx_core/loader/single_gpu_model_builder.py
@@ -9,11 +9,13 @@ from torch import nn
from ltx_core.loader.fuse_loras import FuseRule, apply_loras, bf16_fuse_rule
from ltx_core.loader.helpers import (
+ LazyModelMetadata,
as_path_list,
+ assign_state_dict,
create_meta_model,
load_state_dict,
- read_model_config,
- read_model_metadata,
+ module_tensors,
+ to_device,
)
from ltx_core.loader.module_ops import ModuleOps
from ltx_core.loader.primitives import (
@@ -37,6 +39,8 @@ logger: logging.Logger = logging.getLogger(__name__)
def _check_uninitialized(model: nn.Module) -> list[str]:
"""Return names of any parameters/buffers still on meta device."""
+ if not any(t.is_meta for t in module_tensors(model)):
+ return []
names = []
for name, param in model.named_parameters():
if str(param.device) == "meta":
@@ -95,7 +99,7 @@ def _load_model_weights(
sd = model_sd.sd
if dtype is not None:
sd = _cast_floating_sd(sd, dtype)
- meta_model.load_state_dict(sd, strict=False, assign=True)
+ assign_state_dict(meta_model, sd)
return
lora_state_dicts = [load_state_dict([lora.path], loader, registry, lora_load_device, lora.sd_ops) for lora in loras]
@@ -116,7 +120,7 @@ def _load_model_weights(
fused_sd = final_sd.sd
if dtype is not None:
fused_sd = _cast_floating_sd(fused_sd, dtype)
- meta_model.load_state_dict(fused_sd, strict=False, assign=True)
+ assign_state_dict(meta_model, fused_sd)
class SingleGPUModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType], LoRAAdaptableProtocol):
@@ -164,6 +168,7 @@ class SingleGPUModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType],
self._registry = registry if registry is not None else ModelRegistry(cache_models=True, cache_weights=False)
self._lora_load_device = lora_load_device if lora_load_device is not None else torch.device("cpu")
self._fuse_rule = fuse_rule
+ self._metadata = LazyModelMetadata(model_path, self._model_loader)
@property
def model_sd_ops(self) -> SDOps | None:
@@ -186,6 +191,10 @@ class SingleGPUModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType],
# Registry may retain a CPU SD; each build H2Ds into fresh GPU storages.
return False
+ @property
+ def builds_per_shape(self) -> bool:
+ return False
+
@property
def model_path(self) -> str | tuple[str, ...]:
return self._model_path
@@ -242,10 +251,10 @@ class SingleGPUModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType],
return clone
def model_config(self) -> dict:
- return read_model_config(self._model_path, self._model_loader)
+ return self.model_metadata().get("config", {})
def model_metadata(self) -> dict:
- return read_model_metadata(self._model_path, self._model_loader)
+ return self._metadata.get()
def meta_model(self, metadata: dict, module_ops: tuple[ModuleOps, ...]) -> ModelType:
struct_key = module_registry_key(
@@ -291,4 +300,4 @@ class SingleGPUModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType],
if uninitialized:
logger.warning(f"Uninitialized parameters or buffers: {uninitialized}")
return meta_model
- return meta_model.to(device)
+ return to_device(meta_model, device)
diff --git a/packages/ltx-core/src/ltx_core/modality_tiling.py b/packages/ltx-core/src/ltx_core/modality_tiling.py
index 8d988ba..6ad6f28 100644
--- a/packages/ltx-core/src/ltx_core/modality_tiling.py
+++ b/packages/ltx-core/src/ltx_core/modality_tiling.py
@@ -1,7 +1,6 @@
"""Video modality tiling helpers.
-Provides :class:`VideoModalityTilingHelper` — a stateless helper that
-tiles and blends video :class:`Modality` token sequences by
-spatial/temporal region. Tile geometry is represented by the existing
+Provides :class:`VideoModalityTilingHelper`, which tiles and blends video
+:class:`Modality` token sequences by spatial/temporal region. Tile geometry is represented by the existing
:class:`Tile` NamedTuple from :mod:`ltx_core.tiling`; no distributed
primitives are required.
"""
@@ -9,12 +8,13 @@ primitives are required.
from __future__ import annotations
import logging
-from collections.abc import Sequence
-from dataclasses import dataclass, replace
+from collections.abc import Callable, Sequence
+from dataclasses import dataclass, field, replace
from functools import partial
import torch
+from ltx_core.devices import upload_from_host
from ltx_core.model.transformer.modality import Modality
from ltx_core.tiling import (
DimensionTilingConfig,
@@ -32,6 +32,22 @@ from ltx_core.types import VideoLatentShape
logger = logging.getLogger(__name__)
+@dataclass(frozen=True)
+class _TilePlan:
+ """One tile's token layout for one positions tensor; device tables reused at every step.
+ The layout is fixed once planned; only the per-``(dtype, device)`` weight caches fill in lazily.
+ """
+
+ keep_indices: torch.Tensor
+ gen_indices: torch.Tensor
+ cond_indices: torch.Tensor | None
+ num_tile_gen: int
+ cond_blend_weights: torch.Tensor | None
+ positions: torch.Tensor
+ blend_weights: dict[tuple[torch.dtype, torch.device], torch.Tensor] = field(default_factory=dict)
+ cond_weights: dict[tuple[torch.dtype, torch.device], torch.Tensor] = field(default_factory=dict)
+
+
@dataclass(frozen=True)
class TilingContext:
"""Opaque context produced by :meth:`VideoModalityTilingHelper.tile_modality`.
@@ -47,6 +63,7 @@ class TilingContext:
"""``(num_kept_cond,)`` — weight for each kept conditioning token,
equal to ``1 / num_tiles_that_keep_this_token``. ``None`` when
there are no conditioning tokens."""
+ plan: _TilePlan
def seam_split(
@@ -77,11 +94,17 @@ def seam_split(
class VideoModalityTilingHelper:
- """Stateless helper that tiles and blends video :class:`Modality` sequences.
+ """Tiles and blends video :class:`Modality` sequences.
Constructed once with a :class:`TileCountConfig` and
:class:`VideoLatentTools`. Tiles are computed at construction and
available via the :attr:`tiles` property. Use :meth:`tile_modality`
and :meth:`blend` with any tile from that list.
+ Every tile's token layout depends only on ``modality.positions``. The first
+ :meth:`tile_modality` call for a positions tensor plans all tiles at once (its
+ host syncs drain the GPU queue a single time, before the sequence's first
+ forward) and later calls with the same tensor -- a denoising loop carries it
+ unchanged across steps -- read those plans. The helper holds the tensor while
+ its plans are live, so the identity check cannot see a recycled object.
Passing ``seams`` (interior latent-frame indices) lets a temporal split land on those
cells instead of blended overlaps (:func:`seam_split`); leftover segments go to the
leading tiles. Missing or non-interior seams keep the blended split. A regular keyframe
@@ -117,6 +140,9 @@ class VideoModalityTilingHelper:
splitters=[frames, height, width],
mappers=[frames_mapper, identity_mapping_operation, identity_mapping_operation],
)
+ self._plan_positions: torch.Tensor | None = None
+ self._plan_num_total = -1
+ self._plans: dict[tuple[int, bool], _TilePlan] = {}
@property
def tiles(self) -> list[Tile]:
@@ -139,26 +165,8 @@ class VideoModalityTilingHelper:
A ``(tiled_modality, context)`` tuple. Pass *context* to
:meth:`blend` together with the model output.
"""
- device = modality.positions.device
- gen_indices = self._generated_token_indices(tile, device=device)
- num_total = modality.latent.shape[1]
-
- cond_blend_weights: torch.Tensor | None = None
- if num_total > self._num_generated_tokens:
- keep_per_tile_cond = self._all_tiles_cond_keep(modality) # (num_tiles, num_cond) bool
- tile_idx = next((i for i, t in enumerate(self._tiles) if t.in_coords == tile.in_coords), None)
- if tile_idx is None:
- raise RuntimeError(
- f"Tile with in_coords={tile.in_coords} is not in this helper's tile set; "
- f"pass a tile obtained from `helper.tiles`."
- )
- my_cond_keep = keep_per_tile_cond[tile_idx]
- cond_indices = self._num_generated_tokens + my_cond_keep.nonzero(as_tuple=False).squeeze(1)
- keep_indices = torch.cat([gen_indices, cond_indices])
- total_keepers = keep_per_tile_cond.sum(dim=0).float() # (num_cond,)
- cond_blend_weights = 1.0 / total_keepers[my_cond_keep]
- else:
- keep_indices = gen_indices
+ plan = self._plan(modality, tile, normalize_positions=normalize_positions)
+ keep_indices = plan.keep_indices
tile_attention_mask = None
if modality.attention_mask is not None:
@@ -171,24 +179,20 @@ class VideoModalityTilingHelper:
if modality.keyframes_mask is not None:
tile_keyframes_mask = modality.keyframes_mask[:, keep_indices]
- positions = modality.positions[:, :, keep_indices, :]
- if normalize_positions:
- num_tile_gen = self._tile_generated_token_count(tile)
- gen_pos = positions[:, :, :num_tile_gen, :] # (B, 3, num_tile_gen, 2)
- offset = gen_pos[..., 0].amin(dim=2, keepdim=True).unsqueeze(-1) # (B, 3, 1, 1)
- positions = positions - offset
-
tiled = replace(
modality,
latent=modality.latent[:, keep_indices, :],
- timesteps=modality.timesteps[:, keep_indices],
- positions=positions,
+ timesteps=modality.timesteps.select_tokens(keep_indices),
+ positions=plan.positions,
attention_mask=tile_attention_mask,
keyframes_mask=tile_keyframes_mask,
)
return tiled, TilingContext(
- keep_indices=keep_indices, num_total_tokens=num_total, cond_blend_weights=cond_blend_weights
+ keep_indices=keep_indices,
+ num_total_tokens=modality.latent.shape[1],
+ cond_blend_weights=plan.cond_blend_weights,
+ plan=plan,
)
# -- blend -------------------------------------------------------------
@@ -218,10 +222,8 @@ class VideoModalityTilingHelper:
The output tensor with the blended tile added at the correct
positions.
"""
+ plan = context.plan
batch, _, dim = tile_to_blend.shape
- num_tile_gen = self._tile_generated_token_count(tile)
- gen_indices = self._generated_token_indices(tile, device=tile_to_blend.device)
-
num_total_tokens = context.num_total_tokens
expected_shape = (batch, num_total_tokens, dim)
@@ -232,23 +234,89 @@ class VideoModalityTilingHelper:
else:
result = torch.zeros(*expected_shape, device=tile_to_blend.device, dtype=tile_to_blend.dtype)
- # Blend mask is (tile_F, tile_H, tile_W) — one weight per token in row-major order.
- blend_weights = tile.blend_mask.reshape(-1).to(device=tile_to_blend.device, dtype=tile_to_blend.dtype)
- tile_gen = tile_to_blend[:, :num_tile_gen, :] * blend_weights[None, :, None]
-
- result[:, gen_indices, :] += tile_gen
+ # Blend mask is (tile_F, tile_H, tile_W) — one weight per token in row-major order. The dense mask is
+ # built on the host only when the plan has no copy on this device and dtype yet.
+ blend_weights = _device_weights(plan.blend_weights, lambda: tile.blend_mask.reshape(-1), tile_to_blend)
+ result[:, plan.gen_indices, :] += tile_to_blend[:, : plan.num_tile_gen, :] * blend_weights[None, :, None]
# Scatter kept conditioning tokens, weighted by 1/N where N is
# the number of tiles that keep each token (so they sum to 1).
- if num_total_tokens > self._num_generated_tokens and context.cond_blend_weights is not None:
- cond_indices = context.keep_indices[context.keep_indices >= self._num_generated_tokens]
- weights = context.cond_blend_weights.to(device=tile_to_blend.device, dtype=tile_to_blend.dtype)
- result[:, cond_indices, :] += tile_to_blend[:, num_tile_gen:, :] * weights[None, :, None]
+ if num_total_tokens > self._num_generated_tokens and plan.cond_blend_weights is not None:
+ cond_blend_weights = plan.cond_blend_weights
+ weights = _device_weights(plan.cond_weights, lambda: cond_blend_weights, tile_to_blend)
+ result[:, plan.cond_indices, :] += tile_to_blend[:, plan.num_tile_gen :, :] * weights[None, :, None]
return result
# -- private -----------------------------------------------------------
+ def _plan(self, modality: Modality, tile: Tile, *, normalize_positions: bool) -> _TilePlan:
+ """*tile*'s plan for this sequence; a new positions tensor or token count plans every tile again."""
+ num_total = modality.latent.shape[1]
+ if modality.positions is not self._plan_positions or num_total != self._plan_num_total:
+ self._plan_positions = modality.positions
+ self._plan_num_total = num_total
+ self._plans = {}
+ tile_idx = next((i for i, t in enumerate(self._tiles) if t.in_coords == tile.in_coords), None)
+ if tile_idx is None:
+ raise RuntimeError(
+ f"Tile with in_coords={tile.in_coords} is not in this helper's tile set; "
+ f"pass a tile obtained from `helper.tiles`."
+ )
+ key = (tile_idx, normalize_positions)
+ if key not in self._plans:
+ # The one host read of a plan build: how many conditioning tokens each tile keeps is data dependent.
+ keep_per_tile_cond = None
+ cond_keepers = None
+ if num_total > self._num_generated_tokens:
+ keep_per_tile_cond = self._all_tiles_cond_keep(modality).cpu()
+ cond_keepers = keep_per_tile_cond.sum(dim=0).float() # (num_cond,)
+ for idx, t in enumerate(self._tiles):
+ cond_keep = keep_per_tile_cond[idx] if keep_per_tile_cond is not None else None
+ self._plans[(idx, normalize_positions)] = self._build_plan(
+ modality, t, normalize_positions, cond_keep, cond_keepers
+ )
+ return self._plans[key]
+
+ def _build_plan(
+ self,
+ modality: Modality,
+ tile: Tile,
+ normalize_positions: bool,
+ cond_keep: torch.Tensor | None,
+ cond_keepers: torch.Tensor | None,
+ ) -> _TilePlan:
+ """*tile*'s plan. ``cond_keep`` is the tile's ``(num_cond,)`` keep row, ``cond_keepers`` how many tiles
+ keep each conditioning token; both host tensors, ``None`` without conditioning tokens."""
+ device = modality.positions.device
+ gen_indices = self._generated_token_indices(tile, device=device)
+ cond_indices = None
+ cond_blend_weights = None
+ if cond_keep is not None and cond_keepers is not None:
+ # The keep row is on the host: the indices and weights are derived there and uploaded
+ # through pinned memory, so no step of the plan waits on the device.
+ host_cond_indices = self._num_generated_tokens + cond_keep.nonzero(as_tuple=False).squeeze(1)
+ cond_indices = upload_from_host(host_cond_indices, device)
+ keep_indices = torch.cat([gen_indices, cond_indices])
+ cond_blend_weights = 1.0 / cond_keepers[cond_keep]
+ else:
+ keep_indices = gen_indices
+
+ num_tile_gen = self._tile_generated_token_count(tile)
+ positions = modality.positions[:, :, keep_indices, :]
+ if normalize_positions:
+ gen_pos = positions[:, :, :num_tile_gen, :] # (B, 3, num_tile_gen, 2)
+ offset = gen_pos[..., 0].amin(dim=2, keepdim=True).unsqueeze(-1) # (B, 3, 1, 1)
+ positions = positions - offset
+ return _TilePlan(
+ keep_indices=keep_indices,
+ gen_indices=gen_indices,
+ cond_indices=cond_indices,
+ num_tile_gen=num_tile_gen,
+ cond_blend_weights=cond_blend_weights,
+ positions=positions,
+ )
+
def _tile_generated_token_count(self, tile: Tile) -> int:
"""Number of generated tokens in *tile*."""
frame_slice, height_slice, width_slice = tile.in_coords
@@ -300,3 +368,18 @@ class VideoModalityTilingHelper:
overlaps_all_dims = overlaps.all(dim=2) # (num_tiles, B, num_cond)
has_negative_time = (cond_positions[:, 0, :, 0] < 0)[None] # (1, B, num_cond)
return (overlaps_all_dims | has_negative_time).any(dim=1) # (num_tiles, num_cond)
+
+
+def _device_weights(
+ cache: dict[tuple[torch.dtype, torch.device], torch.Tensor],
+ weights: Callable[[], torch.Tensor],
+ like: torch.Tensor,
+) -> torch.Tensor:
+ """``weights()`` on ``like``'s device and dtype, built and uploaded once per plan through pinned memory."""
+ key = (like.dtype, like.device)
+ if key not in cache:
+ host = weights().to(dtype=like.dtype)
+ if like.device.type == "cuda" and host.device.type == "cpu":
+ host = host.pin_memory()
+ cache[key] = host.to(device=like.device, non_blocking=True)
+ return cache[key]
diff --git a/packages/ltx-core/src/ltx_core/model/disposable.py b/packages/ltx-core/src/ltx_core/model/disposable.py
index 496a734..9cfceab 100644
--- a/packages/ltx-core/src/ltx_core/model/disposable.py
+++ b/packages/ltx-core/src/ltx_core/model/disposable.py
@@ -24,19 +24,12 @@ class Disposable(DisposableProtocol):
if not isinstance(self, nn.Module):
raise TypeError(f"{type(self).__name__} must be an nn.Module to dispose")
- persistent = set(self.state_dict().keys())
- for name, param in list(self.named_parameters()):
- parent_name, _, attr = name.rpartition(".")
- parent = self.get_submodule(parent_name) if parent_name else self
- setattr(
- parent,
- attr,
- nn.Parameter(torch.empty_like(param, device="meta"), requires_grad=param.requires_grad),
- )
-
- for name, buf in list(self.named_buffers()):
- if name not in persistent:
- continue
- parent_name, _, attr = name.rpartition(".")
- parent = self.get_submodule(parent_name) if parent_name else self
- parent.register_buffer(attr, torch.empty_like(buf, device="meta"), persistent=True)
+ for module in self.modules():
+ for name, param in module._parameters.items():
+ if param is not None:
+ module._parameters[name] = nn.Parameter(
+ torch.empty_like(param, device="meta"), requires_grad=param.requires_grad
+ )
+ for name, buf in module._buffers.items():
+ if buf is not None and name not in module._non_persistent_buffers_set:
+ module._buffers[name] = torch.empty_like(buf, device="meta")
diff --git a/packages/ltx-core/src/ltx_core/model/transformer/__init__.py b/packages/ltx-core/src/ltx_core/model/transformer/__init__.py
index 7d2936b..bd811f6 100644
--- a/packages/ltx-core/src/ltx_core/model/transformer/__init__.py
+++ b/packages/ltx-core/src/ltx_core/model/transformer/__init__.py
@@ -1,6 +1,6 @@
"""Transformer model components."""
-from ltx_core.model.transformer.modality import Modality
+from ltx_core.model.transformer.modality import Modality, TokenTimesteps
from ltx_core.model.transformer.model import LTXModel, X0Model
from ltx_core.model.transformer.model_configurator import (
LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP,
@@ -18,5 +18,6 @@ __all__ = [
"LTXModelConfigurator",
"LTXVideoOnlyModelConfigurator",
"Modality",
+ "TokenTimesteps",
"X0Model",
]
diff --git a/packages/ltx-core/src/ltx_core/model/transformer/attention.py b/packages/ltx-core/src/ltx_core/model/transformer/attention.py
index 4d108c7..c8e7f1f 100644
--- a/packages/ltx-core/src/ltx_core/model/transformer/attention.py
+++ b/packages/ltx-core/src/ltx_core/model/transformer/attention.py
@@ -166,6 +166,13 @@ class FlashAttention3(AttentionCallable):
return out
+def _flash_attention_4_output(result: torch.Tensor | tuple[torch.Tensor, ...]) -> torch.Tensor:
+ """Normalize FA4 return value: tensor (current releases) or legacy (out, aux) tuple."""
+ if isinstance(result, tuple):
+ return result[0]
+ return result
+
+
class FlashAttention4(AttentionCallable):
label = "FlashAttention4"
@@ -186,7 +193,8 @@ class FlashAttention4(AttentionCallable):
q, k, v = (t.view(b, -1, heads, dim_head) for t in (q, k, v))
- out, _ = flash_attn_4_func(q.to(v.dtype), k.to(v.dtype), v)
+ result = flash_attn_4_func(q.to(v.dtype), k.to(v.dtype), v)
+ out = _flash_attention_4_output(result)
out = out.reshape(b, -1, heads * dim_head)
return out
@@ -360,8 +368,12 @@ class AttentionFunction(Enum):
Every non-AUTOMATIC variant raises :class:`RuntimeError` when the backend
isn't usable on this machine -- missing package or SDPA backend rejected
on this hardware (e.g. cuDNN under ``torch.use_deterministic_algorithms``).
- Opting in means "this kernel or fail loudly". ``AUTOMATIC`` returns the
- cached :func:`automatic_attention` instance so every build shares one callable.
+ Opting in means "this kernel or fail loudly", with one exception: under the
+ ``ltx_kernels_inductor`` and ``ltx_kernels_eager`` compile backends, an unmasked
+ call from ``PYTORCH`` or an ``SDPA_*`` variant runs ``ltx_kernels::sdpa_select``,
+ which picks cuDNN or flash attention per shape regardless of the pinned backend.
+ ``AUTOMATIC`` returns the cached :func:`automatic_attention` instance so every
+ build shares one callable.
"""
match self:
case AttentionFunction.AUTOMATIC:
diff --git a/packages/ltx-core/src/ltx_core/model/transformer/compiling.py b/packages/ltx-core/src/ltx_core/model/transformer/compiling.py
index aa4d2dd..4c1b863 100644
--- a/packages/ltx-core/src/ltx_core/model/transformer/compiling.py
+++ b/packages/ltx-core/src/ltx_core/model/transformer/compiling.py
@@ -11,7 +11,11 @@ from ltx_core.loader.module_ops import ModuleOps
from ltx_core.loader.sd_ops import SDOps
from ltx_core.model.transformer.cudagraph_capture import CudaGraphRunner
from ltx_core.model.transformer.model import LTXModel
-from ltx_core.model.transformer.transformer_args import BlockPerturbationsProcessor, TransformerArgs
+from ltx_core.model.transformer.transformer_args import (
+ AdaLNTimestepEmbedding,
+ BlockPerturbationsProcessor,
+ TransformerArgs,
+)
# Defaults applied inside the patched forward. Overriding via CompilationConfig
# replaces these wholesale; it does not merge.
@@ -105,20 +109,20 @@ class CompiledBlockPerturbationsProcessor(BlockPerturbationsProcessor):
torch._dynamo.mark_dynamic(args.self_attention_mask, 3)
if args.context_mask is not None:
torch._dynamo.mark_dynamic(args.context_mask, 2)
- # `timesteps` / `embedded_timestep` are per-token when conditioning sets a
- # per-position denoise mask, in which case their dim 1 equals the seq length
- # and must vary with it. When they're a single timestep broadcast across the
- # sequence (dim 1 == 1), leaving them static lets Dynamo keep the size-1
- # broadcast.
- if args.timesteps.shape[1] > 1:
- torch._dynamo.mark_dynamic(args.timesteps, 1)
- if args.embedded_timestep.shape[1] > 1:
- torch._dynamo.mark_dynamic(args.embedded_timestep, 1)
+ if args.timestep_embedding.token_indices is not None:
+ torch._dynamo.mark_dynamic(args.timestep_embedding.token_indices, 1)
+ # The table dimension is the sequence length for dense per-token conditioning
+ # and the candidate count for indexed conditioning. Both can vary between
+ # requests. A single row broadcasts and must remain static.
+ if args.timestep_embedding.modulation.shape[1] > 1:
+ torch._dynamo.mark_dynamic(args.timestep_embedding.modulation, 1)
+ if args.timestep_embedding.embedding.shape[1] > 1:
+ torch._dynamo.mark_dynamic(args.timestep_embedding.embedding, 1)
# `cross_scale_shift_timestep` is the cross-attn AdaLN scale/shift input
- # derived from the own-modality per-token timesteps (denoise_mask * sigma),
- # so its dim 1 equals the seq length when conditioning is per-token.
+ # derived from the own-modality timesteps, so its dim 1 is the sequence length
+ # for dense conditioning and the candidate count for indexed conditioning.
# `cross_gate_timestep` is the cross-modality sigma scalar -- dim 1 is 1
- # and broadcasts, leave it static. Same guard pattern as `timesteps`.
+ # and broadcasts, leave it static.
if args.cross_scale_shift_timestep is not None and args.cross_scale_shift_timestep.shape[1] > 1:
torch._dynamo.mark_dynamic(args.cross_scale_shift_timestep, 1)
@@ -161,11 +165,12 @@ class CompiledBlockPerturbationsProcessor(BlockPerturbationsProcessor):
def _register_transformer_args_pytree() -> None:
- """Register ``TransformerArgs`` as a pytree so :class:`CudaGraphRunner` can flatten it to
- tensor leaves for copy-in/out. Idempotent.
+ """Register ``TransformerArgs`` and its ``AdaLNTimestepEmbedding`` as pytrees so
+ :class:`CudaGraphRunner` can flatten them to tensor leaves for its capture key. Idempotent.
"""
- if TransformerArgs not in pytree.SUPPORTED_NODES:
- pytree.register_dataclass(TransformerArgs)
+ for node in (TransformerArgs, AdaLNTimestepEmbedding):
+ if node not in pytree.SUPPORTED_NODES:
+ pytree.register_dataclass(node)
@contextlib.contextmanager
diff --git a/packages/ltx-core/src/ltx_core/model/transformer/cudagraph_capture.py b/packages/ltx-core/src/ltx_core/model/transformer/cudagraph_capture.py
index 25f55c6..98dbd00 100644
--- a/packages/ltx-core/src/ltx_core/model/transformer/cudagraph_capture.py
+++ b/packages/ltx-core/src/ltx_core/model/transformer/cudagraph_capture.py
@@ -12,13 +12,17 @@ from __future__ import annotations
import dataclasses
import math
-from collections.abc import Callable, Hashable
+from collections.abc import Callable, Hashable, Iterator
+from contextlib import contextmanager
+from contextvars import ContextVar
from typing import TYPE_CHECKING
import torch
import torch.utils._pytree as pytree
from ltx_core.guidance.perturbations import BatchedPerturbationConfig
+from ltx_core.model.transformer.transformer_args import AdaLNTimestepEmbedding
+from ltx_core.types import HostDeviceTensorPair
if TYPE_CHECKING:
# torch imports this under TYPE_CHECKING too (a runtime top-level import cycles).
@@ -31,9 +35,25 @@ _BlockLoopFn = Callable[
"tuple[TransformerArgs | None, TransformerArgs | None]",
]
_ShapeKey = tuple[tuple[int, ...], ...]
-# One capture identity: input shapes plus the block's perturbation graph signature (what the block
-# specialises on), so a perturbed pass keys to -- and recaptures -- its own graph.
-_CaptureKey = tuple[_ShapeKey, Hashable]
+# One capture identity: input shapes, the block's perturbation graph signature (what the block
+# specialises on), so a perturbed pass keys to -- and recaptures -- its own graph, and the
+# enclosing `capture_key_scope`'s key.
+_CaptureKey = tuple[_ShapeKey, Hashable, Hashable]
+
+_scope_key: ContextVar[Hashable] = ContextVar("cudagraph_capture_scope_key", default=None)
+
+
+@contextmanager
+def capture_key_scope(key: Hashable) -> Iterator[None]:
+ """Add ``key`` to the capture identity of every :class:`CudaGraphRunner` call inside the scope.
+ For host state the block loop branches on that its input shapes do not carry, such as the
+ sequence-parallel padding layout: calls under different keys capture and replay their own graphs.
+ """
+ token = _scope_key.set(key)
+ try:
+ yield
+ finally:
+ _scope_key.reset(token)
@dataclasses.dataclass
@@ -132,6 +152,14 @@ class _InputBufferPool:
return static
if isinstance(value, tuple):
return tuple(self.copy_value_to_buffer(f"{field}.{i}", t, tokens) for i, t in enumerate(value))
+ if isinstance(value, AdaLNTimestepEmbedding):
+ return dataclasses.replace(
+ value,
+ **{
+ f.name: self.copy_value_to_buffer(f"{field}.{f.name}", getattr(value, f.name), tokens)
+ for f in dataclasses.fields(value)
+ },
+ )
return value
def copy_args_to_buffers(self, args: TransformerArgs) -> TransformerArgs:
@@ -153,7 +181,9 @@ class _InputBufferPool:
def copy_perturbations_to_buffers(self, perturbations: BatchedPerturbationConfig) -> BatchedPerturbationConfig:
static_block_masks = self.copy_value_to_buffer("block_masks", perturbations.block_masks, tokens=None)
assert isinstance(static_block_masks, torch.Tensor)
- return BatchedPerturbationConfig.from_masks(static_block_masks, perturbations.block_masks_cpu)
+ return BatchedPerturbationConfig.from_masks(
+ HostDeviceTensorPair(host_tensor=perturbations.block_masks_cpu, device_tensor=static_block_masks)
+ )
class CudaGraphRunner:
@@ -227,11 +257,12 @@ class CudaGraphRunner:
def _cache_key(
self, video: TransformerArgs | None, audio: TransformerArgs | None, perturbations: BatchedPerturbationConfig
) -> _CaptureKey:
- """Capture identity: input shapes + the block's perturbation graph signature. Keying on the
- signature makes a pass recapture exactly when it would recompile (and no more) -- the perturbed
- pass no longer silently replays the clean same-shape graph."""
+ """Capture identity: input shapes + the block's perturbation graph signature + the enclosing
+ :func:`capture_key_scope` key. Keying on the signature makes a pass recapture exactly when it
+ would recompile (and no more) -- the perturbed pass no longer silently replays the clean
+ same-shape graph."""
shape = _shape_key(video, audio, perturbations.block_masks)
- return shape, self._block_input_processor.graph_signature(perturbations)
+ return shape, self._block_input_processor.graph_signature(perturbations), _scope_key.get()
@staticmethod
def _configure_shared_input_pool(pool: _InputBufferPool | None, max_tokens: int) -> _InputBufferPool:
diff --git a/packages/ltx-core/src/ltx_core/model/transformer/modality.py b/packages/ltx-core/src/ltx_core/model/transformer/modality.py
index 872307d..ec44fcd 100644
--- a/packages/ltx-core/src/ltx_core/model/transformer/modality.py
+++ b/packages/ltx-core/src/ltx_core/model/transformer/modality.py
@@ -6,6 +6,56 @@ from dataclasses import dataclass
import torch
+@dataclass(frozen=True)
+class TokenTimesteps:
+ """Timestep values and the optional per-token index into their compact table.
+ ``values`` is ``(B, T)`` when dense or ``(B, K)`` when ``indices`` supplies
+ the ``(B, T)`` token layout. ``zero_index`` is the column of a compact table known, on the
+ host, to hold timestep zero in every row, or ``None`` when the table has no such column.
+ """
+
+ values: torch.Tensor
+ indices: torch.Tensor | None = None
+ zero_index: int | None = None
+
+ def per_token(self) -> torch.Tensor:
+ """The timestep of each token, with a trailing singleton dimension."""
+ if self.indices is None:
+ return self.values.unsqueeze(-1) if self.values.ndim == 2 else self.values
+ return torch.gather(self.values, 1, self.indices.to(torch.int64)).unsqueeze(-1)
+
+ def select_tokens(self, index: slice | torch.Tensor) -> TokenTimesteps:
+ if self.indices is None:
+ return TokenTimesteps(self.values[:, index])
+ return TokenTimesteps(self.values, self.indices[:, index], self.zero_index)
+
+ def pad_tokens(self, pad: int) -> TokenTimesteps:
+ """Append ``pad`` tokens with timestep zero.
+ Padded tokens stay visible as keys to video-to-audio cross-attention, so their timestep
+ reaches real outputs and must match dense padding. An indexed table keeps its shape when
+ ``zero_index`` names a zero column, and the padded tokens point at it; otherwise it gains
+ one zero column, which becomes ``zero_index``.
+ """
+ if self.indices is None:
+ pad_shape = list(self.values.shape)
+ pad_shape[1] = pad
+ return TokenTimesteps(torch.cat([self.values, self.values.new_zeros(pad_shape)], dim=1))
+ values, zero_index = self.values, self.zero_index
+ if zero_index is None:
+ values = torch.cat([values, values.new_zeros(values.shape[0], 1)], dim=1)
+ zero_index = self.values.shape[1]
+ pad_indices = self.indices.new_full((self.indices.shape[0], pad), zero_index)
+ return TokenTimesteps(values, torch.cat([self.indices, pad_indices], dim=1), zero_index)
+
+ def split(self, sizes: list[int]) -> list[TokenTimesteps]:
+ values = self.values.split(sizes, dim=0)
+ indices = self.indices.split(sizes, dim=0) if self.indices is not None else [None] * len(sizes)
+ return [
+ TokenTimesteps(chunk_values, chunk_indices, self.zero_index)
+ for chunk_values, chunk_indices in zip(values, indices, strict=True)
+ ]
+
+
@dataclass(frozen=True)
class Modality:
"""
@@ -16,7 +66,7 @@ class Modality:
latent: Patchified latent tokens, shape ``(B, T, D)`` where *B* is
the batch size, *T* is the total number of tokens (noisy +
conditioning), and *D* is the input dimension.
- timesteps: Per-token timestep embeddings, shape ``(B, T)``.
+ timesteps: Per-token timestep values or a compact table and its token indices.
positions: Per-token patch coordinates used to build the RoPE
frequencies. With the default ``use_middle_indices_grid=True``,
shape is ``(B, n_pos_dims, T, 2)`` where ``n_pos_dims=3`` for
@@ -38,8 +88,8 @@ class Modality:
attention. ``None`` means unrestricted (full) attention between
all tokens. Built incrementally by conditioning items; see
:class:`~ltx_core.conditioning.types.attention_strength_wrapper.ConditioningItemAttentionStrengthWrapper`.
- keyframes_mask: Optional per-token marker, shape ``(B, T, 1)`` -- the same layout as
- ``timesteps`` -- non-zero for tokens whose latent encodes a *single standalone pixel
+ keyframes_mask: Optional per-token marker, shape ``(B, T, 1)`` -- aligned with the latent
+ token axis -- non-zero for tokens whose latent encodes a *single standalone pixel
frame* rather than the usual multi-frame span. That is the target's first latent frame
(the video encoder is causal) plus any generated keyframe slots. Selects the tokens
that receive the model's learned keyframe absolute-position embedding. ``None`` means
@@ -51,7 +101,7 @@ class Modality:
torch.Tensor
) # Shape: (B, T, D) where B is the batch size, T is the number of tokens, and D is input dimension
sigma: torch.Tensor # Shape: (B,). Current sigma value, used for cross-attention timestep calculation.
- timesteps: torch.Tensor # Shape: (B, T) where T is the number of timesteps
+ timesteps: TokenTimesteps
# Shape: (B, n_pos_dims, T, 2) by default (use_middle_indices_grid=True);
# n_pos_dims=3 for video, 1 for audio; last dim holds [start, end) patch bounds.
# Legacy form (B, n_pos_dims, T) when use_middle_indices_grid=False.
@@ -65,11 +115,13 @@ class Modality:
def split(self, sizes: list[int]) -> list[Modality]:
"""Split along the batch dimension into chunks of the given sizes."""
n = len(sizes)
- split_fields: dict[str, list[torch.Tensor | None] | list[bool]] = {}
+ split_fields: dict[str, list[torch.Tensor | TokenTimesteps | None] | list[bool]] = {}
for f in dataclasses.fields(self):
value = getattr(self, f.name)
if isinstance(value, torch.Tensor):
split_fields[f.name] = list(value.split(sizes, dim=0))
+ elif isinstance(value, TokenTimesteps):
+ split_fields[f.name] = value.split(sizes)
elif value is None or isinstance(value, bool):
split_fields[f.name] = [value] * n
else:
diff --git a/packages/ltx-core/src/ltx_core/model/transformer/model.py b/packages/ltx-core/src/ltx_core/model/transformer/model.py
index 9783d5b..5def3b9 100644
--- a/packages/ltx-core/src/ltx_core/model/transformer/model.py
+++ b/packages/ltx-core/src/ltx_core/model/transformer/model.py
@@ -12,6 +12,7 @@ from ltx_core.model.transformer.modality import Modality
from ltx_core.model.transformer.rope import LTXRopeType
from ltx_core.model.transformer.transformer import BasicAVTransformerBlock, TransformerConfig
from ltx_core.model.transformer.transformer_args import (
+ AdaLNTimestepEmbedding,
BlockPerturbationsProcessor,
MultiModalTransformerArgsPreprocessor,
TransformerArgs,
@@ -471,9 +472,10 @@ class LTXModel(torch.nn.Module, Disposable):
norm_out: torch.nn.LayerNorm,
proj_out: torch.nn.Linear,
x: torch.Tensor,
- embedded_timestep: torch.Tensor,
+ timestep_embedding: AdaLNTimestepEmbedding,
) -> torch.Tensor:
"""Process output for LTXV."""
+ embedded_timestep = timestep_embedding.per_token_embedding()
# Apply scale-shift modulation
scale_shift_values = (
scale_shift_table[None, None].to(device=x.device, dtype=x.dtype) + embedded_timestep[:, :, None]
@@ -515,7 +517,11 @@ class LTXModel(torch.nn.Module, Disposable):
# Process output
vx = (
self._process_output(
- self.scale_shift_table, self.norm_out, self.proj_out, video_out.x, video_out.embedded_timestep
+ self.scale_shift_table,
+ self.norm_out,
+ self.proj_out,
+ video_out.x,
+ video_out.timestep_embedding,
)
if video_out is not None
else None
@@ -526,7 +532,7 @@ class LTXModel(torch.nn.Module, Disposable):
self.audio_norm_out,
self.audio_proj_out,
audio_out.x,
- audio_out.embedded_timestep,
+ audio_out.timestep_embedding,
)
if audio_out is not None
else None
@@ -595,6 +601,6 @@ class X0Model(torch.nn.Module, Disposable):
Denoised video and audio
"""
vx, ax = self.velocity_model(video, audio, perturbations)
- denoised_video = to_denoised(video.latent, vx, video.timesteps) if vx is not None else None
- denoised_audio = to_denoised(audio.latent, ax, audio.timesteps) if ax is not None else None
+ denoised_video = to_denoised(video.latent, vx, video.timesteps.per_token()) if vx is not None else None
+ denoised_audio = to_denoised(audio.latent, ax, audio.timesteps.per_token()) if ax is not None else None
return denoised_video, denoised_audio
diff --git a/packages/ltx-core/src/ltx_core/model/transformer/transformer.py b/packages/ltx-core/src/ltx_core/model/transformer/transformer.py
index 02ab32d..4d5983c 100644
--- a/packages/ltx-core/src/ltx_core/model/transformer/transformer.py
+++ b/packages/ltx-core/src/ltx_core/model/transformer/transformer.py
@@ -6,7 +6,7 @@ from ltx_core.model.transformer.adaln import adaln_embedding_coefficient
from ltx_core.model.transformer.attention import Attention, AttentionCallable, AttentionOps
from ltx_core.model.transformer.feed_forward import FeedForward
from ltx_core.model.transformer.rope import LTXRopeType
-from ltx_core.model.transformer.transformer_args import TransformerArgs
+from ltx_core.model.transformer.transformer_args import AdaLNTimestepEmbedding, TransformerArgs
from ltx_core.utils import rms_norm
@@ -124,31 +124,45 @@ class BasicAVTransformerBlock(torch.nn.Module):
self.norm_eps = norm_eps
def get_ada_values(
- self, scale_shift_table: torch.Tensor, batch_size: int, timestep: torch.Tensor, indices: slice
+ self,
+ scale_shift_table: torch.Tensor,
+ batch_size: int,
+ timestep_embedding: AdaLNTimestepEmbedding,
+ indices: slice,
+ modulation: torch.Tensor | None = None,
) -> tuple[torch.Tensor, ...]:
num_ada_params = scale_shift_table.shape[0]
-
- ada_values = (
- scale_shift_table[indices].unsqueeze(0).unsqueeze(0).to(device=timestep.device, dtype=timestep.dtype)
- + timestep.reshape(batch_size, timestep.shape[1], num_ada_params, -1)[:, :, indices, :]
+ hidden_dim = scale_shift_table.shape[1]
+ modulation = timestep_embedding.modulation if modulation is None else modulation
+ modulation = modulation.reshape(batch_size, -1, num_ada_params, hidden_dim)[:, :, indices, :]
+ modulation = timestep_embedding.gather(modulation)
+ return (
+ scale_shift_table[indices].unsqueeze(0).unsqueeze(0).to(device=modulation.device, dtype=modulation.dtype)
+ + modulation
).unbind(dim=2)
- return ada_values
def get_av_ca_ada_values(
self,
scale_shift_table: torch.Tensor,
batch_size: int,
+ timestep_embedding: AdaLNTimestepEmbedding,
scale_shift_timestep: torch.Tensor,
gate_timestep: torch.Tensor,
scale_shift_indices: slice,
num_scale_shift_values: int = 4,
) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
scale_shift_ada_values = self.get_ada_values(
- scale_shift_table[:num_scale_shift_values, :], batch_size, scale_shift_timestep, scale_shift_indices
+ scale_shift_table[:num_scale_shift_values, :],
+ batch_size,
+ timestep_embedding,
+ scale_shift_indices,
+ scale_shift_timestep,
)
- gate_ada_values = self.get_ada_values(
- scale_shift_table[num_scale_shift_values:, :], batch_size, gate_timestep, slice(None, None)
+ gate_table = scale_shift_table[num_scale_shift_values:, :].to(
+ device=gate_timestep.device, dtype=gate_timestep.dtype
)
+ gate_modulation = gate_timestep.reshape(batch_size, -1, 1, gate_table.shape[-1])
+ gate_ada_values = (gate_table[None, None] + gate_modulation).unbind(dim=2)
scale, shift = (t.squeeze(2) for t in scale_shift_ada_values)
(gate,) = (t.squeeze(2) for t in gate_ada_values)
@@ -162,7 +176,7 @@ class BasicAVTransformerBlock(torch.nn.Module):
attn: AttentionCallable,
scale_shift_table: torch.Tensor,
prompt_scale_shift_table: torch.Tensor | None,
- timestep: torch.Tensor,
+ timestep_embedding: AdaLNTimestepEmbedding,
prompt_timestep: torch.Tensor | None,
context_mask: torch.Tensor | None,
cross_attention_adaln: bool = False,
@@ -172,7 +186,10 @@ class BasicAVTransformerBlock(torch.nn.Module):
the PostSA composite); this method does not normalize again.
"""
if cross_attention_adaln:
- shift_q, scale_q, gate = self.get_ada_values(scale_shift_table, x_normed.shape[0], timestep, slice(6, 9))
+ shift_q, scale_q = self.get_ada_values(
+ scale_shift_table, x_normed.shape[0], timestep_embedding, slice(6, 8)
+ )
+ (gate,) = self.get_ada_values(scale_shift_table, x_normed.shape[0], timestep_embedding, slice(8, 9))
return apply_cross_attention_adaln(
x_normed,
context,
@@ -205,7 +222,7 @@ class BasicAVTransformerBlock(torch.nn.Module):
if run_vx:
vshift_msa, vscale_msa, vgate_msa = self.get_ada_values(
- self.scale_shift_table, vx.shape[0], video.timesteps, slice(0, 3)
+ self.scale_shift_table, vx.shape[0], video.timestep_embedding, slice(0, 3)
)
norm_vx = rms_norm(vx, eps=self.norm_eps) * (1 + vscale_msa) + vshift_msa
del vshift_msa, vscale_msa
@@ -226,7 +243,7 @@ class BasicAVTransformerBlock(torch.nn.Module):
self.attn2,
self.scale_shift_table,
getattr(self, "prompt_scale_shift_table", None),
- video.timesteps,
+ video.timestep_embedding,
video.prompt_timestep,
video.context_mask,
cross_attention_adaln=self.cross_attention_adaln,
@@ -235,7 +252,7 @@ class BasicAVTransformerBlock(torch.nn.Module):
if run_ax:
ashift_msa, ascale_msa, agate_msa = self.get_ada_values(
- self.audio_scale_shift_table, ax.shape[0], audio.timesteps, slice(0, 3)
+ self.audio_scale_shift_table, ax.shape[0], audio.timestep_embedding, slice(0, 3)
)
norm_ax = rms_norm(ax, eps=self.norm_eps) * (1 + ascale_msa) + ashift_msa
@@ -256,7 +273,7 @@ class BasicAVTransformerBlock(torch.nn.Module):
self.audio_attn2,
self.audio_scale_shift_table,
getattr(self, "audio_prompt_scale_shift_table", None),
- audio.timesteps,
+ audio.timestep_embedding,
audio.prompt_timestep,
audio.context_mask,
cross_attention_adaln=self.cross_attention_adaln,
@@ -273,6 +290,7 @@ class BasicAVTransformerBlock(torch.nn.Module):
scale_ca_video_a2v, shift_ca_video_a2v, gate_out_a2v = self.get_av_ca_ada_values(
self.scale_shift_table_a2v_ca_video,
vx.shape[0],
+ video.timestep_embedding,
video.cross_scale_shift_timestep,
video.cross_gate_timestep,
slice(0, 2),
@@ -283,6 +301,7 @@ class BasicAVTransformerBlock(torch.nn.Module):
scale_ca_audio_a2v, shift_ca_audio_a2v, _ = self.get_av_ca_ada_values(
self.scale_shift_table_a2v_ca_audio,
ax.shape[0],
+ audio.timestep_embedding,
audio.cross_scale_shift_timestep,
audio.cross_gate_timestep,
slice(0, 2),
@@ -305,6 +324,7 @@ class BasicAVTransformerBlock(torch.nn.Module):
scale_ca_audio_v2a, shift_ca_audio_v2a, gate_out_v2a = self.get_av_ca_ada_values(
self.scale_shift_table_a2v_ca_audio,
ax.shape[0],
+ audio.timestep_embedding,
audio.cross_scale_shift_timestep,
audio.cross_gate_timestep,
slice(2, 4),
@@ -314,6 +334,7 @@ class BasicAVTransformerBlock(torch.nn.Module):
scale_ca_video_v2a, shift_ca_video_v2a, _ = self.get_av_ca_ada_values(
self.scale_shift_table_a2v_ca_video,
vx.shape[0],
+ video.timestep_embedding,
video.cross_scale_shift_timestep,
video.cross_gate_timestep,
slice(2, 4),
@@ -335,7 +356,7 @@ class BasicAVTransformerBlock(torch.nn.Module):
if run_vx:
vshift_mlp, vscale_mlp, vgate_mlp = self.get_ada_values(
- self.scale_shift_table, vx.shape[0], video.timesteps, slice(3, 6)
+ self.scale_shift_table, vx.shape[0], video.timestep_embedding, slice(3, 6)
)
vx_scaled = rms_norm(vx, eps=self.norm_eps) * (1 + vscale_mlp) + vshift_mlp
vx = vx + self.ff(vx_scaled) * vgate_mlp
@@ -344,7 +365,7 @@ class BasicAVTransformerBlock(torch.nn.Module):
if run_ax:
ashift_mlp, ascale_mlp, agate_mlp = self.get_ada_values(
- self.audio_scale_shift_table, ax.shape[0], audio.timesteps, slice(3, 6)
+ self.audio_scale_shift_table, ax.shape[0], audio.timestep_embedding, slice(3, 6)
)
ax_scaled = rms_norm(ax, eps=self.norm_eps) * (1 + ascale_mlp) + ashift_mlp
ax = ax + self.audio_ff(ax_scaled) * agate_mlp
diff --git a/packages/ltx-core/src/ltx_core/model/transformer/transformer_args.py b/packages/ltx-core/src/ltx_core/model/transformer/transformer_args.py
index 566bbd3..f5ff226 100644
--- a/packages/ltx-core/src/ltx_core/model/transformer/transformer_args.py
+++ b/packages/ltx-core/src/ltx_core/model/transformer/transformer_args.py
@@ -43,13 +43,35 @@ def apply_keyframes_absolute_embedding(
return hidden_states + mask * embedding.to(dtype=hidden_states.dtype)
+@dataclass(frozen=True)
+class AdaLNTimestepEmbedding:
+ """Projected AdaLN timesteps and the token index that selects their rows.
+ Transformer blocks consume ``modulation`` and carry ``embedding`` to the model
+ output head, which applies the final scale and shift.
+ Compact-table indices have the batch offset folded in and address the first two
+ dimensions of any projected table passed to ``gather``.
+ """
+
+ modulation: torch.Tensor
+ embedding: torch.Tensor
+ token_indices: torch.Tensor | None = None
+
+ def gather(self, values: torch.Tensor) -> torch.Tensor:
+ """Expand a compact table to one row per token."""
+ if self.token_indices is None:
+ return values
+ return values.reshape(-1, *values.shape[2:])[self.token_indices]
+
+ def per_token_embedding(self) -> torch.Tensor:
+ return self.gather(self.embedding)
+
+
@dataclass(frozen=True)
class TransformerArgs:
x: torch.Tensor
context: torch.Tensor
context_mask: torch.Tensor
- timesteps: torch.Tensor
- embedded_timestep: torch.Tensor
+ timestep_embedding: AdaLNTimestepEmbedding
positional_embeddings: tuple[torch.Tensor, torch.Tensor]
cross_positional_embeddings: tuple[torch.Tensor, torch.Tensor] | None
cross_scale_shift_timestep: torch.Tensor | None
@@ -226,14 +248,13 @@ class TransformerArgsPreprocessor:
# Convert [0, 1] attention mask to additive log-space bias:
# 1.0 -> log(1.0) = 0.0 (no bias, full attention)
# 0.0 -> finfo.min (fully masked)
+ # Must stay elementwise: boolean indexing would sync the host on every forward.
finfo = torch.finfo(x_dtype)
- eps = finfo.tiny
-
- bias = torch.full_like(attention_mask, finfo.min, dtype=x_dtype)
- positive = attention_mask > 0
- if positive.any():
- bias[positive] = torch.log(attention_mask[positive].clamp(min=eps)).to(x_dtype)
-
+ bias = torch.where(
+ attention_mask > 0,
+ torch.log(attention_mask.clamp(min=finfo.tiny)).to(x_dtype),
+ torch.full_like(attention_mask, finfo.min, dtype=x_dtype),
+ )
return bias.unsqueeze(1) # (B, 1, T_q, T_k) for head broadcast
def _prepare_positional_embeddings(
@@ -269,8 +290,15 @@ class TransformerArgsPreprocessor:
x = apply_keyframes_absolute_embedding(x, modality.keyframes_mask, self.keyframes_embedding_provider)
batch_size = x.shape[0]
timestep, embedded_timestep = self._prepare_timestep(
- modality.timesteps, self.adaln, batch_size, modality.latent.dtype
+ modality.timesteps.values, self.adaln, batch_size, modality.latent.dtype
)
+ timestep_indices = modality.timesteps.indices
+ if timestep_indices is not None:
+ # Fold the batch into the index so a consumer resolves batch and value with one
+ # gather over the table flattened to (B * K, ...). Every consumer of
+ # `AdaLNTimestepEmbedding` reads them in that flattened form.
+ batch_offsets = torch.arange(batch_size, device=x.device, dtype=torch.int32)
+ timestep_indices = timestep_indices + batch_offsets[:, None] * timestep.shape[1]
prompt_timestep = None
if self.prompt_adaln is not None:
prompt_timestep, _ = self._prepare_timestep(
@@ -291,8 +319,7 @@ class TransformerArgsPreprocessor:
x=x,
context=context,
context_mask=attention_mask,
- timesteps=timestep,
- embedded_timestep=embedded_timestep,
+ timestep_embedding=AdaLNTimestepEmbedding(timestep, embedded_timestep, timestep_indices),
positional_embeddings=pe,
cross_positional_embeddings=None,
cross_scale_shift_timestep=None,
@@ -356,7 +383,7 @@ class MultiModalTransformerArgsPreprocessor:
return transformer_args
if cross_modality.sigma.numel() > 1:
- if cross_modality.sigma.shape[0] != modality.timesteps.shape[0]:
+ if cross_modality.sigma.shape[0] != modality.timesteps.values.shape[0]:
raise ValueError("Cross modality sigma must have the same batch size as the modality")
if cross_modality.sigma.ndim != 1:
raise ValueError("Cross modality sigma must be a 1D tensor")
@@ -371,7 +398,7 @@ class MultiModalTransformerArgsPreprocessor:
)
cross_scale_shift_timestep, cross_gate_timestep = self._prepare_cross_attention_timestep(
- modality_timesteps=modality.timesteps,
+ modality_timesteps=modality.timesteps.values,
cross_modality_sigma=cross_modality.sigma,
timestep_scale_multiplier=self.simple_preprocessor.timestep_scale_multiplier,
batch_size=transformer_args.x.shape[0],
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/convolution.py b/packages/ltx-core/src/ltx_core/model/video_vae/convolution.py
index 7c3957a..6003e3e 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/convolution.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/convolution.py
@@ -8,6 +8,25 @@ from torch.nn import functional as F
from ltx_core.model.video_vae.enums import PaddingModeType
+def memory_format_of(t: torch.Tensor, prefer_channels_last_3d: bool = False) -> torch.memory_format:
+ """Pick the memory format for a workspace allocation.
+ When ``prefer_channels_last_3d`` is True and ``t`` is 5D, return
+ ``channels_last_3d`` regardless of ``t``'s current strides -- the
+ workspace's ``.copy_(t)`` will transcribe the data into the new layout.
+ This is needed because intermediate tensors inside the memory-efficient decoder (after
+ ``rearrange`` + slice + residual add in ``_upsample_forward_efficient``)
+ are not NHWC-contiguous, so an auto-detect helper would silently fall
+ back to NCHW for every workspace after the first upsample.
+ Otherwise fall back to inspecting ``t``: ``channels_last_3d`` if ``t``
+ already uses it, else contiguous.
+ """
+ if prefer_channels_last_3d and t.dim() == 5:
+ return torch.channels_last_3d
+ if t.dim() == 5 and t.is_contiguous(memory_format=torch.channels_last_3d):
+ return torch.channels_last_3d
+ return torch.contiguous_format
+
+
def make_conv_nd( # noqa: PLR0913
dims: Union[int, Tuple[int, int]],
in_channels: int,
@@ -302,15 +321,23 @@ class CausalConv3d(nn.Module):
)
def forward(self, x: torch.Tensor, causal: bool = True) -> torch.Tensor:
- if causal:
- first_frame_pad = x[:, :, :1, :, :].repeat((1, 1, self.time_kernel_size - 1, 1, 1))
- x = torch.concatenate((first_frame_pad, x), dim=2)
- else:
- first_frame_pad = x[:, :, :1, :, :].repeat((1, 1, (self.time_kernel_size - 1) // 2, 1, 1))
- last_frame_pad = x[:, :, -1:, :, :].repeat((1, 1, (self.time_kernel_size - 1) // 2, 1, 1))
- x = torch.concatenate((first_frame_pad, x, last_frame_pad), dim=2)
- x = self.conv(x)
- return x
+ front = self.time_kernel_size - 1 if causal else (self.time_kernel_size - 1) // 2
+ back = 0 if causal else front
+ if front == 0 and back == 0:
+ return self.conv(x)
+ # Edge frames are replicated into one buffer laid out like ``x``, so channels-last
+ # activations reach the conv without a layout round trip.
+ frames = x.shape[2]
+ padded = torch.empty(
+ (*x.shape[:2], front + frames + back, *x.shape[3:]),
+ dtype=x.dtype,
+ device=x.device,
+ memory_format=memory_format_of(x),
+ )
+ padded[:, :, front : front + frames] = x
+ padded[:, :, :front] = x[:, :, :1]
+ padded[:, :, front + frames :] = x[:, :, -1:]
+ return self.conv(padded)
@property
def weight(self) -> torch.Tensor:
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/decode_workspace.py b/packages/ltx-core/src/ltx_core/model/video_vae/decode_workspace.py
new file mode 100644
index 0000000..51b20e5
--- /dev/null
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/decode_workspace.py
@@ -0,0 +1,91 @@
+"""The DiffVAE decoder's view of the per-decode buffer workspace (:mod:`ltx_kernels.vae.workspace`).
+The workspace exists only with ``ltx-kernels`` installed, the fused stage-5 blocks it serves; without it
+every helper here is a plain allocation and :func:`tile_workspace` returns ``None``.
+"""
+
+from __future__ import annotations
+
+import contextlib
+from functools import cache
+from types import ModuleType
+
+import torch
+
+
+@cache
+def _workspace_module() -> ModuleType | None:
+ try:
+ from ltx_kernels.vae import workspace # noqa: PLC0415
+ except ImportError:
+ return None
+ return workspace
+
+
+def available() -> bool:
+ """Whether ``ltx-kernels`` (whose fused blocks the workspace serves) is importable."""
+ return _workspace_module() is not None
+
+
+def out_slack_rows() -> int:
+ """Rows a fused-block output buffer carries past its volume (``ltx_kernels.vae.OUT_SLACK_ROWS``)."""
+ from ltx_kernels.vae import OUT_SLACK_ROWS # noqa: PLC0415
+
+ return int(OUT_SLACK_ROWS)
+
+
+def workspace_active() -> bool:
+ """Whether a decode workspace is active on this thread."""
+ ws = _workspace_module()
+ return ws is not None and ws.active_decode_workspace() is not None
+
+
+def volume(shape: tuple[int, ...], *, dtype: torch.dtype, device: torch.device) -> torch.Tensor:
+ """``torch.empty(shape)`` leased from the active workspace until its tile ends."""
+ ws = _workspace_module()
+ if ws is None:
+ return torch.empty(shape, dtype=dtype, device=device)
+ return ws.workspace_volume(shape, dtype=dtype, device=device)
+
+
+def phase_empty(
+ shape: tuple[int, ...],
+ *,
+ dtype: torch.dtype,
+ device: torch.device,
+ not_aliasing: tuple[torch.Tensor, ...] = (),
+) -> torch.Tensor:
+ """``torch.empty(shape)`` carved from the active workspace's idle volumes (a plain allocation otherwise)."""
+ ws = _workspace_module()
+ if ws is None:
+ return torch.empty(shape, dtype=dtype, device=device)
+ return ws.workspace_phase_empty(shape, dtype=dtype, device=device, not_aliasing=not_aliasing)
+
+
+def phase_reset() -> None:
+ """Every phase tensor is dead: the idle volumes may be carved again."""
+ ws = _workspace_module()
+ if ws is not None:
+ ws.workspace_phase_reset()
+
+
+def release_tile() -> None:
+ """A tile's leased volumes are free for the next tile."""
+ ws = _workspace_module()
+ active = None if ws is None else ws.active_decode_workspace()
+ if active is not None:
+ active.release_tile()
+
+
+def tile_workspace(large_volume_numel: int) -> object | None:
+ """A workspace for one decode whose large volume slots hold ``large_volume_numel`` elements, or ``None``
+ without the fused kernels it serves."""
+ ws = _workspace_module()
+ return None if ws is None else ws.DecodeWorkspace(large_volume_numel=large_volume_numel)
+
+
+def scope(workspace: object | None) -> contextlib.AbstractContextManager:
+ """Activate ``workspace`` (from :func:`tile_workspace`) for the block; a null context for ``None``."""
+ ws = _workspace_module()
+ if workspace is None or ws is None:
+ return contextlib.nullcontext(None)
+ return ws.decode_workspace(workspace)
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_tiling.py b/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_tiling.py
index 3f9523e..5280308 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_tiling.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_tiling.py
@@ -6,6 +6,7 @@ geometry/schedule/mask pieces that tiling uses.
from __future__ import annotations
import itertools
+import logging
import math
from dataclasses import dataclass
from typing import List, Literal, Sequence, Tuple
@@ -30,6 +31,8 @@ from ltx_core.tiling import (
)
from ltx_core.types import VIDEO_SCALE_FACTORS, SpatioTemporalScaleFactors
+logger = logging.getLogger(__name__)
+
ResizeAxisMode = Literal["repeat_last", "symmetric"]
_GIB: int = 1 << 30
@@ -65,12 +68,13 @@ class _StageFiveBudget:
# the fusion it holds several live full-volume intermediates instead of one Inductor buffer,
# measured as 11 -> 15.
# - BLACKWELL_DSL: the CuTe kernel has its own joint entry point, so the fused path serves
-# keyframes too and nothing changes.
+# keyframes too, but the keyframe decode's eager det blocks and its keyframe planes raise the
+# measured working set to 4.0 (2.5 leaves a streamed multi-GPU decode out of memory warm).
_BUDGET_BY_MODE: dict[DiffVAEMode, _StageFiveBudget] = {
DiffVAEMode.COMBINED_COMPILE: _StageFiveBudget(coef=11, coef_keyframes=15, reserve_bytes=2 * _GIB),
DiffVAEMode.CHUNKED_COMPILE: _StageFiveBudget(coef=7, coef_keyframes=5, reserve_bytes=2 * _GIB),
DiffVAEMode.CHUNKED_EAGER: _StageFiveBudget(coef=5, coef_keyframes=5, reserve_bytes=1 * _GIB),
- DiffVAEMode.BLACKWELL_DSL: _StageFiveBudget(coef=2.5, coef_keyframes=2.5, reserve_bytes=2 * _GIB),
+ DiffVAEMode.BLACKWELL_DSL: _StageFiveBudget(coef=2.5, coef_keyframes=4.0, reserve_bytes=2 * _GIB),
}
_DEFAULT_ELEMENT_SIZE: int = 2 # bf16 features → fp16 accumulator / bf16 stage-5
_ACCUMULATOR_CHANNELS: int = 3 # RGB pixel blend buffer (decoder out_channels)
@@ -303,7 +307,7 @@ def recommended_decode_tiling_config( # noqa: PLR0913
production bf16 features use fp16 accumulators (2), matching
:func:`accumulator_element_size`. Stage-5 uses the same element size x
``stage5_channels`` x ``coef``, which :func:`stage5_mem_coef` reads off the
- mode and ``keyframes`` (11 / 7 / 5 / 2.5 by mode; 15 / 5 / 5 / 2.5 with a
+ mode and ``keyframes`` (11 / 7 / 5 / 2.5 by mode; 15 / 5 / 5 / 4.0 with a
keyframe stream, which runs eager blocks).
Args beyond the geometry:
keyframes: this decode carries a keyframe stream (joint attention, eager blocks).
@@ -411,6 +415,17 @@ def recommended_decode_tiling_config( # noqa: PLR0913
scored.sort()
_waste, _vol, _ntiles, tile_t, tile_h, tile_w = scored[0]
+ logger.info(
+ "DiffVAE AUTO tiling: %df x %dx%d (overlap T=%d HW=%d) mode=%s keyframes=%s free_bytes=%d",
+ tile_t,
+ tile_h,
+ tile_w,
+ overlap_t,
+ overlap_hw,
+ mode.value,
+ keyframes,
+ int(free_bytes),
+ )
return TileSizeConfig(
frames=DimensionSizeConfig(tile_size=tile_t, overlap=overlap_t),
height=DimensionSizeConfig(tile_size=tile_h, overlap=overlap_hw),
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_video_decoder.py b/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_video_decoder.py
index 1ffd659..98567eb 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_video_decoder.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_video_decoder.py
@@ -10,18 +10,25 @@ from typing import Iterator, List, Literal, Tuple
import torch
from torch import nn
+from ltx_core.devices import upload_from_host
from ltx_core.model.disposable import Disposable
from ltx_core.model.transformer.timestep_embedding import PixArtAlphaCombinedTimestepSizeEmbeddings
-from ltx_core.model.video_vae import diffusion_tiling
+from ltx_core.model.video_vae import decode_workspace, diffusion_tiling
from ltx_core.model.video_vae.keyframes import (
DecodeKeyframes,
+ KeyframePlanes,
KeyframeStream,
keyframe_clip_times,
planes_for_tile,
remaining_time_strides,
upsample_keyframe_planes,
)
-from ltx_core.model.video_vae.ops import PerChannelStatistics, patchify, unpatchify
+from ltx_core.model.video_vae.ops import (
+ PerChannelStatistics,
+ apply_rms_norm_uint32_safe,
+ patchify_channels_last,
+ unpatchify,
+)
from ltx_core.model.video_vae.transformer import (
AdaLNZero,
ChannelLinear,
@@ -29,6 +36,8 @@ from ltx_core.model.video_vae.transformer import (
LinearPixelShuffleUpsample,
NABlock,
)
+from ltx_core.model.video_vae.transformer.rope_math import abs_rope_table_cache
+from ltx_core.model.video_vae.transformer.stage5_tail import stage5_tail, stage5_tail_supported
from ltx_core.model.video_vae.video_vae import VideoDecoder, iter_decoded_single_frames
from ltx_core.tiling import (
Tile,
@@ -68,6 +77,52 @@ _DIFF_STAGE5_KERNEL_DEFAULT: Tuple[int, int, int] = (3, 7, 7)
_DIFF_STAGE5_DEPTH_DEFAULT: int = 8
_DIFF_STAGE_DEPTHS_DEFAULT: Tuple[int, ...] = (*_L_STAGE_DEPTHS[:-1], _DIFF_STAGE5_DEPTH_DEFAULT)
+# Rows of the stage-5 volume normalised per in-place ``norm_out`` chunk (unfused tail).
+_NORM_OUT_CHUNK_ROWS: int = 1 << 20
+# Pixel bytes per ``conv_out`` / unpatchify chunk (unfused tail) and per tile-blend chunk.
+_TAIL_CHUNK_BYTES: int = 1 << 29
+
+
+def _release_cached_blocks(device: torch.device) -> None:
+ """Return the caching allocator's free segments, so one decode phase's churn is not reserved on top of
+ the next one's."""
+ if device.type == "cuda":
+ torch.cuda.synchronize(device)
+ torch.cuda.empty_cache()
+
+
+def _blend_tile_into(
+ buffer: torch.Tensor,
+ weights: torch.Tensor | None,
+ local_coords: tuple,
+ pixel_tile: torch.Tensor,
+ masks: tuple,
+) -> None:
+ """The masked accumulate of ``pixel_tile`` into ``buffer`` (and ``weights``), one temporal chunk at a time.
+ Per-element masks and accumulate, so chunking reproduces the whole-tile blend bit for bit.
+ """
+ frames = int(pixel_tile.shape[2])
+ t_lo, t_hi, _ = local_coords[2].indices(buffer.shape[2])
+ if t_hi - t_lo != frames:
+ raise RuntimeError(f"tile covers {frames} frames but its buffer slice spans {t_hi - t_lo}")
+ frame_bytes = pixel_tile[:, :, :1].numel() * 4
+ chunk_t = max(1, min(frames, _TAIL_CHUNK_BYTES // max(frame_bytes, 1)))
+ for f0 in range(0, frames, chunk_t):
+ f1 = min(frames, f0 + chunk_t)
+ coords = (local_coords[0], local_coords[1], slice(t_lo + f0, t_lo + f1), local_coords[3], local_coords[4])
+ chunk_masks = (
+ masks[0],
+ masks[1],
+ masks[2][f0:f1] if masks[2].numel() > 1 else masks[2],
+ masks[3],
+ masks[4],
+ )
+ chunk = pixel_tile[:, :, f0:f1].to(buffer.dtype)
+ buffer[coords] += scale_by_masks_1d(chunk, chunk_masks)
+ if weights is not None:
+ strength = torch.ones(chunk.shape, device=buffer.device, dtype=buffer.dtype)
+ weights[coords] += scale_by_masks_1d(strength, chunk_masks)
+
class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
"""Diffusion-based video VAE decoder (Neighborhood-Attention backbone).
@@ -264,13 +319,13 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
x = self.conv_in(x)
planes = x.shape[1]
if valid is None:
- valid = torch.ones(planes, dtype=torch.bool, device=x.device)
+ valid = torch.ones(planes, dtype=torch.bool)
times = keyframe_clip_times(
keyframes.pixel_frame_indices,
self._keyframe_time_strides[0],
keyframes.clip_start_frame,
)
- return KeyframeStream(x=x, times=times.to(device=x.device), valid=valid.to(device=x.device)).masked()
+ return KeyframeStream.from_host(x, times, valid).masked()
def _run_det_stage_with_keyframes(
self,
@@ -305,11 +360,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
clip_start_frame,
extra_origin=next_time_origin,
)
- return x, KeyframeStream(
- x=keyframe_x,
- times=next_times.to(device=keyframe_x.device),
- valid=keyframes.valid,
- ).masked()
+ return x, dataclasses.replace(keyframes, x=keyframe_x).with_times(next_times).masked()
def forward_stages_1_to_3_with_keyframes(
self,
@@ -383,14 +434,13 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
"""
# Rebuild from global indices rather than trusting the caller's stream: stages 1-3 of a
# full-clip decode are global, and Dist has already folded clip_start into clip times.
- keyframes = dataclasses.replace(
- keyframes,
- times=keyframe_clip_times(
+ keyframes = keyframes.with_times(
+ keyframe_clip_times(
pixel_frame_indices,
self._keyframe_time_strides[3],
clip_start_frame,
extra_origin=stage4_time_origin,
- ).to(device=keyframes.x.device),
+ )
)
if self.deferred_stage4_upsample:
return self._forward_stage_4_deferred_with_keyframes(
@@ -446,11 +496,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
clip_start_frame,
extra_origin=pixel_time_origin,
)
- return x, KeyframeStream(
- x=keyframes.x,
- times=stage5_times.to(device=keyframes.x.device),
- valid=keyframes.valid,
- ).masked()
+ return x, keyframes.with_times(stage5_times).masked()
def forward_stage_4(
self,
@@ -492,8 +538,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
def _context_and_x_for_diff_step(self, context: torch.Tensor, x_t: torch.Tensor) -> torch.Tensor:
"""Build block-ready ``[context | conv_in_x_t(patched x)]`` for ``forward_diff_step``."""
- noised_pixels_patched = patchify(x_t, patch_size_hw=self.patch_size, patch_size_t=1)
- x = self.conv_in_x_t(noised_pixels_patched.permute(0, 2, 3, 4, 1))
+ x = self.conv_in_x_t(patchify_channels_last(x_t, self.patch_size))
return torch.cat([context, x], dim=-1)
def _keyframe_context_and_x_for_diff_step(
@@ -506,23 +551,21 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
``keyframe_x_t`` is ``(B, C_pix, P, H_pix, W_pix)`` -- the keyframe planes' own noised
pixels, one pixel frame per plane, through the *shared* ``conv_in_x_t``.
"""
- patched = patchify(keyframe_x_t, patch_size_hw=self.patch_size, patch_size_t=1)
- x = self.conv_in_x_t(patched.permute(0, 2, 3, 4, 1))
+ x = self.conv_in_x_t(patchify_channels_last(keyframe_x_t, self.patch_size))
x = x * keyframe_valid[None, :, None, None, None]
return torch.cat([keyframe_context, x], dim=-1)
def _x_for_diff_step(self, x_t: torch.Tensor) -> torch.Tensor:
"""Conv-processed noised pixels only (deferred-context path)."""
- noised_pixels_patched = patchify(x_t, patch_size_hw=self.patch_size, patch_size_t=1)
- return self.conv_in_x_t(noised_pixels_patched.permute(0, 2, 3, 4, 1))
+ return self.conv_in_x_t(patchify_channels_last(x_t, self.patch_size))
def _keyframe_x_for_diff_step(self, keyframe_x_t: torch.Tensor, keyframe_valid: torch.Tensor) -> torch.Tensor:
"""Mask-zeroed keyframe noised pixels only (deferred-context path).
- Contiguous by construction: the chunked pathway mutates this buffer in place.
+ Contiguous by construction: the chunked pathway mutates this buffer in place. Masked in
+ place: ``x`` is this call's own fresh projection, whose storage the fused block chain recycles.
"""
- patched = patchify(keyframe_x_t, patch_size_hw=self.patch_size, patch_size_t=1)
- x = self.conv_in_x_t(patched.permute(0, 2, 3, 4, 1))
- return (x * keyframe_valid[None, :, None, None, None]).contiguous()
+ x = self.conv_in_x_t(patchify_channels_last(keyframe_x_t, self.patch_size))
+ return x.contiguous().mul_(keyframe_valid[None, :, None, None, None].to(x.dtype))
def forward_diff_step(
self,
@@ -553,8 +596,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
context_and_x: torch.Tensor,
keyframe_context_and_x: torch.Tensor,
t: torch.Tensor,
- keyframe_times: torch.Tensor,
- keyframe_valid: torch.Tensor,
+ keyframe_planes: KeyframePlanes,
) -> tuple[torch.Tensor, torch.Tensor]:
"""One dual-stream stage-5 step. Returns ``(video_pred, keyframe_pred)`` in pixel space.
The keyframe stream at stage 5 is a genuine second *pixel* diffusion stream -- its own
@@ -565,7 +607,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
discarded: callers use the video prediction only.
Both buffers follow ``forward_diff_step``'s ``[context | x]`` layout and the same
``copy_``-into-a-view discipline. Video T/H/W stay dynamic; the keyframe plane axis is
- specialized, since ``keyframe_times`` / ``keyframe_valid`` pin it.
+ specialized, since ``keyframe_planes`` pin it.
"""
x_half = context_and_x[..., self.context_channels :]
keyframe_half = keyframe_context_and_x[..., self.context_channels :]
@@ -575,7 +617,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
if self.mark_dynamic_shapes:
for dim in (1, 2, 3):
torch._dynamo.mark_dynamic(context_and_x, dim)
- # Keyframe dim 1 is the plane count, not T, and keyframe_times / keyframe_valid pin it.
+ # Keyframe dim 1 is the plane count, not T, and keyframe_planes pin it.
# Marking it dynamic and then specializing to P raises ConstraintViolationError.
for dim in (2, 3):
torch._dynamo.mark_dynamic(keyframe_context_and_x, dim)
@@ -585,8 +627,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
context_and_x,
keyframe_context_and_x,
modulation,
- keyframe_times,
- keyframe_valid,
+ keyframe_planes,
)
x_half.copy_(x_out)
keyframe_half.copy_(keyframe_out)
@@ -594,11 +635,32 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
return self._pixels_from_stage5(x_half), self._pixels_from_stage5(keyframe_half)
def _pixels_from_stage5(self, x: torch.Tensor) -> torch.Tensor:
- """Shared stage-5 tail: ``norm_out`` -> ``conv_out`` -> channels-first -> unpatchify."""
- x = self.norm_out(x)
- x = self.conv_out(x)
- x = x.permute(0, 4, 1, 2, 3).contiguous()
- return unpatchify(x, patch_size_hw=self.patch_size, patch_size_t=1)
+ """Shared stage-5 tail: ``norm_out`` -> ``conv_out`` -> channels-first -> unpatchify.
+ Without the fused tail, ``norm_out`` runs row chunk by row chunk into a contiguous ``x`` (the
+ block chain's own output, read by nothing after this) and ``conv_out`` + unpatchify one
+ temporal chunk at a time into the pixel volume: same values, no volume-sized transients.
+ Each eager RMSNorm launch stays under 2**32 elements (torch < 2.13 CUDA kernels wrap
+ flattened offsets).
+ """
+ if stage5_tail_supported(x, self.norm_out, self.conv_out, self.patch_size):
+ return stage5_tail(x, self.norm_out, self.conv_out, self.patch_size)
+ if x.is_contiguous():
+ rows = x.view(-1, x.shape[-1])
+ for r0 in range(0, rows.shape[0], _NORM_OUT_CHUNK_ROWS):
+ chunk = rows[r0 : r0 + _NORM_OUT_CHUNK_ROWS]
+ chunk.copy_(apply_rms_norm_uint32_safe(self.norm_out, chunk))
+ else:
+ x = apply_rms_norm_uint32_safe(self.norm_out, x)
+ b, t, h5, w5, _ = x.shape
+ p = self.patch_size
+ pixels = torch.empty((b, self.out_channels, t, h5 * p, w5 * p), device=x.device, dtype=x.dtype)
+ frame_bytes = b * h5 * w5 * self.out_channels * p * p * x.element_size()
+ chunk_t = max(1, min(t, _TAIL_CHUNK_BYTES // max(frame_bytes, 1)))
+ for t0 in range(0, t, chunk_t):
+ t1 = min(t, t0 + chunk_t)
+ y = self.conv_out(x[:, t0:t1]).permute(0, 4, 1, 2, 3).contiguous()
+ pixels[:, :, t0:t1].copy_(unpatchify(y, patch_size_hw=self.patch_size, patch_size_t=1))
+ return pixels
def forward_diff_step_deferred(
self,
@@ -632,10 +694,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
for block in self.diff_blocks:
x = block.forward_x_ctx(x, stage4_feat, modulation, drop_leading_frame=drop_leading_frame)
- x = self.norm_out(x)
- x = self.conv_out(x)
- x = x.permute(0, 4, 1, 2, 3).contiguous()
- return unpatchify(x, patch_size_hw=self.patch_size, patch_size_t=1)
+ return self._pixels_from_stage5(x)
def forward_diff_step_deferred_with_keyframes(
self,
@@ -644,8 +703,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
keyframe_x: torch.Tensor,
keyframe_stage4_feat: torch.Tensor,
t: torch.Tensor,
- keyframe_times: torch.Tensor,
- keyframe_valid: torch.Tensor,
+ keyframe_planes: KeyframePlanes,
*,
drop_leading_frame: bool = True,
) -> tuple[torch.Tensor, torch.Tensor]:
@@ -680,8 +738,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
keyframe_x,
keyframe_stage4_feat,
modulation,
- keyframe_times,
- keyframe_valid,
+ keyframe_planes,
drop_leading_frame=drop_leading_frame,
)
else:
@@ -692,8 +749,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
keyframe_x,
keyframe_stage4_feat,
modulation,
- keyframe_times,
- keyframe_valid,
+ keyframe_planes,
drop_leading_frame=drop_leading_frame,
)
@@ -838,25 +894,28 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
x_t = _noise(canvas_t) if x_t_tile_init is None else x_t_tile_init
keyframe_x_t = _noise(keyframes.num_planes)
- def _step(t_now: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
+ def _step(t_now: torch.Tensor, *, final: bool = False) -> tuple[torch.Tensor, torch.Tensor]:
+ nonlocal x_t, keyframe_x_t
if self.deferred_stage4_upsample:
+ x_in = self._x_for_diff_step(x_t)
+ keyframe_x_in = self._keyframe_x_for_diff_step(keyframe_x_t, keyframes.planes.valid.device_tensor)
+ if final and self.model_output_type == "x0":
+ # An x0 model's last step never reads the noise again; release both canvases now.
+ x_t = keyframe_x_t = None # type: ignore[assignment]
return self.forward_diff_step_deferred_with_keyframes(
- self._x_for_diff_step(x_t),
+ x_in,
context_tile,
- self._keyframe_x_for_diff_step(keyframe_x_t, keyframes.valid),
+ keyframe_x_in,
keyframes.x,
t_now,
- keyframes.times,
- keyframes.valid,
+ keyframes.planes,
drop_leading_frame=is_origin,
)
context_and_x = self._context_and_x_for_diff_step(context_tile, x_t)
keyframe_context_and_x = self._keyframe_context_and_x_for_diff_step(
- keyframes.x, keyframe_x_t, keyframes.valid
- )
- return self.forward_diff_step_with_keyframes(
- context_and_x, keyframe_context_and_x, t_now, keyframes.times, keyframes.valid
+ keyframes.x, keyframe_x_t, keyframes.planes.valid.device_tensor
)
+ return self.forward_diff_step_with_keyframes(context_and_x, keyframe_context_and_x, t_now, keyframes.planes)
_, num_steps = timestep.shape
for i in range(num_steps - 1):
@@ -867,7 +926,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
keyframe_x_t = self._euler_step(keyframe_x_t, keyframe_out.to(torch.float32), t_now, t_next)
t_now = timestep[:, -1]
- video_out, _ = _step(t_now)
+ video_out, _ = _step(t_now, final=True)
if self.model_output_type == "x0":
return video_out
return self._euler_step(x_t, video_out.to(torch.float32), t_now, torch.zeros_like(t_now))
@@ -900,7 +959,11 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
group_shape = full_video_shape._replace(frames=group_temporal_len)
full_torch_shape = full_video_shape.to_torch_shape()
accum_dtype = torch.float16 if feat_s4.dtype == torch.bfloat16 else feat_s4.dtype
- buffer = torch.zeros(group_shape.to_torch_shape(), device=feat_s4.device, dtype=accum_dtype)
+ # One tile with all-ones masks covering the group *is* the blend (0 + x * 1).
+ sole_tile = len(tiles) == 1 and complementary and all(bool((mask == 1).all()) for mask in tiles[0].masks_1d)
+ buffer = (
+ None if sole_tile else torch.zeros(group_shape.to_torch_shape(), device=feat_s4.device, dtype=accum_dtype)
+ )
weights: torch.Tensor | None = None if complementary else torch.zeros_like(buffer)
local_temporal_slice = slice(0, group_temporal_len)
@@ -925,9 +988,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
f"{pixel_hi - 1 + clip_start_frame}] selected no keyframe planes "
f"out of {int(pixel_frame_indices.shape[0])}; planes_for_tile always keeps at least one"
)
- tile_stream = stream.select_planes(keep.to(stream.valid.device)).crop_spatial(
- tile.in_coords[2], tile.in_coords[3]
- )
+ tile_stream = stream.select_planes(keep).crop_spatial(tile.in_coords[2], tile.in_coords[3])
# Not debug-only: the decode below needs this tile's plane positions.
tile_indices = pixel_frame_indices[keep.to(pixel_frame_indices.device)]
if logger.isEnabledFor(logging.INFO):
@@ -978,15 +1039,23 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
pixel_time_origin=float(pixel_lo),
clip_start_frame=clip_start_frame,
)
+ # The tile's pixels are a fresh tensor; its leased stage-5 volumes go back to the workspace.
+ decode_workspace.release_tile()
content_pixel_shape = diffusion_tiling.pixel_tile_shape(full_torch_shape, tile.out_coords)
pixel_tile = diffusion_tiling.crop_pixels_to_content(
pixel_tile,
content_pixel_shape[2],
content_pixel_shape[3],
content_pixel_shape[4],
- ).to(buffer.dtype)
-
- masks = tuple(m.to(device=buffer.device, dtype=torch.float32) for m in tile.masks_1d)
+ )
+ if sole_tile and pixel_tile.shape == group_shape.to_torch_shape():
+ pixel_tile = pixel_tile.to(accum_dtype)
+ _release_cached_blocks(pixel_tile.device)
+ return pixel_tile, None
+ if buffer is None:
+ buffer = torch.zeros(group_shape.to_torch_shape(), device=feat_s4.device, dtype=accum_dtype)
+
+ masks = tuple(upload_from_host(m.to(dtype=torch.float32), buffer.device) for m in tile.masks_1d)
local_coords = (
tile.out_coords[0],
tile.out_coords[1],
@@ -994,13 +1063,28 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
tile.out_coords[3],
tile.out_coords[4],
)
- buffer[local_coords] += scale_by_masks_1d(pixel_tile, masks)
- if weights is not None:
- strength = torch.ones(pixel_tile.shape, device=buffer.device, dtype=buffer.dtype)
- weights[local_coords] += scale_by_masks_1d(strength, masks)
+ _blend_tile_into(buffer, weights, local_coords, pixel_tile, masks)
+ # The tiles' transients are gone; hand their segments back before the caller emits the group.
+ del pixel_tile
+ _release_cached_blocks(buffer.device)
return buffer, weights
+ def _tile_workspace(self, tiles: List[Tile], feat_s4: torch.Tensor, content_s4_frames: int) -> object | None:
+ """The per-decode buffer workspace the fused stage-5 blocks draw from; ``None`` without ``ltx-kernels``.
+ Large-volume slots hold the biggest stage-5 volume any tile asks for, so they are allocated once.
+ """
+ st, sh, sw = (int(v) for v in self.upsamples[3].stride)
+ rows = 0
+ for tile in tiles:
+ feat_tile, _, _, _ = diffusion_tiling.slice_stage4_tile(feat_s4, tile, content_frames=content_s4_frames)
+ _, t, h, w, _ = feat_tile.shape
+ rows = max(rows, t * st * h * sh * w * sw)
+ if not decode_workspace.available():
+ return None
+ channels = int(self.conv_in_x_t.out_features)
+ return decode_workspace.tile_workspace((rows + decode_workspace.out_slack_rows()) * channels)
+
def _decode_groups_with_keyframes( # noqa: PLR0913, PLR0915
self,
feat_s4: torch.Tensor,
@@ -1022,7 +1106,8 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
is retained between iterations, and a group's exclusive frames are yielded before the
next group decodes. That keeps residency at roughly two tile extents rather than a
whole video, which matters more here than on the plain path -- a keyframe decode also
- carries a second pixel stream through stage 5.
+ carries a second pixel stream through stage 5. Every group's tiles draw their large
+ stage-5 buffers from one workspace allocated for the decode.
"""
full_video_shape = (
VideoLatentShape.from_torch_shape(latent.shape)
@@ -1068,6 +1153,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
scaled_w_pad = diffusion_tiling.scale_axis_pad(w_pad, self.video_downscale_factors.width)
overlap_stub: torch.Tensor | None = None
overlap_stub_weights: torch.Tensor | None = None
+ workspace = self._tile_workspace(tiles, feat_s4, s4_t)
def _emit(buf: torch.Tensor, wts: torch.Tensor | None, global_start: int) -> torch.Tensor | None:
"""Finalize, crop to content, and lay out one emitted run of frames."""
@@ -1080,7 +1166,24 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
if wts is not None:
floor = diffusion_tiling._weight_floor(wts.dtype)
chunk = chunk / wts[:, :, :frames_keep].clamp(min=floor)
- chunk = diffusion_tiling.crop_pixels_to_content(
+ if as_fhwc:
+ # Crop first, then cast and re-lay out in one copy: one frame-run transient instead of two.
+ cropped = diffusion_tiling.crop_pixels_to_content(
+ chunk,
+ frames_keep,
+ content_pixel.height,
+ content_pixel.width,
+ h_pad=scaled_h_pad,
+ w_pad=scaled_w_pad,
+ )[0]
+ out = torch.empty(
+ (cropped.shape[1], cropped.shape[2], cropped.shape[3], cropped.shape[0]),
+ device=cropped.device,
+ dtype=latent.dtype,
+ )
+ out.copy_(cropped.permute(1, 2, 3, 0))
+ return out
+ return diffusion_tiling.crop_pixels_to_content(
chunk.to(latent.dtype),
frames_keep,
content_pixel.height,
@@ -1088,7 +1191,6 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
h_pad=scaled_h_pad,
w_pad=scaled_w_pad,
)
- return chunk[0].permute(1, 2, 3, 0).contiguous() if as_fhwc else chunk
for group_index, group in enumerate(groups):
curr_temporal_slice = group_slices[group_index]
@@ -1099,20 +1201,24 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
curr_temporal_slice.start,
curr_temporal_slice.stop,
)
- buffer, weights = self._decode_temporal_group_isolated_with_keyframes(
- group,
- feat_s4,
- stream,
- pixel_frame_indices,
- s4_t,
- x_t_init,
- timestep,
- full_video_shape,
- curr_temporal_slice,
- generator,
- complementary=complementary,
- clip_start_frame=clip_start_frame,
- )
+ # Each group's tiles have their own transient sizes; hand back the previous group's free segments first.
+ _release_cached_blocks(feat_s4.device)
+ # Same-extent tiles and blocks build their abs-RoPE tables once per group.
+ with decode_workspace.scope(workspace), abs_rope_table_cache(self):
+ buffer, weights = self._decode_temporal_group_isolated_with_keyframes(
+ group,
+ feat_s4,
+ stream,
+ pixel_frame_indices,
+ s4_t,
+ x_t_init,
+ timestep,
+ full_video_shape,
+ curr_temporal_slice,
+ generator,
+ complementary=complementary,
+ clip_start_frame=clip_start_frame,
+ )
if overlap_stub is not None:
overlap_len = int(overlap_stub.shape[2])
@@ -1137,6 +1243,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
)
if emitted is not None:
yield emitted
+ del emitted
# Retain only the trailing overlap for the next group's handoff.
overlap_stub = buffer[:, :, exclusive_len:].clone()
if not complementary:
@@ -1273,7 +1380,11 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
group_shape = full_video_shape._replace(frames=group_temporal_len)
full_torch_shape = full_video_shape.to_torch_shape()
accum_dtype = torch.float16 if feat_s4.dtype == torch.bfloat16 else feat_s4.dtype
- buffer = torch.zeros(group_shape.to_torch_shape(), device=feat_s4.device, dtype=accum_dtype)
+ # One tile with all-ones masks covering the group *is* the blend (0 + x * 1).
+ sole_tile = len(tiles) == 1 and complementary and all(bool((mask == 1).all()) for mask in tiles[0].masks_1d)
+ buffer = (
+ None if sole_tile else torch.zeros(group_shape.to_torch_shape(), device=feat_s4.device, dtype=accum_dtype)
+ )
weights: torch.Tensor | None = None if complementary else torch.zeros_like(buffer)
local_temporal_slice = slice(0, group_temporal_len)
@@ -1325,9 +1436,13 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):
content_pixel_shape[2],
content_pixel_shape[3],
content_pixel_shape[4],
- ).to(buffer.dtype)
+ ).to(accum_dtype)
+ if sole_tile and pixel_tile.shape == group_shape.to_torch_shape():
+ return pixel_tile, None
+ if buffer is None:
+ buffer = torch.zeros(group_shape.to_torch_shape(), device=feat_s4.device, dtype=accum_dtype)
- masks = tuple(m.to(device=buffer.device, dtype=torch.float32) for m in tile.masks_1d)
+ masks = tuple(upload_from_host(m.to(dtype=torch.float32), buffer.device) for m in tile.masks_1d)
local_coords = (
tile.out_coords[0],
tile.out_coords[1],
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/keyframes.py b/packages/ltx-core/src/ltx_core/model/video_vae/keyframes.py
index 565dfcd..fa6ee04 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/keyframes.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/keyframes.py
@@ -13,12 +13,16 @@ vector per axis. That keeps the RoPE call and the slot tables batch-independent.
from __future__ import annotations
+import dataclasses
from dataclasses import dataclass
-from typing import Sequence
+from typing import NamedTuple, Sequence
import torch
from einops import rearrange
+from ltx_core.devices import upload_from_host
+from ltx_core.types import HostDeviceTensorPair
+
#: Keyframe planes visible to one video query (and video frames visible to one keyframe
#: query). Fixed at 2 upstream -- its backward kernel hardcodes at most two slots.
KEYFRAME_CONTEXT_SLOTS = 2
@@ -110,35 +114,71 @@ class DecodeKeyframes:
return int(self.latents.shape[2])
+class KeyframePlanes(NamedTuple):
+ """Where a keyframe plane stack sits in time and which planes are live.
+ Each field is a :class:`~ltx_core.types.HostDeviceTensorPair`: the attention kernels read the
+ device tensor, and everything that picks planes or builds plane tables reads the host one, so a
+ decode never reads plane metadata back from the device.
+ Attributes:
+ times: ``(P,)`` float32 plane position in *this stage's* temporal units and *local to
+ the current tile* -- the same origin the video stream's RoPE uses. Both streams
+ must share one origin or the joint softmax sees wrong offsets.
+ valid: ``(P,)`` bool. Invalid planes are masked out of every softmax and their
+ activations are re-zeroed after each upsample.
+ """
+
+ times: HostDeviceTensorPair
+ valid: HostDeviceTensorPair
+
+ @classmethod
+ def on_device(cls, times: torch.Tensor, valid: torch.Tensor, device: torch.device) -> KeyframePlanes:
+ """The planes on ``device`` with their host mirror; host inputs are uploaded without waiting."""
+ return cls(HostDeviceTensorPair.on_device(times, device), HostDeviceTensorPair.on_device(valid, device))
+
+
@dataclass(frozen=True)
class KeyframeStream:
"""The keyframe half of the dual stream at one decoder stage.
Attributes:
x: ``(B, P, H, W, C)`` channels-last activations. ``H``/``W`` always match the
video stream at the same stage; ``P`` is invariant across the whole decode.
- times: ``(P,)`` float32 plane position in *this stage's* temporal units and
- *local to the current tile* -- the same origin the video stream's RoPE uses.
- Both streams must share one origin or the joint softmax sees wrong offsets.
- valid: ``(P,)`` bool. Invalid planes are masked out of every softmax and their
- activations are re-zeroed after each upsample.
+ planes: the planes' :class:`KeyframePlanes`, on ``x``'s device with their host mirror.
"""
x: torch.Tensor
- times: torch.Tensor
- valid: torch.Tensor
+ planes: KeyframePlanes
+
+ @classmethod
+ def from_host(cls, x: torch.Tensor, times: torch.Tensor, valid: torch.Tensor) -> KeyframeStream:
+ """A stream on ``x``'s device; host ``times`` / ``valid`` are uploaded without waiting for it.
+ Device inputs are read back once here, so later plane selection stays on the host.
+ """
+ return cls(x=x, planes=KeyframePlanes.on_device(times, valid, x.device))
def masked(self) -> KeyframeStream:
"""Re-zero invalid planes' activations (channels-last plane axis)."""
- return KeyframeStream(x=self.x * self.valid[None, :, None, None, None], times=self.times, valid=self.valid)
+ return dataclasses.replace(self, x=self.x * self.planes.valid.device_tensor[None, :, None, None, None])
def select_planes(self, keep: torch.Tensor) -> KeyframeStream:
- """Subset the plane axis, keeping ``x``/``times``/``valid`` in step.
+ """Subset the plane axis, keeping ``x`` and both copies of the planes in step.
``keep`` is a ``(P,)`` bool mask. Used by tiled decode, which gives each tile only the
- planes near it -- see :func:`planes_for_tile`.
+ planes near it -- see :func:`planes_for_tile`. The selection is made on the host and the
+ device side follows by uploaded indices, so a host ``keep`` never waits for the device
+ (a device one is read back to size the result).
"""
if keep.shape != (self.num_planes,):
raise ValueError(f"keep must be ({self.num_planes},) bool, got {tuple(keep.shape)}")
- return KeyframeStream(x=self.x[:, keep], times=self.times[keep], valid=self.valid[keep])
+ keep_host = keep.cpu()
+ index_host = keep_host.nonzero().flatten()
+ index = upload_from_host(index_host, self.x.device)
+ times, valid = self.planes
+ return KeyframeStream(
+ x=self.x.index_select(1, index),
+ planes=KeyframePlanes(
+ HostDeviceTensorPair(times.host_tensor[keep_host], times.device_tensor.index_select(0, index)),
+ HostDeviceTensorPair(valid.host_tensor[keep_host], valid.device_tensor.index_select(0, index)),
+ ),
+ )
def crop_spatial(self, height: slice, width: slice) -> KeyframeStream:
"""Crop H/W with the *same* slices the video stream's tile used.
@@ -146,7 +186,13 @@ class KeyframeStream:
reads as ghosting rather than as an obvious failure -- the same hazard as the spatial
padding rule.
"""
- return KeyframeStream(x=self.x[:, :, height, width, :], times=self.times, valid=self.valid)
+ return dataclasses.replace(self, x=self.x[:, :, height, width, :])
+
+ def with_times(self, times: torch.Tensor) -> KeyframeStream:
+ """The same planes with ``times`` for another stage or origin (a host tensor is uploaded without waiting)."""
+ return dataclasses.replace(
+ self, planes=self.planes._replace(times=HostDeviceTensorPair.on_device(times, self.x.device))
+ )
@property
def num_planes(self) -> int:
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/memory_efficient_decode.py b/packages/ltx-core/src/ltx_core/model/video_vae/memory_efficient_decode.py
index eaff636..afea9d0 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/memory_efficient_decode.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/memory_efficient_decode.py
@@ -55,7 +55,7 @@ from torch.nn import functional as F
from ltx_core.loader.module_ops import ModuleOps
from ltx_core.loader.sd_ops import KeyValueOperationResult, SDOps
from ltx_core.model.common.normalization import PixelNorm
-from ltx_core.model.video_vae.convolution import CausalConv3d
+from ltx_core.model.video_vae.convolution import CausalConv3d, memory_format_of
from ltx_core.model.video_vae.ops import unpatchify
from ltx_core.model.video_vae.resnet import ResnetBlock3D, UNetMidBlock3D
from ltx_core.model.video_vae.sampling import DepthToSpaceUpsample
@@ -69,25 +69,6 @@ if TYPE_CHECKING:
# ---------------------------------------------------------------------------
-def _memory_format_of(t: torch.Tensor, prefer_channels_last_3d: bool = False) -> torch.memory_format:
- """Pick the memory format for a workspace allocation.
- When ``prefer_channels_last_3d`` is True and ``t`` is 5D, return
- ``channels_last_3d`` regardless of ``t``'s current strides -- the
- workspace's ``.copy_(t)`` will transcribe the data into the new layout.
- This is needed because intermediate tensors inside the decoder (after
- ``rearrange`` + slice + residual add in ``_upsample_forward_efficient``)
- are not NHWC-contiguous, so an auto-detect helper would silently fall
- back to NCHW for every workspace after the first upsample.
- Otherwise fall back to inspecting ``t``: ``channels_last_3d`` if ``t``
- already uses it, else contiguous.
- """
- if prefer_channels_last_3d and t.dim() == 5:
- return torch.channels_last_3d
- if t.dim() == 5 and t.is_contiguous(memory_format=torch.channels_last_3d):
- return torch.channels_last_3d
- return torch.contiguous_format
-
-
def _find_temporal_split_size(num_frames: int) -> int:
"""Find chunk size for in-place temporal convolution.
The chunk size ensures the last chunk has at least 3 frames
@@ -164,7 +145,7 @@ def inplace_conv3d_temporal_chunked(workspace: torch.Tensor, conv: nn.Conv3d) ->
workspace.shape[4],
device=workspace.device,
dtype=workspace.dtype,
- memory_format=_memory_format_of(workspace),
+ memory_format=memory_format_of(workspace),
)
o_buf = torch.empty_like(x_buf)
@@ -223,7 +204,7 @@ def _causal_pad(x: torch.Tensor, pad_size: int) -> torch.Tensor:
x.shape[4],
device=x.device,
dtype=x.dtype,
- memory_format=_memory_format_of(x),
+ memory_format=memory_format_of(x),
)
padded[:, :, pad_size:].copy_(x)
for i in range(pad_size):
@@ -386,7 +367,7 @@ def _midblock_forward_efficient(
hidden_states.shape[4],
device=hidden_states.device,
dtype=hidden_states.dtype,
- memory_format=_memory_format_of(hidden_states, prefer_channels_last_3d),
+ memory_format=memory_format_of(hidden_states, prefer_channels_last_3d),
)
for resnet in block.res_blocks:
@@ -430,7 +411,7 @@ def _upsample_forward_efficient(
if causal:
x = _causal_pad_free_and_conv(x, block.conv)
else:
- mem_fmt = _memory_format_of(x, prefer_channels_last_3d)
+ mem_fmt = memory_format_of(x, prefer_channels_last_3d)
workspace = torch.empty(
x.shape[0],
max(in_channels, out_channels),
@@ -480,7 +461,7 @@ def _final_norm_and_conv_out(
conv_out = conv_out_mod.conv
feature_channels = sample.shape[1]
- mem_fmt = _memory_format_of(sample, prefer_channels_last_3d)
+ mem_fmt = memory_format_of(sample, prefer_channels_last_3d)
workspace = torch.empty(
sample.shape[0],
max(feature_channels, conv_out.out_channels),
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/ops.py b/packages/ltx-core/src/ltx_core/model/video_vae/ops.py
index 7eccea5..c959cdd 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/ops.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/ops.py
@@ -1,4 +1,9 @@
+from __future__ import annotations
+
+from collections.abc import Sequence
+
import torch
+import torch.nn.functional as F
from einops import rearrange
from torch import nn
@@ -32,6 +37,16 @@ def patchify(x: torch.Tensor, patch_size_hw: int, patch_size_t: int = 1) -> torc
return x
+def patchify_channels_last(x: torch.Tensor, patch_size_hw: int) -> torch.Tensor:
+ """``patchify(x, patch_size_hw).permute(0, 2, 3, 4, 1)`` for a 5D ``x``, materialized contiguous.
+ Same element mapping as that pair, in one copy instead of a copy plus the one a
+ following ``Linear`` makes of the permuted view -- and a contiguous input lets the
+ ``Linear`` fold its bias into the matmul. At ``patch_size_hw == 1`` ``rearrange`` alone
+ returns a permuted view of ``x``.
+ """
+ return rearrange(x, "b c f (h q) (w r) -> b f h w (c r q)", q=patch_size_hw, r=patch_size_hw).contiguous()
+
+
def unpatchify(x: torch.Tensor, patch_size_hw: int, patch_size_t: int = 1) -> torch.Tensor:
"""
Rearrange channels back into spatial dimensions. Inverse of patchify - moves pixels from
@@ -82,3 +97,77 @@ class PerChannelStatistics(nn.Module):
return (x - self.get_buffer("mean-of-means").view(1, -1, 1, 1, 1).to(x)) / self.get_buffer("std-of-means").view(
1, -1, 1, 1, 1
).to(x)
+
+
+def apply_rms_norm_uint32_safe(norm: nn.RMSNorm, x: torch.Tensor) -> torch.Tensor:
+ """``norm(x)`` via :func:`rms_norm_uint32_safe`."""
+ return rms_norm_uint32_safe(x, tuple(norm.normalized_shape), norm.weight, norm.eps)
+
+
+def rms_norm_uint32_safe(
+ x: torch.Tensor,
+ normalized_shape: Sequence[int],
+ weight: torch.Tensor | None = None,
+ eps: float = 1e-6,
+) -> torch.Tensor:
+ """Last-dim RMSNorm that never feeds CUDA a volume of ``2**32``+ elements.
+ PyTorch < 2.13 computes ``row * N`` in 32-bit in the layer_norm / RMSNorm
+ kernels (pytorch/pytorch#181600). On 2.13+ this is ``F.rms_norm``.
+ """
+ torch_mm = tuple(int(p) for p in torch.__version__.split("+")[0].split(".")[:2])
+ max_elements = (1 << 32) if torch_mm < (2, 13) else (1 << 62)
+ return rms_norm_chunked(x, normalized_shape, weight, eps, max_elements=max_elements)
+
+
+def rms_norm_chunked(
+ x: torch.Tensor,
+ normalized_shape: Sequence[int],
+ weight: torch.Tensor | None = None,
+ eps: float = 1e-6,
+ *,
+ max_elements: int,
+) -> torch.Tensor:
+ """Last-dim RMSNorm with each launch strictly below ``max_elements``.
+ Prefers slicing T (dim 1) on ``ndim >= 3``. If one T-plane is already at or
+ above the limit, flattens to rows and chunks dim 0. Writes into a preallocated
+ output so peak extra memory is one window, not every slice plus ``torch.cat``.
+ """
+ rms_args = (normalized_shape, weight, eps)
+ if x.numel() < max_elements:
+ return F.rms_norm(x, *rms_args)
+ if x.ndim >= 3:
+ t = int(x.shape[1])
+ per_frame = x.numel() // max(t, 1)
+ if per_frame < max_elements:
+ chunk_t = _chunk_extent(t, per_frame, max_elements)
+ if chunk_t >= t:
+ return F.rms_norm(x, *rms_args)
+ return _rms_norm_into(torch.empty_like(x), x, 1, chunk_t, *rms_args)
+ if x.ndim < 2:
+ return F.rms_norm(x, *rms_args)
+ flat = x.reshape(-1, x.shape[-1])
+ n = int(flat.shape[0])
+ per_row = max(int(flat.shape[-1]), 1)
+ chunk = _chunk_extent(n, per_row, max_elements)
+ if chunk >= n:
+ return F.rms_norm(x, *rms_args)
+ out = _rms_norm_into(torch.empty_like(flat), flat, 0, chunk, *rms_args)
+ return out if x.ndim == 2 else out.view(x.shape)
+
+
+def _chunk_extent(length: int, per_item: int, max_elements: int) -> int:
+ """Largest slice along ``length`` whose numel stays strictly below ``max_elements``."""
+ if length <= 0:
+ return 1
+ return max(1, (max_elements - 1) // max(per_item, 1))
+
+
+def _rms_norm_into(out: torch.Tensor, x: torch.Tensor, dim: int, chunk: int, *rms_args: object) -> torch.Tensor:
+ """Last-dim RMSNorm along ``dim``, writing each slice into a preallocated ``out``."""
+ n = int(x.shape[dim])
+ idx: list[slice | int] = [slice(None)] * x.ndim
+ for i in range(0, n, chunk):
+ idx[dim] = slice(i, min(i + chunk, n))
+ key = tuple(idx)
+ out[key] = F.rms_norm(x[key], *rms_args)
+ return out
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/sampling.py b/packages/ltx-core/src/ltx_core/model/video_vae/sampling.py
index 5d8f742..24643b6 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/sampling.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/sampling.py
@@ -5,7 +5,7 @@ import torch
from einops import rearrange
from torch import nn
-from .convolution import make_conv_nd
+from .convolution import make_conv_nd, memory_format_of
from .enums import PaddingModeType
@@ -36,6 +36,7 @@ class SpaceToDepthDownsample(nn.Module):
x: torch.Tensor,
causal: bool = True,
) -> torch.Tensor:
+ memory_format = memory_format_of(x)
if self.stride[0] == 2:
x = torch.cat([x[:, :, :1, :, :], x], dim=2) # duplicate first frames for padding
@@ -60,9 +61,12 @@ class SpaceToDepthDownsample(nn.Module):
p3=self.stride[2],
)
- x = x + x_in
-
- return x
+ # The space-to-depth rearranges come out NCHW; summing into a buffer laid out like the input
+ # keeps channels-last activations channels-last for the next block's convs. ``out=`` has no
+ # autograd support, so a graph-recording call pays one extra copy instead.
+ if x.requires_grad or x_in.requires_grad:
+ return (x + x_in).contiguous(memory_format=memory_format)
+ return torch.add(x, x_in, out=torch.empty_like(x, memory_format=memory_format))
class DepthToSpaceUpsample(nn.Module):
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/attention.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/attention.py
index 5a73311..85b5087 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/attention.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/attention.py
@@ -13,16 +13,25 @@ from typing import TYPE_CHECKING, Protocol
import torch
from torch import nn
-from ltx_core.model.video_vae.transformer.det_attn_rope import det_qkv_rope, det_qkv_rope_at_times
+from ltx_core.model.video_vae.transformer.det_attn_rope import (
+ det_project_qkv_eager,
+ det_project_qkv_lean,
+ det_qkv_rope,
+ det_qkv_rope_at_times,
+ det_rope_qk_,
+ linear_into_phase_buffer,
+)
+from ltx_core.model.video_vae.transformer.det_rms_norm import DetRMSNorm
from ltx_core.model.video_vae.transformer.qkv import QKVProjections
from ltx_core.model.video_vae.transformer.rope_math import (
DEFAULT_ABS_ROPE_NUM_TILES,
+ AbsRopeTableCache,
default_rope_dim_split,
rope_inv_freqs,
)
if TYPE_CHECKING:
- from ltx_core.model.video_vae.keyframes import KeyframeStream
+ from ltx_core.model.video_vae.keyframes import KeyframePlanes, KeyframeStream
try:
import natten
@@ -59,7 +68,8 @@ class JointNAAttentionCallable(Protocol):
Same conventions as :class:`NAAttentionCallable` -- Q/K/V already normed, scaled and
absolutely RoPE'd -- with a second stream shaped ``(B, P, H, W, NH, HD)`` whose plane
axis sits in video's temporal slot. Both streams' RoPE must share one origin, so
- ``keyframe_times`` are tile-local. Returns one output per stream.
+ ``keyframe_planes.times`` are tile-local. Backends that build plane tables on the host read the
+ planes' host tensors; the others read the device ones. Returns one output per stream.
NATTEN and the CuTe DSL kernel cannot express a joint window, so this is a separate
slot from ``attention_function`` rather than a widening of it: it keeps the shipping
keyframe-less hot path untouched, and it is immune to the install-order hazard that
@@ -75,8 +85,7 @@ class JointNAAttentionCallable(Protocol):
keyframe_q: torch.Tensor,
keyframe_k: torch.Tensor,
keyframe_v: torch.Tensor,
- keyframe_times: torch.Tensor,
- keyframe_valid: torch.Tensor,
+ keyframe_planes: KeyframePlanes,
) -> tuple[torch.Tensor, torch.Tensor]: ...
@@ -146,6 +155,8 @@ class NeighborhoodAttention3D(nn.Module):
self.rope_base = rope_base
self.rope_num_tiles = DEFAULT_ABS_ROPE_NUM_TILES
self.rope_compute_dtype = torch.float32
+ # The decode scope's abs-RoPE table cache while one is open (see ``abs_rope_table_cache``).
+ self.rope_table_cache: AbsRopeTableCache | None = None
# Kept for the chunked opaque residual (string arg); callable is the swap surface.
self.natten_backend: str | None = None
self.attention_function: NAAttentionCallable = NattenAttention()
@@ -159,8 +170,8 @@ class NeighborhoodAttention3D(nn.Module):
self.qkv = QKVProjections(dim)
self.proj = nn.Linear(dim, dim, bias=True)
- self.q_norm = nn.RMSNorm(head_dim, eps=1e-6)
- self.k_norm = nn.RMSNorm(head_dim, eps=1e-6)
+ self.q_norm = DetRMSNorm(head_dim, eps=1e-6)
+ self.k_norm = DetRMSNorm(head_dim, eps=1e-6)
# W-chunking configuration (consumed by ``chunked.attn``).
self.w_chunks = 1 # 1 = no chunking
@@ -186,7 +197,12 @@ class NeighborhoodAttention3D(nn.Module):
f"got (T,H,W)=({t},{h},{w}) vs kernel={self.kernel_size}"
)
- q, k, v = det_qkv_rope(self, x)
+ if torch.compiler.is_compiling():
+ q, k, v = det_qkv_rope(self, x)
+ else:
+ # Eager: norms (and Q's scale) and RoPE in place on the fresh projections; same values.
+ q, k, v = det_project_qkv_eager(self, x)
+ q, k = det_rope_qk_(self, q.contiguous(), k.contiguous())
# natten's CUTLASS kernel silently produces wrong output if inputs are non-contiguous.
q, k, v = q.contiguous(), k.contiguous(), v.contiguous()
out = self.attention_function(self, q, k, v)
@@ -210,8 +226,17 @@ class NeighborhoodAttention3D(nn.Module):
batch, t, h, w, _ = x.shape
planes = keyframes.x.shape[1]
- q, k, v = det_qkv_rope(self, x)
- keyframe_q, keyframe_k, keyframe_v = det_qkv_rope_at_times(self, keyframes.x, keyframes.times)
+ eager = not torch.compiler.is_compiling()
+ if eager:
+ # Projections in workspace phase buffers, norms, scale and RoPE in place; same values.
+ q, k, v = det_project_qkv_lean(self, x)
+ del x
+ q, k = det_rope_qk_(self, q, k)
+ else:
+ q, k, v = det_qkv_rope(self, x)
+ keyframe_q, keyframe_k, keyframe_v = det_qkv_rope_at_times(
+ self, keyframes.x, keyframes.planes.times.device_tensor
+ )
q, k, v = q.contiguous(), k.contiguous(), v.contiguous()
keyframe_q = keyframe_q.contiguous()
keyframe_k = keyframe_k.contiguous()
@@ -225,10 +250,12 @@ class NeighborhoodAttention3D(nn.Module):
keyframe_q,
keyframe_k,
keyframe_v,
- keyframes.times,
- keyframes.valid,
+ keyframes.planes,
)
- out = self.proj(out.reshape(batch, t, h, w, self.dim))
+ # Q/K/V are dead once the attention has run; three volumes need not stay live across the projection.
+ del q, k, v, keyframe_q, keyframe_k, keyframe_v
+ out = out.reshape(batch, t, h, w, self.dim)
+ out = linear_into_phase_buffer(self.proj, out) if eager else self.proj(out)
keyframe_out = self.proj(keyframe_out.reshape(batch, planes, h, w, self.dim))
return out, dataclasses.replace(keyframes, x=keyframe_out)
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/blocks.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/blocks.py
index 152dd4d..4acf092 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/blocks.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/blocks.py
@@ -12,9 +12,11 @@ from typing import TYPE_CHECKING
import torch
from torch import nn
+from ltx_core.model.video_vae import decode_workspace
from ltx_core.model.video_vae.transformer.attention import NeighborhoodAttention3D
+from ltx_core.model.video_vae.transformer.det_rms_norm import DetRMSNorm
from ltx_core.model.video_vae.transformer.layers import AdaLNZero
-from ltx_core.model.video_vae.transformer.swiglu import SwiGLU, plain_mlp
+from ltx_core.model.video_vae.transformer.swiglu import SwiGLU, plain_mlp, plain_mlp_accumulate
if TYPE_CHECKING:
from ltx_core.model.video_vae.keyframes import KeyframeStream
@@ -37,17 +39,16 @@ class NABlock(nn.Module):
rope_dim_split: tuple[int, int, int] | None = None,
) -> None:
super().__init__()
- self.norm1 = nn.RMSNorm(dim, eps=1e-6)
+ self.norm1 = DetRMSNorm(dim, eps=1e-6)
self.attn = NeighborhoodAttention3D(dim, kernel_size, head_dim=head_dim, rope_dim_split=rope_dim_split)
- self.norm2 = nn.RMSNorm(dim, eps=1e-6)
+ self.norm2 = DetRMSNorm(dim, eps=1e-6)
hidden = (int(dim * mlp_ratio) + 15) // 16 * 16
self.mlp = SwiGLU(dim, hidden)
def forward(self, x: torch.Tensor) -> torch.Tensor:
- """Channels-last in/out: (B, T, H, W, C)."""
+ """Channels-last in/out: (B, T, H, W, C). The norms pick their kernel themselves (:class:`DetRMSNorm`)."""
x = x + self.attn(self.norm1(x))
- x = plain_mlp(x, self.mlp, self.norm2, self.mlp.tile)
- return x
+ return plain_mlp(x, self.mlp, self.norm2, self.mlp.tile)
def forward_with_keyframes(
self,
@@ -60,15 +61,28 @@ class NABlock(nn.Module):
the decoder re-zeroes after each upsample instead, matching upstream, so a masked
plane's hidden state may drift within a stage. It is masked out of every softmax
regardless, so this is cosmetic; reproducing it keeps us comparable.
+ Eager calls carve the video stream's per-block temporaries from the decode workspace's
+ idle volumes (none outlives the block, so the arena is reset at both block boundaries)
+ and add the residuals into the fresh attention and MLP outputs; same values.
"""
+ if torch.compiler.is_compiling():
+ attn_out, keyframe_attn = self.attn.forward_with_keyframes(
+ self.norm1(x), dataclasses.replace(keyframes, x=self.norm1(keyframes.x))
+ )
+ x = x + attn_out
+ keyframe_x = keyframes.x + keyframe_attn.x
+ x = plain_mlp(x, self.mlp, self.norm2, self.mlp.tile)
+ keyframe_x = plain_mlp(keyframe_x, self.mlp, self.norm2, self.mlp.tile)
+ return x, dataclasses.replace(keyframes, x=keyframe_x)
+ decode_workspace.phase_reset()
attn_out, keyframe_attn = self.attn.forward_with_keyframes(
- self.norm1(x),
- dataclasses.replace(keyframes, x=self.norm1(keyframes.x)),
+ self.norm1(x), dataclasses.replace(keyframes, x=self.norm1(keyframes.x))
)
- x = x + attn_out
- keyframe_x = keyframes.x + keyframe_attn.x
- x = plain_mlp(x, self.mlp, self.norm2, self.mlp.tile)
- keyframe_x = plain_mlp(keyframe_x, self.mlp, self.norm2, self.mlp.tile)
+ x = attn_out.add_(x)
+ keyframe_x = keyframe_attn.x.add_(keyframes.x)
+ x = plain_mlp_accumulate(x, self.mlp, self.norm2, self.mlp.tile)
+ keyframe_x = plain_mlp_accumulate(keyframe_x, self.mlp, self.norm2, self.mlp.tile)
+ decode_workspace.phase_reset()
return x, dataclasses.replace(keyframes, x=keyframe_x)
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/attn.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/attn.py
index 79da5e0..32cd029 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/attn.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/attn.py
@@ -25,6 +25,8 @@ import torch
import torch.nn.functional as F
from torch import nn
+from ltx_core.model.video_vae.keyframes import KeyframePlanes
+from ltx_core.model.video_vae.ops import rms_norm_uint32_safe
from ltx_core.model.video_vae.transformer.attention import (
JointNAAttentionCallable,
NAAttentionCallable,
@@ -37,6 +39,7 @@ from ltx_core.model.video_vae.transformer.rope_math import (
rot_abs_axis_impl,
t_positions,
)
+from ltx_core.types import HostDeviceTensorPair
# Bound by ``chunked()`` for the duration of the opaque custom_op. Dynamo only sees
# the op call site; the Python body reads these at runtime.
@@ -155,10 +158,10 @@ def _slab_qkv( # noqa: PLR0913
eps = 1e-6
head_shape = (batch, t, h, ext_w, num_heads, head_dim)
q = F.linear(y, q_w, q_b).view(head_shape)
- q = F.rms_norm(q, (head_dim,), q_norm_w, eps) * attn_scale
+ q = rms_norm_uint32_safe(q, (head_dim,), q_norm_w, eps) * attn_scale
q = _apply_in_slab_abs_rope(q, rope_split, inv_freqs, w_pos, compute_dtype=rope_dtype, t_pos=t_pos)
k = F.linear(y, k_w, k_b).view(head_shape)
- k = F.rms_norm(k, (head_dim,), k_norm_w, eps)
+ k = rms_norm_uint32_safe(k, (head_dim,), k_norm_w, eps)
k = _apply_in_slab_abs_rope(k, rope_split, inv_freqs, w_pos, compute_dtype=rope_dtype, t_pos=t_pos)
v = F.linear(y, v_w, v_b).view(head_shape)
return q, k, v
@@ -208,7 +211,7 @@ def _attn_on_chunk_impl( # noqa: PLR0913
inv_freqs = (inv_t, inv_h, inv_w)
rope_split = (d_t, d_h, d_w)
- y = F.rms_norm(x_chunk, (dim,), norm_weight, 1e-6)
+ y = rms_norm_uint32_safe(x_chunk, (dim,), norm_weight, 1e-6)
y = y * (1.0 + scale) + shift
q, k, v = _slab_qkv(
@@ -244,6 +247,8 @@ def _joint_attn_on_chunk_impl( # noqa: PLR0913
w_pos: torch.Tensor,
keyframe_times: torch.Tensor,
keyframe_valid: torch.Tensor,
+ keyframe_times_host: torch.Tensor,
+ keyframe_valid_host: torch.Tensor,
scale: torch.Tensor,
shift: torch.Tensor,
norm_weight: torch.Tensor,
@@ -286,9 +291,9 @@ def _joint_attn_on_chunk_impl( # noqa: PLR0913
weights = (q_w, q_b, k_w, k_b, v_w, v_b, q_norm_w, k_norm_w)
meta = (rope_split, inv_freqs, num_heads, head_dim, attn_scale, rope_dtype)
- y = F.rms_norm(x_chunk, (dim,), norm_weight, 1e-6)
+ y = rms_norm_uint32_safe(x_chunk, (dim,), norm_weight, 1e-6)
y = y * (1.0 + scale) + shift
- keyframe_y = F.rms_norm(keyframe_chunk, (dim,), norm_weight, 1e-6)
+ keyframe_y = rms_norm_uint32_safe(keyframe_chunk, (dim,), norm_weight, 1e-6)
keyframe_y = keyframe_y * (1.0 + scale) + shift
q, k, v = _slab_qkv(y, w_pos, *weights, *meta)
@@ -305,8 +310,10 @@ def _joint_attn_on_chunk_impl( # noqa: PLR0913
keyframe_q.contiguous(),
keyframe_k.contiguous(),
keyframe_v.contiguous(),
- times,
- keyframe_valid,
+ KeyframePlanes(
+ HostDeviceTensorPair(keyframe_times_host.to(dtype=torch.float32), times),
+ HostDeviceTensorPair(keyframe_valid_host, keyframe_valid),
+ ),
)
del q, k, v, keyframe_q, keyframe_k, keyframe_v
return (
@@ -661,6 +668,8 @@ def _na_residual_w_chunked_joint_op( # noqa: PLR0913
shift: torch.Tensor,
keyframe_times: torch.Tensor,
keyframe_valid: torch.Tensor,
+ keyframe_times_host: torch.Tensor,
+ keyframe_valid_host: torch.Tensor,
norm_weight: torch.Tensor,
q_w: torch.Tensor,
q_b: torch.Tensor,
@@ -743,6 +752,8 @@ def _na_residual_w_chunked_joint_op( # noqa: PLR0913
w_pos,
times,
valid,
+ keyframe_times_host,
+ keyframe_valid_host,
scale_,
shift_,
*rest, # type: ignore[arg-type]
@@ -770,6 +781,8 @@ def _na_residual_w_chunked_joint_fake( # noqa: PLR0913
shift: torch.Tensor,
keyframe_times: torch.Tensor,
keyframe_valid: torch.Tensor,
+ keyframe_times_host: torch.Tensor,
+ keyframe_valid_host: torch.Tensor,
norm_weight: torch.Tensor,
q_w: torch.Tensor,
q_b: torch.Tensor,
@@ -805,6 +818,8 @@ def _na_residual_w_chunked_joint_fake( # noqa: PLR0913
shift,
keyframe_times,
keyframe_valid,
+ keyframe_times_host,
+ keyframe_valid_host,
norm_weight,
q_w,
q_b,
@@ -842,8 +857,7 @@ def chunked_with_keyframes(
norm: nn.RMSNorm,
scale: torch.Tensor,
shift: torch.Tensor,
- keyframe_times: torch.Tensor,
- keyframe_valid: torch.Tensor,
+ keyframe_planes: KeyframePlanes,
) -> tuple[torch.Tensor, torch.Tensor]:
"""In-place dual-stream W-chunked residual through one joint softmax per slab.
Keyframe counterpart of :func:`chunked`, using ``attn.joint_attention_function``
@@ -858,8 +872,10 @@ def chunked_with_keyframes(
keyframe_x,
scale,
shift,
- keyframe_times,
- keyframe_valid,
+ keyframe_planes.times.device_tensor,
+ keyframe_planes.valid.device_tensor,
+ keyframe_planes.times.host_tensor,
+ keyframe_planes.valid.host_tensor,
*packed,
attn.w_chunks,
)
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/block.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/block.py
index e8e49b7..924a593 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/block.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/block.py
@@ -2,6 +2,8 @@
from __future__ import annotations
+from typing import TYPE_CHECKING
+
import torch
from ltx_core.model.video_vae.transformer.blocks import DiffusionNABlock
@@ -14,6 +16,9 @@ from ltx_core.model.video_vae.transformer.chunked.context import deferred_keyfra
from ltx_core.model.video_vae.transformer.chunked.mlp import residual_modulating_mlp
from ltx_core.model.video_vae.transformer.layers import LinearPixelShuffleUpsample
+if TYPE_CHECKING:
+ from ltx_core.model.video_vae.keyframes import KeyframePlanes
+
class ChunkedDiffusionNABlock(DiffusionNABlock):
"""Deferred-context diffusion block.
@@ -47,8 +52,7 @@ class ChunkedDiffusionNABlock(DiffusionNABlock):
shift_msa: torch.Tensor,
scale_mlp: torch.Tensor,
shift_mlp: torch.Tensor,
- keyframe_times: torch.Tensor,
- keyframe_valid: torch.Tensor,
+ keyframe_planes: KeyframePlanes,
) -> tuple[torch.Tensor, torch.Tensor]:
"""Dual-stream chunked NA residual + modulating SwiGLU, both in place.
Invalid keyframe planes are re-zeroed on the way out, matching
@@ -63,12 +67,11 @@ class ChunkedDiffusionNABlock(DiffusionNABlock):
self.norm1,
scale_msa,
shift_msa,
- keyframe_times,
- keyframe_valid,
+ keyframe_planes,
)
x = residual_modulating_mlp(x, self.mlp, self.norm2, scale_mlp, shift_mlp, self.mlp.tile)
keyframe_x = residual_modulating_mlp(keyframe_x, self.mlp, self.norm2, scale_mlp, shift_mlp, self.mlp.tile)
- keyframe_x.mul_(keyframe_valid[None, :, None, None, None])
+ keyframe_x.mul_(keyframe_planes.valid.device_tensor[None, :, None, None, None])
return x, keyframe_x
def forward_x_ctx_with_keyframes(
@@ -78,8 +81,7 @@ class ChunkedDiffusionNABlock(DiffusionNABlock):
keyframe_x: torch.Tensor,
keyframe_stage4_feat: torch.Tensor,
modulation: tuple[torch.Tensor, ...],
- keyframe_times: torch.Tensor,
- keyframe_valid: torch.Tensor,
+ keyframe_planes: KeyframePlanes,
*,
drop_leading_frame: bool = True,
) -> tuple[torch.Tensor, torch.Tensor]:
@@ -118,8 +120,7 @@ class ChunkedDiffusionNABlock(DiffusionNABlock):
shift_msa,
scale_mlp,
shift_mlp,
- keyframe_times,
- keyframe_valid,
+ keyframe_planes,
)
def forward_x_ctx(
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/mlp.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/mlp.py
index 7131697..515c1b1 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/mlp.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/mlp.py
@@ -6,6 +6,7 @@ import torch
import torch.nn.functional as F
from torch import nn
+from ltx_core.model.video_vae.ops import rms_norm_uint32_safe
from ltx_core.model.video_vae.transformer.swiglu import (
SwiGLUTileSpec,
_fused_gate_up_swiglu_triton,
@@ -87,7 +88,7 @@ def _chunked_residual_modulated_impl(
n = end - start
xc = x_flat[start:end]
y = y_buf[:n]
- y.copy_(F.rms_norm(xc, (dim,), norm_weight, eps))
+ y.copy_(rms_norm_uint32_safe(xc, (dim,), norm_weight, eps))
y.mul_(1.0 + s).add_(sh)
ws = workspace[:n]
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/attn.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/attn.py
index efcb05b..8b1e339 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/attn.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/attn.py
@@ -5,9 +5,12 @@ Does not share a residual body with ``chunked`` — only the NA module weights.
from __future__ import annotations
+from typing import TYPE_CHECKING
+
import torch
from torch import nn
+from ltx_core.model.video_vae.ops import apply_rms_norm_uint32_safe
from ltx_core.model.video_vae.transformer.attention import NeighborhoodAttention3D
from ltx_core.model.video_vae.transformer.rope_math import (
h_positions,
@@ -15,6 +18,9 @@ from ltx_core.model.video_vae.transformer.rope_math import (
t_positions,
)
+if TYPE_CHECKING:
+ from ltx_core.model.video_vae.keyframes import KeyframePlanes
+
# Transparent path: innermost per-axis rotation is a nested compile region.
_rot_abs_axis = torch.compiler.nested_compile_region(rot_abs_axis_impl)
@@ -93,8 +99,8 @@ def _qkv_nested_rope(
its (possibly fractional) plane times there.
"""
q, k, v = attn.project_qkv(x)
- q = attn.q_norm(q) * attn.scale
- k = attn.k_norm(k)
+ q = apply_rms_norm_uint32_safe(attn.q_norm, q) * attn.scale
+ k = apply_rms_norm_uint32_safe(attn.k_norm, k)
inv_freqs = (
attn.rope_inv_t.to(device=x.device),
attn.rope_inv_h.to(device=x.device),
@@ -127,12 +133,11 @@ def full_with_keyframes(
norm: nn.RMSNorm,
scale: torch.Tensor,
shift: torch.Tensor,
- keyframe_times: torch.Tensor,
- keyframe_valid: torch.Tensor,
+ keyframe_planes: KeyframePlanes,
) -> tuple[torch.Tensor, torch.Tensor]:
"""Dual-stream AdaLN residual attention: one joint softmax, shared weights.
- Plain tensors rather than a ``KeyframeStream`` here: this is the stage-5 hot path, and
- rebuilding a dataclass per block inside a compiled region buys nothing.
+ The planes' metadata rather than a ``KeyframeStream`` here: this is the stage-5 hot path,
+ and rebuilding a dataclass per block inside a compiled region buys nothing.
Both streams take the *same* ``scale``/``shift``. Upstream computes a separate keyframe
modulation, but the two are identical unless per-frame timestep conditioning supplies a
conditioning mask, which we deliberately do not implement yet.
@@ -147,11 +152,11 @@ def full_with_keyframes(
batch, t, h, w, _ = x.shape
planes = keyframe_x.shape[1]
- y = norm(x) * (1.0 + scale) + shift
- keyframe_y = norm(keyframe_x) * (1.0 + scale) + shift
+ y = apply_rms_norm_uint32_safe(norm, x) * (1.0 + scale) + shift
+ keyframe_y = apply_rms_norm_uint32_safe(norm, keyframe_x) * (1.0 + scale) + shift
q, k, v = _qkv_nested_rope(attn, y)
- keyframe_q, keyframe_k, keyframe_v = _qkv_nested_rope(attn, keyframe_y, keyframe_times)
+ keyframe_q, keyframe_k, keyframe_v = _qkv_nested_rope(attn, keyframe_y, keyframe_planes.times.device_tensor)
q, k, v = q.contiguous(), k.contiguous(), v.contiguous()
keyframe_q = keyframe_q.contiguous()
keyframe_k = keyframe_k.contiguous()
@@ -165,8 +170,7 @@ def full_with_keyframes(
keyframe_q,
keyframe_k,
keyframe_v,
- keyframe_times,
- keyframe_valid,
+ keyframe_planes,
)
x = x + attn.proj(out.reshape(batch, t, h, w, attn.dim))
keyframe_x = keyframe_x + attn.proj(keyframe_out.reshape(batch, planes, h, w, attn.dim))
@@ -181,7 +185,7 @@ def full(
shift: torch.Tensor,
) -> torch.Tensor:
"""``x + NA(modulate(norm(x)))`` with nested full-volume abs-RoPE."""
- y = norm(x) * (1.0 + scale) + shift
+ y = apply_rms_norm_uint32_safe(norm, x) * (1.0 + scale) + shift
batch, t, h, w, _ = y.shape
kt, kh, kw = attn.kernel_size
if t < kt or h < kh or w < kw:
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/block.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/block.py
index 89d237f..c73fa90 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/block.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/block.py
@@ -2,6 +2,8 @@
from __future__ import annotations
+from typing import TYPE_CHECKING
+
import torch
from ltx_core.model.video_vae.transformer.blocks import DiffusionNABlock
@@ -10,6 +12,9 @@ from ltx_core.model.video_vae.transformer.combined.attn import full_with_keyfram
from ltx_core.model.video_vae.transformer.combined.context import combined as inject_context
from ltx_core.model.video_vae.transformer.combined.mlp import residual_mlp
+if TYPE_CHECKING:
+ from ltx_core.model.video_vae.keyframes import KeyframePlanes
+
class CombinedDiffusionNABlock(DiffusionNABlock):
"""Combined-context diffusion block: ``forward`` / ``forward_combined``."""
@@ -19,8 +24,7 @@ class CombinedDiffusionNABlock(DiffusionNABlock):
context_and_x: torch.Tensor,
keyframe_context_and_x: torch.Tensor,
modulation: tuple[torch.Tensor, ...],
- keyframe_times: torch.Tensor,
- keyframe_valid: torch.Tensor,
+ keyframe_planes: KeyframePlanes,
) -> tuple[torch.Tensor, torch.Tensor]:
"""Dual-stream block; returns the two updated x halves (not the concat buffers).
Each stream gets its own ``context_proj(context)`` injection from its own
@@ -39,12 +43,11 @@ class CombinedDiffusionNABlock(DiffusionNABlock):
self.norm1,
scale_msa,
shift_msa,
- keyframe_times,
- keyframe_valid,
+ keyframe_planes,
)
x = residual_mlp(x, self.mlp, self.norm2, scale_mlp, shift_mlp, self.mlp.tile)
keyframe_x = residual_mlp(keyframe_x, self.mlp, self.norm2, scale_mlp, shift_mlp, self.mlp.tile)
- return x, keyframe_x * keyframe_valid[None, :, None, None, None]
+ return x, keyframe_x * keyframe_planes.valid.device_tensor[None, :, None, None, None]
def forward_combined(
self,
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/mlp.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/mlp.py
index 386d113..e706610 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/mlp.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/mlp.py
@@ -5,6 +5,7 @@ from __future__ import annotations
import torch
from torch import nn
+from ltx_core.model.video_vae.ops import apply_rms_norm_uint32_safe
from ltx_core.model.video_vae.transformer.layers import modulate
from ltx_core.model.video_vae.transformer.swiglu import SwiGLUTileSpec, swiglu_tiled
@@ -18,7 +19,7 @@ def residual_mlp(
tile: SwiGLUTileSpec,
) -> torch.Tensor:
"""Combined*: ``x + swiglu_tiled(modulate(norm(x), scale, shift))``."""
- y = modulate(norm(x), scale, shift)
+ y = modulate(apply_rms_norm_uint32_safe(norm, x), scale, shift)
if y.numel() == 0:
return x
return x + swiglu_tiled(y, mlp.w_gate.weight, mlp.w_up.weight, mlp.w_down.weight, tile)
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/det_attn_rope.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/det_attn_rope.py
index 22b772b..1d1dcfa 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/det_attn_rope.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/det_attn_rope.py
@@ -17,8 +17,20 @@ from __future__ import annotations
import torch
+try:
+ from ltx_kernels.vae.rope_dsl import abs_rope_supported, run_abs_rope
+except ImportError: # ltx-kernels is optional; the slab path below covers every device
+ abs_rope_supported = None
+ run_abs_rope = None
+
+from ltx_core.model.video_vae import decode_workspace
+from ltx_core.model.video_vae.ops import apply_rms_norm_uint32_safe
from ltx_core.model.video_vae.transformer.rope_math import (
DEFAULT_ABS_ROPE_NUM_TILES,
+ AbsRopeTableCache,
+ RopeAxis,
+ RopePositions,
+ cached_rope_tables,
h_positions,
rot_abs_axis_impl,
t_positions,
@@ -33,25 +45,40 @@ def _apply_opaque_rope_slab(
w_pos: torch.Tensor,
compute_dtype: torch.dtype,
t_pos: torch.Tensor | None = None,
+ w_pos_key: RopePositions | None = None,
+ cache: AbsRopeTableCache | None = None,
) -> torch.Tensor:
"""Rotate one W-extent with raw abs-RoPE (runs inside the opaque op).
``t_pos`` overrides the default integer ``arange`` on the first axis; the keyframe
- stream passes its (possibly fractional) plane times there.
+ stream passes its (possibly fractional) plane times there. ``w_pos_key`` names ``w_pos``
+ by value for ``cache``.
"""
d_t, d_h, _ = rope_split
inv_t, inv_h, inv_w = inv_freqs
t = x.shape[1]
h = x.shape[2]
positions_t = t_positions(t, x.device) if t_pos is None else t_pos
- xt = rot_abs_axis_impl(x[..., :d_t], positions_t, inv_t, axis=1, compute_dtype=compute_dtype)
+ xt = rot_abs_axis_impl(
+ x[..., :d_t],
+ positions_t,
+ inv_t,
+ axis=1,
+ compute_dtype=compute_dtype,
+ pos_key=RopePositions(RopeAxis.T, t) if t_pos is None else None,
+ cache=cache,
+ )
xh = rot_abs_axis_impl(
x[..., d_t : d_t + d_h],
h_positions(h, x.device),
inv_h,
axis=2,
compute_dtype=compute_dtype,
+ pos_key=RopePositions(RopeAxis.H, h),
+ cache=cache,
+ )
+ xw = rot_abs_axis_impl(
+ x[..., d_t + d_h :], w_pos, inv_w, axis=3, compute_dtype=compute_dtype, pos_key=w_pos_key, cache=cache
)
- xw = rot_abs_axis_impl(x[..., d_t + d_h :], w_pos, inv_w, axis=3, compute_dtype=compute_dtype)
return torch.cat([xt, xh, xw], dim=-1)
@@ -63,12 +90,18 @@ def _apply_opaque_tiled_rope(
num_tiles: int,
compute_dtype: torch.dtype,
t_pos: torch.Tensor | None = None,
+ cache: AbsRopeTableCache | None = None,
) -> torch.Tensor:
- """Fixed-``num_tiles`` W split + per-slab rotation (body of the opaque op).
+ """Fixed-``num_tiles`` W split + per-slab rotation (body of the opaque op); tables from ``cache`` if given.
+ With ``ltx-kernels`` installed, a CUDA bf16 volume at the fp32 compute dtype is rotated in
+ one launch instead (:func:`ltx_kernels.vae.rope_dsl.run_abs_rope`); the split only bounds
+ the slab path's fp32 temporaries, so it does not change the angles.
The keyframe stream shares the video stream's W extent at every stage, so passing the
same ``num_tiles`` yields identical slab boundaries and identical ``w_pos`` -- which
is what keeps the two streams' W phases comparable inside the joint softmax.
"""
+ if compute_dtype == torch.float32 and abs_rope_supported is not None and abs_rope_supported(x, rope_split):
+ return run_abs_rope(x, rope_split, _rope_tables(x, inv_freqs, t_pos, cache))
slabs = torch.chunk(x, num_tiles, dim=3)
w_off = 0
parts: list[torch.Tensor] = []
@@ -83,12 +116,53 @@ def _apply_opaque_tiled_rope(
w_pos=w_pos,
compute_dtype=compute_dtype,
t_pos=t_pos,
+ w_pos_key=RopePositions(RopeAxis.W, w_slab, w_off),
+ cache=cache,
)
)
w_off = w_off + w_slab
return torch.cat(parts, dim=3)
+def _rope_tables(
+ x: torch.Tensor,
+ inv_freqs: tuple[torch.Tensor, torch.Tensor, torch.Tensor],
+ t_pos: torch.Tensor | None,
+ cache: AbsRopeTableCache | None,
+) -> tuple[torch.Tensor, ...]:
+ """``(cos_t, sin_t, cos_h, sin_h, cos_w, sin_w)`` over ``x``'s T/H/W, as the slab path computes them.
+ torch's own fp32 ``cos`` / ``sin`` of ``pos * inv``: the slab offsets make the W positions
+ the plain ``arange(W)``, so these are exactly the angles :func:`rot_abs_axis_impl` uses.
+ """
+ _, t, h, w, _, _ = x.shape
+
+ def build() -> tuple[torch.Tensor, ...]:
+ positions = (
+ t_positions(t, x.device) if t_pos is None else t_pos.to(device=x.device, dtype=torch.float32),
+ h_positions(h, x.device),
+ torch.arange(w, dtype=torch.float32, device=x.device),
+ )
+ tables: list[torch.Tensor] = []
+ for pos, inv in zip(positions, inv_freqs, strict=True):
+ angles = pos[:, None] * inv[None, :]
+ tables += [angles.cos(), angles.sin()]
+ return tuple(tables)
+
+ # Integer positions are named by their extents; explicit plane times are not cached.
+ key = (
+ None
+ if t_pos is not None
+ else (
+ RopePositions(RopeAxis.T, t),
+ RopePositions(RopeAxis.H, h),
+ RopePositions(RopeAxis.W, w),
+ *(inv.data_ptr() for inv in inv_freqs),
+ x.device,
+ )
+ )
+ return cached_rope_tables(cache, key, build)
+
+
@torch.library.custom_op("ltx_core::abs_rope", mutates_args=())
def _abs_rope_op(
x: torch.Tensor,
@@ -197,23 +271,40 @@ def _abs_rope_at_t_fake(
return torch.empty(x.shape, device=x.device, dtype=x.dtype)
-def _rope_config(attn: object, x: torch.Tensor) -> tuple[tuple[torch.Tensor, ...], int, torch.dtype]:
- """``(inv_freqs, num_tiles, compute_dtype)`` read off an attention module."""
- inv_freqs = (
- attn.rope_inv_t.to(device=x.device), # type: ignore[attr-defined]
- attn.rope_inv_h.to(device=x.device), # type: ignore[attr-defined]
- attn.rope_inv_w.to(device=x.device), # type: ignore[attr-defined]
- )
+def _rope_config(
+ attn: object, x: torch.Tensor
+) -> tuple[tuple[torch.Tensor, ...], int, torch.dtype, AbsRopeTableCache | None]:
+ """``(inv_freqs, num_tiles, compute_dtype, cache)`` read off an attention module.
+ ``cache`` is the :class:`AbsRopeTableCache` the module holds for an enclosing
+ :func:`~ltx_core.model.video_vae.transformer.rope_math.abs_rope_table_cache` scope, if this thread may
+ read it (never while tracing or capturing). With one, the inverse frequencies are the cache's tensors of
+ the module's ``rope_dim_split`` and ``rope_base``: the module's ``rope_inv_*`` buffers are
+ :func:`rope_inv_freqs` of those (non-persistent, so never loaded), cast with the module, so the values
+ are the same and every block of one configuration shares one set of cached tables.
+ """
+ buffers = (attn.rope_inv_t, attn.rope_inv_h, attn.rope_inv_w) # type: ignore[attr-defined]
+ # Checked before the slot is read, so a traced graph never guards on the scope's cache.
+ cache = None if torch.compiler.is_compiling() else getattr(attn, "rope_table_cache", None)
+ if cache is not None and not cache.usable():
+ cache = None
+ if cache is None:
+ inv_freqs = tuple(inv.to(device=x.device) for inv in buffers)
+ else:
+ base = float(attn.rope_base) # type: ignore[attr-defined]
+ inv_freqs = tuple(
+ cache.inv_freqs(dim, base, inv.dtype, x.device)
+ for dim, inv in zip(attn.rope_dim_split, buffers, strict=True) # type: ignore[attr-defined]
+ )
num_tiles = getattr(attn, "rope_num_tiles", DEFAULT_ABS_ROPE_NUM_TILES)
compute_dtype = getattr(attn, "rope_compute_dtype", torch.float32)
- return inv_freqs, num_tiles, compute_dtype
+ return inv_freqs, num_tiles, compute_dtype, cache
def _det_project_qkv(attn: object, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
"""Shared Q/K/V proj + norm + scale for :func:`det_qkv_rope` and ``_at_times``."""
q, k, v = attn.project_qkv(x) # type: ignore[attr-defined]
- q = attn.q_norm(q) # type: ignore[attr-defined]
- k = attn.k_norm(k) # type: ignore[attr-defined]
+ q = apply_rms_norm_uint32_safe(attn.q_norm, q) # type: ignore[attr-defined]
+ k = apply_rms_norm_uint32_safe(attn.k_norm, k) # type: ignore[attr-defined]
q = q * attn.scale # type: ignore[attr-defined]
return q, k, v
@@ -232,11 +323,26 @@ def det_qkv_rope_at_times(
"""
if t_pos.ndim != 1 or t_pos.shape[0] != x.shape[1]:
raise ValueError(f"t_pos must be ({x.shape[1]},) to match the plane axis, got {tuple(t_pos.shape)}")
- q, k, v = _det_project_qkv(attn, x)
+ q, k, v = _det_project_qkv(attn, x) if torch.compiler.is_compiling() else det_project_qkv_eager(attn, x)
- inv_freqs, num_tiles, compute_dtype = _rope_config(attn, x)
+ inv_freqs, num_tiles, compute_dtype, cache = _rope_config(attn, x)
d_t, d_h, d_w = attn.rope_dim_split # type: ignore[attr-defined]
positions = t_pos.to(device=x.device, dtype=torch.float32)
+ if cache is not None:
+ # Eager under a table scope: the op's body itself, so the H/W tables come from the scope's cache.
+ rotated = [
+ _apply_opaque_tiled_rope(
+ tensor,
+ (d_t, d_h, d_w),
+ inv_freqs,
+ num_tiles=num_tiles,
+ compute_dtype=compute_dtype,
+ t_pos=positions,
+ cache=cache,
+ )
+ for tensor in (q, k)
+ ]
+ return rotated[0], rotated[1], v
rotated = [
_abs_rope_at_t_op(
tensor,
@@ -256,19 +362,84 @@ def det_qkv_rope_at_times(
def det_qkv_rope(attn: object, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
"""Q/K/V proj + norm/scale + opaque full-volume abs-RoPE."""
q, k, v = _det_project_qkv(attn, x)
- inv_freqs, num_tiles, compute_dtype = _rope_config(attn, x)
- q = _apply_opaque_abs_rope(
- q,
- attn.rope_dim_split, # type: ignore[attr-defined]
- inv_freqs,
- num_tiles=num_tiles,
- compute_dtype=compute_dtype,
- )
- k = _apply_opaque_abs_rope(
- k,
- attn.rope_dim_split, # type: ignore[attr-defined]
- inv_freqs,
- num_tiles=num_tiles,
- compute_dtype=compute_dtype,
- )
+ inv_freqs, num_tiles, compute_dtype, cache = _rope_config(attn, x)
+ rope_split = attn.rope_dim_split # type: ignore[attr-defined]
+ if cache is not None:
+ # Eager under a table scope: the op's body itself, with the scope's cached tables.
+ q, k = (
+ _apply_opaque_tiled_rope(
+ t, rope_split, inv_freqs, num_tiles=num_tiles, compute_dtype=compute_dtype, cache=cache
+ )
+ for t in (q, k)
+ )
+ return q, k, v
+ q = _apply_opaque_abs_rope(q, rope_split, inv_freqs, num_tiles=num_tiles, compute_dtype=compute_dtype)
+ k = _apply_opaque_abs_rope(k, rope_split, inv_freqs, num_tiles=num_tiles, compute_dtype=compute_dtype)
+ return q, k, v
+
+
+def linear_into_phase_buffer(linear: torch.nn.Linear, x: torch.Tensor) -> torch.Tensor:
+ """``linear(x)`` for a contiguous channels-last ``x``, into a workspace phase buffer when one is active.
+ Exactly ``addmm(bias, x.view(-1, K), W.t())`` with an ``out=``, so the values are those of ``nn.Linear``.
+ """
+ if not x.is_contiguous() or linear.bias is None:
+ return linear(x)
+ rows = x.view(-1, linear.in_features)
+ out = decode_workspace.phase_empty((rows.shape[0], linear.out_features), dtype=x.dtype, device=x.device)
+ torch.addmm(linear.bias, rows, linear.weight.t(), out=out)
+ return out.view(*x.shape[:-1], linear.out_features)
+
+
+def det_project_qkv_lean(attn: object, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+ """:func:`_det_project_qkv` for the eager dual-stream det blocks: projections into workspace phase
+ buffers, norms (and Q's scale) in place. Same values."""
+ batch, t, h, w, _ = x.shape
+ qkv = attn.qkv # type: ignore[attr-defined]
+ shape = (batch, t, h, w, attn.num_heads, attn.head_dim) # type: ignore[attr-defined]
+ q = linear_into_phase_buffer(qkv.to_q, x).view(shape)
+ k = linear_into_phase_buffer(qkv.to_k, x).view(shape)
+ v = linear_into_phase_buffer(qkv.to_v, x).view(shape)
+ q = attn.q_norm.forward_(q, scale=attn.scale) # type: ignore[attr-defined]
+ k = attn.k_norm.forward_(k) # type: ignore[attr-defined]
return q, k, v
+
+
+def det_project_qkv_eager(attn: object, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:
+ """:func:`_det_project_qkv` on fresh projections with the norms (and Q's scale) in place. Same values."""
+ q, k, v = attn.project_qkv(x) # type: ignore[attr-defined]
+ q = attn.q_norm.forward_(q, scale=attn.scale) # type: ignore[attr-defined]
+ k = attn.k_norm.forward_(k) # type: ignore[attr-defined]
+ return q, k, v
+
+
+def det_rope_qk_(attn: object, q: torch.Tensor, k: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:
+ """The RoPE half of :func:`det_qkv_rope`, rotating contiguous ``q`` and ``k`` in place.
+ One :func:`~ltx_kernels.vae.rope_dsl.run_abs_rope` launch per tensor where it applies, else W slab by
+ W slab, each slab read in full before it is overwritten; both equal the out-of-place rotation bit for
+ bit. Eager callers only: the compiled det path keeps the pure ``ltx_core::abs_rope`` op.
+ """
+ inv_freqs, num_tiles, compute_dtype, cache = _rope_config(attn, q)
+ rope_split = attn.rope_dim_split # type: ignore[attr-defined]
+ for x in (q, k):
+ if not x.is_contiguous():
+ raise ValueError("det_rope_qk_ needs contiguous Q/K (fresh projections)")
+ if compute_dtype == torch.float32 and abs_rope_supported is not None and abs_rope_supported(x, rope_split):
+ run_abs_rope(x, rope_split, _rope_tables(x, inv_freqs, None, cache), out=x)
+ continue
+ w_off = 0
+ for slab in torch.chunk(x, num_tiles, dim=3):
+ w_slab = slab.shape[3]
+ w_pos = torch.arange(w_slab, dtype=torch.float32, device=x.device) + w_off
+ slab.copy_(
+ _apply_opaque_rope_slab(
+ slab,
+ rope_split,
+ inv_freqs,
+ w_pos=w_pos,
+ compute_dtype=compute_dtype,
+ w_pos_key=RopePositions(RopeAxis.W, w_slab, w_off),
+ cache=cache,
+ )
+ )
+ w_off = w_off + w_slab
+ return q, k
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/det_rms_norm.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/det_rms_norm.py
new file mode 100644
index 0000000..0082e65
--- /dev/null
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/det_rms_norm.py
@@ -0,0 +1,68 @@
+"""RMSNorm of the DiffVAE deterministic blocks: the fused row kernel or the uint32-safe norm, chosen per call."""
+
+from __future__ import annotations
+
+import torch
+from torch import nn
+
+from ltx_core.model.video_vae import decode_workspace
+from ltx_core.model.video_vae.ops import apply_rms_norm_uint32_safe
+from ltx_core.model.video_vae.transformer.row_rms_norm import row_rms_norm_, row_rms_norm_supported
+
+# Rows of a Q/K volume (or of a block input) normalised per chunk when the row kernel does not apply.
+_NORM_CHUNK_ROWS: int = 1 << 22
+
+
+class DetRMSNorm(nn.RMSNorm):
+ """``nn.RMSNorm`` (same parameters and state dict) for the det blocks' block and Q/K norms.
+ Traced, it is the uint32-safe functional norm (:func:`~ltx_core.model.video_vae.ops.rms_norm_uint32_safe`),
+ so compiled graphs keep the functional op. Eager, it goes through the fused row kernel
+ (:mod:`~ltx_core.model.video_vae.transformer.row_rms_norm`; ATen's arithmetic, the fp32 sum of squares may
+ round differently, 1 bf16 ulp on ~2e-6 of the elements) where that applies, else the uint32-safe norm
+ row chunk by row chunk; :meth:`forward` writes into a decode-workspace phase buffer when one is active,
+ :meth:`forward_` back into its input.
+ """
+
+ def _fused(self, rows: torch.Tensor) -> bool:
+ """Whether the fused row kernel serves this norm on 2-D ``rows``."""
+ if self.weight is None or self.eps is None or tuple(self.normalized_shape) != (rows.shape[-1],):
+ return False
+ return bool(row_rms_norm_supported(rows, self.weight))
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ """``norm(x)`` over the last dim, into a workspace phase buffer when one is active, else a fresh tensor."""
+ if torch.compiler.is_compiling() or not x.is_contiguous():
+ return apply_rms_norm_uint32_safe(self, x)
+ rows_in = x.view(-1, x.shape[-1])
+ fused = self._fused(rows_in)
+ if not decode_workspace.workspace_active():
+ if not fused:
+ return apply_rms_norm_uint32_safe(self, x)
+ out = torch.empty_like(x)
+ row_rms_norm_(rows_in, self.weight, float(self.eps), out=out.view(-1, x.shape[-1]))
+ return out
+ out = decode_workspace.phase_empty(tuple(x.shape), dtype=x.dtype, device=x.device)
+ rows_out = out.view(-1, x.shape[-1])
+ if fused:
+ row_rms_norm_(rows_in, self.weight, float(self.eps), out=rows_out)
+ return out
+ for r0 in range(0, rows_in.shape[0], _NORM_CHUNK_ROWS):
+ rows_out[r0 : r0 + _NORM_CHUNK_ROWS].copy_(
+ apply_rms_norm_uint32_safe(self, rows_in[r0 : r0 + _NORM_CHUNK_ROWS])
+ )
+ return out
+
+ def forward_(self, x: torch.Tensor, *, scale: float | None = None) -> torch.Tensor:
+ """``norm(x)`` (then ``* scale``, a second bf16 rounding as ``x.mul_(scale)`` after the norm is) written
+ back into a contiguous ``x``, RMSNorm being per row; a new tensor for a non-contiguous ``x``. Eager only."""
+ if not x.is_contiguous():
+ y = apply_rms_norm_uint32_safe(self, x)
+ return y if scale is None else y.mul_(scale)
+ rows = x.view(-1, x.shape[-1])
+ if self._fused(rows):
+ row_rms_norm_(rows, self.weight, float(self.eps), scale=scale)
+ return x
+ for r0 in range(0, rows.shape[0], _NORM_CHUNK_ROWS):
+ chunk = rows[r0 : r0 + _NORM_CHUNK_ROWS]
+ chunk.copy_(apply_rms_norm_uint32_safe(self, chunk))
+ return x if scale is None else x.mul_(scale)
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/attn.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/attn.py
index 99cc85e..2f17c94 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/attn.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/attn.py
@@ -35,16 +35,17 @@ def require_softmax_bound(attn: NeighborhoodAttention3D) -> torch.Tensor:
Apply registers a NaN 0-d buffer; SDOps overwrite it at load. Rejecting meta, empty,
and non-finite values catches both a missed key under ``assign=True`` (meta left
behind) and an apply-without-SDOps path that left the NaN sentinel on CUDA.
- The non-finite probe is skipped while ``torch.compile`` is tracing: ``isfinite().all()``
- is data-dependent and forces specialization that conflicts with the decoder's hard
- ``mark_dynamic`` on T/H/W. Meta/empty checks still run under compile; the NaN sentinel
- is caught on eager first-call warmups and non-compiled paths.
+ The non-finite probe is an asynchronous assert: it raises at once for a CPU buffer, and
+ as a device-side assert for a CUDA one, so the per-block call never blocks the host on
+ the GPU. It is skipped while ``torch.compile`` is tracing: ``isfinite().all()`` is
+ data-dependent and forces specialization that conflicts with the decoder's hard
+ ``mark_dynamic`` on T/H/W. Meta/empty checks still run under compile.
"""
bound = getattr(attn, NA_SOFTMAX_BOUND_BUFFER, None)
if bound is None or bound.is_meta or bound.numel() == 0:
raise RuntimeError(_SOFTMAX_BOUND_UNSET)
- if not torch.compiler.is_compiling() and not torch.isfinite(bound).all():
- raise RuntimeError(_SOFTMAX_BOUND_UNSET)
+ if not torch.compiler.is_compiling():
+ torch._assert_async(torch.isfinite(bound).all(), _SOFTMAX_BOUND_UNSET)
return bound
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/block.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/block.py
index 223805a..2d71a45 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/block.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/block.py
@@ -2,11 +2,15 @@
from __future__ import annotations
+from typing import TYPE_CHECKING, NamedTuple
+
import torch
+from ltx_core.devices import upload_from_host
from ltx_core.model.video_vae.transformer.blocks import DiffusionNABlock
from ltx_core.model.video_vae.transformer.dsl_kernels.attn import require_softmax_bound
from ltx_core.model.video_vae.transformer.layers import AdaLNZero, LinearPixelShuffleUpsample
+from ltx_core.types import HostDeviceTensorPair
#: The stage-4 upsample fused into ``context_proj``, packed by SDOps as one Linear per
#: pixel-shuffle subpixel: ``(P * C, Ctx)`` / ``(P * C,)``. Registered empty by
@@ -14,6 +18,19 @@ from ltx_core.model.video_vae.transformer.layers import AdaLNZero, LinearPixelSh
FUSED_CTX_WEIGHT_BUFFER = "fused_ctx_weight"
FUSED_CTX_BIAS_BUFFER = "fused_ctx_bias"
+if TYPE_CHECKING:
+ from ltx_core.model.video_vae.keyframes import KeyframePlanes
+
+
+class LiveKeyframes(NamedTuple):
+ """The anchor stream of one fused launch: only its live planes, since the kernel has no notion of validity."""
+
+ x: torch.Tensor
+ stage4_feat: torch.Tensor
+ """The *pre*-upsample stage-4 feature, like the video's."""
+ times: HostDeviceTensorPair
+ """The planes' times; the kernel builds its plane tables from the host tensor, without a device read."""
+
def fused_ctx_weights(block: DiffusionNABlock) -> "tuple[torch.Tensor, torch.Tensor] | None":
"""This block's packed fused-context weights, or ``None`` if they never landed.
@@ -94,8 +111,7 @@ class DSLDiffusionNABlock(DiffusionNABlock):
keyframe_x: torch.Tensor,
keyframe_stage4_feat: torch.Tensor,
modulation: tuple[torch.Tensor, ...],
- keyframe_times: torch.Tensor,
- keyframe_valid: torch.Tensor,
+ keyframe_planes: KeyframePlanes,
*,
drop_leading_frame: bool = True,
out: torch.Tensor | None = None,
@@ -107,35 +123,43 @@ class DSLDiffusionNABlock(DiffusionNABlock):
``drop_leading_frame`` is the video stream's alone: each plane is a ``T=1`` clip whose
temporal stride collapses, so the kernel always takes the subpixel a one-frame input
shuffles out to. That is a tiling property and must not leak into the anchor arm.
- ``keyframe_valid`` is applied by dropping the invalid planes before the launch and
+ ``keyframe_planes.valid`` is applied by dropping the invalid planes before the launch and
restoring their rows as zeros -- the kernel has no notion of validity, and dropping is
order-preserving, so the surviving planes rank identically under the ``(|dt|, index)``
- tie-break. Zero rows are the contract: an invalid plane's every key is masked.
+ tie-break. Zero rows are the contract: an invalid plane's every key is masked. The live planes
+ and the kernel's plane tables come from the planes' host tensors, so the launch never reads
+ the device.
"""
- keep = keyframe_valid.nonzero().flatten()
- if int(keep.numel()) == int(keyframe_x.shape[1]):
+ keyframe_times, keyframe_valid = keyframe_planes
+ keep_host = keyframe_valid.host_tensor.nonzero().flatten()
+ all_live = int(keep_host.numel()) == int(keyframe_x.shape[1])
+ if all_live:
return self._fused_forward_x_ctx(
x,
stage4_feat,
modulation,
drop_leading_frame=drop_leading_frame,
out=out,
- keyframes=(keyframe_x, keyframe_stage4_feat, keyframe_times),
+ keyframes=LiveKeyframes(keyframe_x, keyframe_stage4_feat, keyframe_times),
out_keyframes=out_keyframes,
)
# Dropping planes shortens the anchor stream, so a buffer sized for the full plane
# count no longer fits it -- and the result is scattered into a fresh tensor anyway.
# The anchor stream forgoes recycling for this launch; the video stream does not.
+ keep = upload_from_host(keep_host, keyframe_x.device)
video_out, live = self._fused_forward_x_ctx(
x,
stage4_feat,
modulation,
drop_leading_frame=drop_leading_frame,
out=out,
- keyframes=(
+ keyframes=LiveKeyframes(
keyframe_x.index_select(1, keep),
keyframe_stage4_feat.index_select(1, keep),
- keyframe_times.index_select(0, keep),
+ HostDeviceTensorPair(
+ keyframe_times.host_tensor.index_select(0, keep_host),
+ keyframe_times.device_tensor.index_select(0, keep),
+ ),
),
)
restored = keyframe_x.new_zeros(keyframe_x.shape)
@@ -152,15 +176,14 @@ class DSLDiffusionNABlock(DiffusionNABlock):
*,
drop_leading_frame: bool,
out: torch.Tensor | None,
- keyframes: "tuple[torch.Tensor, torch.Tensor, torch.Tensor] | None" = None,
+ keyframes: LiveKeyframes | None = None,
out_keyframes: torch.Tensor | None = None,
) -> "torch.Tensor | tuple[torch.Tensor, torch.Tensor]":
"""Marshal weights into :func:`ltx_kernels.vae.run_block_fna_dsl`.
- ``keyframes`` is ``(x, stage4_feat, times)`` for the anchor stream, and passing it
- makes the launch dual-stream and the return a pair. ``out_keyframes`` is that
- stream's recycled output buffer, the anchor-side counterpart of ``out``. Its stage-4 feature is the
- *pre*-upsample one, like the video's: the kernel runs the deferred hop over each
- plane in isolation, taking the temporal subpixel a one-frame input shuffles out to.
+ ``keyframes`` is the anchor stream, and passing it makes the launch dual-stream and the return a
+ pair. ``out_keyframes`` is that stream's recycled output buffer, the anchor-side counterpart of
+ ``out``. The kernel runs the deferred hop over each plane in isolation, taking the temporal
+ subpixel a one-frame input shuffles out to.
"""
from ltx_kernels.vae import run_block_fna_dsl # noqa: PLC0415
@@ -214,9 +237,10 @@ class DSLDiffusionNABlock(DiffusionNABlock):
{}
if keyframes is None
else {
- "x_keyframes": keyframes[0],
- "stage4_keyframes": keyframes[1],
- "keyframe_times": keyframes[2],
+ "x_keyframes": keyframes.x,
+ "stage4_keyframes": keyframes.stage4_feat,
+ "keyframe_times": keyframes.times.device_tensor,
+ "keyframe_times_host": keyframes.times.host_tensor,
"out_keyframes": out_keyframes,
}
),
@@ -227,4 +251,4 @@ class DSLDiffusionNABlock(DiffusionNABlock):
if keyframes is None:
return result.to(dtype=x.dtype)
video_out, keyframe_out = result
- return video_out.to(dtype=x.dtype), keyframe_out.to(dtype=keyframes[0].dtype)
+ return video_out.to(dtype=x.dtype), keyframe_out.to(dtype=keyframes.x.dtype)
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/chain.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/chain.py
index c3bcd19..eb177d1 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/chain.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/chain.py
@@ -2,22 +2,23 @@
from __future__ import annotations
+from typing import TYPE_CHECKING
+
import torch
from torch import nn
+from ltx_core.model.video_vae import decode_workspace
from ltx_core.model.video_vae.transformer.layers import ChannelLinear
-
-def _out_slack_rows() -> int:
- from ltx_kernels.vae import OUT_SLACK_ROWS # noqa: PLC0415
-
- return int(OUT_SLACK_ROWS)
+if TYPE_CHECKING:
+ from ltx_core.model.video_vae.keyframes import KeyframePlanes
def alloc_block_volume(shape: tuple[int, ...], *, device: torch.device, dtype: torch.dtype) -> torch.Tensor:
- """A ``(1,T,H,W,C)`` tensor whose storage carries the fused kernel's lane slack."""
+ """A ``(1,T,H,W,C)`` tensor whose storage carries the fused kernel's lane slack, leased from the active
+ decode workspace until its tile ends (``torch.empty`` style without one)."""
_, t, h, w, c = shape
- rows = torch.empty((t * h * w + _out_slack_rows(), c), device=device, dtype=dtype)
+ rows = decode_workspace.volume((t * h * w + decode_workspace.out_slack_rows(), c), device=device, dtype=dtype)
return rows[: t * h * w].view(1, t, h, w, c)
@@ -38,7 +39,7 @@ def _recyclable_rows(volume: torch.Tensor) -> torch.Tensor | None:
return None
channels = volume.shape[-1]
n_rows = volume.numel() // channels
- want = n_rows + _out_slack_rows()
+ want = n_rows + decode_workspace.out_slack_rows()
have = volume.untyped_storage().nbytes() // (volume.element_size() * channels)
if not volume.is_contiguous() or have < want:
return None
@@ -91,8 +92,7 @@ class DSLDiffusionBlockChain(nn.ModuleList):
keyframe_x: torch.Tensor,
keyframe_stage4_feat: torch.Tensor,
modulation: tuple[torch.Tensor, ...],
- keyframe_times: torch.Tensor,
- keyframe_valid: torch.Tensor,
+ keyframe_planes: KeyframePlanes,
*,
drop_leading_frame: bool = True,
) -> tuple[torch.Tensor, torch.Tensor]:
@@ -112,8 +112,7 @@ class DSLDiffusionBlockChain(nn.ModuleList):
keyframe_x,
keyframe_stage4_feat,
modulation,
- keyframe_times,
- keyframe_valid,
+ keyframe_planes,
drop_leading_frame=drop_leading_frame,
out=spare,
out_keyframes=keyframe_spare,
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/joint_attn.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/joint_attn.py
index 69d090d..a751cc5 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/joint_attn.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/joint_attn.py
@@ -22,6 +22,8 @@ in the kernel:
from __future__ import annotations
+from typing import TYPE_CHECKING
+
import torch
from ltx_core.model.video_vae.transformer.attention import (
@@ -34,6 +36,9 @@ from ltx_core.model.video_vae.transformer.dsl_kernels.attn import (
require_softmax_bound,
)
+if TYPE_CHECKING:
+ from ltx_core.model.video_vae.keyframes import KeyframePlanes
+
@torch.library.custom_op("ltx_core::na_attention_joint_dsl", mutates_args=())
def na_attention_joint_dsl( # noqa: PLR0913 -- a custom_op schema takes only flat tensors/ints
@@ -57,9 +62,14 @@ def na_attention_joint_dsl( # noqa: PLR0913 -- a custom_op schema takes only fl
from ltx_kernels.vae import run_na_attention_bound # noqa: PLC0415
from ltx_kernels.vae.keyframe_slots import nearest_keyframe_slots # noqa: PLC0415
+ from ltx_core.devices import upload_from_host # noqa: PLC0415
+
planes = int(keyframe_q.shape[1])
- keep = keyframe_valid.nonzero().flatten()
- live_times = keyframe_times.index_select(0, keep)
+ # Plane selection and slot tables are host work; free when the times and validity arrive on
+ # the host, one read of each otherwise.
+ keep_host = keyframe_valid.detach().cpu().nonzero().flatten()
+ live_times = keyframe_times.detach().cpu().index_select(0, keep_host)
+ keep = upload_from_host(keep_host, q.device)
video_out, keyframe_out = run_na_attention_bound(
q,
k,
@@ -143,13 +153,14 @@ class DSLJointAttention(JointNAAttentionCallable):
keyframe_q: torch.Tensor,
keyframe_k: torch.Tensor,
keyframe_v: torch.Tensor,
- keyframe_times: torch.Tensor,
- keyframe_valid: torch.Tensor,
+ keyframe_planes: KeyframePlanes,
) -> tuple[torch.Tensor, torch.Tensor]:
from ltx_core.model.video_vae.keyframes import KEYFRAME_CONTEXT_SLOTS # noqa: PLC0415
if not self._available or not dsl_joint_supported(attn, q):
- return self._fallback(attn, q, k, v, keyframe_q, keyframe_k, keyframe_v, keyframe_times, keyframe_valid)
+ return self._fallback(attn, q, k, v, keyframe_q, keyframe_k, keyframe_v, keyframe_planes)
+ # The op builds its plane tables on the host, so it gets the planes' host tensors.
+ keyframe_times, keyframe_valid = keyframe_planes.times.host_tensor, keyframe_planes.valid.host_tensor
dtype = keyframe_v.dtype
return na_attention_joint_dsl(
q.to(dtype=dtype),
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/fallback_na/__init__.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/fallback_na/__init__.py
index 68f8225..923cf35 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/fallback_na/__init__.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/fallback_na/__init__.py
@@ -19,6 +19,7 @@ from ltx_core.model.video_vae.transformer.fallback_na.joint_eager import (
)
if TYPE_CHECKING:
+ from ltx_core.model.video_vae.keyframes import KeyframePlanes
from ltx_core.model.video_vae.transformer.attention import NAAttentionCallable, NeighborhoodAttention3D
logger = logging.getLogger(__name__)
@@ -227,8 +228,7 @@ class TritonJointAttention:
keyframe_q: torch.Tensor,
keyframe_k: torch.Tensor,
keyframe_v: torch.Tensor,
- keyframe_times: torch.Tensor,
- keyframe_valid: torch.Tensor,
+ keyframe_planes: KeyframePlanes,
) -> tuple[torch.Tensor, torch.Tensor]:
from ltx_core.model.video_vae.keyframes import KEYFRAME_CONTEXT_SLOTS # noqa: PLC0415
@@ -239,8 +239,8 @@ class TritonJointAttention:
)
return na_attention_joint_triton(
*_joint_dtypes(q, k, v, keyframe_q, keyframe_k, keyframe_v),
- keyframe_times,
- keyframe_valid,
+ keyframe_planes.times.device_tensor,
+ keyframe_planes.valid.device_tensor,
list(attn.kernel_size),
KEYFRAME_CONTEXT_SLOTS,
)
@@ -261,15 +261,14 @@ class EagerJointAttention:
keyframe_q: torch.Tensor,
keyframe_k: torch.Tensor,
keyframe_v: torch.Tensor,
- keyframe_times: torch.Tensor,
- keyframe_valid: torch.Tensor,
+ keyframe_planes: KeyframePlanes,
) -> tuple[torch.Tensor, torch.Tensor]:
from ltx_core.model.video_vae.keyframes import KEYFRAME_CONTEXT_SLOTS # noqa: PLC0415
return na_attention_joint_eager(
*_joint_dtypes(q, k, v, keyframe_q, keyframe_k, keyframe_v),
- keyframe_times,
- keyframe_valid,
+ keyframe_planes.times.device_tensor,
+ keyframe_planes.valid.device_tensor,
list(attn.kernel_size),
KEYFRAME_CONTEXT_SLOTS,
)
@@ -296,11 +295,10 @@ class AutoJointAttention:
keyframe_q: torch.Tensor,
keyframe_k: torch.Tensor,
keyframe_v: torch.Tensor,
- keyframe_times: torch.Tensor,
- keyframe_valid: torch.Tensor,
+ keyframe_planes: KeyframePlanes,
) -> tuple[torch.Tensor, torch.Tensor]:
backend = self._triton if self._triton is not None and q.is_cuda else self._eager
- return backend(attn, q, k, v, keyframe_q, keyframe_k, keyframe_v, keyframe_times, keyframe_valid)
+ return backend(attn, q, k, v, keyframe_q, keyframe_k, keyframe_v, keyframe_planes)
class EagerSdpaAttention:
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/layers.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/layers.py
index f41a1b3..e44e01b 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/layers.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/layers.py
@@ -9,6 +9,9 @@ import torch.nn.functional as F
from einops import rearrange
from torch import nn
+# Bytes of projection output per chunk of LinearPixelShuffleUpsample's eager forward.
+_SHUFFLE_CHUNK_BYTES: int = 1 << 30
+
class ChannelLinear(nn.Linear):
"""``nn.Linear`` exposing ``in_channels``/``out_channels`` for config introspection."""
@@ -47,18 +50,41 @@ class LinearPixelShuffleUpsample(nn.Module):
frame of its own to drop, since the one duplicate frame in the full
(untiled) tensor belongs solely to the origin chunk.
"""
- x = self.proj(x)
- x = rearrange(
- x,
- "b t h w (c p1 p2 p3) -> b (t p1) (h p2) (w p3) c",
- p1=self.stride[0],
- p2=self.stride[1],
- p3=self.stride[2],
- )
+ if torch.compiler.is_compiling():
+ x = self.proj(x)
+ x = rearrange(
+ x,
+ "b t h w (c p1 p2 p3) -> b (t p1) (h p2) (w p3) c",
+ p1=self.stride[0],
+ p2=self.stride[1],
+ p3=self.stride[2],
+ )
+ else:
+ x = self._proj_shuffle(x)
if self.stride[0] == 2 and drop_leading_frame:
x = x[:, 1:, :, :, :]
return x
+ def _proj_shuffle(self, x: torch.Tensor) -> torch.Tensor:
+ """``rearrange(proj(x), "b t h w (c p1 p2 p3) -> b (t p1) (h p2) (w p3) c")``, chunked over ``t``.
+ A per-row Linear plus a permutation, so the chunked result is the one-shot result; one output
+ volume is live plus at most ``_SHUFFLE_CHUNK_BYTES`` of projection.
+ """
+ b, t, h, w, _ = x.shape
+ p1, p2, p3 = self.stride
+ c = self.out_channels
+ out = torch.empty((b, t * p1, h * p2, w * p3, c), device=x.device, dtype=x.dtype)
+ # (b, t, p1, h, p2, w, p3, c): the output with every shuffled axis split back out.
+ out_split = out.view(b, t, p1, h, p2, w, p3, c)
+ frame_bytes = b * h * w * self.proj_out_channels * x.element_size()
+ chunk = max(1, min(t, _SHUFFLE_CHUNK_BYTES // max(frame_bytes, 1)))
+ for t0 in range(0, t, chunk):
+ t1 = min(t, t0 + chunk)
+ y = self.proj(x[:, t0:t1])
+ y = y.view(b, t1 - t0, h, w, c, p1, p2, p3).permute(0, 1, 5, 2, 6, 3, 7, 4)
+ out_split[:, t0:t1].copy_(y)
+ return out
+
class AdaLNZero(nn.Module):
"""Per-block AdaLN-Zero modulation: ``t_emb`` -> 7 (scale/shift/gate) chunks.
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/rope_math.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/rope_math.py
index a671a8c..482e12e 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/rope_math.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/rope_math.py
@@ -2,12 +2,111 @@
from __future__ import annotations
+import enum
+import threading
+from collections.abc import Callable, Hashable, Iterator
+from contextlib import contextmanager
+from dataclasses import dataclass
+
import numpy as np
import torch
DEFAULT_ABS_ROPE_NUM_TILES = 4
+class RopeAxis(enum.Enum):
+ """An axis of the channels-last ``(B, T, H, W, ...)`` volume abs-RoPE rotates."""
+
+ T = "t"
+ H = "h"
+ W = "w"
+
+
+@dataclass(frozen=True)
+class RopePositions:
+ """The integer positions ``offset + arange(extent)`` on ``axis``, named by value for the table cache."""
+
+ axis: RopeAxis
+ extent: int
+ offset: int = 0
+
+
+class AbsRopeTableCache:
+ """The abs-RoPE inputs one decode scope reuses: cos/sin tables by key and inverse frequencies by configuration.
+ Opened by :func:`abs_rope_table_cache`, which hands it to the decoder's attention modules for the scope.
+ Only the thread that opened it reads it, so every table is used on the stream order it was built in. Keys
+ name positions by value (:class:`RopePositions`) and inverse frequencies by this cache's own tensors, so a
+ cached table is exactly the one a fresh build would produce.
+ """
+
+ def __init__(self) -> None:
+ self._owner = threading.get_ident()
+ self._tables: dict[Hashable, tuple[torch.Tensor, ...]] = {}
+ self._inv_freqs: dict[tuple[int, float, torch.dtype, torch.device], torch.Tensor] = {}
+
+ def usable(self) -> bool:
+ """Whether this thread may read the cache: the opening thread, outside CUDA graph capture. Traced code
+ never reaches the cache (see ``det_attn_rope._rope_config``)."""
+ if threading.get_ident() != self._owner:
+ return False
+ return not (torch.cuda.is_available() and torch.cuda.is_current_stream_capturing())
+
+ def tables(self, key: Hashable, build: Callable[[], tuple[torch.Tensor, ...]]) -> tuple[torch.Tensor, ...]:
+ """``build()``, kept under ``key`` for the rest of the scope."""
+ tables = self._tables.get(key)
+ if tables is None:
+ tables = build()
+ self._tables[key] = tables
+ return tables
+
+ def inv_freqs(self, dim: int, base: float, dtype: torch.dtype, device: torch.device) -> torch.Tensor:
+ """:func:`rope_inv_freqs` as ``dtype`` on ``device``, one tensor per configuration for the scope.
+ Attention blocks of one configuration rotate with this tensor instead of their own equal-valued
+ buffers, so table keys (which name the frequencies by ``data_ptr``) match across blocks.
+ """
+ key = (dim, base, dtype, device)
+ inv = self._inv_freqs.get(key)
+ if inv is None:
+ inv = rope_inv_freqs(dim, base).to(device=device, dtype=dtype)
+ self._inv_freqs[key] = inv
+ return inv
+
+
+@contextmanager
+def abs_rope_table_cache(module: object) -> Iterator[AbsRopeTableCache]:
+ """Give ``module``'s attention blocks (submodules with a ``rope_table_cache`` slot) one
+ :class:`AbsRopeTableCache` for the scope, so same-extent tiles and blocks build their tables once.
+ A decoder that is not an ``nn.Module`` has no such blocks and gets a cache nothing reads.
+ Re-entrant on one thread (an inner scope uses the outer cache); the cache is dropped when the scope
+ that opened it exits. Blocks already holding another thread's cache keep it, and this thread builds
+ its tables per call there.
+ """
+ modules = module.modules() if isinstance(module, torch.nn.Module) else ()
+ holders = [m for m in modules if hasattr(m, "rope_table_cache")]
+ for holder in holders:
+ if holder.rope_table_cache is not None and holder.rope_table_cache.usable():
+ yield holder.rope_table_cache
+ return
+ cache = AbsRopeTableCache()
+ claimed = [m for m in holders if m.rope_table_cache is None]
+ for holder in claimed:
+ holder.rope_table_cache = cache
+ try:
+ yield cache
+ finally:
+ for holder in claimed:
+ holder.rope_table_cache = None
+
+
+def cached_rope_tables(
+ cache: AbsRopeTableCache | None, key: Hashable | None, build: Callable[[], tuple[torch.Tensor, ...]]
+) -> tuple[torch.Tensor, ...]:
+ """``build()``, reused under ``key`` from ``cache``; always rebuilt without a cache or a key."""
+ if cache is None or key is None:
+ return build()
+ return cache.tables(key, build)
+
+
def t_positions(t: int, device: torch.device) -> torch.Tensor:
return torch.arange(t, dtype=torch.float32, device=device)
@@ -44,17 +143,25 @@ def rot_abs_axis_impl(
axis: int,
*,
compute_dtype: torch.dtype,
+ pos_key: RopePositions | None = None,
+ cache: AbsRopeTableCache | None = None,
) -> torch.Tensor:
- """Absolute RoPE on one axis chunk ``xc[..., D]`` (D even) → new tensor."""
+ """Absolute RoPE on one axis chunk ``xc[..., D]`` (D even) → new tensor.
+ ``pos_key`` names ``pos`` by value, so the tables come from ``cache`` when one is given.
+ """
out_dtype = xc.dtype
pairs = xc.reshape(*xc.shape[:-1], xc.shape[-1] // 2, 2)
xe = pairs[..., 0].to(compute_dtype)
xo = pairs[..., 1].to(compute_dtype)
- shape = [1, 1, 1, 1, 1, inv.shape[0]]
- shape[axis] = pos.shape[0]
- ang = (pos[:, None] * inv[None, :]).reshape(shape)
- c = ang.cos().to(compute_dtype)
- s = ang.sin().to(compute_dtype)
+
+ def build() -> tuple[torch.Tensor, torch.Tensor]:
+ shape = [1, 1, 1, 1, 1, inv.shape[0]]
+ shape[axis] = pos.shape[0]
+ ang = (pos[:, None] * inv[None, :]).reshape(shape)
+ return ang.cos().to(compute_dtype), ang.sin().to(compute_dtype)
+
+ key = None if pos_key is None else ("axis", axis, pos_key, compute_dtype, inv.data_ptr(), inv.numel(), pos.device)
+ c, s = cached_rope_tables(cache, key, build)
re = xe * c - xo * s
ro = xe * s + xo * c
out = torch.stack([re, ro], dim=-1).reshape(xc.shape)
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/row_rms_norm.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/row_rms_norm.py
new file mode 100644
index 0000000..d6048a0
--- /dev/null
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/row_rms_norm.py
@@ -0,0 +1,106 @@
+# ruff: noqa: ANN001, ANN201, ANN202, N803
+
+"""Row RMSNorm for the DiffVAE in Triton: the in-place det-stage norms, and the norm the stage-5 tail fuses.
+ATen's arithmetic (fp32 ``rstd = rsqrt(mean(x^2) + eps)``, ``y = x * rstd * w``, one rounding to the activation
+dtype) at full bandwidth; the fp32 sum of squares may round differently (<1 bf16 ulp on a small fraction of rows).
+:func:`rms_rstd` is its row reduction, shared by :func:`row_rms_norm_` and
+:mod:`~ltx_core.model.video_vae.transformer.stage5_tail`.
+"""
+
+from __future__ import annotations
+
+import torch
+
+try:
+ import triton
+ import triton.language as tl
+except (ImportError, OSError): # pragma: no cover - platform without Triton
+ triton = None # type: ignore[assignment]
+ tl = None # type: ignore[assignment]
+
+_MAX_WIDTH = 1024
+_BLOCK_ELEMS = 8192
+
+
+def row_rms_norm_supported(x: torch.Tensor, weight: torch.Tensor) -> bool:
+ """Contiguous 2-D bf16 CUDA rows of width <= 1024, with a contiguous 1-D floating weight of that width on
+ the same device, and Triton."""
+ if triton is None or not x.is_cuda or x.dim() != 2 or not x.is_contiguous() or x.dtype != torch.bfloat16:
+ return False
+ if weight.dim() != 1 or weight.shape[0] != x.shape[1] or weight.device != x.device:
+ return False
+ if not weight.is_floating_point() or not weight.is_contiguous():
+ return False
+ return 1 <= x.shape[1] <= _MAX_WIDTH
+
+
+if triton is not None:
+
+ @triton.jit
+ def rms_rstd(x, N: tl.constexpr, eps):
+ """``rsqrt(mean(x^2) + eps)`` per row of fp32 ``(ROWS, BLOCK_N)`` rows of width ``N`` (zero-padded past it).
+ The caller scales: ``x * rstd[:, None] * w[None, :]``, then rounds.
+ """
+ return tl.rsqrt(tl.sum(x * x, axis=1) / N + eps)
+
+ @triton.jit
+ def _row_rms_norm_kernel(
+ x_ptr,
+ y_ptr,
+ w_ptr,
+ n_rows,
+ eps,
+ scale,
+ N: tl.constexpr,
+ BLOCK_N: tl.constexpr,
+ ROWS: tl.constexpr,
+ HAS_SCALE: tl.constexpr,
+ ):
+ pid = tl.program_id(0)
+ row0 = pid * ROWS
+ rows = row0 + tl.arange(0, ROWS)
+ cols = tl.arange(0, BLOCK_N)
+ row_ok = rows < n_rows
+ col_ok = cols < N
+ mask = row_ok[:, None] & col_ok[None, :]
+ offs = rows[:, None].to(tl.int64) * N + cols[None, :]
+ x = tl.load(x_ptr + offs, mask=mask, other=0.0).to(tl.float32)
+ w = tl.load(w_ptr + cols, mask=col_ok, other=0.0).to(tl.float32)
+ y = x * rms_rstd(x, N, eps)[:, None] * w[None, :]
+ if HAS_SCALE:
+ # The caller's ``q.mul_(scale)`` on the bf16 norm output: round first, then scale, then round.
+ y = y.to(tl.bfloat16).to(tl.float32) * scale
+ tl.store(y_ptr + offs, y.to(tl.bfloat16), mask=mask)
+
+
+def row_rms_norm_(
+ x: torch.Tensor, weight: torch.Tensor, eps: float, *, scale: float | None = None, out: torch.Tensor | None = None
+) -> torch.Tensor:
+ """``rms_norm(x, weight, eps)`` (``* scale`` afterwards, as a second bf16 rounding) row by row.
+ ``x`` is ``(rows, N)`` contiguous bf16 (see :func:`row_rms_norm_supported`). Written over ``x`` itself,
+ or into ``out`` (same shape/dtype/contiguity, must not partially overlap ``x``). Returns the written tensor.
+ """
+ rows, n = x.shape
+ y = x if out is None else out
+ if out is not None and (out.shape != x.shape or out.dtype != x.dtype or not out.is_contiguous()):
+ raise ValueError("out must be a contiguous tensor of x's shape and dtype")
+ if rows == 0:
+ return y
+ block_n = triton.next_power_of_2(n)
+ rows_per_cta = max(1, min(64, _BLOCK_ELEMS // block_n))
+ grid = (triton.cdiv(rows, rows_per_cta),)
+ w = weight if weight.dtype in (torch.bfloat16, torch.float32) else weight.to(torch.float32)
+ _row_rms_norm_kernel[grid](
+ x,
+ y,
+ w,
+ rows,
+ float(eps),
+ float(scale) if scale is not None else 1.0,
+ N=n,
+ BLOCK_N=block_n,
+ ROWS=rows_per_cta,
+ HAS_SCALE=scale is not None,
+ num_warps=4 if block_n <= 256 else 8,
+ )
+ return y
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/stage5_tail.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/stage5_tail.py
new file mode 100644
index 0000000..9c97fbc
--- /dev/null
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/stage5_tail.py
@@ -0,0 +1,152 @@
+# ruff: noqa: ANN001, N803, N806, PLR0913
+
+"""DiffVAE stage-5 tail in one pass: ``norm_out`` -> ``conv_out`` -> unpatchify.
+The eager tail makes four passes over the stage-5 volume (the normalised copy, the
+projection, a channels-first copy, the depth-to-space copy) and holds two extra
+full-size tensors. Here each block of tokens is normalised in fp32, rounded to the
+activation dtype where ``nn.RMSNorm`` rounds, projected on the tensor cores with the
+bias in the epilogue, and every output channel is stored straight to its pixel.
+"""
+
+from __future__ import annotations
+
+import torch
+
+try:
+ import triton
+ import triton.language as tl
+except ImportError: # pragma: no cover
+ triton = None # type: ignore[assignment]
+ tl = None # type: ignore[assignment]
+
+if triton is not None:
+ from ltx_core.model.video_vae.transformer.row_rms_norm import rms_rstd
+
+ @triton.jit
+ def _stage5_tail_kernel(
+ x_ptr,
+ norm_w_ptr,
+ proj_w_ptr,
+ proj_b_ptr,
+ out_ptr,
+ n_tokens,
+ F,
+ Hh,
+ Ww,
+ eps,
+ C: tl.constexpr,
+ OUT: tl.constexpr,
+ OUT_PAD: tl.constexpr,
+ PATCH: tl.constexpr,
+ BLOCK_T: tl.constexpr,
+ ) -> None:
+ """``out[b, c, f, h*P+q, w*P+r] = conv_out(norm_out(x))[b, f, h, w, c*P*P + r*P + q]``.
+ int64 offsets: the stage-5 volume passes 2**31 elements at 1080p.
+ """
+ tokens = tl.program_id(0).to(tl.int64) * BLOCK_T + tl.arange(0, BLOCK_T).to(tl.int64)
+ token_mask = tokens < n_tokens
+ channels = tl.arange(0, C)
+ x = tl.load(x_ptr + tokens[:, None] * C + channels[None, :], mask=token_mask[:, None], other=0.0)
+ x = x.to(tl.float32)
+ rstd = rms_rstd(x, C, eps)
+ normed = (x * rstd[:, None] * tl.load(norm_w_ptr + channels).to(tl.float32)[None, :]).to(x_ptr.dtype.element_ty)
+
+ outs = tl.arange(0, OUT_PAD)
+ out_mask = outs < OUT
+ w = tl.load(proj_w_ptr + outs[None, :] * C + channels[:, None], mask=out_mask[None, :], other=0.0)
+ acc = tl.dot(normed, w)
+ acc += tl.load(proj_b_ptr + outs, mask=out_mask, other=0.0).to(tl.float32)[None, :]
+
+ # (T, c*P*P + r*P + q) -> (c*P + q, t*P + r): each row is a run of consecutive pixels
+ # along W, so the store coalesces.
+ C_PAD: tl.constexpr = OUT_PAD // (PATCH * PATCH)
+ vals = tl.reshape(acc.to(out_ptr.dtype.element_ty), (BLOCK_T, C_PAD, PATCH, PATCH))
+ vals = tl.reshape(tl.permute(vals, (1, 3, 0, 2)), (C_PAD * PATCH, BLOCK_T * PATCH))
+ rows = tl.arange(0, C_PAD * PATCH)
+ cols = tl.arange(0, BLOCK_T * PATCH)
+ c = rows // PATCH
+ q = rows % PATCH
+ t = tl.program_id(0).to(tl.int64) * BLOCK_T + (cols // PATCH).to(tl.int64)
+ r = cols % PATCH
+ ww = t % Ww
+ hh = (t // Ww) % Hh
+ f = (t // Ww // Hh) % F
+ b = t // Ww // Hh // F
+ H = Hh * PATCH
+ W = Ww * PATCH
+ offsets = (
+ (
+ ((b[None, :] * (OUT // (PATCH * PATCH)) + c[:, None]) * F + f[None, :]) * H
+ + hh[None, :] * PATCH
+ + q[:, None]
+ )
+ * W
+ + ww[None, :] * PATCH
+ + r[None, :]
+ )
+ tl.store(out_ptr + offsets, vals, mask=(c < OUT // (PATCH * PATCH))[:, None] & (t < n_tokens)[None, :])
+
+
+def stage5_tail_supported(x: torch.Tensor, norm: torch.nn.RMSNorm, proj: torch.nn.Linear, patch_size: int) -> bool:
+ """Whether :func:`stage5_tail` takes this ``(B, T, H, W, C)`` channels-last volume.
+ Not while ``torch.compile`` traces: the compiled DiffVAE modes leave the tail to Inductor.
+ """
+ c = x.shape[-1]
+ return (
+ triton is not None
+ and not torch.compiler.is_compiling()
+ and x.is_cuda
+ and x.dim() == 5
+ and x.dtype in (torch.bfloat16, torch.float16)
+ and x.is_contiguous()
+ and c >= 16
+ and c & (c - 1) == 0
+ and norm.weight is not None
+ and proj.bias is not None
+ and proj.weight.dtype == x.dtype
+ and proj.out_features % (patch_size * patch_size) == 0
+ )
+
+
+def _block_tokens(device: torch.device) -> int:
+ """Tokens per program of the stage-5 tail kernel for ``device``'s architecture."""
+ major, _ = torch.cuda.get_device_capability(device)
+ if major == 10:
+ # Datacenter Blackwell. Swept at a 1080p stage-5 volume on sm_100.
+ return 128
+ return 32
+
+
+def stage5_tail(x: torch.Tensor, norm: torch.nn.RMSNorm, proj: torch.nn.Linear, patch_size: int) -> torch.Tensor:
+ """``unpatchify(proj(norm(x)).permute(0, 4, 1, 2, 3))`` as one kernel; ``(B, C_out, T, H*P, W*P)``.
+ Check :func:`stage5_tail_supported` first.
+ """
+ batch, frames, hh, ww, c = x.shape
+ out_features = proj.out_features
+ out = torch.empty(
+ (batch, out_features // (patch_size * patch_size), frames, hh * patch_size, ww * patch_size),
+ dtype=x.dtype,
+ device=x.device,
+ )
+ n_tokens = batch * frames * hh * ww
+ eps = norm.eps if norm.eps is not None else torch.finfo(x.dtype).eps
+ block_tokens = _block_tokens(x.device)
+ with torch.cuda.device(x.device):
+ _stage5_tail_kernel[(triton.cdiv(n_tokens, block_tokens),)](
+ x,
+ norm.weight,
+ proj.weight,
+ proj.bias,
+ out,
+ n_tokens,
+ frames,
+ hh,
+ ww,
+ eps,
+ C=c,
+ OUT=out_features,
+ OUT_PAD=max(16, triton.next_power_of_2(out_features)),
+ PATCH=patch_size,
+ BLOCK_T=block_tokens,
+ )
+ return out
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/swiglu.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/swiglu.py
index 21b92a2..fc235c9 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/swiglu.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/swiglu.py
@@ -228,6 +228,28 @@ if _TRITON_AVAILABLE:
)
+@dataclass(frozen=True)
+class _GateUpLaunch:
+ """Launch config of the fused gate+up kernel.
+ Every config accumulates each output in the same K order, so all are bit-identical.
+ """
+
+ blocks: tuple[int, int, int] # (BLOCK_M, BLOCK_N, BLOCK_K)
+ num_warps: int
+ num_stages: int
+
+
+def _gate_up_launch(device: torch.device) -> _GateUpLaunch:
+ """Fused gate+up kernel launch config for ``device``'s architecture."""
+ major, _ = torch.cuda.get_device_capability(device)
+ if major == 10:
+ # Datacenter Blackwell. Swept at the DiffVAE det-stage MLP shapes (dim 512..2048,
+ # 16k-row tiles) on sm_100.
+ return _GateUpLaunch(blocks=(256, 128, 64), num_warps=8, num_stages=3)
+ # Triton's default warps and stages.
+ return _GateUpLaunch(blocks=(64, 64, 32), num_warps=4, num_stages=3)
+
+
_DUAL_GATE_UP_DIM_MIN: Final[int] = 256
_DUAL_GATE_UP_DIM_MAX: Final[int] = 2048
@@ -270,7 +292,8 @@ def _fused_gate_up_swiglu_triton(
)
m, k = x.shape
n = w_gate.shape[0]
- block_m, block_n, block_k = 64, 64, 32
+ launch = _gate_up_launch(x.device)
+ block_m, block_n, block_k = launch.blocks
grid = (triton.cdiv(m, block_m), triton.cdiv(n, block_n))
with torch.cuda.device(x.device):
_fused_gate_up_swiglu_kernel[grid](
@@ -292,6 +315,8 @@ def _fused_gate_up_swiglu_triton(
BLOCK_M=block_m,
BLOCK_N=block_n,
BLOCK_K=block_k,
+ num_warps=launch.num_warps,
+ num_stages=launch.num_stages,
)
@@ -462,14 +487,24 @@ def swiglu_tiled(
return _swiglu_tiled_op(x, w_gate, w_up, w_down, tile_size, num_tiles, bool(use_triton))
-def plain_mlp(x: torch.Tensor, mlp: nn.Module, norm: nn.RMSNorm, tile: SwiGLUTileSpec) -> torch.Tensor:
- """Det ``NABlock``: ``x + swiglu_tiled(norm(x))`` (no AdaLN)."""
+def plain_mlp(x: torch.Tensor, mlp: nn.Module, norm: nn.Module, tile: SwiGLUTileSpec) -> torch.Tensor:
+ """Det ``NABlock``: ``x + swiglu_tiled(norm(x))`` (no AdaLN); ``norm`` is the block's
+ :class:`~ltx_core.model.video_vae.transformer.det_rms_norm.DetRMSNorm`."""
y = norm(x)
if y.numel() == 0:
return x
return x + swiglu_tiled(y, mlp.w_gate.weight, mlp.w_up.weight, mlp.w_down.weight, tile)
+def plain_mlp_accumulate(x: torch.Tensor, mlp: nn.Module, norm: nn.Module, tile: SwiGLUTileSpec) -> torch.Tensor:
+ """:func:`plain_mlp` with the residual added into the MLP's own fresh output: same values (addition
+ commutes exactly), one volume fewer live."""
+ y = norm(x)
+ if y.numel() == 0:
+ return x
+ return swiglu_tiled(y, mlp.w_gate.weight, mlp.w_up.weight, mlp.w_down.weight, tile).add_(x)
+
+
class SwiGLU(nn.Module):
"""Gated MLP weights: ``w_down(silu(w_gate(x)) * w_up(x))``."""
diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/video_vae.py b/packages/ltx-core/src/ltx_core/model/video_vae/video_vae.py
index c3f8d8d..565495d 100644
--- a/packages/ltx-core/src/ltx_core/model/video_vae/video_vae.py
+++ b/packages/ltx-core/src/ltx_core/model/video_vae/video_vae.py
@@ -10,7 +10,7 @@ from ltx_core.model.common.normalization import PixelNorm
from ltx_core.model.disposable import Disposable
from ltx_core.model.transformer.attention import AttentionCallable, AttentionFunction
from ltx_core.model.video_vae.attention import AttnBlock3D
-from ltx_core.model.video_vae.convolution import make_conv_nd
+from ltx_core.model.video_vae.convolution import make_conv_nd, memory_format_of
from ltx_core.model.video_vae.enums import LogVarianceType, NormLayerType, PaddingModeType
from ltx_core.model.video_vae.keyframes import DecodeKeyframes
from ltx_core.model.video_vae.ops import PerChannelStatistics, patchify
@@ -291,6 +291,9 @@ class VideoEncoder(nn.Module, Disposable):
# This reduces H,W by patch_size and increases channels, making convolutions more efficient
# Example: (B, 3, F, 512, 512) -> (B, 48, F, 128, 128) with patch_size=4
sample = patchify(sample, patch_size_hw=self.patch_size, patch_size_t=1)
+ # Activations follow the weights' layout: channels-last weights (CHANNELS_LAST_3D_WEIGHTS)
+ # keep the whole encoder on NHWC conv kernels.
+ sample = sample.contiguous(memory_format=memory_format_of(self.conv_in.weight))
sample = self.conv_in(sample)
for down_block in self.down_blocks:
@@ -335,7 +338,7 @@ class VideoEncoder(nn.Module, Disposable):
# Split into means and logvar, then normalize means
means, _ = torch.chunk(sample, 2, dim=1)
- return self.per_channel_statistics.normalize(means)
+ return self.per_channel_statistics.normalize(means).contiguous()
def tiled_encode(
self,
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/_broadcast.py b/packages/ltx-core/src/ltx_core/multigpu/broadcast.py
similarity index 55%
rename from packages/ltx-pipelines/src/ltx_pipelines/multigpu/_broadcast.py
rename to packages/ltx-core/src/ltx_core/multigpu/broadcast.py
index dd13306..f5b2777 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/_broadcast.py
+++ b/packages/ltx-core/src/ltx_core/multigpu/broadcast.py
@@ -1,7 +1,8 @@
-"""Store-based broadcast signaling between the relay rank and workers.
+"""Store-based broadcast signaling between a driver rank and workers.
Workers poll an incrementing counter in the process group's store instead of
-blocking directly on a collective, so an idle fleet doesn't trip the NCCL
-watchdog timeout while waiting for the next job.
+blocking directly on a collective. An idle worker therefore neither trips the
+NCCL watchdog timeout nor keeps an NCCL receive kernel spinning on its GPU
+while the driver is still producing what it will broadcast.
"""
from __future__ import annotations
@@ -14,7 +15,7 @@ import torch.distributed as dist
class BroadcastCoordinator:
- """Coordinates broadcast signaling between the relay and workers via a distributed store.
+ """Coordinates broadcast signaling between the driver and workers via a distributed store.
Uses an incrementing counter to signal when a broadcast is ready. Workers
poll for counter changes to avoid NCCL timeout issues during idle periods.
The counter wraps at a large value to prevent overflow.
@@ -24,7 +25,7 @@ class BroadcastCoordinator:
"""Initialize the broadcast coordinator.
Args:
store: Distributed store (TCPStore, PrefixStore, etc.) for coordination across ranks.
- is_driver: Whether this rank is the relay (signals broadcasts).
+ is_driver: Whether this rank is the driver (signals broadcasts).
"""
self.store = store
self.is_driver = is_driver
@@ -45,23 +46,36 @@ class BroadcastCoordinator:
Raises:
RuntimeError: If called on a non-driver rank.
"""
+ self.notify()
+ yield
+
+ def notify(self) -> None:
+ """Increment the signal counter so workers waiting on it proceed (driver only).
+ Raises:
+ RuntimeError: If called on a non-driver rank.
+ """
if not self.is_driver:
- raise RuntimeError("broadcast_context can only be called on driver rank")
+ raise RuntimeError("only the driver rank can signal a broadcast")
self.signal = (self.signal + 1) % self.signal_max
self.store.set(self.broadcast_key, str(self.signal))
- yield
- def wait_for_signal_change(self, poll_interval: float = 0.01) -> None:
- """Wait until the signal changes from the last seen value (worker only).
+ def wait_for_signal_change(self, poll_interval: float = 0.01, timeout: float | None = None) -> None:
+ """Wait for the driver's next signal (worker only).
+ Each call consumes exactly one signal. If the driver has signaled several times since
+ the previous call, this returns at once and the next call consumes the next signal, so
+ a worker that polls late never skips one.
Args:
poll_interval: Seconds to sleep between polls (default: 0.01).
+ timeout: Seconds to wait before raising; ``None`` waits indefinitely.
+ Raises:
+ TimeoutError: If no new signal arrives within ``timeout``.
"""
- while True:
- current = self.get_current_signal()
- if current != self.last_seen_signal:
- self.last_seen_signal = current
- return
+ deadline = None if timeout is None else time.monotonic() + timeout
+ while self.get_current_signal() == self.last_seen_signal:
+ if deadline is not None and time.monotonic() > deadline:
+ raise TimeoutError(f"no broadcast signal from the driver within {timeout} s")
time.sleep(poll_interval)
+ self.last_seen_signal = (self.last_seen_signal + 1) % self.signal_max
def get_current_signal(self) -> int:
"""Get the current signal value from the store."""
diff --git a/packages/ltx-core/src/ltx_core/multigpu/gemma/broadcast_wrapper.py b/packages/ltx-core/src/ltx_core/multigpu/gemma/broadcast_wrapper.py
index e8ce742..999ad43 100644
--- a/packages/ltx-core/src/ltx_core/multigpu/gemma/broadcast_wrapper.py
+++ b/packages/ltx-core/src/ltx_core/multigpu/gemma/broadcast_wrapper.py
@@ -6,8 +6,13 @@ import torch
import torch.distributed as dist
from ltx_core.model.disposable import Disposable
+from ltx_core.multigpu.broadcast import BroadcastCoordinator
from ltx_core.text_encoders.gemma.encoders.base_encoder import LTXGemmaTextEncoder
+# Bounds a wait on the source rank, which includes its Gemma load on a cold job and any prompt enhancement.
+# Matches NCCL's default collective timeout, which bounded the receive these waits precede.
+_SRC_WAIT_TIMEOUT_S = 600.0
+
class BroadcastGemmaWrapper(torch.nn.Module, Disposable):
"""Encoder/group plumbing, prompt enhancement, and result broadcast.
@@ -21,6 +26,10 @@ class BroadcastGemmaWrapper(torch.nn.Module, Disposable):
``dist.get_group_rank``.
dtype: Target dtype for output tensors.
device: Target device for output tensors.
+ coordinator: Store signal whose driver is ``src_rank``. When set, the other ranks wait for it on
+ the host before each broadcast from ``src_rank``, so their NCCL receive kernels do not spin
+ on a GPU the source rank's encoder is still running on. A wait raises ``TimeoutError``
+ after ``_SRC_WAIT_TIMEOUT_S``, e.g. when the source rank failed before broadcasting.
"""
def __init__(
@@ -30,6 +39,7 @@ class BroadcastGemmaWrapper(torch.nn.Module, Disposable):
src_rank: int,
dtype: torch.dtype = torch.bfloat16,
device: torch.device | None = None,
+ coordinator: BroadcastCoordinator | None = None,
) -> None:
super().__init__()
if device is None and torch.cuda.is_available():
@@ -40,6 +50,7 @@ class BroadcastGemmaWrapper(torch.nn.Module, Disposable):
self._rank = dist.get_rank(broadcast_group)
self._dtype = dtype
self._device = device
+ self._coordinator = coordinator
def encode(self, prompts: list[str]) -> list[tuple[tuple[torch.Tensor, ...], torch.Tensor]]:
"""Encode a batch of prompts to per-prompt hidden states; implemented by subclasses."""
@@ -86,6 +97,7 @@ class BroadcastGemmaWrapper(torch.nn.Module, Disposable):
return self._broadcast_str(result)
def _broadcast_str(self, value: str | None) -> str:
+ self._wait_for_src()
obj_list: list[str | None] = [value]
dist.broadcast_object_list(obj_list, src=self._src_rank, group=self._group)
result = obj_list[0]
@@ -99,6 +111,7 @@ class BroadcastGemmaWrapper(torch.nn.Module, Disposable):
src_rank: int,
) -> tuple[tuple[torch.Tensor, ...], torch.Tensor]:
"""Broadcast hidden states + attention mask via NCCL from ``src_rank``."""
+ self._wait_for_src(*(hidden_states or ()), attention_mask)
if self._rank == src_rank:
meta = [{"hs_shapes": [h.shape for h in hidden_states], "mask_shape": attention_mask.shape}]
else:
@@ -118,3 +131,14 @@ class BroadcastGemmaWrapper(torch.nn.Module, Disposable):
dist.broadcast(attention_mask, src=src_rank, group=self._group)
return hidden_states, attention_mask
+
+ def _wait_for_src(self, *src_outputs: torch.Tensor | None) -> None:
+ """Hold the other ranks on the host until the source rank has computed ``src_outputs``."""
+ if self._coordinator is None:
+ return
+ if self._rank != self._src_rank:
+ self._coordinator.wait_for_signal_change(poll_interval=0.001, timeout=_SRC_WAIT_TIMEOUT_S)
+ return
+ for device in {t.device for t in src_outputs if t is not None and t.is_cuda}:
+ torch.cuda.synchronize(device)
+ self._coordinator.notify()
diff --git a/packages/ltx-core/src/ltx_core/multigpu/sharded_sd.py b/packages/ltx-core/src/ltx_core/multigpu/sharded_sd.py
index 6eb55ee..6dd5c6a 100644
--- a/packages/ltx-core/src/ltx_core/multigpu/sharded_sd.py
+++ b/packages/ltx-core/src/ltx_core/multigpu/sharded_sd.py
@@ -1,5 +1,5 @@
"""Sharded state dict with distributed weight backup.
-Each rank stores ~1/N of the model weights. Bucketed broadcasts
+Each rank stores ~1/N of the model weights. Bucketed all-gather rounds
restore weights into a target state dict using only a small,
caller-provided staging buffer.
"""
@@ -7,7 +7,8 @@ caller-provided staging buffer.
from __future__ import annotations
import hashlib
-from dataclasses import dataclass
+from collections.abc import Iterable
+from dataclasses import dataclass, replace
import torch
import torch.distributed as dist
@@ -23,6 +24,16 @@ def _nbytes(t: torch.Tensor) -> int:
return t.numel() * t.element_size()
+# Staging slots start on a multiple of this many bytes, which keeps the multi-tensor copies' loads aligned.
+_SLOT_ALIGN = 256
+
+
+def _as_bytes(t: torch.Tensor, key: str) -> torch.Tensor:
+ if not t.is_contiguous():
+ raise RuntimeError(f"ShardedSD: tensor '{key}' is not contiguous")
+ return t.view(torch.uint8).view(-1)
+
+
@dataclass
class ShardedSD:
"""Sharded state dict with distributed weight backup.
@@ -32,7 +43,7 @@ class ShardedSD:
rollback, checkpoint recovery).
- Deterministic ownership: ``MD5(key) % world_size``
- Local storage only for owned keys (VRAM ≈ 1/world_size of model)
- - Bucketed broadcast using a single small staging buffer
+ - Bucketed all-gather rounds through a single small staging buffer
Usage::
backup = ShardedSD.from_state_dict(model.state_dict(), group)
staging = torch.empty(64 * 1024 * 1024, dtype=torch.uint8, device=device)
@@ -101,63 +112,71 @@ class ShardedSD:
_owner_groups=owner_groups,
)
+ def subset(self, keys: Iterable[str]) -> ShardedSD:
+ """The same shards restricted to ``keys`` (no copies): restoring it writes only those keys.
+ Every rank must pass the same keys, as for :meth:`broadcast_shards_into`.
+ """
+ wanted = frozenset(keys)
+ return replace(
+ self,
+ keys=tuple(k for k in self.keys if k in wanted),
+ key_sizes={k: v for k, v in self.key_sizes.items() if k in wanted},
+ owner_of={k: v for k, v in self.owner_of.items() if k in wanted},
+ local_shard={k: v for k, v in self.local_shard.items() if k in wanted},
+ _owner_groups={r: [k for k in ks if k in wanted] for r, ks in self._owner_groups.items()},
+ )
+
def broadcast_shards_into(
self,
target_sd: dict[str, torch.Tensor],
staging: torch.Tensor,
) -> None:
- """Broadcast stored weights from sharded backup into *target_sd*.
+ """Restore the stored weights from the sharded backup into *target_sd*.
This is a **cooperative operation** — all ranks in the process group
must call it simultaneously.
- *staging* is a caller-owned ``uint8`` scratch buffer; its size sets the
- broadcast granularity (tensors larger than it split across rounds). It
- may be shared by instances that never broadcast at the same time. Writes
- directly into existing tensors in *target_sd*.
+ *staging* is a caller-owned ``uint8`` scratch buffer: one send slot plus ``world``
+ receive slots are carved from it, and tensors larger than a slot split across rounds.
+ It may be shared by instances that never broadcast at the same time. In round ``i``
+ every rank packs its ``i``-th bucket and one ``all_gather_into_tensor`` ships all of
+ them, so every link direction carries data at once; the pack and the unpack of a round
+ are single multi-tensor copies. Writes directly into existing tensors in *target_sd*.
"""
if staging.dtype != torch.uint8 or staging.numel() == 0:
raise ValueError("staging must be a non-empty uint8 buffer")
- for owner, klist in self._owner_groups.items():
- if klist:
- self._broadcast_group(owner, klist, target_sd, staging)
-
- def _broadcast_group(
- self,
- owner: int,
- keys: list[str],
- target_sd: dict[str, torch.Tensor],
- staging: torch.Tensor,
- ) -> None:
- """Pack & broadcast params from *owner*, splitting tensors across rounds."""
- rounds = self._plan_rounds(keys, staging.numel())
-
- for round_chunks in rounds:
- filled = 0
- if self.rank == owner:
- for k, offset, chunk_size in round_chunks:
- src = self.local_shard[k]
- if not src.is_contiguous():
- raise RuntimeError(f"ShardedSD: local shard tensor '{k}' is not contiguous")
- src_bytes = src.view(torch.uint8).view(-1)
- staging[filled : filled + chunk_size].copy_(
- src_bytes[offset : offset + chunk_size], non_blocking=True
- )
- filled += chunk_size
- else:
- filled = sum(chunk_size for (_, _, chunk_size) in round_chunks)
-
- if filled == 0:
- continue
- view = staging[:filled]
- dist.broadcast(view, src=owner, group=self.group)
-
- cursor = 0
- for k, offset, chunk_size in round_chunks:
- dst = target_sd[k]
- if not dst.is_contiguous():
- raise RuntimeError(f"ShardedSD: target tensor '{k}' is not contiguous")
- dst_bytes = dst.view(torch.uint8).view(-1)
- dst_bytes[offset : offset + chunk_size].copy_(staging[cursor : cursor + chunk_size], non_blocking=True)
- cursor += chunk_size
+ slot = (staging.numel() // (self.world + 1)) // _SLOT_ALIGN * _SLOT_ALIGN
+ if slot <= 0:
+ raise ValueError(f"staging buffer of {staging.numel()} bytes is too small for {self.world} ranks")
+ plans = {owner: self._plan_rounds(keys, slot) for owner, keys in self._owner_groups.items()}
+ n_rounds = max((len(rounds) for rounds in plans.values()), default=0)
+ send = staging[:slot]
+ recv = staging[slot : slot + self.world * slot]
+ for i in range(n_rounds):
+ mine = plans[self.rank][i] if i < len(plans[self.rank]) else []
+ if mine:
+ pack_dst: list[torch.Tensor] = []
+ pack_src: list[torch.Tensor] = []
+ filled = 0
+ for key, offset, chunk in mine:
+ pack_dst.append(send[filled : filled + chunk])
+ pack_src.append(_as_bytes(self.local_shard[key], key)[offset : offset + chunk])
+ filled += chunk
+ torch._foreach_copy_(pack_dst, pack_src)
+ dist.all_gather_into_tensor(recv, send, group=self.group)
+ unpack_dst: list[torch.Tensor] = []
+ unpack_src: list[torch.Tensor] = []
+ for owner, rounds in plans.items():
+ if i >= len(rounds):
+ continue
+ cursor = owner * slot
+ for key, offset, chunk in rounds[i]:
+ dst = target_sd[key]
+ # A zero-copy shard is the target tensor itself: nothing to write back.
+ if owner != self.rank or dst.data_ptr() != self.local_shard[key].data_ptr():
+ unpack_dst.append(_as_bytes(dst, key)[offset : offset + chunk])
+ unpack_src.append(recv[cursor : cursor + chunk])
+ cursor += chunk
+ if unpack_dst:
+ torch._foreach_copy_(unpack_dst, unpack_src)
def _plan_rounds(self, keys: list[str], capacity: int) -> list[list[tuple[str, int, int]]]:
"""Build rounds that pack a *capacity*-byte buffer, splitting tensors if needed.
diff --git a/packages/ltx-core/src/ltx_core/multigpu/transformer/attention.py b/packages/ltx-core/src/ltx_core/multigpu/transformer/attention.py
index 2f9e7e2..5da68fd 100644
--- a/packages/ltx-core/src/ltx_core/multigpu/transformer/attention.py
+++ b/packages/ltx-core/src/ltx_core/multigpu/transformer/attention.py
@@ -60,6 +60,11 @@ class AttentionManager:
)
self.group = group
self._all2all_timeout_seconds = _DEFAULT_ALL2ALL_TIMEOUT_SECONDS
+ # Padded tokens in the current forward's video sequence, one at the end of each of the last
+ # `key_padding` shards (see `tile_modality_for_rank`), which the redistributions cut from the
+ # gathered keys. Set by the SP model wrapper per forward; host state the block loop branches
+ # on, so Dynamo guards on it and the wrapper adds it to the CUDA-graph capture key.
+ self.key_padding = 0
@property
def all2all_timeout_seconds(self) -> float:
@@ -90,6 +95,24 @@ class AttentionManager:
return out
+def drop_padding_keys(t: torch.Tensor, world_size: int, padding: int, dim: int = 1) -> torch.Tensor:
+ """Drop the padded tokens from a gathered sequence in shard order along ``dim``.
+ Each of the last ``padding`` of the ``world_size`` equal shards ends with one padded token (the
+ layout of :func:`~ltx_core.multigpu.transformer.sequence_parallel.tile_modality_for_rank`). With
+ one, it is the last token and the result is a view.
+ """
+ if padding == 0:
+ return t
+ dim %= t.ndim
+ tokens = t.shape[dim]
+ if padding == 1:
+ return t.narrow(dim, 0, tokens - 1)
+ shard = tokens // world_size
+ full = (world_size - padding) * shard
+ short = t.narrow(dim, full, tokens - full).unflatten(dim, (padding, shard)).narrow(dim + 1, 0, shard - 1)
+ return torch.cat([t.narrow(dim, 0, full), short.flatten(dim, dim + 1)], dim=dim)
+
+
class _All2AllRedistribute:
"""Shared redistribute/gather pipeline for self-attention SP wrappers.
Folds the head dim view-and-shuffle so the masked and unmasked variants only
@@ -113,6 +136,10 @@ class _All2AllRedistribute:
t_q, t_k, t_v = self.manager.send_recv_qkv(q, k, v)
local_heads = heads // self.manager.world_size
+ # The gathered keys hold the sharding's padded tokens; cut them here, so attention only ever
+ # sees real keys. Queries keep them: their rows are dropped when the outputs are gathered.
+ t_k = drop_padding_keys(t_k, self.manager.world_size, self.manager.key_padding)
+ t_v = drop_padding_keys(t_v, self.manager.world_size, self.manager.key_padding)
# `flatten` / `unflatten` collapse only the head dims, avoiding a `-1` in the
# seq position -- that would otherwise be ambiguous if the seq is 0 for a
@@ -146,6 +173,12 @@ class All2AllAttention(AttentionCallable):
class MaskedAll2AllAttention(MaskedAttentionCallable):
+ """Masked SP attention, for a caller's own mask.
+ The mask is in the shard order the keys are gathered in
+ (:func:`~ltx_core.multigpu.transformer.sequence_parallel.tile_modality_for_rank`); its columns for
+ the padded keys the redistribution cuts are cut with them.
+ """
+
def __init__(self, manager: AttentionManager, original_attention: MaskedAttentionCallable):
self._sp = _All2AllRedistribute(manager)
self.original_attention = original_attention
@@ -159,6 +192,7 @@ class MaskedAll2AllAttention(MaskedAttentionCallable):
mask: torch.Tensor,
) -> torch.Tensor:
t_q, t_k, t_v, local_heads, head_dim = self._sp.redistribute(q, k, v, heads)
+ mask = drop_padding_keys(mask, self._sp.manager.world_size, self._sp.manager.key_padding, dim=-1)
hidden_states = self.original_attention(q=t_q, k=t_k, v=t_v, heads=local_heads, mask=mask)
return self._sp.gather(hidden_states, local_heads, head_dim)
@@ -193,6 +227,9 @@ class _AudioAll2AllRedistribute:
t_q = q[:, :, heads_per_rank * rank : heads_per_rank * (rank + 1), :].clone()
t_k = self.manager.all2all_k.send_recv_heads(k, copy_out=self.manager.copy_out)
t_v = self.manager.all2all_v.send_recv_heads(v, copy_out=self.manager.copy_out)
+ # The keys are the sharded video sequence: cut its padded tokens, as for self-attention.
+ t_k = drop_padding_keys(t_k, self.manager.world_size, self.manager.key_padding)
+ t_v = drop_padding_keys(t_v, self.manager.world_size, self.manager.key_padding)
# `flatten` / `unflatten` collapse only the head dims, avoiding a `-1` in the
# seq position -- that would otherwise be ambiguous if the seq is 0 for a
diff --git a/packages/ltx-core/src/ltx_core/multigpu/transformer/sequence_parallel.py b/packages/ltx-core/src/ltx_core/multigpu/transformer/sequence_parallel.py
index 6624263..81498fd 100644
--- a/packages/ltx-core/src/ltx_core/multigpu/transformer/sequence_parallel.py
+++ b/packages/ltx-core/src/ltx_core/multigpu/transformer/sequence_parallel.py
@@ -16,6 +16,7 @@ import torch
from ltx_core.guidance.perturbations import BatchedPerturbationConfig
from ltx_core.loader.module_ops import ModuleOps
from ltx_core.model.transformer.attention import Attention
+from ltx_core.model.transformer.cudagraph_capture import capture_key_scope
from ltx_core.model.transformer.modality import Modality
from ltx_core.model.transformer.model import LTXModel
from ltx_core.model.transformer.transformer import BasicAVTransformerBlock
@@ -72,11 +73,6 @@ def pad_modality_for_uniform_sharding(
latent_pad = torch.zeros(b, pad, modality.latent.shape[2], dtype=dtype, device=device)
latent = torch.cat([modality.latent, latent_pad], dim=1)
- timesteps_pad_shape = list(modality.timesteps.shape)
- timesteps_pad_shape[1] = pad
- timesteps_pad = torch.zeros(timesteps_pad_shape, dtype=modality.timesteps.dtype, device=modality.timesteps.device)
- timesteps = torch.cat([modality.timesteps, timesteps_pad], dim=1)
-
positions_pad_shape = list(modality.positions.shape)
positions_pad_shape[2] = pad
positions_pad = torch.zeros(positions_pad_shape, dtype=modality.positions.dtype, device=modality.positions.device)
@@ -113,7 +109,7 @@ def pad_modality_for_uniform_sharding(
padded = replace(
modality,
latent=latent,
- timesteps=timesteps,
+ timesteps=modality.timesteps.pad_tokens(pad),
positions=positions,
attention_mask=attention_mask,
keyframes_mask=keyframes_mask,
@@ -132,51 +128,87 @@ def compute_sequence_offsets(token_counts: list[int]) -> list[int]:
return [0, *accumulate(token_counts[:-1])]
+def spread_padding_order(num_tokens: int, padded_tokens: int, world_size: int, device: torch.device) -> torch.Tensor:
+ """Source row of each position of a padded sequence laid out shard by shard, with the padding spread.
+ ``padded_tokens`` is ``num_tokens`` rounded up to a multiple of ``world_size``, with the padded
+ rows at its end (:func:`pad_modality_for_uniform_sharding`). Every shard holds
+ ``padded_tokens // world_size`` rows: the first shards only real tokens, each of the last
+ ``padded_tokens - num_tokens`` shards one real token fewer and then one padded row. Real tokens
+ keep their order, so the real rows of the shards, concatenated, are the original sequence.
+ Built on ``device`` from ``arange``, without a host sync.
+ """
+ shard = padded_tokens // world_size
+ full = world_size - (padded_tokens - num_tokens)
+ position = torch.arange(padded_tokens, device=device)
+ rank, offset = position // shard, position % shard
+ short = (rank - full).clamp(min=0)
+ real = rank * shard - short + offset
+ return torch.where((rank < full) | (offset < shard - 1), real, num_tokens + short)
+
+
def tile_modality_for_rank(
modality: Modality,
rank: int,
world_size: int,
+ num_tokens: int | None = None,
) -> tuple[Modality, list[int]]:
"""
Tile a modality's tensors for a specific GPU rank.
Splits the sequence dimension (dim 1 for latent/timesteps, dim 2 for positions)
across GPUs, returning the local tile for the given rank.
+ ``num_tokens`` is the real token count of a modality :func:`pad_modality_for_uniform_sharding`
+ padded. The padding is then spread one row per shard over the last shards
+ (:func:`spread_padding_order`), so no rank holds more than one padded token, and the
+ attention mask is reordered the same way: the sequence every rank gathers in the all-to-all
+ is in shard order.
Args:
modality: The modality to tile.
rank: Current GPU rank.
world_size: Total number of GPUs.
+ num_tokens: Real (unpadded) token count; ``None`` when the modality is not padded.
Returns:
- Tuple of (tiled_modality, token_counts_per_rank).
+ Tuple of (tiled_modality, real token count per rank).
"""
total_tokens = modality.latent.shape[1]
token_counts = compute_sequence_partition(total_tokens, world_size)
- offsets = compute_sequence_offsets(token_counts)
-
- start = offsets[rank]
- end = start + token_counts[rank]
+ shard = token_counts[rank]
+ start = rank * shard
+ padding = 0 if num_tokens is None else total_tokens - num_tokens
+ if padding == 0:
+ rows: slice | torch.Tensor = slice(start, start + shard)
+ attention_mask = modality.attention_mask
+ else:
+ order = spread_padding_order(num_tokens, total_tokens, world_size, modality.latent.device)
+ rows = order[start : start + shard]
+ attention_mask = modality.attention_mask
+ if attention_mask is not None:
+ attention_mask = attention_mask.index_select(-1, order)
+ if attention_mask.shape[-2] > 1:
+ attention_mask = attention_mask.index_select(-2, order)
+ token_counts = [shard] * (world_size - padding) + [shard - 1] * padding
# Tile latent: (B, T, D) -> (B, T_local, D)
- tiled_latent = modality.latent[:, start:end, :]
-
- # Tile timesteps: (B, T) -> (B, T_local)
- tiled_timesteps = modality.timesteps[:, start:end]
+ tiled_latent = modality.latent[:, rows, :]
# Tile positions: (B, 3, T, 2) -> (B, 3, T_local, 2)
- tiled_positions = modality.positions[:, :, start:end, :]
+ tiled_positions = modality.positions[:, :, rows]
# Tile keyframes_mask: (B, T, 1) -> (B, T_local, 1). Sharding splits the sequence, so each
# rank must carry the marker for its own slice -- the embedding is applied per rank, before
# the all-to-all.
tiled_keyframes_mask = None
if modality.keyframes_mask is not None:
- tiled_keyframes_mask = modality.keyframes_mask[:, start:end]
+ tiled_keyframes_mask = modality.keyframes_mask[:, rows]
+ # Sharding narrows the per-token timesteps; `select_tokens` applies the slice to whichever
+ # tensor carries the token dimension.
tiled_modality = replace(
modality,
latent=tiled_latent,
- timesteps=tiled_timesteps,
+ timesteps=modality.timesteps.select_tokens(rows),
positions=tiled_positions,
keyframes_mask=tiled_keyframes_mask,
+ attention_mask=attention_mask,
)
return tiled_modality, token_counts
@@ -293,13 +325,17 @@ class SequenceParallelModelWrapper(torch.nn.Module):
if video is None:
return self.model(video, audio, perturbations)
- # Pad the video seq dim up to a multiple of world_size so all ranks get
- # equal shards. The attention mask we attach makes the padded keys
- # invisible to attention; padded rows are sliced off after the gather.
+ # Pad the video seq dim up to a multiple of world_size so all ranks get equal shards. The
+ # attention redistributions cut the padded keys (`key_padding`), so a modality without a mask
+ # of its own goes in without the padding mask: attention only ever sees real keys. The
+ # padded rows are dropped when the outputs are gathered.
+ has_mask = video.attention_mask is not None
video, t_orig = pad_modality_for_uniform_sharding(video, self.attention_manager.world_size)
+ if not has_mask:
+ video = replace(video, attention_mask=None)
video_tile, token_counts = tile_modality_for_rank(
- video, self.attention_manager.rank, self.attention_manager.world_size
+ video, self.attention_manager.rank, self.attention_manager.world_size, num_tokens=t_orig
)
total_tokens = sum(token_counts)
if total_tokens > self.attention_manager.max_tokens:
@@ -307,11 +343,18 @@ class SequenceParallelModelWrapper(torch.nn.Module):
f"Total video token count ({total_tokens}) exceeds attention_manager max_tokens "
f"({self.attention_manager.max_tokens}). Use a smaller resolution or fewer frames."
)
- torch.distributed.barrier(self.attention_manager.group)
- video, audio = self.model(video_tile, audio, perturbations)
+ # This must remain a device-side collective: the first all2all requires aligned ranks,
+ # while the host must remain free to queue the next forward.
+ torch.distributed.all_reduce(
+ torch.zeros(1, device=video_tile.latent.device), group=self.attention_manager.group
+ )
+ # The padding layout is host state the redistributions branch on (`key_padding`) but no input
+ # shape carries: Dynamo guards on it, and the scope keys the CUDA-graph captures on it.
+ padding = video.latent.shape[1] - t_orig
+ self.attention_manager.key_padding = padding
+ with capture_key_scope(("sequence_parallel_padding", padding)):
+ video, audio = self.model(video_tile, audio, perturbations)
+ # Each rank's real tokens lead its shard, so gathering the real counts drops the padding and
+ # restores the caller's order and length.
video = gather_output_tokens(video, token_counts, self.attention_manager.group)
- # Unpad: drop the rows we added in `pad_modality_for_uniform_sharding` to make
- # the seq dim divisible by world_size, restoring the caller's original length.
- if video.shape[1] != t_orig:
- video = video[:, :t_orig, :]
return video, audio
diff --git a/packages/ltx-core/src/ltx_core/multigpu/vae/distributed_decoder.py b/packages/ltx-core/src/ltx_core/multigpu/vae/distributed_decoder.py
index 20b73f6..7377521 100644
--- a/packages/ltx-core/src/ltx_core/multigpu/vae/distributed_decoder.py
+++ b/packages/ltx-core/src/ltx_core/multigpu/vae/distributed_decoder.py
@@ -16,17 +16,24 @@ management — they are independent concerns.
from __future__ import annotations
+import contextlib
import logging
+import os
+import queue
+import threading
from collections.abc import Callable, Iterator, Sequence
from dataclasses import dataclass
+from typing import Any, NamedTuple
import torch
import torch.distributed as dist
from einops import rearrange
from torch.multiprocessing import Queue
+from ltx_core.devices import upload_from_host
from ltx_core.model.disposable import Disposable
from ltx_core.model.video_vae.keyframes import DecodeKeyframes
+from ltx_core.model.video_vae.transformer.rope_math import abs_rope_table_cache
from ltx_core.model.video_vae.video_vae import (
VideoDecoder,
clip_generators,
@@ -80,7 +87,7 @@ def dist_rank_tiles(
rank: int,
world_size: int,
) -> list[Tile]:
- """Round-robin Dist tiles for ``rank`` (same assignment as :meth:`DistributedVideoDecoder._decode_tiles`)."""
+ """Round-robin Dist tiles for ``rank`` (the assignment :meth:`DistributedVideoDecoder.decode_video` decodes)."""
if world_size < 1:
raise ValueError(f"world_size must be >= 1, got {world_size}")
if not 0 <= rank < world_size:
@@ -89,6 +96,231 @@ def dist_rank_tiles(
return [t for i, t in enumerate(all_tiles) if i % world_size == rank]
+# How long the driver waits for the ranks' end markers after the last batch, and for its queue
+# receiver thread once the decode ended; only a worker that died without its end marker runs it out.
+# LTX_MGPU_DECODE_RECEIVER_TIMEOUT_SECONDS overrides the 60 s default.
+_RECEIVER_JOIN_TIMEOUT_SECONDS = float(os.environ.get("LTX_MGPU_DECODE_RECEIVER_TIMEOUT_SECONDS", "60"))
+
+
+def _planes_with_shared_rope_tables(
+ decoder: torch.nn.Module, latents: list[torch.Tensor], gens: list[torch.Generator | None]
+) -> Iterator[torch.Tensor]:
+ """``iter_decoded_single_frames`` under one abs-RoPE table scope: every plane has the same extent, so
+ its tables are built once. Consume it on one thread."""
+ with abs_rope_table_cache(decoder):
+ yield from iter_decoded_single_frames(decoder, latents, gens)
+
+
+class _StreamAssembler:
+ """The driver's assembly of streamed chunks into temporal batches, consumed lazily.
+ Worker chunks arrive over the queue on a receiver thread and the driver's own tiles are
+ decoded on a producer thread, each chunk handed over with an event recorded on the stream
+ that made it. :meth:`batches` blends a batch on a side stream that waits only on those
+ events, as soon as every tile overlapping it has delivered its frames, and synchronizes the
+ side stream before yielding it: the batch is complete on every stream, so its consumer (the
+ MP4 encoder) can work on it while the decode keeps running on the default stream. A batch that
+ ``device_fn`` places on another GPU is blended on a side stream of that GPU, which reads the
+ chunks through the driver's side stream, and both are synchronized. Chunks are
+ blended in (rank, tile, frame) order -- what the whole-tile path summed in -- so the pixels
+ do not depend on arrival order.
+ """
+
+ def __init__(
+ self,
+ tiles_by_rank: dict[int, list[Tile]],
+ full_shape: VideoPixelShape,
+ num_temporal_batches: int,
+ weights: torch.Tensor | None,
+ device_fn: Callable[[int], str | torch.device],
+ *,
+ mp_queue: Queue, # type: ignore[type-arg]
+ driver_rank: int,
+ ) -> None:
+ self.tiles_by_rank = tiles_by_rank
+ self.keys = sorted((rank, index) for rank, tiles in tiles_by_rank.items() for index in range(len(tiles)))
+ self.chunks: dict[tuple[int, int], list[DecodedTile]] = {key: [] for key in self.keys}
+ self.delivered = dict.fromkeys(self.keys, 0)
+ self.full_shape = full_shape
+ self.grid = _temporal_batches(full_shape.frames, num_temporal_batches)
+ self.weights = weights
+ self.device_fn = device_fn
+ self.mp_queue = mp_queue
+ self.driver_rank = driver_rank
+ self.remote_ranks = sorted(rank for rank in tiles_by_rank if rank != driver_rank)
+ self.device = torch.device("cuda", torch.cuda.current_device()) if torch.cuda.is_available() else None
+ self.inference = torch.is_inference_mode_enabled()
+ self.handoff: queue.SimpleQueue = queue.SimpleQueue()
+ self.threads: list[threading.Thread] = []
+ self.batch_streams: dict[int, torch.cuda.Stream] = {}
+
+ def start(self, own: Iterator[tuple[int, DecodedTile]]) -> None:
+ """Start the receiver thread (worker chunks) and the producer thread (``own``, this rank's chunks)."""
+ if self.remote_ranks:
+ # A daemon, so a worker that died without its end marker cannot pin the process at exit.
+ receiver = threading.Thread(target=self._receive, name="dist-vae-recv", daemon=True)
+ receiver.start()
+ self.threads.append(receiver)
+ producer = threading.Thread(target=self._produce, args=(own,), name="dist-vae-decode")
+ producer.start()
+ self.threads.append(producer)
+
+ def _device_ctx(self) -> contextlib.AbstractContextManager:
+ return torch.cuda.device(self.device) if self.device is not None else contextlib.nullcontext()
+
+ def _produce(self, own: Iterator[tuple[int, DecodedTile]]) -> None:
+ try:
+ with torch.inference_mode(self.inference), self._device_ctx():
+ for index, chunk in own:
+ event = None
+ if chunk.pixels.is_cuda:
+ event = torch.cuda.Event()
+ event.record()
+ self.handoff.put(("chunk", self.driver_rank, index, chunk, event))
+ self.handoff.put(("end", self.driver_rank, None, None, None))
+ except BaseException as exc: # re-raised on the consuming thread
+ self.handoff.put(("error", self.driver_rank, None, exc, None))
+
+ def _receive(self) -> None:
+ pending = set(self.remote_ranks)
+ try:
+ # Unpickling a CUDA IPC tensor makes the current stream wait on its producer's event; a
+ # stream of this thread's own keeps that wait off the default stream the decode runs on.
+ stream = torch.cuda.Stream(device=self.device) if self.device is not None else None
+ stream_ctx = torch.cuda.stream(stream) if stream is not None else contextlib.nullcontext()
+ with self._device_ctx(), stream_ctx:
+ while pending:
+ rank, index, frames_start, pixels = self.mp_queue.get()
+ if index is None:
+ pending.discard(rank)
+ kind = "error" if frames_start == _DECODE_FAILED else "end"
+ self.handoff.put((kind, rank, None, None, None))
+ continue
+ event = None
+ if stream is not None and pixels.is_cuda:
+ event = torch.cuda.Event()
+ event.record(stream)
+ chunk = DecodedTile(pixels, self.tiles_by_rank[rank][index], frames_start)
+ self.handoff.put(("chunk", rank, index, chunk, event))
+ except BaseException as exc: # re-raised on the consuming thread
+ self.handoff.put(("error", -1, None, exc, None))
+
+ def _missing(self, batch: slice) -> bool:
+ for key in self.keys:
+ frames = self.tiles_by_rank[key[0]][key[1]].out_coords[0]
+ if frames.start < batch.stop and frames.start + self.delivered[key] < min(batch.stop, frames.stop):
+ return True
+ return False
+
+ def batches(self) -> Iterator[torch.Tensor]:
+ """The temporal batches in playback order, each complete on every stream when yielded."""
+ stream = torch.cuda.Stream(device=self.device) if self.device is not None else None
+ pending = {self.driver_rank, *self.remote_ranks}
+ try:
+ for b, batch in enumerate(self.grid):
+ while self._missing(batch):
+ if not pending:
+ raise RuntimeError("Dist decode: every rank finished before the batches were covered")
+ kind, rank, index, payload, event = self.handoff.get()
+ if kind != "chunk":
+ self._finish(pending, kind, rank, payload)
+ continue
+ if event is not None and stream is not None:
+ stream.wait_event(event)
+ payload.pixels.record_stream(stream)
+ self.chunks[(rank, index)].append(payload)
+ self.delivered[(rank, index)] = payload.frames_start + payload.pixels.shape[0]
+ yield self._blend(b, batch, stream)
+ self._drain(pending)
+ finally:
+ self.join()
+
+ @staticmethod
+ def _finish(pending: set[int], kind: str, rank: int, payload: object) -> None:
+ """Take ``rank``'s end marker off ``pending``; raise if it is a failure."""
+ pending.discard(rank)
+ if kind == "error":
+ raise RuntimeError(f"Dist decode: rank {rank} failed decoding its tiles") from (
+ payload if isinstance(payload, BaseException) else None
+ )
+
+ def _drain(self, pending: set[int]) -> None:
+ """Wait for the end markers still due after the last batch, raising on a failure marker.
+ A rank can deliver every frame the batches need and fail afterwards (its decode yielded
+ the wrong frame count, or raised on the way out); its failure marker comes after its
+ chunks, so the decode is complete only once every rank reported.
+ """
+ while pending:
+ try:
+ kind, rank, _index, payload, _event = self.handoff.get(timeout=_RECEIVER_JOIN_TIMEOUT_SECONDS)
+ except queue.Empty:
+ logger.warning(
+ "Dist decode: ranks %s sent no end marker %.0f s after the last batch; check their logs",
+ sorted(pending),
+ _RECEIVER_JOIN_TIMEOUT_SECONDS,
+ )
+ return
+ if kind != "chunk":
+ self._finish(pending, kind, rank, payload)
+
+ def _batch_stream(self, device: torch.device) -> torch.cuda.Stream | None:
+ """Side stream of the GPU ``device`` names when that is not the driver's GPU, else ``None``."""
+ if self.device is None or device.type != "cuda":
+ return None
+ index = self.device.index if device.index is None else device.index
+ if index == self.device.index:
+ return None
+ if index not in self.batch_streams:
+ self.batch_streams[index] = torch.cuda.Stream(device=index)
+ return self.batch_streams[index]
+
+ def _blend(self, b: int, batch: slice, stream: torch.cuda.Stream | None) -> torch.Tensor:
+ chunks = [chunk for key in self.keys for chunk in self.chunks[key]]
+ device = torch.device(self.device_fn(b))
+ batch_stream = self._batch_stream(device)
+ stream_ctx = torch.cuda.stream(stream) if stream is not None else contextlib.nullcontext()
+ # Entered last, so it is the current device and stream for the batch's own work, while the
+ # driver's side stream (waiting on the chunks' events) stays current for its GPU: copies of
+ # the chunks off the driver GPU run there, and the batch stream waits for them.
+ batch_ctx = torch.cuda.stream(batch_stream) if batch_stream is not None else contextlib.nullcontext()
+ with torch.inference_mode(self.inference), self._device_ctx(), stream_ctx, batch_ctx:
+ out = _assemble_batch(chunks, batch, self.full_shape.height, self.full_shape.width, device, self.weights)
+ # The batch must be complete when handed out, and its chunks read before their memory is reused.
+ if batch_stream is not None:
+ batch_stream.synchronize()
+ if stream is not None:
+ stream.synchronize()
+ for key in self.keys:
+ first = self.tiles_by_rank[key[0]][key[1]].out_coords[0].start
+ self.chunks[key] = [c for c in self.chunks[key] if first + c.frames_start + c.pixels.shape[0] > batch.stop]
+ return out
+
+ def join(self) -> None:
+ """Wait for the threads, which end on their own once the last chunk went out."""
+ for thread in self.threads:
+ thread.join(timeout=_RECEIVER_JOIN_TIMEOUT_SECONDS if thread.daemon else None)
+ if thread.is_alive():
+ logger.warning(
+ "Dist decode thread %s is still waiting after %.0f s: a worker rank ended without its "
+ "end marker; check that rank's log",
+ thread.name,
+ _RECEIVER_JOIN_TIMEOUT_SECONDS,
+ )
+ self.threads = []
+
+
+# A worker's last queue message after its chunks: its decode finished, or raised.
+_DECODE_DONE = 0
+_DECODE_FAILED = -1
+
+
+class OwnedSingleFrames(NamedTuple):
+ """One rank's share of a split single-frame decode (see ``DistributedVideoDecoder.owned_single_frames``)."""
+
+ indices: list[int]
+ planes: Iterator[torch.Tensor]
+ gather: Callable[[dict[int, Any]], dict[int, Any]]
+
+
def dist_rank_plane_indices(count: int, rank: int, world_size: int) -> list[int]:
"""Round-robin share of ``count`` independent single-frame clips owned by ``rank``.
The same assignment rule as :func:`dist_rank_tiles`, on a list of whole clips instead of
@@ -140,12 +372,15 @@ def dist_rank_tile_pixel_shape(
class DecodedTile:
"""A VAE-decoded tile with pixel-space placement.
Attributes:
- pixels: ``[F_tile, H_tile, W_tile, C]`` in the decoder's native dtype.
+ pixels: ``[F, H_tile, W_tile, C]`` in the decoder's native dtype: the tile's frames
+ ``frames_start .. frames_start + F`` (all of them for a whole tile).
pixel_tile: Carries ``out_coords`` (f, h, w slices) and separable ``masks_1d``.
+ frames_start: Offset of ``pixels[0]`` within the tile's frames.
"""
pixels: torch.Tensor
pixel_tile: Tile
+ frames_start: int = 0
# ------------------------------------------------------------------
@@ -161,6 +396,48 @@ def _to_decoded_tile(
return DecodedTile(pixels=pixels, pixel_tile=tile)
+def _temporal_batches(total_frames: int, num_temporal_batches: int) -> list[slice]:
+ batch_size = (total_frames + num_temporal_batches - 1) // num_temporal_batches
+ batches = [slice(b * batch_size, min((b + 1) * batch_size, total_frames)) for b in range(num_temporal_batches)]
+ return [b for b in batches if b.stop > b.start]
+
+
+def _assemble_batch(
+ tiles: Sequence[DecodedTile],
+ batch_range: slice,
+ output_height: int,
+ output_width: int,
+ device: str | torch.device,
+ weights: torch.Tensor | None,
+) -> torch.Tensor:
+ """Blend the frames of ``tiles`` (whole tiles or chunks of them) that fall in ``batch_range``."""
+ dtype = tiles[0].pixels.dtype
+ output = torch.zeros(
+ batch_range.stop - batch_range.start, output_height, output_width, 3, device=device, dtype=dtype
+ )
+ for tile in tiles:
+ f_slice, h_slice, w_slice = tile.pixel_tile.out_coords
+ first = f_slice.start + tile.frames_start
+ overlap = slice(max(batch_range.start, first), min(batch_range.stop, first + tile.pixels.shape[0]))
+ if overlap.start >= overlap.stop:
+ continue
+ pixel_frames = slice(overlap.start - first, overlap.stop - first)
+ mask_frames = slice(overlap.start - f_slice.start, overlap.stop - f_slice.start)
+ out_frames = slice(overlap.start - batch_range.start, overlap.stop - batch_range.start)
+ # Blend weights stay float32 so the multiply promotes bf16/fp16 pixels.
+ mf, mh, mw = (
+ upload_from_host(m.to(dtype=torch.float32), torch.device(device)) for m in tile.pixel_tile.masks_1d
+ )
+ pix = tile.pixels[pixel_frames].to(device=device, non_blocking=True)
+ # Channel axis is not on the pixel Tile; length-1 ones broadcasts over C.
+ output[out_frames, h_slice, w_slice, :] += scale_by_masks_1d(
+ pix, (mf[mask_frames], mh, mw, torch.ones(1, device=device, dtype=torch.float32))
+ )
+ if weights is not None:
+ output.div_(weights[batch_range.start : batch_range.stop].to(device=device)[:, :, :, None])
+ return output
+
+
def gather_frames(
tiles: list[DecodedTile],
total_frames: int,
@@ -180,42 +457,8 @@ def gather_frames(
"""
if device_fn is None:
device_fn = lambda b: f"cuda:{b % world_size}" # noqa: E731
-
- batch_size = (total_frames + num_temporal_batches - 1) // num_temporal_batches
-
- for b in range(num_temporal_batches):
- batch_range = slice(b * batch_size, min((b + 1) * batch_size, total_frames))
- batch_len = batch_range.stop - batch_range.start
- if batch_len <= 0:
- break
-
- device = device_fn(b)
- dtype = tiles[0].pixels.dtype
- output = torch.zeros(batch_len, output_height, output_width, 3, device=device, dtype=dtype)
-
- for tile in tiles:
- f_slice, h_slice, w_slice = tile.pixel_tile.out_coords
-
- overlap = slice(max(batch_range.start, f_slice.start), min(batch_range.stop, f_slice.stop))
- if overlap.start >= overlap.stop:
- continue
-
- tile_frames = slice(overlap.start - f_slice.start, overlap.stop - f_slice.start)
- out_frames = slice(overlap.start - batch_range.start, overlap.stop - batch_range.start)
-
- # Blend weights stay float32 so the multiply promotes bf16/fp16 pixels.
- mf, mh, mw = (m.to(device=device, dtype=torch.float32) for m in tile.pixel_tile.masks_1d)
- mf = mf[tile_frames]
- pix = tile.pixels[tile_frames].to(device=device, non_blocking=True)
- # Channel axis is not on the pixel Tile; length-1 ones broadcasts over C.
- output[out_frames, h_slice, w_slice, :] += scale_by_masks_1d(
- pix, (mf, mh, mw, torch.ones(1, device=device, dtype=torch.float32))
- )
-
- if weights is not None:
- batch_weights = weights[batch_range.start : batch_range.stop].to(device=device)
- output.div_(batch_weights[:, :, :, None])
- yield output
+ for b, batch_range in enumerate(_temporal_batches(total_frames, num_temporal_batches)):
+ yield _assemble_batch(tiles, batch_range, output_height, output_width, device_fn(b), weights)
# ------------------------------------------------------------------
@@ -274,6 +517,7 @@ class DistributedVideoDecoder(torch.nn.Module, Disposable):
if num_temporal_batches is not None and num_temporal_batches < 1:
raise ValueError(f"num_temporal_batches must be >= 1, got {num_temporal_batches}")
self.num_temporal_batches = self.world_size if num_temporal_batches is None else num_temporal_batches
+ self._assembler: _StreamAssembler | None = None
@property
def video_downscale_factors(self) -> SpatioTemporalScaleFactors:
@@ -302,53 +546,119 @@ class DistributedVideoDecoder(torch.nn.Module, Disposable):
device_fn: Callable[[int], str | torch.device] | None = None,
**_kwargs: object,
) -> Iterator[torch.Tensor]:
- """Distributed decode — all ranks decode, driver assembles.
- Not a generator so that worker side-effects (decode + queue.put)
- execute eagerly regardless of whether the caller iterates.
- 1. Each rank decodes its latent tile (with optional intra-GPU tiling).
- 2. Workers send their :class:`DecodedTile` to the driver via the queue.
- 3. The driver collects all tiles, blends overlaps, and returns
- temporal batches distributed across GPUs.
+ """Distributed decode -- all ranks decode, the driver assembles while the decode runs.
+ Not a generator, so that a worker's side effects (decode + queue.put) run eagerly whether or
+ not the caller iterates.
+ 1. Each rank decodes its latent tiles (with optional intra-GPU tiling), chunk by chunk.
+ 2. Workers put each chunk on the queue as soon as it is decoded, then a done marker -- or a
+ failure marker if their decode raises, which the driver re-raises instead of waiting.
+ 3. The driver decodes its own tiles on a thread and returns an iterator over the temporal
+ batches, each yielded once every tile overlapping it has delivered those frames and
+ complete on every stream (see :class:`_StreamAssembler`), so whatever consumes them (the
+ MP4 encoder) overlaps the rest of the decode. Chunks no later batch overlaps are released.
"""
latent_shape = VideoLatentShape.from_torch_shape(latent.shape)
scale = self.decoder.video_downscale_factors
full_shape = latent_shape.upscale(scale)
-
- # Phase 1: each rank decodes its assigned tiles.
- my_tiles = self._decode_tiles(latent, latent_shape, scale, generator, tiling_config, keyframes)
-
- # Phase 2: workers send tiles to driver.
+ tiles_by_rank = {
+ rank: dist_rank_tiles(self.vae_tiling, latent_shape, scale, rank, self.world_size)
+ for rank in range(self.world_size)
+ }
+ own = self._decoded_chunks(latent, tiles_by_rank[self.rank], scale, generator, tiling_config, keyframes)
if self.rank != self.driver_rank:
- self.queue.put((self.rank, my_tiles))
+ try:
+ for local_index, chunk in own:
+ self.queue.put((self.rank, local_index, chunk.frames_start, chunk.pixels))
+ except BaseException:
+ # The driver waits on this queue for every tile; tell it this rank's never comes.
+ self.queue.put((self.rank, None, _DECODE_FAILED, None))
+ raise
+ self.queue.put((self.rank, None, _DECODE_DONE, None))
return iter([])
- # Phase 3: driver collects and assembles.
- all_tiles = self._collect_tiles(my_tiles)
- if masks_are_complementary(
- [t.pixel_tile for t in all_tiles],
- (full_shape.frames, full_shape.height, full_shape.width),
- ):
+ pixel_tiles = [tile for rank in sorted(tiles_by_rank) for tile in tiles_by_rank[rank]]
+ if masks_are_complementary(pixel_tiles, (full_shape.frames, full_shape.height, full_shape.width)):
weights = None
else:
logger.warning(
"VAE blend masks are not complementary; falling back to dense [F,H,W] "
"weight normalization (expensive for large videos)."
)
- weights = compute_summed_weights(
- [t.pixel_tile for t in all_tiles],
- (full_shape.frames, full_shape.height, full_shape.width),
- )
- batches = gather_frames(
- all_tiles,
- full_shape.frames,
- full_shape.height,
- full_shape.width,
+ weights = compute_summed_weights(pixel_tiles, (full_shape.frames, full_shape.height, full_shape.width))
+ if device_fn is None:
+ device_fn = lambda b: f"cuda:{b % self.world_size}" # noqa: E731
+ assembler = _StreamAssembler(
+ tiles_by_rank,
+ full_shape,
self.num_temporal_batches,
- self.world_size,
weights,
- device_fn=device_fn,
+ device_fn,
+ mp_queue=self.queue,
+ driver_rank=self.driver_rank,
)
- return batches
+ self._join_assembler()
+ self._assembler = assembler
+ assembler.start(own)
+ return assembler.batches()
+
+ def _join_assembler(self) -> None:
+ if self._assembler is not None:
+ self._assembler.join()
+ self._assembler = None
+
+ def dispose(self) -> None:
+ """Join a streaming decode still running (it ends once its last chunk went out), then dispose."""
+ self._join_assembler()
+ super().dispose()
+
+ def _decoded_chunks(
+ self,
+ latent: torch.Tensor,
+ tiles: list[Tile],
+ scale: SpatioTemporalScaleFactors,
+ generator: torch.Generator | None,
+ tiling_config: TilingConfig | None,
+ keyframes: DecodeKeyframes | None,
+ ) -> Iterator[tuple[int, DecodedTile]]:
+ """Decode this rank's ``tiles`` and yield ``(local tile index, chunk)`` in temporal order.
+ Each tile gets its keyframe planes narrowed to its own window:
+ * spatially, with the same slices as its video latent -- planes are full-frame, so
+ handing them over uncropped offsets every one of them by the tile origin;
+ * temporally, via :meth:`DecodeKeyframes.for_frame_span`, which keeps the planes inside
+ the tile **and the nearest one on each side of it**, leaves ``pixel_frame_indices``
+ **global**, and sets ``clip_start_frame`` to the tile's first pixel. DiffVAE then
+ uses ``t_s(48) - t_s(56)`` for the 8-frame gap instead of rewriting 48 to -8.
+ """
+ if self.vae_tiling.frames.num_tiles > 1 and tiling_config is not None:
+ for tile in tiles:
+ latent_f = tile.in_coords[0].stop - tile.in_coords[0].start
+ pixel_f = (latent_f - 1) * scale.time + 1
+ if _sgpu_has_temporal_tiling(tiling_config, pixel_f):
+ raise ValueError(
+ "Cannot combine multi-GPU temporal tiling (vae_tiling.frames.num_tiles > 1) "
+ "with single-GPU temporal tiling that would split this rank's tile "
+ f"({pixel_f} frames → {tiling_config.video_chunks_number(pixel_f)} chunks). "
+ "Use only one to avoid causal decoding conflicts."
+ )
+ for local_index, tile in enumerate(tiles):
+ latent_slice = latent[:, :, tile.in_coords[0], tile.in_coords[1], tile.in_coords[2]]
+ tile_keyframes = (
+ None
+ if keyframes is None
+ else keyframes.for_frame_span(tile.out_coords[0].start, tile.out_coords[0].stop - 1).crop_spatial(
+ tile.in_coords[1], tile.in_coords[2]
+ )
+ )
+ # Only pass the keyword when there is something to pass: the wrapped decoder may be
+ # any VideoDecoder, including one written against the pre-keyframes signature.
+ extra = {} if tile_keyframes is None else {"keyframes": tile_keyframes}
+ start = 0
+ for pixels in self.decoder.decode_video(latent_slice, tiling_config, generator=generator, **extra):
+ yield local_index, DecodedTile(pixels=pixels, pixel_tile=tile, frames_start=start)
+ start += pixels.shape[0]
+ expected = tile.out_coords[0].stop - tile.out_coords[0].start
+ if start != expected:
+ raise RuntimeError(f"Dist tile decode yielded {start} frames for a {expected}-frame tile")
def decode_single_frames(
self,
@@ -360,28 +670,44 @@ class DistributedVideoDecoder(torch.nn.Module, Disposable):
rank ``r`` decodes planes ``r, r + W, r + 2W, ...`` on the inner SGPU decoder and every
rank ends up holding the whole list, in the order the latents were given. A DiffVAE
plane is expensive enough that decoding all of them on every rank -- what this used to
- do -- wastes ``W - 1`` GPUs per plane.
+ do -- wastes ``W - 1`` GPUs per plane. A caller that only needs something derived from
+ the pixels should use :meth:`owned_single_frames` and gather that instead.
``decode_video`` is still the wrong tool here: it splits one volume, and its workers
yield nothing after queueing their tile.
Not a generator. The all-gather has to run on every rank whether or not the caller
iterates the result, and a rank whose share is empty (fewer planes than ranks, or a
ragged tail) still takes part with an empty payload instead of skipping the collective
-- one absent participant blocks the group until the NCCL timeout.
- Pixels cross the group as CPU tensors: ``all_gather_object`` pickles a CUDA tensor with
- its source device index, so a plane decoded on ``cuda:3`` would be rebuilt on ``cuda:3``
- on every rank. Each plane is moved onto the device of its own latent on the way out, and
- a rank keeps the GPU copy of the planes it decoded itself.
+ Each plane is moved onto the device of its own latent on the way out, and a rank keeps
+ the GPU copy of the planes it decoded itself.
+ """
+ if not latents:
+ return iter(())
+ owned = self.owned_single_frames(latents, generator)
+ mine: dict[int, torch.Tensor] = dict(zip(owned.indices, owned.planes, strict=True))
+ merged = owned.gather(mine)
+ # Lazy on the way out: the collective already ran, and the caller usually consumes one
+ # plane at a time, so only the ranks' own planes stay resident on the GPU.
+ return (merged[index].to(latents[index].device) for index in range(len(latents)))
+
+ def owned_single_frames(
+ self,
+ latents: Sequence[torch.Tensor],
+ generator: torch.Generator | Sequence[torch.Generator | None] | None = None,
+ ) -> OwnedSingleFrames:
+ """This rank's round-robin share of the one-frame clips, decoded, and the group's gather.
+ The pixels stay on this rank; ``gather`` is a collective every rank calls exactly once,
+ with ``{index: value}`` for its own indices, to receive every rank's values. Values cross
+ the group through ``all_gather_object``, as CPU tensors where they are tensors (a pickled
+ CUDA tensor is rebuilt on its source device index on every rank), so callers gather what
+ they derive from the pixels -- e.g. re-encoded latents -- rather than the pixels.
With a per-plane ``generator`` sequence the pixels do not depend on the world size. A
single shared generator (or ``None``) is drawn from independently on each rank, so with
those the noise a given plane gets does.
"""
- if not latents:
- return iter(())
validate_single_frame_latents(latents)
gens = clip_generators(len(latents), generator)
- if self.world_size == 1:
- return iter_decoded_single_frames(self.decoder, latents, gens)
- if len(latents) > 1 and not isinstance(generator, Sequence):
+ if self.world_size > 1 and len(latents) > 1 and not isinstance(generator, Sequence):
logger.warning(
"Dist single-frame decode is splitting %d planes over %d ranks with a shared "
"generator; each rank draws its own noise, so the pixels depend on the world "
@@ -389,7 +715,6 @@ class DistributedVideoDecoder(torch.nn.Module, Disposable):
len(latents),
self.world_size,
)
-
my_indices = dist_rank_plane_indices(len(latents), self.rank, self.world_size)
logger.info(
"Dist single-frame decode: rank %d of %d takes %d of %d planes",
@@ -398,92 +723,30 @@ class DistributedVideoDecoder(torch.nn.Module, Disposable):
len(my_indices),
len(latents),
)
- decoded = iter_decoded_single_frames(
+ planes = _planes_with_shared_rope_tables(
self.decoder, [latents[i] for i in my_indices], [gens[i] for i in my_indices]
)
- mine: dict[int, torch.Tensor] = dict(zip(my_indices, decoded, strict=True))
-
- payload = [(index, pixels.to("cpu")) for index, pixels in mine.items()]
- shares: list[list[tuple[int, torch.Tensor]] | None] = [None] * self.world_size
- dist.all_gather_object(shares, payload, group=self.vae_group)
- merged: dict[int, torch.Tensor] = {}
- for share in shares:
- merged.update(dict(share or ()))
- # This rank's own planes are already on the right device; prefer them over the round trip.
- merged.update(mine)
- missing = [index for index in range(len(latents)) if index not in merged]
- if missing:
- raise RuntimeError(f"Dist single-frame decode lost planes {missing} in the all-gather")
- # Lazy on the way out: the collective already ran, and the caller usually consumes one
- # plane at a time, so only the ranks' own planes stay resident on the GPU.
- return (merged[index].to(latents[index].device) for index in range(len(latents)))
+
+ def gather(owned: dict[int, Any]) -> dict[int, Any]:
+ if self.world_size == 1:
+ return dict(owned)
+ payload = [
+ (index, value.to("cpu") if isinstance(value, torch.Tensor) else value) for index, value in owned.items()
+ ]
+ shares: list[list[tuple[int, Any]] | None] = [None] * self.world_size
+ dist.all_gather_object(shares, payload, group=self.vae_group)
+ merged: dict[int, Any] = {}
+ for share in shares:
+ merged.update(dict(share or ()))
+ # This rank's own values are already where the caller put them; prefer them over the round trip.
+ merged.update(owned)
+ missing = [index for index in range(len(latents)) if index not in merged]
+ if missing:
+ raise RuntimeError(f"Dist single-frame decode lost planes {missing} in the all-gather")
+ return merged
+
+ return OwnedSingleFrames(my_indices, planes, gather)
# ------------------------------------------------------------------
# Private helpers
# ------------------------------------------------------------------
-
- def _decode_tiles(
- self,
- latent: torch.Tensor,
- latent_shape: VideoLatentShape,
- scale: SpatioTemporalScaleFactors,
- generator: torch.Generator | None,
- tiling_config: TilingConfig | None = None,
- keyframes: DecodeKeyframes | None = None,
- ) -> list[DecodedTile]:
- """Decode this rank's assigned latent tiles and convert to :class:`DecodedTile` list.
- Each tile gets its keyframe planes narrowed to its own window:
- * spatially, with the same slices as its video latent -- planes are full-frame, so
- handing them over uncropped offsets every one of them by the tile origin;
- * temporally, via :meth:`DecodeKeyframes.for_frame_span`, which keeps the planes inside
- the tile **and the nearest one on each side of it**, leaves ``pixel_frame_indices``
- **global**, and sets ``clip_start_frame`` to the tile's first pixel. DiffVAE then
- uses ``t_s(48) - t_s(56)`` for the 8-frame gap instead of rewriting 48 to -8.
- """
- my_tiles = dist_rank_tiles(self.vae_tiling, latent_shape, scale, self.rank, self.world_size)
- if self.vae_tiling.frames.num_tiles > 1 and tiling_config is not None:
- for tile in my_tiles:
- latent_f = tile.in_coords[0].stop - tile.in_coords[0].start
- pixel_f = (latent_f - 1) * scale.time + 1
- if _sgpu_has_temporal_tiling(tiling_config, pixel_f):
- raise ValueError(
- "Cannot combine multi-GPU temporal tiling (vae_tiling.frames.num_tiles > 1) "
- "with single-GPU temporal tiling that would split this rank's tile "
- f"({pixel_f} frames → {tiling_config.video_chunks_number(pixel_f)} chunks). "
- "Use only one to avoid causal decoding conflicts."
- )
- decoded = []
- for tile in my_tiles:
- # One MGPU tile only — cat stitches SGPU temporal yields of this tile,
- # not the full video (full assembly is gather_frames on the driver).
- latent_slice = latent[:, :, tile.in_coords[0], tile.in_coords[1], tile.in_coords[2]]
- tile_keyframes = (
- None
- if keyframes is None
- else keyframes.for_frame_span(tile.out_coords[0].start, tile.out_coords[0].stop - 1).crop_spatial(
- tile.in_coords[1], tile.in_coords[2]
- )
- )
- # Only pass the keyword when there is something to pass: the wrapped decoder may be
- # any VideoDecoder, including one written against the pre-keyframes signature.
- extra = {} if tile_keyframes is None else {"keyframes": tile_keyframes}
- chunks = list(self.decoder.decode_video(latent_slice, tiling_config, generator=generator, **extra))
- pixels = torch.cat(chunks, dim=0) # [F, H, W, C] in [0, 1]
- decoded.append(_to_decoded_tile(pixels, tile))
- return decoded
-
- def _collect_tiles(self, driver_tiles: list[DecodedTile]) -> list[DecodedTile]:
- """Collect tiles from all workers via the queue. Returns flat list of all tiles.
- Sorted by rank so the downstream reduction in ``gather_frames``
- (in-place ``+=`` over overlapping pixel regions) processes tiles in a
- fixed order. Queue-arrival order would otherwise vary run-to-run and
- yield 1-ulp bf16 drift from non-associative floating-point summation.
- """
- per_rank: dict[int, list[DecodedTile]] = {self.driver_rank: driver_tiles}
- for _ in range(self.world_size - 1):
- worker_rank, worker_tiles = self.queue.get()
- per_rank[worker_rank] = worker_tiles
- result: list[DecodedTile] = []
- for rank in sorted(per_rank):
- result.extend(per_rank[rank])
- return result
diff --git a/packages/ltx-core/src/ltx_core/quantization/fp8_cast.py b/packages/ltx-core/src/ltx_core/quantization/fp8_cast.py
index dfa8789..d829719 100644
--- a/packages/ltx-core/src/ltx_core/quantization/fp8_cast.py
+++ b/packages/ltx-core/src/ltx_core/quantization/fp8_cast.py
@@ -1,3 +1,5 @@
+import dataclasses
+import functools
from pathlib import Path
import safetensors
@@ -263,6 +265,82 @@ def _fp8_cast_fuse(
fp8_cast_fuse_rule = FuseRule(aggregation_dtype=torch.bfloat16, fuse_fn=_fp8_cast_fuse)
+E4M3_MAX = float(torch.finfo(torch.float8_e4m3fn).max)
+# fp8 x fp8 GEMMs (torch._scaled_mm) need compute capability 8.9+ (Ada, Hopper, Blackwell).
+_FP8_GEMM_MIN_CAPABILITY = (8, 9)
+
+
+def supports_fp8_gemm(device: torch.device) -> bool:
+ """Whether *device* runs the e4m3 x e4m3 GEMMs of :class:`Fp8RowwiseLinear`."""
+ return device.type == "cuda" and torch.cuda.get_device_capability(device) >= _FP8_GEMM_MIN_CAPABILITY
+
+
+class Fp8RowwiseLinear(Fp8CastLinear):
+ """:class:`Fp8CastLinear` that multiplies long activations in e4m3 instead of upcasting the weight.
+ A bf16 input of at least ``_min_rows`` rows on a device with fp8 GEMMs (:func:`supports_fp8_gemm`)
+ is rounded to e4m3 with one scale per row (the row's absolute maximum maps to the e4m3 maximum)
+ and multiplied with the stored e4m3 weight at scale 1 through ``torch._scaled_mm`` (unit scales,
+ fp32 accumulation, bf16 product); the row scales and the bias are then applied in fp32. Other
+ inputs, including any on a device without fp8 GEMMs, take the upcast path. Lossy:
+ the activations keep e4m3's 3 mantissa bits. The cast is plain tensor ops so a compile
+ backend can fuse it into the kernel producing the activation.
+ """
+
+ _min_rows: int
+
+ def forward(self, input: torch.Tensor) -> torch.Tensor: # noqa: A002, type: ignore[override]
+ in_features = input.shape[-1]
+ rows = input.numel() // in_features
+ if (
+ rows < self._min_rows
+ or input.dtype != torch.bfloat16
+ or not supports_fp8_gemm(input.device)
+ or self._with_stochastic_rounding
+ ):
+ return super().forward(input)
+ x = input.reshape(rows, in_features)
+ amax = x.abs().amax(dim=-1, keepdim=True).float()
+ row_scale = torch.where(amax > 0, amax / E4M3_MAX, torch.ones_like(amax))
+ x8 = (x.float() / row_scale).to(torch.float8_e4m3fn)
+ # Unit (tensor-wise) scales select the cuBLASLt fp8 GEMM; per-row scales would select a slower CUTLASS
+ # kernel with per-launch host setup. The row scale and the bias are applied to the bf16 product as plain
+ # tensor ops, which a compile backend fuses into the kernel consuming this Linear's output.
+ unit = torch.ones((), device=input.device, dtype=torch.float32)
+ out = torch._scaled_mm(
+ x8, self.weight.t(), scale_a=unit, scale_b=unit, out_dtype=torch.bfloat16, use_fast_accum=False
+ )
+ out = out.float() * row_scale
+ if self.bias is not None:
+ out = out + self.bias.float()
+ return out.to(torch.bfloat16).view(*input.shape[:-1], self.out_features)
+
+
+def _amend_forward_with_rowwise_fp8_gemm(model: torch.nn.Module, min_rows: int) -> torch.nn.Module:
+ """Retype every :class:`Fp8CastLinear` of *model* to :class:`Fp8RowwiseLinear`."""
+ for m in model.modules():
+ if type(m) is Fp8CastLinear:
+ m.__class__ = Fp8RowwiseLinear
+ m._min_rows = min_rows
+ return model
+
+
+def with_rowwise_fp8_gemm(policy: QuantizationPolicy, min_rows: int = 8192) -> QuantizationPolicy:
+ """*policy* (an fp8-cast policy) with its Linears' long inputs multiplied in e4m3 (:class:`Fp8RowwiseLinear`).
+ Weights and their loading are unchanged, so a stage on this policy shares its weights with one
+ on *policy*. Only for devices where :func:`supports_fp8_gemm` holds. ``min_rows`` is the per-rank
+ activation row count from which the e4m3 GEMM is taken: the video token sequences of long passes,
+ not the audio or text-context projections.
+ """
+ if policy.fuse_rule is not fp8_cast_fuse_rule:
+ raise ValueError("with_rowwise_fp8_gemm needs an fp8-cast policy (weights stored in e4m3)")
+ op = ModuleOps(
+ name=f"fp8_rowwise_gemm_from_{min_rows}_rows",
+ matcher=lambda model: isinstance(model, LTXModel),
+ mutator=functools.partial(_amend_forward_with_rowwise_fp8_gemm, min_rows=min_rows),
+ )
+ return dataclasses.replace(policy, module_ops=(*policy.module_ops, op))
+
+
# Raw safetensors storage prefix shared by every diffusion-transformer
# parameter (and every prequant `*_scale` sibling). Verified against
# ltx-2.3-22b-{dev,distilled}-fp8.safetensors: 2924/2924 and 2992/2992 of
diff --git a/packages/ltx-core/src/ltx_core/tools.py b/packages/ltx-core/src/ltx_core/tools.py
index d3ae087..602da21 100644
--- a/packages/ltx-core/src/ltx_core/tools.py
+++ b/packages/ltx-core/src/ltx_core/tools.py
@@ -12,7 +12,7 @@ from ltx_core.components.patchifiers import (
get_pixel_coords,
)
from ltx_core.components.protocols import Patchifier
-from ltx_core.types import LatentState, SpatioTemporalScaleFactors
+from ltx_core.types import DenoiseMask, LatentState, SpatioTemporalScaleFactors
DEFAULT_SCALE_FACTORS = SpatioTemporalScaleFactors.default()
@@ -48,7 +48,9 @@ class LatentTools(Protocol):
latent_state = latent_state.clone()
latent = self.patchifier.patchify(latent_state.latent)
clean_latent = self.patchifier.patchify(latent_state.clean_latent)
- denoise_mask = self.patchifier.patchify(latent_state.denoise_mask)
+ denoise_mask = latent_state.denoise_mask.with_same_values(
+ self.patchifier.patchify(latent_state.denoise_mask.tensor)
+ )
# keyframes_mask shares denoise_mask's layout, so it converts alongside it; leaving it
# behind would produce a state mixing patchified and unpatchified per-token tensors.
keyframes_mask = (
@@ -70,8 +72,9 @@ class LatentTools(Protocol):
latent = self.patchifier.unpatchify(latent_state.latent, output_shape=self.target_shape)
clean_latent = self.patchifier.unpatchify(latent_state.clean_latent, output_shape=self.target_shape)
denoise_mask = self.patchifier.unpatchify(
- latent_state.denoise_mask, output_shape=self.target_shape.mask_shape()
+ latent_state.denoise_mask.tensor, output_shape=self.target_shape.mask_shape()
)
+ denoise_mask = latent_state.denoise_mask.with_same_values(denoise_mask)
keyframes_mask = (
self.patchifier.unpatchify(latent_state.keyframes_mask, output_shape=self.target_shape.mask_shape())
if latent_state.keyframes_mask is not None
@@ -101,12 +104,12 @@ class LatentTools(Protocol):
num_tokens = self.patchifier.get_token_count(self.target_shape)
latent = latent_state.latent[:, :num_tokens]
clean_latent = latent_state.clean_latent[:, :num_tokens]
- denoise_mask = torch.ones_like(latent_state.denoise_mask)[:, :num_tokens]
+ denoise_mask = torch.ones_like(latent_state.denoise_mask.tensor)[:, :num_tokens]
positions = latent_state.positions[:, :, :num_tokens]
return LatentState(
latent=latent,
- denoise_mask=denoise_mask,
+ denoise_mask=DenoiseMask(denoise_mask, denoise_mask.new_ones((1,))),
positions=positions,
clean_latent=clean_latent,
attention_mask=None,
@@ -155,11 +158,7 @@ class VideoLatentTools(LatentTools):
clean_latent = initial_latent.clone()
- denoise_mask = torch.ones(
- *self.target_shape.mask_shape().to_torch_shape(),
- device=device,
- dtype=torch.float32,
- )
+ denoise_mask = torch.ones(*self.target_shape.mask_shape().to_torch_shape(), dtype=torch.float32)
latent_coords = self.patchifier.get_patch_grid_bounds(
output_shape=self.target_shape,
@@ -176,7 +175,7 @@ class VideoLatentTools(LatentTools):
state = self.patchify(
LatentState(
latent=initial_latent,
- denoise_mask=denoise_mask,
+ denoise_mask=DenoiseMask(denoise_mask),
positions=positions,
clean_latent=clean_latent,
)
@@ -191,7 +190,7 @@ class VideoLatentTools(LatentTools):
independently of whether any keyframe slots exist. This is plain data about the latent, so
it is always populated; models without the keyframe embedding ignore it.
"""
- mask = torch.zeros_like(state.denoise_mask)
+ mask = torch.zeros_like(state.denoise_mask.tensor, device=state.latent.device)
mask[:, : self.tokens_per_latent_frame] = 1.0
return mask
@@ -262,11 +261,7 @@ class AudioLatentTools(LatentTools):
clean_latent = initial_latent.clone()
- denoise_mask = torch.ones(
- *self.target_shape.mask_shape().to_torch_shape(),
- device=device,
- dtype=torch.float32,
- )
+ denoise_mask = torch.ones(*self.target_shape.mask_shape().to_torch_shape(), dtype=torch.float32)
latent_coords = self.patchifier.get_patch_grid_bounds(
output_shape=self.target_shape,
@@ -275,6 +270,9 @@ class AudioLatentTools(LatentTools):
return self.patchify(
LatentState(
- latent=initial_latent, denoise_mask=denoise_mask, positions=latent_coords, clean_latent=clean_latent
+ latent=initial_latent,
+ denoise_mask=DenoiseMask(denoise_mask),
+ positions=latent_coords,
+ clean_latent=clean_latent,
)
)
diff --git a/packages/ltx-core/src/ltx_core/types.py b/packages/ltx-core/src/ltx_core/types.py
index 115b1e3..f50eee2 100644
--- a/packages/ltx-core/src/ltx_core/types.py
+++ b/packages/ltx-core/src/ltx_core/types.py
@@ -3,6 +3,8 @@ from typing import NamedTuple
import torch
+from ltx_core.devices import upload_from_host
+
class VideoPixelShape(NamedTuple):
"""
@@ -200,6 +202,24 @@ class AudioLatentShape(NamedTuple):
)
+@dataclass(frozen=True)
+class HostDeviceTensorPair:
+ """A tensor together with its host mirror."""
+
+ host_tensor: torch.Tensor
+ device_tensor: torch.Tensor
+
+ @classmethod
+ def on_device(cls, tensor: torch.Tensor, device: torch.device) -> "HostDeviceTensorPair":
+ """``tensor``'s host mirror and a copy on ``device``.
+ Reads the device back only when ``tensor`` lives there; a host tensor is uploaded without
+ waiting for the device.
+ """
+ host = tensor.cpu()
+ device_tensor = tensor if tensor.device == device else upload_from_host(host, device)
+ return cls(host_tensor=host, device_tensor=device_tensor)
+
+
@dataclass(frozen=True)
class Audio:
"""
@@ -247,6 +267,42 @@ class GeneratedKeyframeLayout:
return slice(self.first_token, self.first_token + self.num_tokens)
+@dataclass(frozen=True)
+class DenoiseMask:
+ """A per-token denoising mask and the table of its distinct values.
+ Conditioning builds ``tensor`` on the host, editing it in place or replacing it, with
+ ``values`` unset. ``compact`` tabulates the distinct values there, where ``torch.unique`` needs
+ no device synchronization, and moves both to the device; the denoising loop only sees the
+ compacted form, and nothing may edit ``tensor`` after it. ``values`` must hold exactly the
+ values present: a missing one silently modulates its tokens with a neighbouring value's
+ conditioning, and an extra one changes the shape the modulation is projected at.
+ ``zero_index`` is the position of ``0.0`` in ``values``, recorded on the host by ``compact``
+ (``None`` when no token is fully conditioned), so padding can reuse that entry.
+ """
+
+ tensor: torch.Tensor
+ values: torch.Tensor | None = None
+ zero_index: int | None = None
+
+ def compact(self, device: torch.device) -> "DenoiseMask":
+ """Tabulate the distinct mask values on the host and move mask and table to ``device``."""
+ if self.tensor.device.type != "cpu":
+ raise ValueError(f"denoise mask must be built on the host, got {self.tensor.device}")
+ values = torch.unique(self.tensor, sorted=True)
+ zeros = (values == 0).nonzero()
+ zero_index = int(zeros[0, 0]) if zeros.numel() else None
+ return DenoiseMask(upload_from_host(self.tensor, device), upload_from_host(values, device), zero_index)
+
+ def with_same_values(self, tensor: torch.Tensor) -> "DenoiseMask":
+ """Replace the tensor with a reordering or batch repeat of the same token values."""
+ return DenoiseMask(tensor, self.values, self.zero_index)
+
+ def clone(self) -> "DenoiseMask":
+ return DenoiseMask(
+ self.tensor.clone(), self.values.clone() if self.values is not None else None, self.zero_index
+ )
+
+
@dataclass(frozen=True)
class LatentState:
"""
@@ -277,7 +333,7 @@ class LatentState:
"""
latent: torch.Tensor
- denoise_mask: torch.Tensor
+ denoise_mask: DenoiseMask
positions: torch.Tensor
clean_latent: torch.Tensor
attention_mask: torch.Tensor | None = None
diff --git a/packages/ltx-kernels/csrc/all2all/all2all.cpp b/packages/ltx-kernels/csrc/all2all/all2all.cpp
index d3ff803..b9a028d 100644
--- a/packages/ltx-kernels/csrc/all2all/all2all.cpp
+++ b/packages/ltx-kernels/csrc/all2all/all2all.cpp
@@ -61,7 +61,7 @@ All2All::All2All(int rank, int world_size, int num_tokens, int hidden_dim, int n
// Calculate sizes for each region of the shared memory block
int64_t ptrs_bytes = MAX_NUM_PEERS * sizeof(void *);
- int64_t barrier_signal_bytes = MAX_NUM_PEERS * sizeof(int);
+ int64_t barrier_signal_bytes = kNumSignalSlots * sizeof(int);
int64_t barrier_signal_ptrs_bytes = MAX_NUM_PEERS * sizeof(int *);
// Allocate the main shared memory block and create IPC handle
@@ -320,6 +320,97 @@ at::Tensor All2All::allgather(at::Tensor &x, bool copy_output) {
return get_local_buffer_tensor(x, batch_size, out_tokens, out_heads, head_size, copy_output, stream);
}
+void All2All::rope_send_recv_heads(at::Tensor &x, at::Tensor &sin_freqs, at::Tensor &cos_freqs, at::Tensor &weights,
+ double eps, bool barrier) {
+ EP_HOST_ASSERT(x.dim() == 3 and x.scalar_type() == at::ScalarType::BFloat16);
+ EP_HOST_ASSERT(tensor_dtype == at::ScalarType::BFloat16);
+ EP_HOST_ASSERT(x.device().is_cuda() and x.device().index() == rank);
+ EP_HOST_ASSERT(weights.scalar_type() == at::ScalarType::BFloat16);
+ EP_HOST_ASSERT(cos_freqs.dim() == 4 and sin_freqs.dim() == 4);
+ // The kernel walks x and weights from one base pointer each, so a strided input is made contiguous,
+ // as ops/rms_norm_split_rope.cpp does for x.
+ if (!x.is_contiguous()) {
+ x = x.contiguous();
+ }
+ if (!weights.is_contiguous()) {
+ weights = weights.contiguous();
+ }
+ int b = x.size(0);
+ int s = x.size(1);
+ int h = x.size(2);
+ int n = cos_freqs.size(1);
+ EP_HOST_ASSERT(int64_t(b) * s * world_size * (h / world_size) * int64_t(elementSize(tensor_dtype)) <= tensor_bytes);
+
+ // Same freq-table handling as ops/rms_norm_split_rope.cpp: innermost dim contiguous, outer strides
+ // forwarded, a batch-shared table expanded (stride 0) rather than copied.
+ if (cos_freqs.stride(-1) != 1) {
+ cos_freqs = cos_freqs.contiguous();
+ }
+ if (sin_freqs.stride(-1) != 1) {
+ sin_freqs = sin_freqs.contiguous();
+ }
+ if (cos_freqs.size(0) == 1 && b > 1) {
+ cos_freqs = cos_freqs.expand({b, n, cos_freqs.size(2), cos_freqs.size(3)});
+ }
+ if (sin_freqs.size(0) == 1 && b > 1) {
+ sin_freqs = sin_freqs.expand({b, n, sin_freqs.size(2), sin_freqs.size(3)});
+ }
+
+ at::cuda::CUDAGuard device_guard{x.device()};
+ auto stream = at::cuda::getCurrentCUDAStream().stream();
+ all2all_cuda::rope_scatter_heads_launch(x.data_ptr(), sin_freqs.data_ptr(), cos_freqs.data_ptr(), weights.data_ptr(),
+ static_cast<float>(eps), b, s, n, h, cos_freqs.stride(0), cos_freqs.stride(1),
+ cos_freqs.stride(2), sin_freqs.stride(0), sin_freqs.stride(1),
+ sin_freqs.stride(2), buffer_ptrs_gpu, barrier_signal_ptrs_gpu, rank,
+ world_size, timeout_cycles_, barrier, stream);
+}
+
+void All2All::linear_send_recv_heads(at::Tensor &x, at::Tensor &weight, at::Tensor &bias, bool barrier) {
+ // A blockwise-fp8 exchange quantizes in its own copy kernel; this path writes the GEMM's bf16 output.
+ EP_HOST_ASSERT(tensor_dtype == at::ScalarType::BFloat16);
+ EP_HOST_ASSERT(x.dim() == 3 and x.scalar_type() == tensor_dtype);
+ EP_HOST_ASSERT(weight.dim() == 2 and weight.scalar_type() == tensor_dtype);
+ EP_HOST_ASSERT(bias.dim() == 1 and bias.scalar_type() == tensor_dtype);
+ EP_HOST_ASSERT(x.device().is_cuda() and x.device().index() == rank);
+ int b = x.size(0);
+ int s = x.size(1);
+ int h = weight.size(0);
+ EP_HOST_ASSERT(h % world_size == 0);
+ int64_t cols = h / world_size;
+ int64_t total_tokens = int64_t(s) * world_size;
+ EP_HOST_ASSERT(int64_t(b) * total_tokens * cols * int64_t(elementSize(tensor_dtype)) <= tensor_bytes);
+
+ at::cuda::CUDAGuard device_guard{x.device()};
+ auto stream = at::cuda::getCurrentCUDAStream().stream();
+ auto options = x.options();
+ // Peers first, so the NVLink-bound writes start as early as possible.
+ for (int step = 1; step <= world_size; step++) {
+ int target = (rank + step) % world_size;
+ auto w = weight.narrow(0, target * cols, cols);
+ auto bias_slice = bias.narrow(0, target * cols, cols);
+ for (int batch = 0; batch < b; batch++) {
+ int64_t offset = (int64_t(batch) * total_tokens + int64_t(rank) * s) * cols;
+ // A peer buffer is IPC-mapped into this device's address space; pinning the target device skips the
+ // pointer-owner lookup, which would report the peer.
+ auto out =
+ at::for_blob(static_cast<uint8_t *>(buffer_ptrs[target]) + offset * elementSize(tensor_dtype), {s, cols})
+ .options(options)
+ .target_device(x.device())
+ .make_tensor();
+ at::addmm_out(out, bias_slice, x.select(0, batch), w.t());
+ }
+ }
+ if (barrier) {
+ all2all_cuda::fused_barrier_launch(barrier_signal_ptrs_gpu, rank, world_size, timeout_cycles_, stream);
+ }
+}
+
+void All2All::exchange_barrier() {
+ at::cuda::CUDAGuard device_guard{static_cast<c10::DeviceIndex>(rank)};
+ auto stream = at::cuda::getCurrentCUDAStream().stream();
+ all2all_cuda::fused_barrier_launch(barrier_signal_ptrs_gpu, rank, world_size, timeout_cycles_, stream);
+}
+
} // namespace all2all
} // namespace ltx_kernels
@@ -376,6 +467,12 @@ PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
.def("gather_heads", &ltx_kernels::all2all::All2All::gather_heads,
"Inverse All2All to gather heads back to original distribution.")
.def("allgather", &ltx_kernels::all2all::All2All::allgather, "Gathers sequence tokens from all ranks.")
+ .def("rope_send_recv_heads", &ltx_kernels::all2all::All2All::rope_send_recv_heads,
+ "Fused RMS-norm + split RoPE written straight into every rank's head-exchange buffer.")
+ .def("linear_send_recv_heads", &ltx_kernels::all2all::All2All::linear_send_recv_heads,
+ "Linear projection written straight into every rank's head-exchange buffer.")
+ .def("exchange_barrier", &ltx_kernels::all2all::All2All::exchange_barrier,
+ "Waits until every rank's earlier writes on this stream are visible to every rank.")
.def("set_timeout_seconds", &ltx_kernels::all2all::All2All::set_timeout_seconds,
"Sets the barrier timeout in seconds (converted to cycles via the device peak SM clock).");
}
diff --git a/packages/ltx-kernels/csrc/all2all/all2all.hpp b/packages/ltx-kernels/csrc/all2all/all2all.hpp
index 34b0efc..97addd9 100644
--- a/packages/ltx-kernels/csrc/all2all/all2all.hpp
+++ b/packages/ltx-kernels/csrc/all2all/all2all.hpp
@@ -234,6 +234,20 @@ public:
*/
at::Tensor allgather(at::Tensor &x, bool copy_output);
+ /// Fused RMS-norm + split RoPE of ``x`` ``[b, s, h]``, written straight into every rank's buffer in the
+ /// ``send_recv_heads`` layout ``[b, world_size * s, h / world_size]``. With ``barrier``, returns once all
+ /// ranks' writes into the local buffer are visible; without it, a later exchange_barrier() does that.
+ void rope_send_recv_heads(at::Tensor &x, at::Tensor &sin_freqs, at::Tensor &cos_freqs, at::Tensor &weights,
+ double eps, bool barrier);
+
+ /// ``x @ weight.T + bias`` with each rank's output columns written straight into that rank's buffer in
+ /// the ``send_recv_heads`` layout. ``barrier`` as for rope_send_recv_heads().
+ void linear_send_recv_heads(at::Tensor &x, at::Tensor &weight, at::Tensor &bias, bool barrier);
+
+ /// Returns once every rank's writes issued earlier on this stream -- into any instance's buffer --
+ /// are visible to every rank. Completes the exchanges run with ``barrier`` false.
+ void exchange_barrier();
+
/**
* @brief Sets the all2all barrier timeout in seconds.
*
diff --git a/packages/ltx-kernels/csrc/all2all/cuda/api.cuh b/packages/ltx-kernels/csrc/all2all/cuda/api.cuh
index 3d1bb35..c4cf9d0 100644
--- a/packages/ltx-kernels/csrc/all2all/cuda/api.cuh
+++ b/packages/ltx-kernels/csrc/all2all/cuda/api.cuh
@@ -89,6 +89,14 @@ void allgather_launch(void **buffer_ptrs, int **barrier_signal_ptrs, void *x, in
int seqlen, int hidden_dim, int total_tokens, cudaStream_t stream, int num_sms,
at::ScalarType tensor_dtype, uint64_t timeout_cycles);
+void rope_scatter_heads_launch(void *x, void *sin_freqs, void *cos_freqs, void *weights, float eps, int b, int s, int n,
+ int h, long cos_sb, long cos_sn, long cos_ss, long sin_sb, long sin_sn, long sin_ss,
+ void **buffer_ptrs, int **barrier_signal_ptrs, int rank, int world_size,
+ uint64_t timeout_cycles, bool barrier, cudaStream_t stream);
+
+void fused_barrier_launch(int **barrier_signal_ptrs, int rank, int world_size, uint64_t timeout_cycles,
+ cudaStream_t stream);
+
} // namespace all2all_cuda
} // namespace all2all
} // namespace ltx_kernels
diff --git a/packages/ltx-kernels/csrc/all2all/cuda/fused_send.cu b/packages/ltx-kernels/csrc/all2all/cuda/fused_send.cu
new file mode 100644
index 0000000..23c111d
--- /dev/null
+++ b/packages/ltx-kernels/csrc/all2all/cuda/fused_send.cu
@@ -0,0 +1,161 @@
+/**
+ * Producers that write straight into the All2All head-exchange layout.
+ *
+ * ``send_recv_heads`` reads a finished ``[batch, tokens, heads, head_size]`` tensor and copies each
+ * rank's head slice into that rank's IPC buffer. The kernels here fold that copy into the kernel that
+ * produces the tensor, so the tensor is never materialized locally and its peer slice travels over
+ * NVLink while the producer runs:
+ *
+ * - ``rope_scatter_heads``: fused RMS-norm + split RoPE (the math of ops/rms_norm_split_rope) whose
+ * output rows land in the owning rank's buffer at ``[batch, rank * tokens + token, head % hpr, :]``.
+ * - ``fused_barrier``: signals every rank that this rank's writes into its buffer are complete, then
+ * waits for the same from every rank. Used after producers that are not kernels of ours (cuBLAS).
+ *
+ * Both complete only once every rank's writes into the local buffer are visible, as
+ * ``send_recv_heads`` does.
+ */
+
+#include <algorithm>
+#include <cstdio>
+
+#include "../../include/cuda/configs.cuh"
+#include "../../include/cuda/rms_norm_split_rope.cuh"
+#include "../../include/cuda/utils.cuh"
+#include "api.cuh"
+
+#include <ATen/cuda/CUDAContext.h>
+#include <c10/cuda/CUDAException.h>
+#include <cuda_bf16.h>
+
+namespace ltx_kernels {
+namespace all2all {
+namespace all2all_cuda {
+
+using bf16 = __nv_bfloat16;
+
+// Rotation pairs per thread, as in ops/rms_norm_split_rope_cuda.cu.
+constexpr int kScatterPairsPerThread = 16;
+constexpr int kScatterCtasPerSm = 8;
+
+template <int PAIRS, int THREADS_PER_ROW>
+__global__ void __launch_bounds__(THREADS_PER_ROW)
+ rope_scatter_heads(const bf16 *__restrict__ x, const bf16 *__restrict__ sin_freqs,
+ const bf16 *__restrict__ cos_freqs, const bf16 *__restrict__ weights, float eps, int b, int s,
+ int n, int h, long cos_sb, long cos_sn, long cos_ss, long sin_sb, long sin_sn, long sin_ss,
+ void **buffer_ptrs, int **barrier_signal_ptrs, int rank, int world_size, uint64_t timeout_cycles,
+ bool barrier) {
+ constexpr int kWarpsPerRow = THREADS_PER_ROW / 32;
+ constexpr int kRowElems = PAIRS * THREADS_PER_ROW * 2;
+ __shared__ float smem[kWarpsPerRow];
+ // The finished row, staged so it leaves as contiguous 16-byte-per-lane runs: the RoPE pair layout
+ // alone would store 16-byte chunks 32 bytes apart, half-filling every sector it sends over NVLink.
+ __shared__ __align__(16) bf16 row_out[kRowElems];
+
+ const int lane_in_row = threadIdx.x;
+ const int d = h / n;
+ const int half = d / 2;
+ const int pair0 = lane_in_row * PAIRS;
+ const int head = pair0 / half;
+ const int freq0 = pair0 - head * half;
+ const int first_channel = head * d + freq0;
+ const int second_channel = first_channel + half;
+ const int hpr = n / world_size;
+ const int slice = hpr * d; // elements of one row that go to one rank
+ const int64_t total_tokens = int64_t(s) * world_size;
+
+ bf16 first_weight[PAIRS], second_weight[PAIRS];
+ load_run<PAIRS>(weights + first_channel, first_weight);
+ load_run<PAIRS>(weights + second_channel, second_weight);
+
+ for (int token_idx = blockIdx.x; token_idx < b * s; token_idx += gridDim.x) {
+ const int batch = token_idx / s;
+ const int token = token_idx % s;
+ const size_t row = static_cast<size_t>(token_idx) * static_cast<size_t>(h);
+ const long cos_offset = static_cast<long>(batch) * cos_sb + static_cast<long>(head) * cos_sn +
+ static_cast<long>(token) * cos_ss + freq0;
+ const long sin_offset = static_cast<long>(batch) * sin_sb + static_cast<long>(head) * sin_sn +
+ static_cast<long>(token) * sin_ss + freq0;
+
+ float first_rot[PAIRS], second_rot[PAIRS];
+ rms_norm_split_rope_pairs<PAIRS, kWarpsPerRow>(x + row + first_channel, x + row + second_channel,
+ cos_freqs + cos_offset, sin_freqs + sin_offset, first_weight,
+ second_weight, eps, h, lane_in_row, smem, first_rot, second_rot);
+ bf16 first_out[PAIRS], second_out[PAIRS];
+#pragma unroll
+ for (int i = 0; i < PAIRS; i++) {
+ first_out[i] = __float2bfloat16(first_rot[i]);
+ second_out[i] = __float2bfloat16(second_rot[i]);
+ }
+#pragma unroll
+ for (int i = 0; i < PAIRS / 8; i++) {
+ reinterpret_cast<int4 *>(row_out + first_channel)[i] = reinterpret_cast<int4 *>(first_out)[i];
+ reinterpret_cast<int4 *>(row_out + second_channel)[i] = reinterpret_cast<int4 *>(second_out)[i];
+ }
+ __syncthreads();
+
+ // Rank `target` owns heads [target * hpr, (target + 1) * hpr), i.e. row elements
+ // [target * slice, (target + 1) * slice); its buffer is [batch, world_size * s, hpr, d] with this
+ // rank's tokens at [rank * s, (rank + 1) * s).
+ const int64_t dst_row = int64_t(batch) * total_tokens + int64_t(rank) * s + token;
+ for (int chunk = lane_in_row; chunk < kRowElems / 8; chunk += THREADS_PER_ROW) {
+ const int elem = chunk * 8;
+ const int target = elem / slice;
+ bf16 *dst = static_cast<bf16 *>(buffer_ptrs[target]) + dst_row * slice + (elem - target * slice);
+ st_na_global(reinterpret_cast<int4 *>(dst), reinterpret_cast<int4 *>(row_out)[chunk]);
+ }
+ // row_out and smem are rewritten by the next row.
+ __syncthreads();
+ }
+
+ // Every block wrote to every rank, so completion is counted across blocks before anyone signals.
+ // Without the barrier a later fused_barrier on the stream covers these writes.
+ if (barrier) {
+ barrier_last_block_wait_and_reset(barrier_signal_ptrs, barrier_signal_ptrs[rank] + kBlocksDoneSlot, rank,
+ world_size, threadIdx.x, timeout_cycles);
+ }
+}
+
+// One block per target rank: the writes it covers are complete already (they came from earlier
+// kernels on this stream), so each block only has to signal its rank.
+__global__ void fused_barrier(int **barrier_signal_ptrs, int rank, int world_size, uint64_t timeout_cycles) {
+ barrier_wait_and_reset(barrier_signal_ptrs, blockIdx.x, rank, world_size, 1, blockIdx.x, threadIdx.x, timeout_cycles);
+}
+
+void rope_scatter_heads_launch(void *x, void *sin_freqs, void *cos_freqs, void *weights, float eps, int b, int s, int n,
+ int h, long cos_sb, long cos_sn, long cos_ss, long sin_sb, long sin_sn, long sin_ss,
+ void **buffer_ptrs, int **barrier_signal_ptrs, int rank, int world_size,
+ uint64_t timeout_cycles, bool barrier, cudaStream_t stream) {
+ TORCH_CHECK(n > 0 && h % n == 0 && n % world_size == 0, "heads must divide the hidden size and the world size");
+ TORCH_CHECK((h / n) % (2 * kScatterPairsPerThread) == 0, "head_dim/2 must be a multiple of ", kScatterPairsPerThread);
+ // Persistent CTAs walk the rows, so the completion counter sees one atomic per CTA, not per row.
+ int device = 0;
+ int num_sms = 0;
+ C10_CUDA_CHECK(cudaGetDevice(&device));
+ C10_CUDA_CHECK(cudaDeviceGetAttribute(&num_sms, cudaDevAttrMultiProcessorCount, device));
+ const int grid = std::min(b * s, num_sms * kScatterCtasPerSm);
+#define LAUNCH_ROPE_SCATTER(THREADS_PER_ROW) \
+ rope_scatter_heads<kScatterPairsPerThread, (THREADS_PER_ROW)><<<grid, (THREADS_PER_ROW), 0, stream>>>( \
+ static_cast<bf16 *>(x), static_cast<bf16 *>(sin_freqs), static_cast<bf16 *>(cos_freqs), \
+ static_cast<bf16 *>(weights), eps, b, s, n, h, cos_sb, cos_sn, cos_ss, sin_sb, sin_sn, sin_ss, buffer_ptrs, \
+ barrier_signal_ptrs, rank, world_size, timeout_cycles, barrier)
+ if (h == 2048) {
+ LAUNCH_ROPE_SCATTER(64);
+ } else if (h == 4096) {
+ LAUNCH_ROPE_SCATTER(128);
+ } else {
+ TORCH_CHECK(h == 8192, "Hidden dimension must be 2048, 4096, or 8192");
+ LAUNCH_ROPE_SCATTER(256);
+ }
+#undef LAUNCH_ROPE_SCATTER
+ C10_CUDA_KERNEL_LAUNCH_CHECK();
+}
+
+void fused_barrier_launch(int **barrier_signal_ptrs, int rank, int world_size, uint64_t timeout_cycles,
+ cudaStream_t stream) {
+ fused_barrier<<<world_size, 32, 0, stream>>>(barrier_signal_ptrs, rank, world_size, timeout_cycles);
+ C10_CUDA_KERNEL_LAUNCH_CHECK();
+}
+
+} // namespace all2all_cuda
+} // namespace all2all
+} // namespace ltx_kernels
diff --git a/packages/ltx-kernels/csrc/include/cuda/configs.cuh b/packages/ltx-kernels/csrc/include/cuda/configs.cuh
index e7381bc..1f60f5b 100644
--- a/packages/ltx-kernels/csrc/include/cuda/configs.cuh
+++ b/packages/ltx-kernels/csrc/include/cuda/configs.cuh
@@ -39,6 +39,16 @@ constexpr double DEFAULT_BARRIER_TIMEOUT_SECONDS = 10.0;
*/
constexpr int MAX_NUM_PEERS = 8;
+/**
+ * @brief Layout of each rank's barrier signal slots.
+ *
+ * [0, MAX_NUM_PEERS): per-source signal counters. kBlocksDoneSlot: the local count of finished
+ * blocks that barrier_last_block_wait_and_reset keeps. The count is even so the pointer arrays that
+ * follow the slots in the IPC block stay 8-byte aligned.
+ */
+constexpr int kBlocksDoneSlot = MAX_NUM_PEERS;
+constexpr int kNumSignalSlots = MAX_NUM_PEERS + 2;
+
// =============================================================================
// Kernel Configuration
// =============================================================================
diff --git a/packages/ltx-kernels/csrc/include/cuda/rms_norm_split_rope.cuh b/packages/ltx-kernels/csrc/include/cuda/rms_norm_split_rope.cuh
new file mode 100644
index 0000000..8a938d6
--- /dev/null
+++ b/packages/ltx-kernels/csrc/include/cuda/rms_norm_split_rope.cuh
@@ -0,0 +1,92 @@
+#pragma once
+
+/**
+ * @file rms_norm_split_rope.cuh
+ * @brief Per-thread core of fused RMS-norm + split RoPE, shared by the kernels that apply it.
+ *
+ * A row is one token's hidden vector, ``h = heads * d``. The split RoPE rotates pair ``(i, i + d/2)``
+ * of every head, and each thread owns PAIRS consecutive pairs of a single head -- both halves of each,
+ * so rotating needs no exchange with another thread and every cos/sin value is read once.
+ *
+ * The kernels differ in how they walk rows and where the result goes (in place, fp8, or scattered
+ * into all2all buffers); they share this function so their rounding is identical by construction.
+ */
+
+#include <cuda_bf16.h>
+
+namespace ltx_kernels {
+
+/// Loads N consecutive bf16 values as N / 8 16-byte vectors; `source` must be 16-byte aligned.
+template <int N> __device__ __forceinline__ void load_run(const __nv_bfloat16 *source, __nv_bfloat16 values[N]) {
+#pragma unroll
+ for (int i = 0; i < N / 8; i++) {
+ reinterpret_cast<int4 *>(values)[i] = reinterpret_cast<const int4 *>(source)[i];
+ }
+}
+
+/**
+ * @brief One thread's PAIRS rotation pairs of a row, RMS-normalized, weighted and rotated.
+ *
+ * Every thread of the row must call it (it contains a __syncthreads()), including threads of an
+ * inactive row, which pass any valid addresses and discard the result. Returns in fp32 so the
+ * caller picks the output type.
+ *
+ * @param x_first First-half run of this thread's pairs in the input row
+ * @param x_second Second-half run (x_first + d/2)
+ * @param cos_run, sin_run This thread's PAIRS frequency values
+ * @param first_weight, second_weight RMS-norm weights of the two runs
+ * @param eps Added to the mean square before the rsqrt
+ * @param h Row length
+ * @param lane_in_row This thread's index among the row's WARPS_PER_ROW * 32 threads
+ * @param row_smem WARPS_PER_ROW floats of shared memory owned by this row
+ * @param first_out, second_out The rotated runs
+ */
+template <int PAIRS, int WARPS_PER_ROW>
+__device__ __forceinline__ void
+rms_norm_split_rope_pairs(const __nv_bfloat16 *x_first, const __nv_bfloat16 *x_second, const __nv_bfloat16 *cos_run,
+ const __nv_bfloat16 *sin_run, const __nv_bfloat16 (&first_weight)[PAIRS],
+ const __nv_bfloat16 (&second_weight)[PAIRS], float eps, int h, int lane_in_row,
+ float *row_smem, float (&first_out)[PAIRS], float (&second_out)[PAIRS]) {
+ __nv_bfloat16 first_raw[PAIRS], second_raw[PAIRS], cos_raw[PAIRS], sin_raw[PAIRS];
+ load_run<PAIRS>(x_first, first_raw);
+ load_run<PAIRS>(x_second, second_raw);
+ load_run<PAIRS>(cos_run, cos_raw);
+ load_run<PAIRS>(sin_run, sin_raw);
+
+ float first[PAIRS], second[PAIRS], sum_sq = 0.0f;
+#pragma unroll
+ for (int i = 0; i < PAIRS; i++) {
+ first[i] = __bfloat162float(first_raw[i]);
+ second[i] = __bfloat162float(second_raw[i]);
+ sum_sq += first[i] * first[i] + second[i] * second[i];
+ }
+#pragma unroll
+ for (int offset = 16; offset > 0; offset >>= 1) {
+ sum_sq += __shfl_xor_sync(0xffffffff, sum_sq, offset);
+ }
+
+ // One barrier: every thread re-sums the per-warp partials itself. Electing one thread to sum them
+ // and publish the result back needs a second barrier.
+ if ((lane_in_row & 31) == 0) {
+ row_smem[lane_in_row / 32] = sum_sq;
+ }
+ __syncthreads();
+ float total_sum = 0.0f;
+#pragma unroll
+ for (int i = 0; i < WARPS_PER_ROW; i++) {
+ total_sum += row_smem[i];
+ }
+ const float inv_rms = rsqrtf(total_sum / h + eps);
+
+#pragma unroll
+ for (int i = 0; i < PAIRS; i++) {
+ const float a = first[i] * inv_rms * __bfloat162float(first_weight[i]);
+ const float c = second[i] * inv_rms * __bfloat162float(second_weight[i]);
+ const float cs = __bfloat162float(cos_raw[i]);
+ const float sn = __bfloat162float(sin_raw[i]);
+ first_out[i] = a * cs - c * sn;
+ second_out[i] = c * cs + a * sn;
+ }
+}
+
+} // namespace ltx_kernels
diff --git a/packages/ltx-kernels/csrc/include/cuda/utils.cuh b/packages/ltx-kernels/csrc/include/cuda/utils.cuh
index 4a65502..2d1c83b 100644
--- a/packages/ltx-kernels/csrc/include/cuda/utils.cuh
+++ b/packages/ltx-kernels/csrc/include/cuda/utils.cuh
@@ -357,4 +357,65 @@ __device__ __forceinline__ void barrier_wait_and_reset_roundrobin(int **barrier_
}
}
+/**
+ * @brief Barrier for a kernel in which every block writes to every rank.
+ *
+ * barrier_wait_and_reset and its round-robin variant have each block signal the one rank it wrote
+ * to, which assumes a block's writes go to a single rank. When every block writes to every rank,
+ * no single block's signal covers the others' writes. Here each block instead counts itself done
+ * in `blocks_done` (a counter on this rank), and the last block to finish signals every rank once,
+ * then waits for one signal from every rank -- the same per-source counters and counts as
+ * barrier_wait_and_reset with one block per target. Earlier blocks return without waiting, so no
+ * block spins on an SM that unfinished blocks still need.
+ *
+ * Call from every block after its last write. Needs blockDim.x >= world_size.
+ *
+ * @param barrier_signal_ptrs Array of pointers to barrier signal buffers for each rank
+ * @param blocks_done This rank's count of finished blocks; zero on entry, zero again on return
+ * @param rank This GPU's rank
+ * @param world_size Total number of GPUs/ranks
+ * @param thread_id The thread ID within the block (threadIdx.x)
+ * @param timeout_cycles Number of cycles to wait before timeout
+ */
+__device__ __forceinline__ void barrier_last_block_wait_and_reset(int **barrier_signal_ptrs, int *blocks_done, int rank,
+ int world_size, int thread_id,
+ uint64_t timeout_cycles) {
+ __shared__ bool is_last_block;
+
+ // Release: every block fences its own writes before counting itself done.
+ __threadfence_system();
+ __syncthreads();
+ if (thread_id == 0) {
+ is_last_block = atomicAdd(blocks_done, 1) == static_cast<int>(gridDim.x) - 1;
+ }
+ __syncthreads();
+ if (!is_last_block) {
+ return;
+ }
+ if (thread_id == 0) {
+ *blocks_done = 0;
+ }
+ __threadfence_system();
+
+ // Each thread signals one rank and waits for that rank's signal.
+ if (thread_id < world_size) {
+ atomicAdd_system(barrier_signal_ptrs[thread_id] + rank, 1);
+ auto start_time = clock64();
+ while (true) {
+ // Acquire: seeing the signal guarantees the peer's data is visible.
+ int recv_count = ld_acquire_sys_global(barrier_signal_ptrs[rank] + thread_id);
+ if (recv_count == 1) {
+ break;
+ }
+ if (clock64() - start_time >= timeout_cycles) {
+ printf("All2All barrier timeout (last block): rank=%d, waiting_for_source=%d, expected=1, got=%d\n", rank,
+ thread_id, recv_count);
+ trap();
+ }
+ }
+ // Reset barrier for next use
+ atomicSub_system(barrier_signal_ptrs[rank] + thread_id, 1);
+ }
+}
+
} // namespace ltx_kernels
diff --git a/packages/ltx-kernels/csrc/ops/rms_norm_split_rope_cuda.cu b/packages/ltx-kernels/csrc/ops/rms_norm_split_rope_cuda.cu
index 928065b..e247070 100644
--- a/packages/ltx-kernels/csrc/ops/rms_norm_split_rope_cuda.cu
+++ b/packages/ltx-kernels/csrc/ops/rms_norm_split_rope_cuda.cu
@@ -5,6 +5,8 @@
#include <cuda_bf16.h>
#include <cuda_fp8.h>
+#include "cuda/rms_norm_split_rope.cuh"
+
// CUDA kernel template for RMS norm + split RoPE
// out_t can be at::Float8_e4m3fn or at::BFloat16
@@ -21,14 +23,6 @@ using fp8 = __nv_fp8_e4m3;
// kPairsPerThread * threads_per_row * 2 must equal h.
constexpr int kPairsPerThread = 16;
-template<int N>
-__device__ __forceinline__ void _load_run(const bf16* source, bf16 values[N]) {
- #pragma unroll
- for (int i = 0; i < N / 8; i++) {
- reinterpret_cast<int4*>(values)[i] = reinterpret_cast<const int4*>(source)[i];
- }
-}
-
// Frequency tables are laid out logically as [b, n, s, d/2] (what
// apply_split_rotary_emb produces -- a swapaxes view whose physical layout is
// [b, s, n, d/2]). The strides (sb, sn, ss; inner d/2 stride is 1) are forwarded from
@@ -70,40 +64,14 @@ __global__ void _rms_norm_split_rope_kernel(
+ static_cast<long>(head) * sin_sn
+ static_cast<long>(safe_token % s) * sin_ss + freq0;
- bf16 first_raw[PAIRS], second_raw[PAIRS];
- bf16 cos_raw[PAIRS], sin_raw[PAIRS];
bf16 first_weight[PAIRS], second_weight[PAIRS];
- _load_run<PAIRS>(x + row + first_channel, first_raw);
- _load_run<PAIRS>(x + row + second_channel, second_raw);
- _load_run<PAIRS>(cos_freqs + freq_offset, cos_raw);
- _load_run<PAIRS>(sin_freqs + sin_offset, sin_raw);
- _load_run<PAIRS>(weights + first_channel, first_weight);
- _load_run<PAIRS>(weights + second_channel, second_weight);
-
- float first[PAIRS], second[PAIRS], sum_sq = 0.0f;
- #pragma unroll
- for (int i = 0; i < PAIRS; i++) {
- first[i] = float(first_raw[i]);
- second[i] = float(second_raw[i]);
- sum_sq += first[i] * first[i] + second[i] * second[i];
- }
- #pragma unroll
- for (int offset = 16; offset > 0; offset >>= 1) {
- sum_sq += __shfl_xor_sync(0xffffffff, sum_sq, offset);
- }
-
- // One barrier: every thread re-sums the per-warp partials itself. Electing one
- // thread to sum them and publish the result back needs a second barrier.
- if ((lane_in_row & 31) == 0) {
- smem[row_in_block * kWarpsPerRow + lane_in_row / 32] = sum_sq;
- }
- __syncthreads();
- float total_sum = 0.0f;
- #pragma unroll
- for (int i = 0; i < kWarpsPerRow; i++) {
- total_sum += smem[row_in_block * kWarpsPerRow + i];
- }
- const float inv_rms = rsqrtf(total_sum / h + eps);
+ ltx_kernels::load_run<PAIRS>(weights + first_channel, first_weight);
+ ltx_kernels::load_run<PAIRS>(weights + second_channel, second_weight);
+ float first_rot[PAIRS], second_rot[PAIRS];
+ ltx_kernels::rms_norm_split_rope_pairs<PAIRS, kWarpsPerRow>(
+ x + row + first_channel, x + row + second_channel, cos_freqs + freq_offset, sin_freqs + sin_offset,
+ first_weight, second_weight, eps, h, lane_in_row, smem + row_in_block * kWarpsPerRow, first_rot,
+ second_rot);
// at::Float8_e4m3fn / at::BFloat16 do not convert from the native CUDA types, so
// the staging arrays and the output pointer both use the native one.
@@ -111,16 +79,12 @@ __global__ void _rms_norm_split_rope_kernel(
store_t first_out[PAIRS], second_out[PAIRS];
#pragma unroll
for (int i = 0; i < PAIRS; i++) {
- const float a = first[i] * inv_rms * float(first_weight[i]);
- const float c = second[i] * inv_rms * float(second_weight[i]);
- const float cs = float(cos_raw[i]);
- const float sn = float(sin_raw[i]);
if constexpr (std::is_same_v<out_t, at::Float8_e4m3fn>) {
- first_out[i] = fp8(a * cs - c * sn);
- second_out[i] = fp8(c * cs + a * sn);
+ first_out[i] = fp8(first_rot[i]);
+ second_out[i] = fp8(second_rot[i]);
} else {
- first_out[i] = __float2bfloat16(a * cs - c * sn);
- second_out[i] = __float2bfloat16(c * cs + a * sn);
+ first_out[i] = __float2bfloat16(first_rot[i]);
+ second_out[i] = __float2bfloat16(second_rot[i]);
}
}
if (active) {
diff --git a/packages/ltx-kernels/pyproject.toml b/packages/ltx-kernels/pyproject.toml
index 0f15d73..208137c 100644
--- a/packages/ltx-kernels/pyproject.toml
+++ b/packages/ltx-kernels/pyproject.toml
@@ -8,7 +8,7 @@ build-backend = "setuptools.build_meta"
[project]
name = "ltx-kernels"
-version = "1.4.2"
+version = "1.5.0"
dependencies = [
"torch",
"triton>=3.0",
diff --git a/packages/ltx-kernels/setup.py b/packages/ltx-kernels/setup.py
index 8d548d9..3bdcf0c 100644
--- a/packages/ltx-kernels/setup.py
+++ b/packages/ltx-kernels/setup.py
@@ -261,6 +261,7 @@ if __name__ == "__main__":
"csrc/all2all/all2all.cpp",
"csrc/all2all/cuda/all2all_heads.cu",
"csrc/all2all/cuda/allgather.cu",
+ "csrc/all2all/cuda/fused_send.cu",
],
library_dirs=cuda_link_dirs,
extra_compile_args={"cxx": all2all_args, "nvcc": ["-O3"]},
@@ -281,7 +282,7 @@ if __name__ == "__main__":
"csrc/ops/rms_norm_split_rope.cpp",
"csrc/ops/rms_norm_split_rope_cuda.cu",
],
- include_dirs=[str(ROOT / "csrc/ops/include"), *_nvidia_include_dirs()],
+ include_dirs=[str(ROOT / "csrc/ops/include"), str(ROOT / "csrc/include"), *_nvidia_include_dirs()],
library_dirs=cuda_link_dirs,
extra_compile_args={
"cxx": ["-O3", "-std=c++17"],
diff --git a/packages/ltx-kernels/src/ltx_kernels/all_to_all.py b/packages/ltx-kernels/src/ltx_kernels/all_to_all.py
index 8fa5677..75aaf11 100644
--- a/packages/ltx-kernels/src/ltx_kernels/all_to_all.py
+++ b/packages/ltx-kernels/src/ltx_kernels/all_to_all.py
@@ -44,6 +44,70 @@ def _gather_heads_fake(x: torch.Tensor, buffer: torch.Tensor, comm_id: int) -> N
return None
+@custom_op("ltx_kernels::rope_send_recv_heads", mutates_args=("buffer",), device_types="cuda")
+def _rope_send_recv_heads_op(
+ x: torch.Tensor,
+ weights: torch.Tensor,
+ cos_freqs: torch.Tensor,
+ sin_freqs: torch.Tensor,
+ eps: float,
+ buffer: torch.Tensor, # noqa: ARG001
+ comm_id: int,
+ barrier: bool = True,
+) -> None:
+ All2All._runtime_registry[comm_id].rope_send_recv_heads(x, sin_freqs, cos_freqs, weights, eps, barrier)
+
+
+@_rope_send_recv_heads_op.register_fake
+def _rope_send_recv_heads_fake(
+ x: torch.Tensor, # noqa: ARG001
+ weights: torch.Tensor, # noqa: ARG001
+ cos_freqs: torch.Tensor, # noqa: ARG001
+ sin_freqs: torch.Tensor, # noqa: ARG001
+ eps: float, # noqa: ARG001
+ buffer: torch.Tensor, # noqa: ARG001
+ comm_id: int, # noqa: ARG001
+ barrier: bool = True, # noqa: ARG001
+) -> None:
+ return None
+
+
+@custom_op("ltx_kernels::linear_send_recv_heads", mutates_args=("buffer",), device_types="cuda")
+def _linear_send_recv_heads_op(
+ x: torch.Tensor,
+ weight: torch.Tensor,
+ bias: torch.Tensor,
+ buffer: torch.Tensor, # noqa: ARG001
+ comm_id: int,
+ barrier: bool = True,
+) -> None:
+ All2All._runtime_registry[comm_id].linear_send_recv_heads(x, weight, bias, barrier)
+
+
+@_linear_send_recv_heads_op.register_fake
+def _linear_send_recv_heads_fake(
+ x: torch.Tensor, # noqa: ARG001
+ weight: torch.Tensor, # noqa: ARG001
+ bias: torch.Tensor, # noqa: ARG001
+ buffer: torch.Tensor, # noqa: ARG001
+ comm_id: int, # noqa: ARG001
+ barrier: bool = True, # noqa: ARG001
+) -> None:
+ return None
+
+
+@custom_op("ltx_kernels::exchange_barrier", mutates_args=("buffers",), device_types="cuda")
+def _exchange_barrier_op(buffers: list[torch.Tensor], comm_id: int) -> None: # noqa: ARG001
+ # ``buffers`` are the exchanges this barrier completes: declaring them mutated orders every read of
+ # them after it.
+ All2All._runtime_registry[comm_id].exchange_barrier()
+
+
+@_exchange_barrier_op.register_fake
+def _exchange_barrier_fake(buffers: list[torch.Tensor], comm_id: int) -> None: # noqa: ARG001
+ return None
+
+
class All2All:
"""IPC-based All2All communication for distributed head-parallel attention.
This class manages GPU memory buffers and IPC handles to enable efficient
diff --git a/packages/ltx-kernels/src/ltx_kernels/blockwise/_common.py b/packages/ltx-kernels/src/ltx_kernels/blockwise/_common.py
index 478bb68..68d03c5 100644
--- a/packages/ltx-kernels/src/ltx_kernels/blockwise/_common.py
+++ b/packages/ltx-kernels/src/ltx_kernels/blockwise/_common.py
@@ -24,11 +24,13 @@ def _get_tma_aligned_size(n: int, element_size: int) -> int:
return ((n + num_elems_tma - 1) // num_elems_tma) * num_elems_tma
-def _empty_tma_aligned_scales(num_rows: int, num_blocks: int, device: torch.device) -> tuple[torch.Tensor, int]:
- """Allocate MN-major FP32 scales with TMA-padded row storage."""
- tma_aligned_mn = _get_tma_aligned_size(num_rows, 4)
- scales = torch.empty((num_blocks, tma_aligned_mn), dtype=torch.float, device=device).t()[:num_rows]
- return scales, tma_aligned_mn
+def _empty_tma_aligned_scales(num_rows: int, num_blocks: int, device: torch.device) -> torch.Tensor:
+ """Allocate ``(num_blocks, tma_aligned_rows)`` FP32 scale storage.
+ Kernels write this tensor itself; the op returns ``.t()[:num_rows]``, the MN-major
+ ``(num_rows, num_blocks)`` view. Writing through that view instead makes inductor guard
+ on whether ``num_rows`` is TMA-aligned, so a dynamic row count recompiles per alignment.
+ """
+ return torch.empty((num_blocks, _get_tma_aligned_size(num_rows, 4)), dtype=torch.float, device=device)
@jit
diff --git a/packages/ltx-kernels/src/ltx_kernels/blockwise/ops/quantize.py b/packages/ltx-kernels/src/ltx_kernels/blockwise/ops/quantize.py
index 86dfd0e..de73d76 100644
--- a/packages/ltx-kernels/src/ltx_kernels/blockwise/ops/quantize.py
+++ b/packages/ltx-kernels/src/ltx_kernels/blockwise/ops/quantize.py
@@ -33,10 +33,10 @@ def _block_quantize_kernel(
X,
Out_scales,
X_out,
+ tma_aligned_mn: int,
H: tl.constexpr,
NUM_BLOCKS: tl.constexpr,
BLOCK_SIZE: tl.constexpr,
- TMA_ALIGNED_MN: tl.constexpr,
SCALE_MAX: tl.constexpr,
):
token_idx = tl.program_id(0).to(tl.int64)
@@ -48,7 +48,7 @@ def _block_quantize_kernel(
X_out + token_idx * H + BLOCK_SIZE * tl.arange(0, NUM_BLOCKS)[:, None] + tl.arange(0, BLOCK_SIZE)[None, :]
)
tl.store(x_out_ptr, o_quant)
- out_scales_ptr = Out_scales + token_idx + TMA_ALIGNED_MN * tl.arange(0, NUM_BLOCKS)[:, None]
+ out_scales_ptr = Out_scales + token_idx + tma_aligned_mn * tl.arange(0, NUM_BLOCKS)[:, None]
tl.store(out_scales_ptr, o_scales)
@@ -65,7 +65,7 @@ def blockwise_quantize(x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
num_rows = b * s
num_blocks = h // _BLOCK_SIZE
out = torch.empty((num_rows, h), device=x.device, dtype=torch.float8_e4m3fn)
- scales, tma_aligned_mn = _empty_tma_aligned_scales(num_rows, num_blocks, x.device)
+ scales = _empty_tma_aligned_scales(num_rows, num_blocks, x.device)
wrap_triton(_block_quantize_kernel)[(num_rows,)](
x,
scales,
@@ -73,10 +73,10 @@ def blockwise_quantize(x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:
H=h,
BLOCK_SIZE=_BLOCK_SIZE,
NUM_BLOCKS=num_blocks,
- TMA_ALIGNED_MN=tma_aligned_mn,
+ tma_aligned_mn=scales.shape[1],
SCALE_MAX=_FP8_SCALE_MAX,
)
- return out.view(b, s, h), scales
+ return out.view(b, s, h), scales.t()[:num_rows]
@jit
diff --git a/packages/ltx-kernels/src/ltx_kernels/compile/passes/__init__.py b/packages/ltx-kernels/src/ltx_kernels/compile/passes/__init__.py
index e224521..df86570 100644
--- a/packages/ltx-kernels/src/ltx_kernels/compile/passes/__init__.py
+++ b/packages/ltx-kernels/src/ltx_kernels/compile/passes/__init__.py
@@ -8,11 +8,13 @@ from ltx_kernels.compile.passes.fuse_trailing_quantize import (
AdaZeroTrailingQuantizePass,
GatedAttentionTrailingQuantizePass,
)
+from ltx_kernels.compile.passes.fused_send_recv import CoalescedExchangeBarrierPass, FusedSendRecvPass
from ltx_kernels.compile.passes.gated_attention import GatedAttentionPass
from ltx_kernels.compile.passes.grouped_upcast import GroupedUpcastPass
from ltx_kernels.compile.passes.linear_gelu import LinearGELUPass
from ltx_kernels.compile.passes.pre_attention import PreAttentionPass
from ltx_kernels.compile.passes.rms_fma import RMSFmaAdaLNPass, RMSFmaPass
+from ltx_kernels.compile.passes.sdpa_select import SDPASelectPass
def inductor_passes() -> list[PassBase]:
@@ -22,8 +24,10 @@ def inductor_passes() -> list[PassBase]:
adds cost more there than they save, alone and together, by cutting inductor's own
fusion across the block. The quantize passes are inert unless
``ltx_core.quantization.blockwise`` installs the linears that emit
- ``ltx_kernels::blockwise_quantize``. :class:`GroupedUpcastPass` is inert outside the
- fp8-cast policy.
+ ``ltx_kernels::blockwise_quantize``. :class:`FusedSendRecvPass` is inert outside
+ sequence parallelism and binds the op :class:`PreAttentionPass` leaves behind;
+ :class:`CoalescedExchangeBarrierPass` then gives the exchanges it fuses one barrier per attention.
+ :class:`GroupedUpcastPass` is inert outside the fp8-cast policy.
One instance per stream shape (LTX-2.3: video 4096, audio 2048), since a pass pins
the ``hidden_dim`` and ``heads`` literals its pattern carries.
"""
@@ -32,11 +36,14 @@ def inductor_passes() -> list[PassBase]:
PreAttentionPass(hidden_dim=2048, heads=32, eps=1e-6),
GatedAttentionPass(),
LinearGELUPass(),
+ FusedSendRecvPass(),
+ CoalescedExchangeBarrierPass(),
BlockwiseQuantizeCSE(),
AdaZeroTrailingQuantizePass(hidden_dim=4096, eps=1e-6),
AdaZeroTrailingQuantizePass(hidden_dim=2048, eps=1e-6),
GatedAttentionTrailingQuantizePass(),
GroupedUpcastPass(),
+ SDPASelectPass(),
]
@@ -64,6 +71,8 @@ __all__ = [
"AdaZeroPass",
"AdaZeroTrailingQuantizePass",
"BlockwiseQuantizeCSE",
+ "CoalescedExchangeBarrierPass",
+ "FusedSendRecvPass",
"GatedAttentionPass",
"GatedAttentionTrailingQuantizePass",
"GroupedUpcastPass",
@@ -71,6 +80,7 @@ __all__ = [
"PreAttentionPass",
"RMSFmaAdaLNPass",
"RMSFmaPass",
+ "SDPASelectPass",
"eager_passes",
"inductor_passes",
]
diff --git a/packages/ltx-kernels/src/ltx_kernels/compile/passes/fused_send_recv.py b/packages/ltx-kernels/src/ltx_kernels/compile/passes/fused_send_recv.py
new file mode 100644
index 0000000..9b58722
--- /dev/null
+++ b/packages/ltx-kernels/src/ltx_kernels/compile/passes/fused_send_recv.py
@@ -0,0 +1,186 @@
+"""Pass that folds the all2all head exchange into the kernel producing the exchanged tensor."""
+
+import logging
+import operator
+
+import torch
+from torch.fx import GraphModule, Node
+from torch.fx.experimental.symbolic_shapes import statically_known_true
+from torch.fx.passes.infra.pass_base import PassBase, PassResult
+
+logger = logging.getLogger(__name__)
+
+
+def _is_target(node: object, *targets: object) -> bool:
+ return isinstance(node, Node) and node.op == "call_function" and node.target in targets
+
+
+def _sole_user(node: Node) -> Node | None:
+ return next(iter(node.users)) if len(node.users) == 1 else None
+
+
+class FusedSendRecvPass(PassBase):
+ """Rewrite ``send_recv_heads(producer(...).view(b, t, heads, d))`` to a fused producer-exchange op.
+ Producers taken, when the exchange is their only consumer:
+ - ``ltx_kernels::rms_norm_split_rope`` (left by :class:`PreAttentionPass`) ->
+ ``ltx_kernels::rope_send_recv_heads``, which normalizes, rotates and writes each head straight
+ into its owning rank's buffer.
+ - ``linear`` -> ``ltx_kernels::linear_send_recv_heads``, which runs the projection per rank-owned
+ column block with the output in that rank's buffer.
+ Both write the same bytes ``send_recv_heads`` would, into the same buffer, so everything reading
+ the buffer afterwards is unchanged. Must run after :class:`PreAttentionPass`.
+ A plain pass rather than a :class:`SubgraphRewritePass`: ``send_recv_heads`` returns nothing,
+ so a pattern ending in it has no returning node for the matcher to anchor on, and capturing it
+ through Dynamo would run the exchange.
+ """
+
+ def __init__(self) -> None:
+ self.match_count = 0
+
+ def call(self, graph_module: GraphModule) -> PassResult:
+ from ltx_kernels import all_to_all as _all_to_all_ops # noqa: F401, PLC0415
+ from ltx_kernels.ops import rms_norm_split_rope as _rope_ops # noqa: F401, PLC0415
+
+ send_recv = (torch.ops.ltx_kernels.send_recv_heads, torch.ops.ltx_kernels.send_recv_heads.default)
+ rope = torch.ops.ltx_kernels.rms_norm_split_rope.default
+ graph = graph_module.graph
+ modified = False
+ for node in list(graph.nodes):
+ if not _is_target(node, *send_recv) or node.kwargs:
+ continue
+ view, buffer, comm_id = node.args
+ if not (isinstance(view, Node) and view.op == "call_method" and view.target == "view"):
+ continue
+ producer = view.args[0]
+ if _sole_user(view) is not node or not isinstance(producer, Node) or _sole_user(producer) is not view:
+ continue
+ if _is_target(producer, rope) and not producer.kwargs:
+ x, weights, cos, sin, eps = producer.args
+ target, args = torch.ops.ltx_kernels.rope_send_recv_heads.default, (x, weights, cos, sin, eps)
+ elif _is_target(producer, torch._C._nn.linear) and len(producer.args) == 3 and not producer.kwargs:
+ x, weight, bias = producer.args
+ if bias is None:
+ continue
+ target, args = torch.ops.ltx_kernels.linear_send_recv_heads.default, (x, weight, bias)
+ else:
+ continue
+ value = args[0].meta.get("example_value") if isinstance(args[0], Node) else None
+ buffer_value = buffer.meta.get("example_value") if isinstance(buffer, Node) else None
+ if not isinstance(value, torch.Tensor) or value.dtype != torch.bfloat16 or value.ndim != 3:
+ continue
+ # The fused op takes batch and tokens from the producer input, so the exchanged view must
+ # split only the hidden dim; a view regrouping batch and tokens has another buffer layout.
+ heads_view = view.meta.get("example_value")
+ if not (
+ isinstance(heads_view, torch.Tensor)
+ and heads_view.ndim == 4
+ and all(statically_known_true(heads_view.shape[i] == value.shape[i]) for i in (0, 1))
+ ):
+ continue
+ # The fused producers write bf16; a blockwise-fp8 exchange keeps its own copy kernel.
+ if not isinstance(buffer_value, torch.Tensor) or buffer_value.dtype != torch.bfloat16:
+ continue
+ with graph.inserting_before(node):
+ fused = graph.call_function(target, (*args, buffer, comm_id))
+ node.replace_all_uses_with(fused)
+ graph.erase_node(node)
+ graph.erase_node(view)
+ graph.erase_node(producer)
+ self.match_count += 1
+ modified = True
+ if modified:
+ graph.lint()
+ graph_module.recompile()
+ logger.info("FusedSendRecvPass fused %d exchange(s) into their producers", self.match_count)
+ return PassResult(graph_module, modified)
+
+
+# Methods that always return a view of their input.
+_VIEW_METHODS = frozenset({"view", "unflatten", "transpose", "permute", "unsqueeze", "squeeze"})
+# Methods that return a view only when the input's strides allow one, and a copy otherwise.
+_MAYBE_VIEW_METHODS = frozenset({"flatten", "reshape"})
+
+
+def _is_basic_index(index: object) -> bool:
+ """Whether ``tensor[index]`` is basic indexing (a view), not advanced indexing (a copy)."""
+ if isinstance(index, tuple):
+ return all(_is_basic_index(i) for i in index)
+ if isinstance(index, slice):
+ return all(_is_basic_index(bound) for bound in (index.start, index.stop, index.step))
+ if isinstance(index, Node):
+ # A traced size; a tensor index is advanced.
+ index = index.meta.get("example_value")
+ return (
+ index is None or index is Ellipsis or (isinstance(index, (int, torch.SymInt)) and not isinstance(index, bool))
+ )
+
+
+def _shares_storage(node: Node, source: Node) -> bool:
+ """Whether the traced values prove ``node`` is a view of ``source``."""
+ from torch.multiprocessing.reductions import StorageWeakRef # noqa: PLC0415
+
+ value, source_value = node.meta.get("example_value"), source.meta.get("example_value")
+ if not (isinstance(value, torch.Tensor) and isinstance(source_value, torch.Tensor)):
+ return False
+ return StorageWeakRef(value.untyped_storage()) == StorageWeakRef(source_value.untyped_storage())
+
+
+def _is_view(node: Node) -> bool:
+ """Whether ``node`` only re-views its first argument, so reaching it does not read the data."""
+ if node.op == "call_method" and node.target in _VIEW_METHODS:
+ return True
+ if node.op == "call_method" and node.target in _MAYBE_VIEW_METHODS:
+ return isinstance(node.args[0], Node) and _shares_storage(node, node.args[0])
+ return _is_target(node, operator.getitem) and len(node.args) == 2 and _is_basic_index(node.args[1])
+
+
+class CoalescedExchangeBarrierPass(PassBase):
+ """Complete consecutive fused exchanges with one barrier instead of one each.
+ Attention reads its q, k and v buffers only after all three exchanges, so their per-exchange
+ barriers -- a cross-rank round trip each -- can be one ``ltx_kernels::exchange_barrier``, placed
+ before the first node that reads any of their buffers. Views of a buffer are not reads; ops that
+ may copy (advanced indexing, a ``flatten`` the strides do not allow as a view) are, as is a later
+ exchange taking a pending buffer, which then starts the next group. The barrier names the buffers
+ as mutated, so everything that reads them is ordered after it. A group that is not read inside the
+ graph keeps its barriers. Must run after :class:`FusedSendRecvPass`.
+ """
+
+ def __init__(self) -> None:
+ self.match_count = 0
+
+ def call(self, graph_module: GraphModule) -> PassResult:
+ from ltx_kernels import all_to_all as _all_to_all_ops # noqa: F401, PLC0415
+
+ fused = (
+ torch.ops.ltx_kernels.rope_send_recv_heads.default,
+ torch.ops.ltx_kernels.linear_send_recv_heads.default,
+ )
+ graph = graph_module.graph
+ removed = 0
+ pending: list[Node] = []
+ aliases: set[Node] = set()
+ for node in list(graph.nodes):
+ if pending and _is_view(node) and node.args[0] in aliases:
+ aliases.add(node)
+ continue
+ if pending and any(input_node in aliases for input_node in node.all_input_nodes):
+ if len(pending) > 1:
+ for exchange in pending:
+ exchange.update_kwarg("barrier", False)
+ with graph.inserting_before(node):
+ buffers = [exchange.args[-2] for exchange in pending]
+ graph.call_function(
+ torch.ops.ltx_kernels.exchange_barrier.default, (buffers, pending[-1].args[-1])
+ )
+ removed += len(pending) - 1
+ pending, aliases = [], set()
+ # Checked after the reads: an exchange whose inputs are a pending buffer completes that group first.
+ if _is_target(node, *fused) and node.kwargs.get("barrier", True):
+ pending.append(node)
+ aliases.add(node.args[-2])
+ if removed:
+ graph.lint()
+ graph_module.recompile()
+ logger.info("CoalescedExchangeBarrierPass removed %d exchange barrier(s)", removed)
+ self.match_count += removed
+ return PassResult(graph_module, bool(removed))
diff --git a/packages/ltx-kernels/src/ltx_kernels/compile/passes/sdpa_select.py b/packages/ltx-kernels/src/ltx_kernels/compile/passes/sdpa_select.py
new file mode 100644
index 0000000..0189891
--- /dev/null
+++ b/packages/ltx-kernels/src/ltx_kernels/compile/passes/sdpa_select.py
@@ -0,0 +1,39 @@
+"""Pass that routes unmasked SDPA calls to ``ltx_kernels::sdpa_select``."""
+
+from typing import ClassVar
+
+import torch
+from torch.fx.passes.utils.matcher_utils import InternalMatch
+
+from ltx_kernels.compile.subgraph_rewrite_pass import SubgraphRewritePass, is_cuda_bf16
+
+
+class SDPASelectPass(SubgraphRewritePass):
+ """Rewrite the unmasked ``scaled_dot_product_attention(q, k, v)`` to ``ltx_kernels::sdpa_select``.
+ The pattern carries the keyword arguments ``Attention``'s SDPA backends pass, and
+ ``ignore_literals=False`` pins them: only a call with no mask, no dropout and
+ ``is_causal=False`` is the op's to serve. The surrounding ``sdpa_kernel`` priority context
+ is left in place and has no effect on the op, which pins its own backend per shape.
+ """
+
+ ignore_literals: ClassVar[bool] = False
+
+ @staticmethod
+ def pattern_fn(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor:
+ return torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=0.0, is_causal=False)
+
+ @staticmethod
+ def replacement_fn(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor:
+ return torch.ops.ltx_kernels.sdpa_select.default(q, k, v)
+
+ def _is_match_eligible(self, match: InternalMatch) -> bool:
+ inputs = self._matched_inputs(match)
+ if not super()._is_match_eligible(match) or not inputs:
+ return False
+ return is_cuda_bf16(*inputs) and all(t.ndim == 4 for t in inputs)
+
+ def example_inputs(self) -> tuple:
+ return tuple(torch.empty(1, 4, 64, 64, dtype=torch.bfloat16, device="cuda") for _ in range(3))
+
+ def _register_ops(self) -> None:
+ from ltx_kernels.ops import sdpa_select as _sdpa_select_ops # noqa: F401, PLC0415
diff --git a/packages/ltx-kernels/src/ltx_kernels/devices.py b/packages/ltx-kernels/src/ltx_kernels/devices.py
new file mode 100644
index 0000000..86a5ec9
--- /dev/null
+++ b/packages/ltx-kernels/src/ltx_kernels/devices.py
@@ -0,0 +1,19 @@
+"""Host-to-device transfer helpers shared by the kernels.
+``ltx_kernels`` does not depend on ``ltx_core``, so it keeps its own counterpart of
+``ltx_core.devices.upload_from_host``.
+"""
+
+from __future__ import annotations
+
+import torch
+
+
+def upload_from_host(host: torch.Tensor, device: torch.device) -> torch.Tensor:
+ """Copy a host tensor to ``device`` without waiting for the device.
+ A pageable host-to-device copy synchronizes the stream; a pinned one is ordered on it instead,
+ so ``host`` may be modified as soon as this returns. A tensor that is already on a device, or a
+ target that is not CUDA, is moved with ``.to``.
+ """
+ if host.device.type != "cpu" or device.type != "cuda":
+ return host.to(device)
+ return host.pin_memory().to(device, non_blocking=True)
diff --git a/packages/ltx-kernels/src/ltx_kernels/ops/ada_zero.py b/packages/ltx-kernels/src/ltx_kernels/ops/ada_zero.py
index d4f0670..aa31ed6 100644
--- a/packages/ltx-kernels/src/ltx_kernels/ops/ada_zero.py
+++ b/packages/ltx-kernels/src/ltx_kernels/ops/ada_zero.py
@@ -37,11 +37,11 @@ def _ada_zero_kernel( # noqa: PLR0913
norm_scale_token_stride: int,
norm_shift_token_stride: int,
seqlen: int,
+ tma_aligned_mn: int,
eps: float,
H: tl.constexpr,
NUM_BLOCKS: tl.constexpr,
BLOCK_SIZE: tl.constexpr,
- TMA_ALIGNED_MN: tl.constexpr,
QUANTIZE: tl.constexpr,
SCALE_MAX: tl.constexpr,
):
@@ -80,7 +80,7 @@ def _ada_zero_kernel( # noqa: PLR0913
X_out + token_idx * H + BLOCK_SIZE * tl.arange(0, NUM_BLOCKS)[:, None] + tl.arange(0, BLOCK_SIZE)[None, :]
)
tl.store(x_out_ptr, o_quant)
- out_scales_ptr = Out_scales + token_idx + TMA_ALIGNED_MN * tl.arange(0, NUM_BLOCKS)[:, None]
+ out_scales_ptr = Out_scales + token_idx + tma_aligned_mn * tl.arange(0, NUM_BLOCKS)[:, None]
tl.store(out_scales_ptr, o_scales)
else:
x_out_ptr = X_out + token_idx * H + tl.arange(0, H)
@@ -132,7 +132,7 @@ def ada_zero(
H=h,
NUM_BLOCKS=0,
BLOCK_SIZE=_BLOCK_SIZE,
- TMA_ALIGNED_MN=0,
+ tma_aligned_mn=0,
QUANTIZE=False,
SCALE_MAX=_FP8_SCALE_MAX,
)
@@ -160,7 +160,7 @@ def ada_zero_quantized(
scale = scale.expand(b, s, h)
shift = shift.expand(b, s, h)
out = torch.empty((num_rows, h), device=x.device, dtype=torch.float8_e4m3fn)
- scales, tma_aligned_mn = _empty_tma_aligned_scales(num_rows, num_blocks, x.device)
+ scales = _empty_tma_aligned_scales(num_rows, num_blocks, x.device)
wrap_triton(_ada_zero_kernel)[(num_rows,)](
x,
scale,
@@ -176,8 +176,8 @@ def ada_zero_quantized(
H=h,
NUM_BLOCKS=num_blocks,
BLOCK_SIZE=_BLOCK_SIZE,
- TMA_ALIGNED_MN=tma_aligned_mn,
+ tma_aligned_mn=scales.shape[1],
QUANTIZE=True,
SCALE_MAX=_FP8_SCALE_MAX,
)
- return out.view(b, s, h), scales
+ return out.view(b, s, h), scales.t()[:num_rows]
diff --git a/packages/ltx-kernels/src/ltx_kernels/ops/gated_attention.py b/packages/ltx-kernels/src/ltx_kernels/ops/gated_attention.py
index 32f95a6..a78b33e 100644
--- a/packages/ltx-kernels/src/ltx_kernels/ops/gated_attention.py
+++ b/packages/ltx-kernels/src/ltx_kernels/ops/gated_attention.py
@@ -40,12 +40,12 @@ def _gated_attention_kernel( # noqa: PLR0913
gate_batch_stride: int,
gate_token_stride: int,
seqlen: int,
+ tma_aligned_mn: int,
H: tl.constexpr,
NUM_HEADS: tl.constexpr,
DIM_HEAD: tl.constexpr,
NUM_BLOCKS: tl.constexpr,
BLOCK_SIZE: tl.constexpr,
- TMA_ALIGNED_MN: tl.constexpr,
QUANTIZE: tl.constexpr,
SCALE_MAX: tl.constexpr,
):
@@ -74,7 +74,7 @@ def _gated_attention_kernel( # noqa: PLR0913
Out + token_idx * H + BLOCK_SIZE * tl.arange(0, NUM_BLOCKS)[:, None] + tl.arange(0, BLOCK_SIZE)[None, :]
)
tl.store(out_ptr, o_quant)
- scales_ptr = Out_scales + token_idx + TMA_ALIGNED_MN * tl.arange(0, NUM_BLOCKS)[:, None]
+ scales_ptr = Out_scales + token_idx + tma_aligned_mn * tl.arange(0, NUM_BLOCKS)[:, None]
tl.store(scales_ptr, o_scales)
else:
out_ptr = Out + token_idx * H + offsets
@@ -120,7 +120,7 @@ def gated_attention(x: torch.Tensor, gate_logits: torch.Tensor) -> torch.Tensor:
DIM_HEAD=dim_head,
NUM_BLOCKS=num_blocks,
BLOCK_SIZE=_BLOCK_SIZE,
- TMA_ALIGNED_MN=0,
+ tma_aligned_mn=0,
QUANTIZE=False,
SCALE_MAX=_FP8_SCALE_MAX,
)
@@ -149,7 +149,7 @@ def gated_attention_quantized(x: torch.Tensor, gate_logits: torch.Tensor) -> Tup
gate_logits = gate_logits.expand(b, t, num_heads)
out = torch.empty((num_rows, h), device=x.device, dtype=torch.float8_e4m3fn)
- scales, tma_aligned_mn = _empty_tma_aligned_scales(num_rows, num_blocks, x.device)
+ scales = _empty_tma_aligned_scales(num_rows, num_blocks, x.device)
wrap_triton(_gated_attention_kernel)[(num_rows,)](
x,
@@ -164,9 +164,9 @@ def gated_attention_quantized(x: torch.Tensor, gate_logits: torch.Tensor) -> Tup
DIM_HEAD=dim_head,
NUM_BLOCKS=num_blocks,
BLOCK_SIZE=_BLOCK_SIZE,
- TMA_ALIGNED_MN=tma_aligned_mn,
+ tma_aligned_mn=scales.shape[1],
QUANTIZE=True,
SCALE_MAX=_FP8_SCALE_MAX,
)
- return out.view(b, t, h), scales
+ return out.view(b, t, h), scales.t()[:num_rows]
diff --git a/packages/ltx-kernels/src/ltx_kernels/ops/sdpa_select.py b/packages/ltx-kernels/src/ltx_kernels/ops/sdpa_select.py
new file mode 100644
index 0000000..f2eb7c2
--- /dev/null
+++ b/packages/ltx-kernels/src/ltx_kernels/ops/sdpa_select.py
@@ -0,0 +1,88 @@
+"""Unmasked SDPA that runs whichever of cuDNN and flash attention is faster for each shape.
+cuDNN's fused attention is the fastest kernel for the transformer's long self-attention, but
+for a short query against a long key sequence (audio queries over video keys) it falls back to
+an sm80 kernel with no split over the keys, which leaves most SMs idle and runs several times
+slower than flash attention's split-KV kernel.
+The first call per input geometry times both backends and caches the winner; later calls, and
+calls made while a CUDA graph is being captured, reuse it. A geometry first seen during capture
+runs cuDNN, since timing inside a capture is impossible. A geometry neither backend takes runs
+unpinned, so SDPA falls back to its other kernels as the rewritten call would. The choice is
+cached per deterministic-algorithms mode, which decides whether cuDNN is usable at all.
+Candidates are timed as replays of a CUDA graph, as the transformer runs them. Timing eager
+launches measures host launch cost for short attentions, where cuDNN's is higher, and picks
+flash for shapes where cuDNN's kernel is several times faster.
+"""
+
+import torch
+import torch.nn.functional as F
+from torch.nn.attention import SDPBackend, sdpa_kernel
+
+_CANDIDATES: tuple[SDPBackend, ...] = (SDPBackend.CUDNN_ATTENTION, SDPBackend.FLASH_ATTENTION)
+_TIMING_ITERS = 10
+
+_Key = tuple[tuple[int, ...], tuple[int, ...], tuple[int, ...], torch.dtype, int, bool]
+# ``None`` is a geometry neither candidate takes: it runs unpinned.
+_choices: dict[_Key, SDPBackend | None] = {}
+
+
+def _key(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> _Key:
+ deterministic = torch.are_deterministic_algorithms_enabled()
+ return (tuple(q.shape), tuple(k.shape), tuple(v.shape), q.dtype, q.device.index, deterministic)
+
+
+def _run(backend: SDPBackend | None, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor:
+ if backend is None:
+ return F.scaled_dot_product_attention(q, k, v)
+ with sdpa_kernel([backend]):
+ return F.scaled_dot_product_attention(q, k, v)
+
+
+def _time(backend: SDPBackend, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> float:
+ try:
+ # Eager first, so plan building and workspace allocation stay out of the timed graph.
+ _run(backend, q, k, v)
+ except RuntimeError:
+ return float("inf")
+ graph = torch.cuda.CUDAGraph()
+ with torch.cuda.graph(graph):
+ for _ in range(_TIMING_ITERS):
+ _run(backend, q, k, v)
+ graph.replay()
+ start, end = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)
+ start.record()
+ graph.replay()
+ end.record()
+ end.synchronize()
+ return start.elapsed_time(end)
+
+
+def _select(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> SDPBackend | None:
+ key = _key(q, k, v)
+ if key in _choices:
+ return _choices[key]
+ if torch.cuda.is_current_stream_capturing():
+ return SDPBackend.CUDNN_ATTENTION
+ times = {backend: _time(backend, q, k, v) for backend in _CANDIDATES}
+ fastest = min(times, key=times.__getitem__)
+ choice = fastest if times[fastest] != float("inf") else None
+ _choices[key] = choice
+ return choice
+
+
[email protected]_op("ltx_kernels::sdpa_select", mutates_args=(), device_types="cuda")
+def sdpa_select(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor:
+ """Unmasked, non-causal ``scaled_dot_product_attention`` over ``(B, H, S, D)`` inputs."""
+ out = _run(_select(q, k, v), q, k, v)
+ if not out.transpose(1, 2).is_contiguous():
+ out = out.transpose(1, 2).contiguous().transpose(1, 2)
+ return out
+
+
+def _sdpa_select_fake(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor: # noqa: ARG001
+ # A (B, S, H, D)-contiguous buffer viewed as (B, H, S, D): what both backends return for the
+ # head-interleaved inputs attention builds, so the head merge that follows is a free view.
+ b, h, s, _ = q.shape
+ return q.new_empty((b, s, h, v.shape[-1])).transpose(1, 2)
+
+
+sdpa_select.register_fake(_sdpa_select_fake)
diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/aot.py b/packages/ltx-kernels/src/ltx_kernels/vae/aot.py
index 2784a34..6314c6c 100644
--- a/packages/ltx-kernels/src/ltx_kernels/vae/aot.py
+++ b/packages/ltx-kernels/src/ltx_kernels/vae/aot.py
@@ -140,10 +140,11 @@ def _load_function(file_name: str, function_prefix: str) -> CompiledFunction:
@functools.lru_cache(maxsize=8)
def production_kernel_keys(gpu_arch: str) -> tuple[KernelKey, ...]:
"""Return the finite production specialization set for ``gpu_arch``."""
- from ltx_kernels.vae import block_fna_dsl, na_attn_dsl # noqa: PLC0415
+ from ltx_kernels.vae import block_fna_dsl, na_attn_dsl, rope_dsl # noqa: PLC0415
na = na_attn_dsl._compile_kernel
block = block_fna_dsl._compile_kernel
+ rope = rope_dsl._compile_kernel
return (
KernelKey.create(
na,
@@ -205,6 +206,11 @@ def production_kernel_keys(gpu_arch: str) -> tuple[KernelKey, ...]:
slab_box_max=block_fna_dsl.SLAB_BOX_MAX,
keyframes=True,
),
+ KernelKey.create(
+ rope,
+ gpu_arch,
+ rope_dims=(16, 24, 24),
+ ),
)
diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/block_fna_dsl.py b/packages/ltx-kernels/src/ltx_kernels/vae/block_fna_dsl.py
index 7efbc6a..a11ecae 100644
--- a/packages/ltx-kernels/src/ltx_kernels/vae/block_fna_dsl.py
+++ b/packages/ltx-kernels/src/ltx_kernels/vae/block_fna_dsl.py
@@ -25,6 +25,11 @@ Structure, in the order the kernel runs it:
same query tile as the video stream's, with :func:`ltx_kernels.vae.fna_smem.select_tensor`
picking the buffer at runtime. Tracing either of them twice cost ~70 s of backend compile
on a kernel whose plain build is already 25 s.
+* **One body per row pass.** Heads and C-wide chunks are walked by device loops, and each
+ warpgroup reaches its own column slice through an SMEM operand view shifted by whole
+ K-stages (:func:`ltx_kernels.vae.fna_smem._alias`) rather than through a constexpr arm
+ per warpgroup. Only what has to stay static -- slice coordinates within a stage, the
+ KV pipeline's buffer parity -- is still unrolled.
* **The attention half is not here.** The window mask, the pipelined KV loop and the
softmax are :func:`ltx_kernels.vae.fna_attn_core.attention_tile`, shared verbatim
with the standalone NA kernel.
@@ -50,6 +55,7 @@ import torch
from cutlass.cute.nvgpu import cpasync, tcgen05
from cutlass.utils import SmemAllocator
+from ltx_kernels.devices import upload_from_host
from ltx_kernels.vae.aot import (
KERNEL_CACHE,
fake_dynamic_matrix,
@@ -75,10 +81,12 @@ from ltx_kernels.vae.fna_attn_core import (
row_sum,
)
from ltx_kernels.vae.fna_gemm import (
+ GemmJob,
UmmaSharedStorage,
_mma_op,
_reg_shape,
_tmem_load,
+ _tmem_shift,
acquire_tmem,
gemm_tma,
release_tmem,
@@ -129,6 +137,7 @@ from ltx_kernels.vae.keyframe_slots import (
nearest_video_frames,
tile_plane_union,
)
+from ltx_kernels.vae.workspace import workspace_volume
# Channel counts the kernel is compiled and validated for. The shipped DiffVAE only
# ever asks for 256 (stage 5); 128 is kept because it is the cheap shape to validate
@@ -181,6 +190,8 @@ def _check_out(out: torch.Tensor, rows: int, channels: int, x_: torch.Tensor, na
#: Bytes in one N=128 B-operand stage, which is what ``gemm_tma``'s TMA transaction
#: count has to match.
_B_STAGE_BYTES = TILE_N * TILE_K * 2
+# Weight tiles the MLP keeps in flight; four N=128 x K=64 bf16 stages fill a C=256 s_q exactly.
+MLP_B_STAGES = 4
# --------------------------------------------------------------------------
@@ -306,15 +317,23 @@ class _Workspace:
# ---- SMEM operands, all aliased onto one pool; see the map in _fna_kernel ----
s_a: cute.Tensor # the activation / A operand
s_b: cute.Tensor # the streamed weight tile / B operand
- s_q: cute.Tensor # queries, then the MLP's hidden chunks
+ s_q: cute.Tensor # queries; dead after attention
+ s_b4: cute.Tensor # the MLP's weight queue, aliasing s_q
s_k: cute.Tensor
s_v: cute.Tensor
- s_p: cute.Tensor
+ s_p_own: cute.Tensor # this warpgroup's slice of P (s_p), at stage 0
+ s_h_own: cute.Tensor # this warpgroup's slice of the MLP hidden chunk (s_h), at stage 0
+ # Raw storage and operand layouts, for views shifted by a runtime number of K-stages.
+ pool: cute.Tensor # under s_a, s_p, s_k, s_b, s_v and s_h
+ q_pool: cute.Tensor # under s_q and s_b4
+ a_layout: Any
+ q_layout: Any
storage: Any # barrier storage for the pipelines gemm_tma stands up
# ---- MMA operand fragments over those buffers ----
fr_a: Any
fr_q: Any
+ fr_h: Any
fr_k: Any
fr_p: Any
fr_v: Any
@@ -345,6 +364,8 @@ class _Workspace:
rowix: cute.Tensor # in-bounds source row per lane; feeds every gather
rowix_lo: cute.Tensor # the stage-4 cell row per lane, for the fused-context gather
rowix_w: cute.Tensor # destination row per lane; scratch when out of range
+ kv_rowtab: cute.Tensor # (2, M) KV-tile source rows, for the cooperative K/V copies
+ kv_copy_bars: cute.Tensor # K-full x2, P/V-full: the KV loop's operand-ready counters
copy_slot: tuple # this thread's slot in the cooperative copies
red_c: cute.Tensor # row reduction across all warpgroups
red_head: cute.Tensor # row reduction across the warpgroups sharing a head
@@ -498,6 +519,8 @@ def _fna_kernel(
q_layout: cute.ComposedLayout,
k_layout: cute.ComposedLayout,
v_layout: cute.ComposedLayout,
+ b4_layout: cute.ComposedLayout,
+ h_layout: cute.ComposedLayout,
):
tidx, _, _ = cute.arch.thread_idx()
cta_index, _, _ = cute.arch.block_idx()
@@ -541,7 +564,19 @@ def _fna_kernel(
s_k = _alias(pool, k_layout, 2 * M * TILE_K)
s_b = _alias(pool, b_layout, 4 * M * TILE_K)
s_v = _alias(pool, v_layout, 4 * M * TILE_K)
- s_q = smem.allocate_tensor(IO, q_layout.outer, byte_alignment=128, swizzle=q_layout.inner)
+ # s_q holds the queries until attention ends; the MLP then reuses it as its weight queue.
+ # Sized for the larger view: at C=128 the queries take half of what the queue needs.
+ q_pool_elems = max(cute.cosize(q_layout), cute.cosize(b4_layout))
+ q_pool = smem.allocate_tensor(IO, cute.make_layout((q_pool_elems,)), byte_alignment=128)
+ s_q = _alias(q_pool, q_layout)
+ s_b4 = _alias(q_pool, b4_layout)
+ s_h = _alias(pool, h_layout, 4 * M * TILE_K)
+ # Warpgroup ``g``'s ``ACC_COLS`` chunk of a 128-wide row is K-stage ``g`` of it, so these
+ # views -- shifted ``wg`` stages -- give every warpgroup its own chunk at static stage 0,
+ # and a row pass over them is one code copy rather than one per warpgroup.
+ wg_stage = ids.wg * (M * TILE_K)
+ s_p_own = _alias(pool, p_layout, wg_stage)
+ s_h_own = _alias(pool, h_layout, 4 * M * TILE_K + wg_stage)
# Completion barriers: one per Q@K accumulator plus one the P@V chain signals.
# ``attention_tile`` re-initialises them per query tile so the wait phases stay
# compile-time constants.
@@ -550,6 +585,8 @@ def _fna_kernel(
rowix = smem.allocate_tensor(cutlass.Int32, cute.make_layout((M,)), byte_alignment=16)
rowix_lo = smem.allocate_tensor(cutlass.Int32, cute.make_layout((M,)), byte_alignment=16)
rowix_w = smem.allocate_tensor(cutlass.Int32, cute.make_layout((M,)), byte_alignment=16)
+ kv_rowtab = smem.allocate_tensor(cutlass.Int32, cute.make_layout((2, M)), byte_alignment=16)
+ kv_copy_bars = smem.allocate_tensor(cutlass.Int64, cute.make_layout((3,)), byte_alignment=16)
m_sm = smem.allocate_tensor(ACC, cute.make_layout((M, NH)), byte_alignment=16)
l_sm = smem.allocate_tensor(ACC, cute.make_layout((M, NUM_WARPGROUPS, NH)), byte_alignment=16)
red_c = smem.allocate_tensor(ACC, cute.make_layout((M, NUM_WARPGROUPS)), byte_alignment=16)
@@ -616,12 +653,19 @@ def _fna_kernel(
s_a=s_a,
s_b=s_b,
s_q=s_q,
+ s_b4=s_b4,
s_k=s_k,
s_v=s_v,
- s_p=s_p,
+ s_p_own=s_p_own,
+ s_h_own=s_h_own,
+ pool=pool,
+ q_pool=q_pool,
+ a_layout=a_layout,
+ q_layout=q_layout,
storage=storage,
fr_a=mma128.make_fragment_A(s_a),
fr_q=mma128.make_fragment_A(s_q),
+ fr_h=mma128.make_fragment_A(s_h),
fr_k=mma128.make_fragment_B(s_k),
fr_p=mma64.make_fragment_A(s_p),
fr_v=mma64.make_fragment_B(s_v),
@@ -644,6 +688,8 @@ def _fna_kernel(
rowix=rowix,
rowix_lo=rowix_lo,
rowix_w=rowix_w,
+ kv_rowtab=kv_rowtab,
+ kv_copy_bars=kv_copy_bars,
copy_slot=cooperative_copy_slot(tidx),
red_c=red_c,
red_head=red_head,
@@ -692,11 +738,20 @@ def _fna_kernel(
# keyframe build traces only the guarded call and a plain build only the bare
# one. Written as a single call under a runtime-or-constant guard instead, the
# plain build would carry a branch it can never take.
- while tile_id < n_tiles + ws.n_kf * tile_plane:
+ # A keyframe tile carries ``TT`` planes (group ``g`` = planes ``g*TT .. g*TT+TT-1``), so the
+ # projections and MLP run once per group of distinct queries. The group is live in this slab
+ # if any of its planes is.
+ n_groups = (ws.n_kf + shape.tile[0] - 1) // shape.tile[0]
+ while tile_id < n_tiles + n_groups * tile_plane:
live = cutlass.Int32(1)
if tile_id >= n_tiles:
- plane = unflatten_3d(tile_id - n_tiles, tile_plane, tile_row)[0]
- live = cutlass.Int32(1) if ws.kf_slab[plane] == slab.t_index else cutlass.Int32(0)
+ group = unflatten_3d(tile_id - n_tiles, tile_plane, tile_row)[0]
+ live = cutlass.Int32(0)
+ for j in cutlass.range_constexpr(shape.tile[0]):
+ plane_j = group * shape.tile[0] + j
+ if plane_j < ws.n_kf: # noqa: SIM102
+ if ws.kf_slab[plane_j] == slab.t_index:
+ live = cutlass.Int32(1)
if live != 0:
_run_tile(ws, shape, slab, tile_id, n_tiles, tile_plane, tile_row, KF, SLOTS, P_MAX, NKV_KF)
tile_id += n_ctas
@@ -907,17 +962,22 @@ def _fill_slab_caches(ws, shape: cutlass.Constexpr, slab, KF: cutlass.Constexpr
if cutlass.const_expr(shape.ctx_groups > 1):
_fill_a_gather(ws.s_q, src_ctx, ws.rowix_lo, ws.copy_slot, channels, channels)
cute.arch.sync_threads()
- for n in cutlass.range_constexpr(shape.n_chunks):
- _project(
- ws,
- shape,
- ws.w_ctx,
- ws.acc_main[n],
- subpixel * shape.n_chunks + n,
- n_k_tiles=shape.ctx_groups * shape.n_k_tiles,
- frag2=ws.fr_q,
- a_split=shape.n_k_tiles if shape.ctx_groups > 1 else 0,
- )
+ _run_jobs(
+ ws,
+ tuple(
+ _job(
+ ws,
+ shape,
+ ws.w_ctx,
+ ws.acc_main[n],
+ subpixel * shape.n_chunks + n,
+ n_k_tiles=shape.ctx_groups * shape.n_k_tiles,
+ frag2=ws.fr_q,
+ a_split=shape.n_k_tiles if shape.ctx_groups > 1 else 0,
+ )
+ for n in range(shape.n_chunks)
+ ),
+ )
cute.arch.sync_threads()
_fill_a_gather(ws.s_a, src_x, ws.rowix, ws.copy_slot, channels)
cute.arch.sync_threads()
@@ -938,11 +998,26 @@ def _fill_slab_caches(ws, shape: cutlass.Constexpr, slab, KF: cutlass.Constexpr
_apply_adaln(ws, shape, rms_inv, param["msa_weight"], param["msa_shift"])
cute.arch.sync_threads()
- for head in cutlass.range_constexpr(shape.n_heads):
- if cutlass.const_expr(KF):
- _project_head_kv(ws, shape, head, token, rope_offset, cache, cache_stride, kf_time, kf_row)
- else:
- _project_head_kv(ws, shape, head, token, rope_offset)
+ # Heads go in pairs, one pipeline and one wait per pair, into the two accumulators the
+ # residual above has finished with: ``acc_main[0]`` and ``acc_scratch``, ``TILE_N``
+ # columns apart. Both loops are device loops, so the per-head pass is traced once;
+ # the head reaches only the weight N-tile, a TMEM address, bias offsets and cache
+ # rows. ``n_heads`` is even (``attention_tile`` needs it too).
+ for pair in cutlass.range(shape.n_heads // 2):
+ _run_jobs(
+ ws,
+ (
+ _job(ws, shape, ws.w_kv, ws.acc_main[0], 2 * pair),
+ _job(ws, shape, ws.w_kv, ws.acc_scratch, 2 * pair + 1),
+ ),
+ )
+ for j in cutlass.range(2):
+ head = 2 * pair + j
+ acc = _tmem_shift(ws.acc_main[0], j * TILE_N)
+ if cutlass.const_expr(KF):
+ _project_head_kv(ws, shape, head, acc, token, rope_offset, cache, cache_stride, kf_time, kf_row)
+ else:
+ _project_head_kv(ws, shape, head, acc, token, rope_offset)
step += ws.n_ctas
@@ -1084,7 +1159,8 @@ def _subpixel_step(ws, shape: cutlass.Constexpr, slab, step, cells, n_cells):
def _project_head_kv(
ws,
shape: cutlass.Constexpr,
- head: cutlass.Constexpr,
+ head,
+ acc,
cache_row,
box_offset,
dst=None,
@@ -1092,7 +1168,7 @@ def _project_head_kv(
kf_time=None,
kf_row=None,
):
- """Project one head's K and V out of the normed row and store both to the cache.
+ """Finish one head's K and V projection out of ``acc`` and store both to the cache.
The packed K|V weight puts K in the accumulator's first ``HD`` columns and V in
the second, which lines the warpgroup split up with the two paths: the first
``WARPGROUPS_PER_HEAD`` warpgroups take K (bias, RMSNorm, RoPE), the rest take V (bias only).
@@ -1103,15 +1179,15 @@ def _project_head_kv(
``kf_row`` decides that swap at *runtime*, for the merged fill that walks both streams'
rows in one loop. Given, both temporal ropes are traced and one is selected; left ``None``
only the lattice one is, which is what a plain build gets.
+ ``acc`` holds the head's projection, which the caller already ran. ``head`` may be a
+ runtime value: it reaches only bias offsets and cache rows.
"""
lane, wg = ws.ids.lane, ws.ids.wg
areg, params, param = ws.areg, ws.params, shape.param
n_heads, rope_dims = shape.n_heads, shape.rope_dims
- acc = ws.acc_scratch
dst = ws.kv_cache if dst is None else dst
row_stride = ws.cache_row_stride if row_stride is None else row_stride
- _project(ws, shape, ws.w_kv, acc, head)
k_sq = ACC(0.0)
for g in cutlass.range_constexpr(NUM_WARPGROUPS):
if wg == g:
@@ -1162,9 +1238,11 @@ class _TileGeometry(NamedTuple):
why these are runtime selections rather than one branch's two exits.
"""
- kf_tile: Any # this tile is a keyframe plane's; None in a binary without that stream
+ kf_tile: Any # this tile is a keyframe plane group's; None in a binary without that stream
video_tile: Any # its complement, which the deep temporal panel keys off
- plane: Any # the anchor plane a keyframe tile carries, clamped into range
+ plane: Any # this lane's anchor plane (``kf_group * TT + slot_t``), clamped into range
+ kf_group: Any # the plane group a keyframe tile carries (planes ``kf_group*TT ..``)
+ kf_live: Any # per lane: its plane exists and this slab owns it (0/1 Int32)
query_pos: tuple # this lane's (t, h, w) in the volume
read_row: Any # the row ``post`` is gathered from
write_row: Any # where the result lands; a scratch row for an out-of-range lane
@@ -1202,7 +1280,7 @@ def _tile_geometry(
_, vol_h, vol_w = ws.volume
index = tile_id
- kf_tile, video_tile, plane = None, None, None
+ kf_tile, video_tile, plane, kf_group, kf_live = None, None, None, None, None
if cutlass.const_expr(KF):
kf_tile = tile_id >= n_core_tiles
video_tile = tile_id < n_core_tiles
@@ -1245,17 +1323,23 @@ def _tile_geometry(
# clamped because a video tile evaluates this arithmetic too -- it is pure arithmetic
# plus small table lookups, and selecting the results is cheaper than branching
# around them.
- plane = at_most(at_least(local[0], zero), at_least(ws.n_kf - 1, zero))
+ # ``TT`` planes per keyframe tile: slot ``slot_t`` of group ``local[0]`` is plane ``group * TT + slot_t``;
+ # :func:`_run_tile` walks the planes one pass set at a time, each lane gated into its own plane's passes.
+ kf_group = local[0]
slot_t, slot_h, slot_w = ws.query_slot
+ plane_raw = kf_group * shape.tile[0] + slot_t
+ plane = at_most(at_least(plane_raw, zero), at_least(ws.n_kf - 1, zero))
+ kf_live = cutlass.Int32(0)
+ if plane_raw < ws.n_kf: # noqa: SIM102
+ if ws.kf_slab[plane] == slab.t_index:
+ kf_live = cutlass.Int32(1)
q_h = tile_origin[1] + slot_h
q_w = tile_origin[2] + slot_w
r_h = q_h if q_h < vol_h else vol_h - 1
r_w = q_w if q_w < vol_w else vol_w - 1
kf_read = flatten_3d(plane, r_h, r_w, vol_h, vol_w)
kf_write = kf_read
- # One plane per tile, so the borrowed tile's other temporal slots hold duplicates
- # of ``slot_t == 0``; they compute and park on a scratch row.
- if slot_t != 0 or q_h >= vol_h or q_w >= vol_w:
+ if kf_live == 0 or q_h >= vol_h or q_w >= vol_w:
kf_write = ws.n_kf * vol_h * vol_w + lane
# The *panel* still shifts inward -- it only has to contain every key the tile can
# reach, and the inward-shifted box is the larger of the two -- but the per-query
@@ -1299,6 +1383,8 @@ def _tile_geometry(
kf_tile=kf_tile,
video_tile=video_tile,
plane=plane,
+ kf_group=kf_group,
+ kf_live=kf_live,
query_pos=query_pos,
read_row=read_row,
write_row=write_row,
@@ -1385,7 +1471,7 @@ def _run_tile(
if deep:
attention_tile(
(ws.fr_q, ws.fr_k, ws.fr_p, ws.fr_v),
- (ws.s_k, ws.s_v, ws.s_p),
+ (ws.s_k, ws.s_v, ws.s_p_own),
ws.acc_qk,
ws.acc_o,
(ws.qkbar, ws.pvbar),
@@ -1406,6 +1492,8 @@ def _run_tile(
shape.n_kv_tiles,
None,
geom.window_hi,
+ ws.kv_rowtab,
+ ws.kv_copy_bars,
)
# Single-plane passes, into the same (m_sm, l_sm, acc_o). The fixed softmax offset means
@@ -1416,80 +1504,97 @@ def _run_tile(
# keyframe query's own plane, and its <=2 nearest video frames. Which cache a pass reads
# is an address, and the pass count is runtime.
if cutlass.const_expr(KF):
- pass_i = zero
- n_passes = ws.kf_counts[geom.t_idx]
+ # Each plane's key sets (its own plane, then its <=2 nearest video frames) are one pass set, walked
+ # plane by plane with the other lanes gated out; a gated lane's exponentials are exact zeros, so its
+ # softmax is untouched.
+ n_outer = cutlass.Int32(1)
if geom.kf_tile:
- n_passes = 1 + ws.kf_video_counts[geom.plane]
- while pass_i < n_passes:
- # Video tail: this tile's plane union, with each lane gated out of the planes its
- # own ``t`` did not pick -- the plane is a per-tile gather but the nearest-S set
- # is per ``t``, and a tile spans ``TT`` of them.
- source = ws.kf_planes[geom.t_idx * P_MAX + at_most(pass_i, P_MAX - 1)]
- sel = zero
- for s in cutlass.range_constexpr(SLOTS):
- if ws.kf_slots[geom.query_pos[0] * SLOTS + s] == source:
- sel = cutlass.Int32(1)
- video_kv = zero
+ n_outer = cutlass.Int32(shape.tile[0])
+ outer = zero
+ while outer < n_outer:
+ plane_j = zero
+ n_passes = ws.kf_counts[geom.t_idx]
if geom.kf_tile:
- # A keyframe query's passes are tile-uniform -- one plane per tile, and its
- # video frames are a property of the plane -- so no lane gating.
- sel = cutlass.Int32(1)
- if pass_i == 0:
- source = geom.plane
- else:
- source = ws.kf_video[geom.plane * SLOTS + at_most(pass_i - 1, SLOTS - 1)]
- video_kv = cutlass.Int32(1)
- cache = select_tensor(video_kv != 0, ws.kv_cache, ws.kv_cache_kf)
- row_stride = ws.cache_row_stride if video_kv != 0 else ws.kf_cache_row_stride
- # The keyframe cache is (n_kf, box_h, box_w): a plane index is absolute, so its
- # box origin is 0, while a video frame index is relative to the box.
- attention_tile(
- (ws.fr_q, ws.fr_k, ws.fr_p, ws.fr_v),
- (ws.s_k, ws.s_v, ws.s_p),
- ws.acc_qk,
- ws.acc_o,
- (ws.qkbar, ws.pvbar),
- ws.m_sm,
- ws.l_sm,
- ws.areg,
- ws.kvfrag,
- (cache, cache),
- (zero, shape.n_heads * row_stride),
- row_stride,
- (
- source,
- geom.panel_h0,
- geom.panel_w0,
- slab.box_origin[0] if video_kv != 0 else zero,
- slab.box_origin[1],
- slab.box_origin[2],
- slab.box_shape[1],
- slab.box_shape[2],
- ),
- (zero, geom.window[1], geom.window[2]),
- (ids.tidx, lane, ids.wg, ids.warp),
- shape.n_heads,
- 1,
- 1,
- shape.panel[1],
- shape.panel[2],
- 1,
- shape.kernel[1],
- shape.kernel[2],
- NKV_KF,
- sel,
- # One plane deep; the spatial pair is the query's own window, keeping both
- # key sets aligned for the query sharing a softmax over them.
- (cutlass.Int32(1), geom.window_hi[1], geom.window_hi[2]),
- )
- pass_i += 1
+ plane_raw_j = geom.kf_group * shape.tile[0] + outer
+ plane_j = at_most(plane_raw_j, at_least(ws.n_kf - 1, zero))
+ n_passes = 1 + ws.kf_video_counts[plane_j]
+ # A plane past ``n_kf`` or owned by another slab is not ours: no passes, its lanes park on scratch.
+ if plane_raw_j >= ws.n_kf: # noqa: SIM114
+ n_passes = zero
+ elif ws.kf_slab[plane_j] != slab.t_index:
+ n_passes = zero
+ pass_i = zero
+ while pass_i < n_passes:
+ # Video tail: this tile's plane union, with each lane gated out of the planes its
+ # own ``t`` did not pick -- the plane is a per-tile gather but the nearest-S set
+ # is per ``t``, and a tile spans ``TT`` of them.
+ source = ws.kf_planes[geom.t_idx * P_MAX + at_most(pass_i, P_MAX - 1)]
+ sel = zero
+ for s in cutlass.range_constexpr(SLOTS):
+ if ws.kf_slots[geom.query_pos[0] * SLOTS + s] == source:
+ sel = cutlass.Int32(1)
+ video_kv = zero
+ if geom.kf_tile:
+ # Keyframe tile: this pass set belongs to plane ``plane_j`` (the lanes of slot ``outer``).
+ sel = cutlass.Int32(1) if ws.query_slot[0] == outer else zero
+ if pass_i == 0:
+ source = plane_j
+ else:
+ source = ws.kf_video[plane_j * SLOTS + at_most(pass_i - 1, SLOTS - 1)]
+ video_kv = cutlass.Int32(1)
+ cache = select_tensor(video_kv != 0, ws.kv_cache, ws.kv_cache_kf)
+ row_stride = ws.cache_row_stride if video_kv != 0 else ws.kf_cache_row_stride
+ # The keyframe cache is (n_kf, box_h, box_w): a plane index is absolute, so its
+ # box origin is 0, while a video frame index is relative to the box.
+ attention_tile(
+ (ws.fr_q, ws.fr_k, ws.fr_p, ws.fr_v),
+ (ws.s_k, ws.s_v, ws.s_p_own),
+ ws.acc_qk,
+ ws.acc_o,
+ (ws.qkbar, ws.pvbar),
+ ws.m_sm,
+ ws.l_sm,
+ ws.areg,
+ ws.kvfrag,
+ (cache, cache),
+ (zero, shape.n_heads * row_stride),
+ row_stride,
+ (
+ source,
+ geom.panel_h0,
+ geom.panel_w0,
+ slab.box_origin[0] if video_kv != 0 else zero,
+ slab.box_origin[1],
+ slab.box_origin[2],
+ slab.box_shape[1],
+ slab.box_shape[2],
+ ),
+ (zero, geom.window[1], geom.window[2]),
+ (ids.tidx, lane, ids.wg, ids.warp),
+ shape.n_heads,
+ 1,
+ 1,
+ shape.panel[1],
+ shape.panel[2],
+ 1,
+ shape.kernel[1],
+ shape.kernel[2],
+ NKV_KF,
+ sel,
+ # One plane deep; the spatial pair is the query's own window, keeping both
+ # key sets aligned for the query sharing a softmax over them.
+ (cutlass.Int32(1), geom.window_hi[1], geom.window_hi[2]),
+ ws.kv_rowtab,
+ ws.kv_copy_bars,
+ )
+ pass_i += 1
+ outer += 1
# ---------------- out projection, residual, AdaLN --------------------
cute.arch.sync_threads()
- attn_epilogue(ws.acc_o, ws.l_sm, ws.oreg, ws.s_a, lane, ids.wg, shape.n_heads)
+ attn_epilogue(ws.acc_o, ws.l_sm, ws.oreg, ws.pool, ws.a_layout, lane, ids.wg, shape.n_heads)
cute.arch.sync_threads()
- for n in cutlass.range_constexpr(shape.n_chunks):
- _project(ws, shape, ws.w_o, ws.acc_main[n], n)
+ _run_jobs(ws, tuple(_job(ws, shape, ws.w_o, ws.acc_main[n], n) for n in range(shape.n_chunks)))
# s_a is free once the out-projection has consumed it, so ``post`` comes back
# through it coalesced rather than one cache line per lane.
cute.arch.sync_threads()
@@ -1537,10 +1642,20 @@ def _project_queries(ws, shape: cutlass.Constexpr, rope_pos, kf_time=None, kf_ti
"""
lane, wg = ws.ids.lane, ws.ids.wg
areg, params, param = ws.areg, ws.params, shape.param
- acc = ws.acc_scratch
- for hp in cutlass.range_constexpr(shape.n_heads // HEADS_PER_TILE):
- _project(ws, shape, ws.w_q, acc, hp)
+ # Every head pair's GEMM goes through one pipeline and one wait, into the two
+ # accumulators attention has not claimed yet: ``acc_main[0]`` and ``acc_scratch``,
+ # ``TILE_N`` columns apart.
+ n_pairs = shape.n_heads // HEADS_PER_TILE
+ assert n_pairs <= 2, "the query projection has two free accumulators"
+ _run_jobs(ws, tuple(_job(ws, shape, ws.w_q, _tmem_shift(ws.acc_main[0], hp * TILE_N), hp) for hp in range(n_pairs)))
+ # A device loop over the pairs, and each warpgroup handles its own head of the pair: one
+ # code copy of the passes below. The runtime head reaches only SMEM data indices, the
+ # TMEM column and the query stage, which is a view shifted ``head`` stages.
+ assert WARPGROUPS_PER_HEAD == 1, "each warpgroup owns one whole head of a pair"
+ for hp in cutlass.range(n_pairs):
+ head = hp * HEADS_PER_TILE + wg
+ _tmem_load(_tmem_shift(ws.acc_main[0], hp * TILE_N), lane, areg, wg)
# Pass one: bias, and the two row sums the normalisation needs. ``sum z^2``
# gives the RMSNorm scale; ``sum (z * q_norm_w)^2`` gives ``||q||`` after it,
# since the scale factors out. Both come off one pass and one barrier -- as two
@@ -1548,54 +1663,40 @@ def _project_queries(ws, shape: cutlass.Constexpr, rope_pos, kf_time=None, kf_ti
# reduction and spilled.
q_sq = ACC(0.0)
q_weighted_sq = ACC(0.0)
- for g in cutlass.range_constexpr(NUM_WARPGROUPS):
- if wg == g:
- c0 = (g % WARPGROUPS_PER_HEAD) * ACC_COLS
- head = hp * HEADS_PER_TILE + g // WARPGROUPS_PER_HEAD
- _tmem_load(acc, lane, areg, g)
- s = ACC(0.0)
- t = ACC(0.0)
- for d in cutlass.range(ACC_COLS, unroll_full=True):
- z = areg[d] + params[param["q_bias"] + head * HD + c0 + d]
- areg[d] = z
- s += z * z
- zw = z * params[param["q_norm"] + c0 + d]
- t += zw * zw
- q_sq = s
- q_weighted_sq = t
+ for d in cutlass.range(ACC_COLS, unroll_full=True):
+ z = areg[d] + params[param["q_bias"] + head * HD + d]
+ areg[d] = z
+ q_sq += z * z
+ zw = z * params[param["q_norm"] + d]
+ q_weighted_sq += zw * zw
q_sum, q_norm_sq = head_sum2(ws.red_head, lane, wg, q_sq, q_weighted_sq)
q_rms = cute.rsqrt(q_sum / ACC(HD) + ACC(EPS))
- for g in cutlass.range_constexpr(NUM_WARPGROUPS):
- if wg == g:
- c0 = (g % WARPGROUPS_PER_HEAD) * ACC_COLS
- head = hp * HEADS_PER_TILE + g // WARPGROUPS_PER_HEAD
- # This row and head's fixed softmax offset, ``||q||`` times the ``||k||``
- # bound -- see :mod:`ltx_kernels.vae.softmax_bound`. ``||q||`` is the
- # reduced sum scaled after the fact, since the RMSNorm factor is uniform
- # over the head and so comes out of the square root. The ``q_norm`` run
- # already carries the attention scale and the base change, folded in by
- # :func:`_pack_channel_params`.
- if cutlass.const_expr(g % WARPGROUPS_PER_HEAD == 0):
- ws.m_sm[lane, head] = cute.sqrt(q_norm_sq) * q_rms * params[param["k_bound"]] * LOG2E
- for d in cutlass.range(ACC_COLS, unroll_full=True):
- areg[d] = areg[d] * q_rms * params[param["q_norm"] + c0 + d]
- if cutlass.const_expr(kf_time is None):
- apply_rope(areg, *shape.rope_dims, ws.rope, *rope_pos, c0, ACC_COLS)
- elif kf_tile:
- apply_rope_float_t(
- areg,
- *shape.rope_dims,
- ws.rope,
- kf_time,
- rope_pos[1],
- rope_pos[2],
- ws.rope_inv[0],
- c0,
- ACC_COLS,
- )
- else:
- apply_rope(areg, *shape.rope_dims, ws.rope, *rope_pos, c0, ACC_COLS)
- _fill_a_chunk(ws.s_q, lane, areg, head * HD + c0, ACC_COLS)
+ # This row and head's fixed softmax offset, ``||q||`` times the ``||k||``
+ # bound -- see :mod:`ltx_kernels.vae.softmax_bound`. ``||q||`` is the
+ # reduced sum scaled after the fact, since the RMSNorm factor is uniform
+ # over the head and so comes out of the square root. The ``q_norm`` run
+ # already carries the attention scale and the base change, folded in by
+ # :func:`_pack_channel_params`.
+ ws.m_sm[lane, head] = cute.sqrt(q_norm_sq) * q_rms * params[param["k_bound"]] * LOG2E
+ for d in cutlass.range(ACC_COLS, unroll_full=True):
+ areg[d] = areg[d] * q_rms * params[param["q_norm"] + d]
+ if cutlass.const_expr(kf_time is None):
+ apply_rope(areg, *shape.rope_dims, ws.rope, *rope_pos, 0, ACC_COLS)
+ elif kf_tile:
+ apply_rope_float_t(
+ areg,
+ *shape.rope_dims,
+ ws.rope,
+ kf_time,
+ rope_pos[1],
+ rope_pos[2],
+ ws.rope_inv[0],
+ 0,
+ ACC_COLS,
+ )
+ else:
+ apply_rope(areg, *shape.rope_dims, ws.rope, *rope_pos, 0, ACC_COLS)
+ _fill_a_chunk(_alias(ws.q_pool, ws.q_layout, head * (M * TILE_K)), lane, areg, 0, ACC_COLS)
@cute.jit
@@ -1606,57 +1707,63 @@ def _swiglu_mlp(ws, shape: cutlass.Constexpr):
``gemm_tma`` shape serves every C-wide linear. N-tiling reuses that path and lets
SiLU read gate/up from TMEM; token-chunking would fight lane ownership and force a
second tiling scheme just for the MLP.
- ``s_q`` is reused as the hidden-chunk staging buffer -- it is dead from the moment
- the query projection's last GEMM consumed it.
+ The weights stream through ``s_b4``, ``MLP_B_STAGES`` tiles deep over ``s_q`` (dead once
+ attention has read the queries); the hidden chunk is staged in ``s_h``, the two-stage weight
+ slot the MLP no longer uses. Gate and up share one pipeline, as do the two down halves.
"""
lane, wg = ws.ids.lane, ws.ids.wg
areg, breg = ws.areg, ws.breg
- for nt in cutlass.range_constexpr(shape.hidden // TILE_N):
+ # A device loop, so the slice body is in the instruction stream once rather than
+ # ``hidden / TILE_N`` times; the down projection's accumulate flag is a runtime predicate.
+ for nt in cutlass.range(shape.hidden // TILE_N):
# gate and up land in separate accumulators, so SiLU reads both straight from
# TMEM instead of parking M gate values in registers.
- _project(ws, shape, ws.w_gate, ws.acc_main[0], nt)
- _project(ws, shape, ws.w_up, ws.acc_scratch, nt)
- for g in cutlass.range_constexpr(NUM_WARPGROUPS):
- if wg == g:
- _tmem_load(ws.acc_main[0], lane, areg, g)
- _tmem_load(ws.acc_scratch, lane, breg, g)
- for e in cutlass.range(ACC_COLS, unroll_full=True):
- gate = areg[e]
- # Two MUFU per element (ex2 and rcp.approx). The one-MUFU identity
- # ``silu(x) = x/2 (1 + tanh(x/2))`` is tempting and does not hold up:
- # ``tanh.approx.f32`` carries 2^-11 *absolute* error and the down
- # projection sums ``hidden`` of them, which cost 20 dB of PSNR
- # against torch.
- breg[e] = gate * cute.arch.rcp_approx(ACC(1.0) + cute.exp(-gate)) * breg[e]
- _fill_a_chunk(ws.s_q, lane, breg, g * ACC_COLS, ACC_COLS)
+ _run_jobs(
+ ws,
+ (_job(ws, shape, ws.w_gate, ws.acc_main[0], nt), _job(ws, shape, ws.w_up, ws.acc_scratch, nt)),
+ ws.s_b4,
+ MLP_B_STAGES,
+ )
+ _tmem_load(ws.acc_main[0], lane, areg, wg)
+ _tmem_load(ws.acc_scratch, lane, breg, wg)
+ for e in cutlass.range(ACC_COLS, unroll_full=True):
+ gate = areg[e]
+ # Two MUFU per element (ex2 and rcp.approx). The one-MUFU identity
+ # ``silu(x) = x/2 (1 + tanh(x/2))`` is tempting and does not hold up:
+ # ``tanh.approx.f32`` carries 2^-11 *absolute* error and the down
+ # projection sums ``hidden`` of them, which cost 20 dB of PSNR
+ # against torch.
+ breg[e] = gate * cute.arch.rcp_approx(ACC(1.0) + cute.exp(-gate)) * breg[e]
+ _fill_a_chunk(ws.s_h_own, lane, breg, 0, ACC_COLS)
cute.arch.sync_threads()
# Down projection: A is this hidden chunk, B is w_down's matching K-slice,
- # accumulated across chunks into one accumulator per N-tile. It reads ``s_q``,
- # so it takes ``fr_q`` rather than the ``fr_a`` every other projection uses.
+ # accumulated across chunks into one accumulator per N-tile. It reads ``s_h``,
+ # so it takes ``fr_h`` rather than the ``fr_a`` every other projection uses.
atom, tensor = ws.w_down
- for n in cutlass.range_constexpr(shape.n_chunks):
- gemm_tma(
- "n128",
- ws.storage,
- atom,
- tensor,
- ws.s_b,
- ws.fr_q,
- ws.acc_down[n],
- n,
- nt * 2,
- 2,
- 0,
- nt != 0,
- _B_STAGE_BYTES,
- )
+ _run_jobs(
+ ws,
+ tuple(
+ GemmJob(atom, tensor, ws.fr_h, ws.acc_down[n], n, nt * 2, 2, 0, nt != 0) for n in range(shape.n_chunks)
+ ),
+ ws.s_b4,
+ MLP_B_STAGES,
+ )
# --------------------------------------------------------------------------
# Per-row channel passes
+# Each walks the C-wide row in ``TILE_N`` chunks with a device loop, and each warpgroup
+# handles its own ``ACC_COLS`` slice of a chunk, so the body is one code copy. A chunk's
+# accumulator is ``acc[0]`` shifted ``n * TILE_N`` columns, which requires the
+# accumulators to be evenly spaced -- ``acc_main`` and ``acc_down`` both are.
# --------------------------------------------------------------------------
+def _own_chunk(ws, n):
+ """``s_a`` shifted so stage 0 is this warpgroup's slice of chunk ``n``, K-stage ``n * NUM_WARPGROUPS + wg``."""
+ return _alias(ws.pool, ws.a_layout, (n * NUM_WARPGROUPS + ws.ids.wg) * (M * TILE_K))
+
+
@cute.jit
def _add_residual(ws, shape: cutlass.Constexpr, acc, bias_off: cutlass.Constexpr, bias_row=0):
"""``s_a += acc + bias`` in place, returning the new row's sum of squares.
@@ -1667,20 +1774,17 @@ def _add_residual(ws, shape: cutlass.Constexpr, acc, bias_off: cutlass.Constexpr
context's per-subpixel bias is reached; it is a runtime value and folds away at 0.
"""
lane, wg = ws.ids.lane, ws.ids.wg
- s_a, areg, breg, params = ws.s_a, ws.areg, ws.breg, ws.params
+ areg, breg, params = ws.areg, ws.breg, ws.params
total = ACC(0.0)
- for g in cutlass.range_constexpr(NUM_WARPGROUPS):
- if wg == g:
- s = ACC(0.0)
- for n in cutlass.range_constexpr(shape.n_chunks):
- _tmem_load(acc[n], lane, breg, g)
- _read_a_chunk(s_a, lane, areg, n * TILE_N + g * ACC_COLS, ACC_COLS)
- for e in cutlass.range(ACC_COLS, unroll_full=True):
- v = areg[e] + breg[e] + params[bias_off + bias_row + n * TILE_N + g * ACC_COLS + e]
- s += v * v
- areg[e] = v
- _fill_a_chunk(s_a, lane, areg, n * TILE_N + g * ACC_COLS, ACC_COLS)
- total = s
+ for n in cutlass.range(shape.n_chunks):
+ s_a = _own_chunk(ws, n)
+ _tmem_load(_tmem_shift(acc[0], n * TILE_N), lane, breg, wg)
+ _read_a_chunk(s_a, lane, areg, 0, ACC_COLS)
+ for e in cutlass.range(ACC_COLS, unroll_full=True):
+ v = areg[e] + breg[e] + params[bias_off + bias_row + n * TILE_N + wg * ACC_COLS + e]
+ total += v * v
+ areg[e] = v
+ _fill_a_chunk(s_a, lane, areg, 0, ACC_COLS)
return total
@@ -1691,57 +1795,47 @@ def _apply_adaln(ws, shape: cutlass.Constexpr, rms_inv, weight_off: cutlass.Cons
``norm_w * (1 + scale)`` into the weight run.
"""
lane, wg = ws.ids.lane, ws.ids.wg
- s_a, areg, params = ws.s_a, ws.areg, ws.params
- for g in cutlass.range_constexpr(NUM_WARPGROUPS):
- if wg == g:
- for n in cutlass.range_constexpr(shape.n_chunks):
- _read_a_chunk(s_a, lane, areg, n * TILE_N + g * ACC_COLS, ACC_COLS)
- for e in cutlass.range(ACC_COLS, unroll_full=True):
- c = n * TILE_N + g * ACC_COLS + e
- areg[e] = areg[e] * rms_inv * params[weight_off + c] + params[shift_off + c]
- _fill_a_chunk(s_a, lane, areg, n * TILE_N + g * ACC_COLS, ACC_COLS)
+ areg, params = ws.areg, ws.params
+ for n in cutlass.range(shape.n_chunks):
+ s_a = _own_chunk(ws, n)
+ _read_a_chunk(s_a, lane, areg, 0, ACC_COLS)
+ for e in cutlass.range(ACC_COLS, unroll_full=True):
+ c = n * TILE_N + wg * ACC_COLS + e
+ areg[e] = areg[e] * rms_inv * params[weight_off + c] + params[shift_off + c]
+ _fill_a_chunk(s_a, lane, areg, 0, ACC_COLS)
@cute.jit
def _add_accumulator(ws, shape: cutlass.Constexpr, acc):
"""``s_a += acc`` in place -- the MLP's closing residual, no bias and no norm."""
lane, wg = ws.ids.lane, ws.ids.wg
- s_a, areg, breg = ws.s_a, ws.areg, ws.breg
- for g in cutlass.range_constexpr(NUM_WARPGROUPS):
- if wg == g:
- for n in cutlass.range_constexpr(shape.n_chunks):
- _tmem_load(acc[n], lane, areg, g)
- _read_a_chunk(s_a, lane, breg, n * TILE_N + g * ACC_COLS, ACC_COLS)
- for e in cutlass.range(ACC_COLS, unroll_full=True):
- areg[e] = breg[e] + areg[e]
- _fill_a_chunk(s_a, lane, areg, n * TILE_N + g * ACC_COLS, ACC_COLS)
+ areg, breg = ws.areg, ws.breg
+ for n in cutlass.range(shape.n_chunks):
+ s_a = _own_chunk(ws, n)
+ _tmem_load(_tmem_shift(acc[0], n * TILE_N), lane, areg, wg)
+ _read_a_chunk(s_a, lane, breg, 0, ACC_COLS)
+ for e in cutlass.range(ACC_COLS, unroll_full=True):
+ areg[e] = breg[e] + areg[e]
+ _fill_a_chunk(s_a, lane, areg, 0, ACC_COLS)
-def _project(ws, shape, weight, acc, n_idx, accumulate: bool = False, n_k_tiles=None, frag2=None, a_split=0):
+def _job(ws, shape, weight, acc, n_idx, accumulate: bool = False, n_k_tiles=None, frag2=None, a_split=0):
"""One ``M``-row GEMM of the activation in ``s_a`` against a streamed weight.
- Thin wrapper over :func:`gemm_tma` that unpacks the ``(atom, tensor)`` pair and
- fills in the operands every projection in this kernel shares. ``frag2`` continues the
- reduction in a second operand past K-tile ``a_split``, which is how the fused
- context's ``Ctx``-wide row -- too wide for one operand -- stays a single call.
+ Unpacks the ``(atom, tensor)`` pair and fills in the operands every projection in this
+ kernel shares. ``frag2`` continues the reduction in a second operand past K-tile
+ ``a_split``, which is how the fused context's ``Ctx``-wide row -- too wide for one
+ operand -- stays a single GEMM.
"""
atom, tensor = weight
- gemm_tma(
- "n128",
- ws.storage,
- atom,
- tensor,
- ws.s_b,
- ws.fr_a,
- acc,
- n_idx,
- 0,
- shape.n_k_tiles if n_k_tiles is None else n_k_tiles,
- 0,
- accumulate,
- _B_STAGE_BYTES,
- frag2,
- a_split,
- )
+ n_kt = shape.n_k_tiles if n_k_tiles is None else n_k_tiles
+ return GemmJob(atom, tensor, ws.fr_a, acc, n_idx, 0, n_kt, 0, accumulate, frag2, a_split)
+
+
+def _run_jobs(ws, jobs, s_b=None, n_stages=2):
+ """Run independent GEMMs through one :func:`gemm_tma` call: one pipeline, one wait.
+ ``s_b`` / ``n_stages`` select the weight queue; the default is the two-stage ``ws.s_b``.
+ """
+ gemm_tma("n128", ws.storage, ws.s_b if s_b is None else s_b, _B_STAGE_BYTES, jobs, n_stages)
# --------------------------------------------------------------------------
@@ -1945,6 +2039,7 @@ def _keyframe_workspace(
plan: _SlabPlan,
device: torch.device,
out: torch.Tensor | None = None,
+ keyframe_times_host: torch.Tensor | None = None,
) -> _KeyframeWorkspace:
"""Allocate the keyframe caches and precompute the two lookups, or hand back placeholders.
The keyframe cache is the video cache with the box's temporal extent replaced by ``n_kf``:
@@ -2005,12 +2100,15 @@ def _keyframe_workspace(
if int(keyframe_times.numel()) != n_kf:
raise ValueError(f"keyframe_times must have {n_kf} entries, got {int(keyframe_times.numel())}")
- slots = nearest_keyframe_slots(keyframe_times, t)
+ # The plane tables are built on the host: from the caller's host copy of the times when given,
+ # else from one read of ``keyframe_times``.
+ host_times = keyframe_times.detach().cpu() if keyframe_times_host is None else keyframe_times_host
+ slots = nearest_keyframe_slots(host_times, t)
# ``SLOTS * tile[0]``, not ``min(n_kf, ...)``: the stride into ``planes`` is a compile-time
# constant of the tile and must not move with the anchor layout. Slack columns are -1.
planes, _ = tile_plane_union(slots, tile_t=tile[0], max_planes=KEYFRAME_CONTEXT_SLOTS * tile[0])
counts = (planes >= 0).sum(dim=-1).to(torch.int32)
- video = nearest_video_frames(keyframe_times, t)
+ video = nearest_video_frames(host_times, t)
# Which slab owns each plane's query tiles: the one whose core holds the plane's own
# position. Its box is that core grown by the NA radius (>=1 on every stage) and the <=2
# nearest video frames are within a step, so the box always covers them.
@@ -2018,7 +2116,7 @@ def _keyframe_workspace(
# times the tile depth. Dividing by the tile count instead sends most planes to a slab
# index that never comes up, and their query tiles silently never run.
core_tokens = plan.core[0] * tile[0]
- positions = keyframe_times.detach().cpu().round().clamp_(0, t - 1).to(torch.int64)
+ positions = host_times.detach().to(torch.float32).round().clamp_(0, t - 1).to(torch.int64)
owner = (positions // core_tokens).clamp_(0, plan.grid[0] - 1).to(torch.int32)
kf_rows = n_kf * h * w
@@ -2028,11 +2126,11 @@ def _keyframe_workspace(
return _as_rows(x[0].detach().to(dtype=torch.bfloat16)).reshape(-1, ch)
def i32(x: torch.Tensor) -> torch.Tensor:
- return x.reshape(-1).contiguous().to(device=device, dtype=torch.int32)
+ return upload_from_host(x.reshape(-1).to(dtype=torch.int32).contiguous(), device)
x_rows = rows(x_keyframes, channels)
y = (
- torch.empty((kf_rows + M, channels), device=device, dtype=torch.bfloat16)
+ workspace_volume((kf_rows + M, channels), device=device, dtype=torch.bfloat16)
if out is None
# Against ``x_rows`` rather than ``x_keyframes``: what the kernel reads is the row
# view, and for a non-contiguous input that is a copy the caller cannot alias anyway.
@@ -2047,7 +2145,7 @@ def _keyframe_workspace(
_dyn_act(rows(stage4_keyframes, ctx_channels)),
_dyn_act(y),
_dyn_act(torch.empty((2 * num_heads * cache_row_stride, head_dim), device=device, dtype=torch.bfloat16)),
- _dyn(torch.empty(cache_row_stride + M, device=device, dtype=torch.float32)),
+ _dyn(torch.empty((cache_row_stride + M,), device=device, dtype=torch.float32)),
_dyn(keyframe_times.detach().to(device=device, dtype=torch.float32).reshape(-1)),
_dyn(i32(slots)),
_dyn(i32(planes)),
@@ -2100,6 +2198,7 @@ def run_block_fna_dsl(
x_keyframes: torch.Tensor | None = None,
stage4_keyframes: torch.Tensor | None = None,
keyframe_times: torch.Tensor | None = None,
+ keyframe_times_host: torch.Tensor | None = None,
out_keyframes: torch.Tensor | None = None,
) -> "torch.Tensor | tuple[torch.Tensor, torch.Tensor]":
"""Run one fused ``DiffusionNABlock``. ``x`` is ``(1, T, H, W, C)``.
@@ -2125,6 +2224,8 @@ def run_block_fna_dsl(
``out_keyframes`` is the same thing for the anchor stream, sized
``(n_kf*H*W + OUT_SLACK_ROWS, C)``; without it a joint launch allocates that stream's
output itself and the chain's alternation stops one stream short.
+ ``keyframe_times_host`` is a host copy of ``keyframe_times``. The anchor stream's plane
+ tables are built on the host; with the copy that needs no read of the device.
"""
if not block_fna_available(x.device.index or 0):
raise RuntimeError(UNSUPPORTED_MESSAGE)
@@ -2169,8 +2270,9 @@ def run_block_fna_dsl(
# M extra rows absorb the stores of out-of-range lanes in partial tiles -- which is
# why a caller-supplied ``out`` has to carry the same slack.
y_rows = T * H * W + OUT_SLACK_ROWS
+ # Under an active ltx_kernels.vae.workspace the volume is leased per tile.
y_buf = (
- torch.empty((y_rows, C), device=x_.device, dtype=torch.bfloat16)
+ workspace_volume((y_rows, C), device=x_.device, dtype=torch.bfloat16)
if out is None
else _check_out(out, y_rows, C, x_)
)
@@ -2181,7 +2283,7 @@ def run_block_fna_dsl(
kv_cache = torch.empty((2 * num_heads * plan.cache_row_stride, hd), device=x_.device, dtype=torch.bfloat16)
# The cache fill's RMSNorm factor per box token; M extra entries absorb
# out-of-range lanes.
- norm_cache = torch.empty(plan.cache_row_stride + M, device=x_.device, dtype=torch.float32)
+ norm_cache = torch.empty((plan.cache_row_stride + M,), device=x_.device, dtype=torch.float32)
barrier = torch.zeros(2, device=x_.device, dtype=torch.int32)
kf = _keyframe_workspace(
@@ -2199,6 +2301,7 @@ def run_block_fna_dsl(
plan,
x_.device,
out_keyframes,
+ keyframe_times_host,
)
def f32(t: torch.Tensor):
@@ -2420,6 +2523,10 @@ def _launch(
q_layout = sm100_utils.make_smem_layout_a(mma128, TILER_N128, IO, NH)
k_layout = sm100_utils.make_smem_layout_b(mma128, TILER_N128, IO, 2)
v_layout = sm100_utils.make_smem_layout_b(mma64, TILER_N64, IO, 4)
+ # The MLP's weights stream through a deeper queue over s_q, which is dead after attention,
+ # and its hidden chunk is staged in the freed two-stage weight slot instead.
+ b4_layout = sm100_utils.make_smem_layout_b(mma128, TILER_N128, IO, MLP_B_STAGES)
+ h_layout = sm100_utils.make_smem_layout_a(mma128, TILER_N128, IO, TILE_N // TILE_K)
# One TMA atom per weight; ``local_tile`` inside ``gemm_tma`` picks the N-tile and
# K-tile, so one atom serves every call site for that weight.
@@ -2529,6 +2636,8 @@ def _launch(
q_layout,
k_layout,
v_layout,
+ b4_layout,
+ h_layout,
).launch(grid=[n_ctas, 1, 1], block=[THREADS, 1, 1])
diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/fna_attn_core.py b/packages/ltx-kernels/src/ltx_kernels/vae/fna_attn_core.py
index 59e0c70..08f37c2 100644
--- a/packages/ltx-kernels/src/ltx_kernels/vae/fna_attn_core.py
+++ b/packages/ltx-kernels/src/ltx_kernels/vae/fna_attn_core.py
@@ -24,9 +24,11 @@ import torch
from cutlass.cute.nvgpu import tcgen05
from cutlass.cute.runtime import from_dlpack
-from ltx_kernels.vae.fna_gemm import _mma_issue, _tmem_load, _tmem_store
+from ltx_kernels.vae.fna_gemm import _mma_issue, _tmem_load, _tmem_shift, _tmem_store
from ltx_kernels.vae.fna_geometry import at_most, flatten_3d, unflatten_3d
from ltx_kernels.vae.fna_smem import (
+ _alias,
+ _cp_kv_rows,
_fill_a_chunk,
_load_head_row,
_publish_smem,
@@ -41,6 +43,7 @@ from ltx_kernels.vae.fna_types import (
LOG2E,
NUM_WARPGROUPS,
THREADS,
+ TILE_K,
WARPGROUPS_PER_HEAD,
M,
)
@@ -338,13 +341,20 @@ def attention_tile(
NKV: cutlass.Constexpr,
sel=None,
window_hi=None,
+ rowtab=None,
+ copy_bars=None,
):
"""Windowed NA over one query tile's halo panel, for ``NHG`` heads.
Args:
frags: ``(frQ, frK, frP, frV)`` -- the four MMA operand fragments.
- smem: ``(sK, sV, sP)`` -- K and V double-buffered, P single.
+ smem: ``(sK, sV, sP_own)`` -- K and V double-buffered, and this warpgroup's view
+ of the single P buffer: ``sP`` shifted by ``wg`` K-stages
+ (:func:`~ltx_kernels.vae.fna_smem._alias`), so the P fill is one copy for every
+ warpgroup.
acc_qk: two TMEM Q@K accumulators; the pipeline alternates between them.
- acc_o: ``NHG`` TMEM attention-output accumulators, live for the whole loop.
+ acc_o: ``NHG`` TMEM attention-output accumulators, live for the whole loop. They
+ must be ``HD`` columns apart: the head loop reaches head ``nh``'s as
+ ``acc_o[0]`` shifted by ``nh * HD``.
bars: ``(qkbar, pvbar)`` MMA completion barriers, re-initialised per tile.
m_sm: ``[lane, nh]`` fixed softmax offsets, from the caller's ``||q||`` pass.
l_sm: ``[lane, wg, nh]`` softmax denominators, accumulated per warpgroup.
@@ -372,6 +382,14 @@ def attention_tile(
``None`` is NATTEN's inward shift, ``[lt0, lt0 + kt)`` etc., and traces to the
byte-identical pre-keyframe code. The two agree only in the interior, so the
choice belongs to the caller's operator, not to this function.
+ rowtab: optional ``(2, M)`` Int32 SMEM table. Given, K/V tiles go global -> SMEM by
+ cooperative ``cp.async`` (:func:`~ltx_kernels.vae.fna_smem._cp_kv_rows`) instead of
+ through ``kvfrag``: each lane publishes its key's source row in the table, a step
+ ahead, and the copy of step ``s+2``'s K (``s+1``'s V) is issued as soon as its
+ buffer frees. ``None`` traces the register path.
+ copy_bars: with ``rowtab``, three SMEM mbarriers -- ``K full`` per buffer and
+ ``P/V full`` -- that replace the two CTA barriers per step: the threads that
+ wrote an operand fence and arrive, and only the MMA-issuing warp waits.
A keyframe tail is this same call with ``PT=1`` and ``kt=1``: ``panel_key_row`` then
reduces to ``flatten_3d(plane, ...)`` and the mask's ``rdt`` test to ``0 <= 0 < 1``, so
a spatial-only window over one keyframe plane needs no separate code. Because the
@@ -385,9 +403,10 @@ def attention_tile(
for *its own* Q@K, issued a whole step earlier, and before returning it issues
P@V(s) and Q@K(s+1) and stages step s+1's K/V row into the other SMEM buffer. So
a step's source read, its P@V and the next step's Q@K all run underneath the
- current softmax, and only the softmax is on the critical path. Both buffer
- indices are ``nh % 2``, compile-time because ``NHG`` is even, which keeps every
- SMEM slice statically aligned and every barrier phase constant.
+ current softmax, and only the softmax is on the critical path. The heads are walked
+ a pair at a time by a device loop, and both buffer indices are the head's place in
+ its pair -- compile-time, because ``NHG`` is even -- which keeps every SMEM slice
+ statically aligned.
K and V are staged by warpgroups 0 and 1 respectively -- two arms of the same
``wg`` test, which costs nothing (the arms already diverge at the store, K being
K-major and V MN-major) and lets the NA caller pass its own two tensors. A
@@ -395,7 +414,7 @@ def attention_tile(
NVVM backend, not the frontend.
"""
frQ, frK, frP, frV = frags
- sK, sV, sP = smem
+ sK, sV, sP_own = smem
qkbar, pvbar = bars
k_src, v_src = kv_src
k_base, v_base = kv_base
@@ -417,18 +436,41 @@ def attention_tile(
# Re-initialised per query tile: each barrier then sees exactly ``2 * NKV`` waits
# in the loop (plus one after it for barrier 0, which retires the pipeline's
# trailing Q@K), so the phase a given ``nh`` waits on never depends on ``kv``.
+ # Every warp must have finished polling the previous tile's barriers before warp 0
+ # resets them: a warp still waiting on a re-initialised barrier never sees the phase
+ # it expects, and the persistent grid spins forever.
+ cute.arch.sync_threads()
if warp_i == 0:
with cute.arch.elect_one():
cute.arch.mbarrier_init(qkbar.iterator, 1)
cute.arch.mbarrier_init(qkbar.iterator + 1, 1)
cute.arch.mbarrier_init(pvbar.iterator, 1)
+ if cutlass.const_expr(copy_bars is not None):
+ cute.arch.mbarrier_init(copy_bars.iterator, THREADS // 2)
+ cute.arch.mbarrier_init(copy_bars.iterator + 1, THREADS // 2)
+ cute.arch.mbarrier_init(copy_bars.iterator + 2, THREADS)
cute.arch.mbarrier_init_fence()
cute.arch.sync_threads()
key_row = panel_key_row(0, lane, panel, N_KEYS, PANEL_HW, PW)
+ if cutlass.const_expr(rowtab is not None):
+ # Prologue: K of steps 0 and 1 and V of step 0 in flight, step 0's landed.
+ rowtab[0, lane] = key_row
+ rowtab[1, lane] = panel_key_row(1, lane, panel, N_KEYS, PANEL_HW, PW)
+ cute.arch.sync_threads()
+ if wg == 0:
+ _cp_kv_rows(k_src, k_base, rowtab, 0, KVRM, sK, 0, lane, False)
+ cute.arch.cp_async_commit_group()
+ _cp_kv_rows(k_src, k_base + kv_hstride, rowtab, 0, KVRM, sK, 1, lane, False)
+ cute.arch.cp_async_commit_group()
+ cute.arch.cp_async_wait_group(1)
+ elif wg == 1:
+ _cp_kv_rows(v_src, v_base, rowtab, 0, KVRM, sV, 0, lane, True)
+ cute.arch.cp_async_commit_group()
+ cute.arch.cp_async_wait_group(0)
# Prologue: fill buffer 0 with step (0, 0), leave step (0, 1) in registers, and
# put Q@K(0, 0) in flight.
- if wg == 0:
+ elif wg == 0:
_load_head_row(k_src, k_base + key_row * KVRM, kvfrag)
_store_k_row(kvfrag, sK, lane, 0)
_load_head_row(k_src, k_base + kv_hstride + key_row * KVRM, kvfrag)
@@ -494,95 +536,159 @@ def attention_tile(
# Whatever filled the K/V source wrote every panel token exactly once, so this
# is a straight gather -- no per-query projection, norm or RoPE.
next_key_row = panel_key_row(kv + 1, lane, panel, N_KEYS, PANEL_HW, PW)
- for nh in cutlass.range_constexpr(NHG):
- # Buffer this step owns, and the one the next step will own. ``kv`` is a
- # runtime value -- the KV loop is a real device loop -- so the buffer
- # index has to come from ``nh`` alone, which is what keeps every SMEM
- # slice statically aligned. ``NHG`` is even, so ``nh % 2`` is the flat
- # step's parity.
- bf = nh % 2
- nf = 1 - bf
- # Wait phases. ``pvbar`` is waited once per step, so its phase is the step
- # parity. ``qkbar[bf]`` is waited every second step, so its phase advances
- # by ``NHG / 2`` per KV tile: a constant when that is even, one runtime
- # AND when it is not.
- qk_ph = (cutlass.Int32(kv) * (NHG // 2) + nh // 2) % 2
- # Stage the *next* step's K, start the fetch two steps out, then issue the
- # next step's Q@K -- all before touching this step's scores. That ordering
- # is the point: Q@K(s+1) enters the MMA queue one full softmax before step
- # s+1 waits on it, and the global load has a full step to land. Issuing it
- # after the softmax measures within noise of no pipeline at all.
- # sK[nf] currently holds K(s-1), whose Q@K was waited on last step, so it
- # is free; sV[nf] is not, which is why V is staged further down, behind
- # its own P@V barrier.
- if wg == 0:
- _store_k_row(kvfrag, sK, lane, nf)
- if cutlass.const_expr(nh + 2 < NHG):
- _load_head_row(k_src, k_base + (nh + 2) * kv_hstride + key_row * KVRM, kvfrag)
+ # Defined on both paths: the staged regions below capture it even where it is unused.
+ tab = kv % 2
+ if cutlass.const_expr(rowtab is not None):
+ # Read first by step (kv, NHG-2)'s copy, which several barriers follow; the
+ # slot's previous rows (tile kv-1) were last read before this tile began.
+ rowtab[1 - tab, lane] = next_key_row
+ if cutlass.const_expr(copy_bars is not None):
+ # The per-step barriers are gone, so the table needs its own.
+ cute.arch.sync_threads()
+ # Heads go a pair at a time through a device loop, so the step body is traced once
+ # per buffer parity ``j`` rather than once per head. The runtime head reaches only
+ # source rows, ``m_sm``/``l_sm`` indices, the Q operand's stage and TMEM addresses;
+ # every SMEM slice index below is ``j`` or a constant, which keeps them statically
+ # aligned.
+ n_pairs = NHG // 2
+ for hp in cutlass.range(n_pairs):
+ # The look-ahead steps wrap into the next KV tile only from the last pair: their
+ # head drops by ``NHG`` and their key rows are the next tile's.
+ last = cutlass.Int32(1) if hp == n_pairs - 1 else cutlass.Int32(0)
+ ld_row = key_row + (next_key_row - key_row) * last
+ ld_tab = tab + (1 - 2 * tab) * last
+ for j in cutlass.range_constexpr(2):
+ nh = hp * 2 + j
+ # Buffer this step owns, and the one the next step will own. ``kv`` and ``hp``
+ # are runtime values, so the buffer index has to come from ``j`` alone; with
+ # ``NHG`` even, ``j`` is the flat step's parity.
+ bf = j
+ nf = 1 - bf
+ # Wait phases. ``pvbar`` is waited once per step, so its phase is the step
+ # parity. ``qkbar[bf]`` is waited every second step, so its phase advances
+ # by ``NHG / 2`` per KV tile and by one per pair.
+ qk_ph = (cutlass.Int32(kv) * n_pairs + hp) % 2
+ # Head of the step one out (its Q operand and, on the cp.async path, its V and
+ # the table slot that V's rows sit in) and of the step two out (its K, and its
+ # V on the register path).
+ if cutlass.const_expr(j == 0):
+ next_head = nh + 1
+ next_tab = tab
else:
- _load_head_row(k_src, k_base + (nh + 2 - NHG) * kv_hstride + next_key_row * KVRM, kvfrag)
- _publish_smem()
- _mma_issue("n128", frQ, frK, acc_qk[nf], 1, (nh + 1) % NHG, nf, False, qkbar.iterator + nf)
- # Now wait for *this* step's Q@K, issued one step ago.
- cute.arch.mbarrier_wait(qkbar.iterator + bf, qk_ph)
- # P@V(s-1) has drained, which frees both sP and the V buffer this step is
- # about to stage into. The wait sits here rather than after the
- # exponentials, where it would have more slack: moving it down measures
- # worse, because the V store has to follow it and the global load that
- # follows the store then loses a whole exp loop of latency hiding.
- cute.arch.mbarrier_wait(pvbar.iterator, bf)
- if wg == 1:
- _store_v_row(kvfrag, sV, lane, 2 * nf)
- if cutlass.const_expr(nh + 2 < NHG):
- _load_head_row(v_src, v_base + (nh + 2) * kv_hstride + key_row * KVRM, kvfrag)
+ next_head = nh + 1 - NHG * last
+ next_tab = ld_tab
+ ld_head = nh + 2 - NHG * last
+ # Stage the *next* step's K, start the fetch two steps out, then issue the
+ # next step's Q@K -- all before touching this step's scores. That ordering
+ # is the point: Q@K(s+1) enters the MMA queue one full softmax before step
+ # s+1 waits on it, and the global load has a full step to land. Issuing it
+ # after the softmax measures within noise of no pipeline at all.
+ # sK[nf] currently holds K(s-1), whose Q@K was waited on last step, so it
+ # is free; sV[nf] is not, which is why V is staged further down, behind
+ # its own P@V barrier.
+ if cutlass.const_expr(rowtab is not None):
+ # K(s+1) was issued a step ago, as soon as sK[nf] freed.
+ if wg == 0:
+ cute.arch.cp_async_wait_group(0)
+ if cutlass.const_expr(copy_bars is not None):
+ cute.arch.fence_proxy("async.shared", space="cta")
+ cute.arch.mbarrier_arrive(copy_bars.iterator + nf)
+ elif wg == 0:
+ _store_k_row(kvfrag, sK, lane, nf)
+ _load_head_row(k_src, k_base + ld_head * kv_hstride + ld_row * KVRM, kvfrag)
+ if cutlass.const_expr(copy_bars is not None):
+ if warp_i == 0:
+ cute.arch.mbarrier_wait(copy_bars.iterator + nf, qk_ph)
else:
- _load_head_row(v_src, v_base + (nh + 2 - NHG) * kv_hstride + next_key_row * KVRM, kvfrag)
- # Softmax over this warpgroup's ``ACC_COLS`` score columns against a *fixed*
- # row offset (the caller put ``||q|| * max||k||`` in ``m_sm``). Nothing
- # here crosses the warpgroup boundary: no max, no max reduction, no
- # rescale of the O accumulator, no denominator reduction. That deletes two
- # CTA-wide barriers and a TMEM read-modify-write from every one of the
- # ``NKV * NHG`` steps.
- m_new = m_sm[lane, nh]
- # One TMEM read. The chunk index may be a runtime value here because a
- # TMEM address is a register either way; only SMEM slices need constexpr.
- _tmem_load(acc_qk[bf], lane, areg, wg)
- cute.arch.fence_view_async_tmem_load()
- # sP still has to be written for a fully masked group (P@V reads all ``M``
- # columns), but zeroing is far cheaper than 32 exps.
- # The group is 32 keys because **only a warp-uniform skip pays**: a warp
- # shares ``qt``, so a group whose panel rows all sit outside the query's dt
- # window is skipped by the whole warp. Finer groups skip more often on
- # paper, but ``lw0`` varies across a warp's lanes, so both arms issue and
- # only the extra tests are real -- measured 9% slower per nibble.
- sm4 = [ACC(0.0) for _ in range(4)]
- for hb in cutlass.range_constexpr(NBW):
- bw = mask[hb]
- if bw != 0:
- for e in cutlass.range_constexpr(32):
- # Masking the *score* rather than the exponential keeps the ex2
- # and the running sum unconditional, so the sum splits across
- # four accumulators too. ex2 of -1e30 underflows to +0, which
- # is the point.
- v = areg[hb * 32 + e] * LOG2E - m_new
- if (bw >> e) & 1 == 0:
- v = ACC(-1.0e30)
- p = cute.math.exp2(v, fastmath=True)
- areg[hb * 32 + e] = p
- sm4[e % 4] = sm4[e % 4] + p
+ _publish_smem()
+ _mma_issue("n128", frQ, frK, acc_qk[nf], 1, next_head, nf, False, qkbar.iterator + nf)
+ # Now wait for *this* step's Q@K, issued one step ago.
+ cute.arch.mbarrier_wait(qkbar.iterator + bf, qk_ph)
+ if cutlass.const_expr(rowtab is not None): # noqa: SIM102 -- a staged ``if`` cannot share a constexpr test
+ # Q@K(s) is done with sK[bf]: start K(s+2) into it.
+ if wg == 0:
+ _cp_kv_rows(k_src, k_base + ld_head * kv_hstride, rowtab, ld_tab, KVRM, sK, bf, lane, False)
+ cute.arch.cp_async_commit_group()
+ if cutlass.const_expr(rowtab is None):
+ # P@V(s-1) has drained, which frees both sP and the V buffer this step is
+ # about to stage into. The wait sits here rather than after the
+ # exponentials, where it would have more slack: moving it down measures
+ # worse, because the V store has to follow it and the global load that
+ # follows the store then loses a whole exp loop of latency hiding.
+ cute.arch.mbarrier_wait(pvbar.iterator, bf)
+ if wg == 1:
+ _store_v_row(kvfrag, sV, lane, 2 * nf)
+ _load_head_row(v_src, v_base + ld_head * kv_hstride + ld_row * KVRM, kvfrag)
+ # Softmax over this warpgroup's ``ACC_COLS`` score columns against a *fixed*
+ # row offset (the caller put ``||q|| * max||k||`` in ``m_sm``). Nothing
+ # here crosses the warpgroup boundary: no max, no max reduction, no
+ # rescale of the O accumulator, no denominator reduction. That deletes two
+ # CTA-wide barriers and a TMEM read-modify-write from every one of the
+ # ``NKV * NHG`` steps.
+ m_new = m_sm[lane, nh]
+ # One TMEM read. The chunk index may be a runtime value here because a
+ # TMEM address is a register either way; only SMEM slices need constexpr.
+ _tmem_load(acc_qk[bf], lane, areg, wg)
+ cute.arch.fence_view_async_tmem_load()
+ # sP still has to be written for a fully masked group (P@V reads all ``M``
+ # columns), but zeroing is far cheaper than 32 exps.
+ # The group is 32 keys because **only a warp-uniform skip pays**: a warp
+ # shares ``qt``, so a group whose panel rows all sit outside the query's dt
+ # window is skipped by the whole warp. Finer groups skip more often on
+ # paper, but ``lw0`` varies across a warp's lanes, so both arms issue and
+ # only the extra tests are real -- measured 9% slower per nibble.
+ sm4 = [ACC(0.0) for _ in range(4)]
+ for hb in cutlass.range_constexpr(NBW):
+ bw = mask[hb]
+ if bw != 0:
+ for e in cutlass.range_constexpr(32):
+ # Masking the *score* rather than the exponential keeps the ex2
+ # and the running sum unconditional, so the sum splits across
+ # four accumulators too. ex2 of -1e30 underflows to +0, which
+ # is the point.
+ v = areg[hb * 32 + e] * LOG2E - m_new
+ if (bw >> e) & 1 == 0:
+ v = ACC(-1.0e30)
+ p = cute.math.exp2(v, fastmath=True)
+ areg[hb * 32 + e] = p
+ sm4[e % 4] = sm4[e % 4] + p
+ else:
+ for e in cutlass.range_constexpr(32):
+ areg[hb * 32 + e] = ACC(0.0)
+ l_sm[lane, wg, nh] = l_sm[lane, wg, nh] + ((sm4[0] + sm4[1]) + (sm4[2] + sm4[3]))
+ if cutlass.const_expr(rowtab is not None):
+ # P@V(s-1) still reads sP and V(s-1)'s buffer; its remaining time overlapped
+ # the exponentials above. Then V(s+1) can start into the freed buffer.
+ cute.arch.mbarrier_wait(pvbar.iterator, bf)
+ if wg == 1:
+ _cp_kv_rows(
+ v_src, v_base + next_head * kv_hstride, rowtab, next_tab, KVRM, sV, 2 * nf, lane, True
+ )
+ cute.arch.cp_async_commit_group()
+ # Stage 0 of the shifted view is this warpgroup's own ``ACC_COLS`` chunk of sP.
+ _fill_a_chunk(sP_own, lane, areg, 0, ACC_COLS)
+ if cutlass.const_expr(rowtab is not None): # noqa: SIM102 -- a staged ``if`` cannot share a constexpr test
+ # V(s) was issued a step ago; V(s+1), issued above, may still be in flight.
+ if wg == 1:
+ cute.arch.cp_async_wait_group(1)
+ if cutlass.const_expr(copy_bars is not None):
+ # Every thread wrote its P row (and wg1 its V copies): fence, count in, and
+ # let only the issuing warp wait.
+ cute.arch.fence_proxy("async.shared", space="cta")
+ cute.arch.mbarrier_arrive(copy_bars.iterator + 2)
+ if warp_i == 0:
+ cute.arch.mbarrier_wait(copy_bars.iterator + 2, bf)
else:
- for e in cutlass.range_constexpr(32):
- areg[hb * 32 + e] = ACC(0.0)
- l_sm[lane, wg, nh] = l_sm[lane, wg, nh] + ((sm4[0] + sm4[1]) + (sm4[2] + sm4[3]))
- for g in cutlass.range_constexpr(NUM_WARPGROUPS):
- if wg == g:
- _fill_a_chunk(sP, lane, areg, g * ACC_COLS, ACC_COLS)
- # Publishes sP for the P@V below and the V buffer for the next step's.
- _publish_smem()
- _mma_issue("pv", frP, frV, acc_o[nh], 2, 0, 2 * bf, True, pvbar.iterator)
+ # Publishes sP for the P@V below and the V buffer for the next step's.
+ _publish_smem()
+ _mma_issue("pv", frP, frV, _tmem_shift(acc_o[0], nh * HD), 2, 0, 2 * bf, True, pvbar.iterator)
key_row = next_key_row
kv += 1
+ if cutlass.const_expr(rowtab is not None):
+ # The look-ahead copies past the last step land in buffers nothing reads; they still
+ # have to finish before the caller reuses the pool.
+ cute.arch.cp_async_wait_group(0)
# Retire the pipeline. P@V is the last MMA of the last step, so its barrier proves
# every accumulator is final -- and, being issued after it, that the trailing
# Q@K's commit has landed too, which is what makes re-initialising the barriers
@@ -591,13 +697,15 @@ def attention_tile(
@cute.jit
-def attn_epilogue(acc_o, l_sm, oreg, sO, lane, wg, NHG: cutlass.Constexpr):
- """Divide the attention accumulators by the denominator, into ``sO``.
- ``sO`` is an A-operand-shaped SMEM buffer whose channel ``nh * HD + d`` is head
- ``nh``'s output: the fused kernel hands it ``sA`` and feeds the out-projection
- straight from it, the NA kernel hands it a buffer it then scatters to global.
+def attn_epilogue(acc_o, l_sm, oreg, o_buf, o_layout, lane, wg, NHG: cutlass.Constexpr):
+ """Divide the attention accumulators by the denominator, into the operand ``sO``.
+ ``sO`` is the A-operand view ``o_layout`` over the raw storage ``o_buf``, and its
+ channel ``nh * HD + d`` is head ``nh``'s output: the fused kernel hands it ``sA`` and
+ feeds the out-projection straight from it, the NA kernel hands it a buffer it then
+ scatters to global. ``HD`` is one K-stage, so the device head loop reaches head
+ ``nh`` through a view shifted ``nh`` stages and every slice coordinate stays static.
"""
- for nh in cutlass.range_constexpr(NHG):
+ for nh in cutlass.range(NHG):
# Each warpgroup summed its own slice of every score row.
lden = l_sm[lane, 0, nh]
for g in cutlass.range_constexpr(1, NUM_WARPGROUPS):
@@ -605,12 +713,15 @@ def attn_epilogue(acc_o, l_sm, oreg, sO, lane, wg, NHG: cutlass.Constexpr):
linv = ACC(0.0)
if lden > ACC(0.0):
linv = cute.arch.rcp_approx(lden)
+ _tmem_load(_tmem_shift(acc_o[0], nh * HD), lane, oreg, wg, HEAD_COLS)
+ for d in cutlass.range_constexpr(HEAD_COLS):
+ oreg[d] = oreg[d] * linv
+ s_head = _alias(o_buf, o_layout, nh * (M * TILE_K))
+ # A ``HEAD_COLS`` chunk is half a stage, not a whole one, so the warpgroup split
+ # stays a constexpr arm here.
for g in cutlass.range_constexpr(NUM_WARPGROUPS):
if wg == g:
- _tmem_load(acc_o[nh], lane, oreg, g, HEAD_COLS)
- for d in cutlass.range_constexpr(HEAD_COLS):
- oreg[d] = oreg[d] * linv
- _fill_a_chunk(sO, lane, oreg, nh * HD + g * HEAD_COLS, HEAD_COLS)
+ _fill_a_chunk(s_head, lane, oreg, g * HEAD_COLS, HEAD_COLS)
@cute.jit
@@ -619,9 +730,7 @@ def attn_zero_accumulators(acc_o, l_sm, oreg, lane, wg, NHG: cutlass.Constexpr):
for d in cutlass.range_constexpr(HEAD_COLS):
oreg[d] = ACC(0.0)
for nh in cutlass.range_constexpr(NHG):
- for g in cutlass.range_constexpr(NUM_WARPGROUPS):
- if wg == g:
- _tmem_store(acc_o[nh], lane, oreg, g, HEAD_COLS)
+ _tmem_store(acc_o[nh], lane, oreg, wg, HEAD_COLS)
l_sm[lane, wg, nh] = ACC(0.0)
cute.arch.fence_view_async_tmem_store()
cute.arch.sync_threads()
diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/fna_gemm.py b/packages/ltx-kernels/src/ltx_kernels/vae/fna_gemm.py
index bc1bdc8..9e2210c 100644
--- a/packages/ltx-kernels/src/ltx_kernels/vae/fna_gemm.py
+++ b/packages/ltx-kernels/src/ltx_kernels/vae/fna_gemm.py
@@ -5,6 +5,8 @@ Extracted from :mod:`ltx_kernels.vae.fna_attn_core`. Constants:
# NOTE: no `from __future__ import annotations` -- cute.struct needs real types.
+from typing import Any, NamedTuple
+
import cutlass
import cutlass.cute as cute
import cutlass.pipeline as pipeline
@@ -113,6 +115,15 @@ def _tmem_store(acc, lane, regs, i=0, w=ACC_COLS):
cute.copy(st, regs, dst[None, None, i])
+def _tmem_shift(acc, cols):
+ """``acc`` moved ``cols`` TMEM columns along, with its layout.
+ How a runtime index picks one of an evenly spaced run of accumulators -- a head's
+ ``acc_o``, an N-tile's ``acc_main`` -- without unrolling the loop that indexes it: a
+ TMEM address is a register, so ``cols`` may be a runtime value.
+ """
+ return cute.make_tensor(acc.iterator + cols, acc.layout)
+
+
def _reg_shape(tiled_mma, acc, lane, n_cols, w=ACC_COLS):
"""Shape every register array must use: the TMEM copy's ``partition_D``.
An identity tensor stands in for the destination -- it carries the layout without
@@ -168,43 +179,54 @@ def _mma_issue(
tcgen05.commit(mbar)
+class GemmJob(NamedTuple):
+ """One ``M=128`` GEMM of a :func:`gemm_tma` call: B from ``(tma_atom, m_b)`` against ``fr_a``.
+ ``fr_a2`` continues the reduction in a second A operand from K-tile ``a_split`` on, for a
+ reduction wider than one operand can hold. ``kt0`` / ``n_kt`` pick the K-tiles of B.
+ ``n_idx``, ``kt0`` and ``accumulate`` may be device values (a device loop's index and the
+ expressions built from it); the other counts are trace-time ints.
+ """
+
+ tma_atom: Any
+ m_b: Any
+ fr_a: Any
+ acc: Any
+ n_idx: Any
+ kt0: Any
+ n_kt: int
+ a_st: int
+ accumulate: Any
+ fr_a2: Any = None
+ a_split: int = 0
+
+
@cute.jit
def gemm_tma(
kind: cutlass.Constexpr,
storage,
- tma_atom,
- mB,
sB,
- frA,
- acc,
- n_idx,
- kt0: cutlass.Constexpr,
- n_kt: cutlass.Constexpr,
- a_st: cutlass.Constexpr,
- accumulate: cutlass.Constexpr,
stage_bytes: cutlass.Constexpr,
- frA2=None,
- a_split: cutlass.Constexpr = 0,
+ jobs: cutlass.Constexpr,
+ num_stages: cutlass.Constexpr = 2,
):
- """One ``M=128`` GEMM with the B operand streamed in by TMA.
+ """Independent ``M=128`` GEMMs with their B operands streamed in by TMA.
A is already in SMEM (the kernel computes it); only B is fetched. The
- ``PipelineTmaUmma`` cycles ``sB``'s two stages, so one call covers every K-tile of
- a C-wide (or Hidd-wide) reduction with the next tile's copy in flight during the
- current tile's MMA.
- ``frA2`` continues the reduction in a second A operand from K-tile ``a_split`` on, for a
- reduction wider than one operand can hold. It is not a convenience: standing up the two
- pipelines and waiting out the accumulator costs more than the K-tiles themselves at
- these shapes, and two *calls* would pay that fixed cost twice. ``kt`` is compile-time,
- so the choice folds away.
+ ``PipelineTmaUmma`` cycles ``sB``'s ``num_stages`` stages over every job's K-tiles as
+ one sequence, keeping ``num_stages - 1`` copies -- the next job's first ones included --
+ in flight during the current tile's MMA, and the CTA waits for the accumulators once.
+ Standing up the two pipelines and that final wait cost more than the K-tiles
+ themselves at these shapes, which is why independent GEMMs share a call.
+ ``jobs`` is a trace-time tuple of :class:`GemmJob`.
"""
tiled_mma = cute.make_tiled_mma(_mma_op(kind))
tiler = _mma_tiler(kind)
warp_idx = cute.arch.make_warp_uniform(cute.arch.warp_idx())
if warp_idx == 0:
- cpasync.prefetch_descriptor(tma_atom)
+ for job in jobs:
+ cpasync.prefetch_descriptor(job.tma_atom)
ab_prod, ab_cons = pipeline.PipelineTmaUmma.create(
- num_stages=2,
+ num_stages=num_stages,
producer_group=pipeline.CooperativeGroup(pipeline.Agent.Thread),
consumer_group=pipeline.CooperativeGroup(pipeline.Agent.Thread),
tx_count=stage_bytes,
@@ -217,39 +239,57 @@ def gemm_tma(
barrier_storage=storage.acc_mbar_ptr.data_ptr(),
).make_participants()
- gB = cute.local_tile(mB, tiler, (0, n_idx, None), proj=(None, 1, 1))
- tCgB = tiled_mma.get_slice(0).partition_B(gB)
- tBsB, tBgB = cpasync.tma_partition(
- tma_atom,
- 0,
- cute.make_layout(1),
- cute.group_modes(sB, 0, 3),
- cute.group_modes(tCgB, 0, 3),
- )
+ parts = []
+ for job in jobs:
+ gB = cute.local_tile(job.m_b, tiler, (0, job.n_idx, None), proj=(None, 1, 1))
+ tCgB = tiled_mma.get_slice(0).partition_B(gB)
+ parts.append(
+ cpasync.tma_partition(
+ job.tma_atom,
+ 0,
+ cute.make_layout(1),
+ cute.group_modes(sB, 0, 3),
+ cute.group_modes(tCgB, 0, 3),
+ )
+ )
frB = tiled_mma.make_fragment_B(sB)
- n_kblk = cute.size(frA, mode=[2])
+ steps = [(j, kt) for j, job in enumerate(jobs) for kt in range(job.n_kt)]
- tiled_mma.set(tcgen05.Field.ACCUMULATE, accumulate)
+ tiled_mma.set(tcgen05.Field.ACCUMULATE, jobs[0].accumulate)
if warp_idx == 0:
acc_empty = acc_prod.acquire_and_advance()
- # One-deep prefetch: K-tile k+1's copy is issued before the wait on k, so
- # sB's two stages actually overlap TMA with MMA.
- e = ab_prod.acquire_and_advance()
- cute.copy(tma_atom, tBgB[(None, kt0)], tBsB[(None, e.index)], tma_bar_ptr=e.barrier)
- for kt in cutlass.range_constexpr(n_kt):
- if kt + 1 < n_kt:
+ # ``num_stages - 1`` copies ahead: step i+look's copy is issued before the wait on
+ # step i, so sB's stages actually overlap TMA with MMA.
+ look = num_stages - 1
+ for i in cutlass.range_constexpr(min(look, len(steps))):
+ pj, pkt = steps[i]
+ pre = ab_prod.acquire_and_advance()
+ cute.copy(
+ jobs[pj].tma_atom,
+ parts[pj][1][(None, jobs[pj].kt0 + pkt)],
+ parts[pj][0][(None, pre.index)],
+ tma_bar_ptr=pre.barrier,
+ )
+ for i in cutlass.range_constexpr(len(steps)):
+ if cutlass.const_expr(i + look < len(steps)):
+ nj, nkt = steps[i + look]
nxt = ab_prod.acquire_and_advance()
cute.copy(
- tma_atom,
- tBgB[(None, kt0 + kt + 1)],
- tBsB[(None, nxt.index)],
+ jobs[nj].tma_atom,
+ parts[nj][1][(None, jobs[nj].kt0 + nkt)],
+ parts[nj][0][(None, nxt.index)],
tma_bar_ptr=nxt.barrier,
)
+ j, kt = steps[i]
+ job = jobs[j]
+ if cutlass.const_expr(j > 0 and kt == 0):
+ tiled_mma.set(tcgen05.Field.ACCUMULATE, job.accumulate)
f = ab_cons.wait_and_advance()
- src = frA if cutlass.const_expr(a_split == 0 or kt < a_split) else frA2
- a_kt = a_st + kt if cutlass.const_expr(a_split == 0 or kt < a_split) else kt - a_split
- for kb in cutlass.range_constexpr(n_kblk):
- cute.gemm(tiled_mma, acc, src[None, None, kb, a_kt], frB[None, None, kb, f.index], acc)
+ first_operand = job.a_split == 0 or kt < job.a_split
+ src = job.fr_a if cutlass.const_expr(first_operand) else job.fr_a2
+ a_kt = job.a_st + kt if cutlass.const_expr(first_operand) else kt - job.a_split
+ for kb in cutlass.range_constexpr(cute.size(job.fr_a, mode=[2])):
+ cute.gemm(tiled_mma, job.acc, src[None, None, kb, a_kt], frB[None, None, kb, f.index], job.acc)
tiled_mma.set(tcgen05.Field.ACCUMULATE, True)
f.release()
acc_empty.commit()
diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/fna_smem.py b/packages/ltx-kernels/src/ltx_kernels/vae/fna_smem.py
index 0ccbc2f..0285e38 100644
--- a/packages/ltx-kernels/src/ltx_kernels/vae/fna_smem.py
+++ b/packages/ltx-kernels/src/ltx_kernels/vae/fna_smem.py
@@ -7,6 +7,7 @@ TMA/UMMA helpers can stay focused. Constants: :mod:`ltx_kernels.vae.fna_types`.
import cutlass
import cutlass.cute as cute
+from cutlass.cute.nvgpu import cpasync
from ltx_kernels.vae.fna_types import ACC, HD, IO, VEC, VEC_ROWS, M
@@ -27,8 +28,11 @@ def _fill_a_chunk(sA, row, vals, c0, n):
"""Write ``vals`` into channels ``[c0, c0+n)`` of A-operand row ``row``.
``c0`` and ``n`` must be compile-time and multiples of ``_KRUN``: a dynamic
coordinate loses the slice's static alignment and ``autovec_copy`` falls back to
- scalar stores. That is why callers spell the warpgroup split as a constexpr loop
- over ``g`` guarded by ``wg == g`` rather than as an arithmetic offset.
+ scalar stores. A runtime offset that is a whole K-stage -- a warpgroup's chunk, a
+ head, a loop-carried chunk index -- goes into the view instead (:func:`_alias` with
+ a runtime ``off``), so ``c0`` stays static. A sub-stage split, such as a
+ ``HEAD_COLS`` half of a head, still needs a constexpr loop over ``g`` guarded by
+ ``wg == g``.
"""
frag = cute.make_rmem_tensor(cute.make_layout(_KRUN), IO)
for k in range(n // _KRUN):
@@ -172,6 +176,39 @@ def _store_v_row(frag, sV, key, st_off):
cute.autovec_copy(frag, sV[((None, key % 64 % 16), 0, key % 64 // 16, st_off + key // 64)])
+# Cooperative cp.async staging of a KV tile: four lanes per key row, one ``_KRUN`` (32 B) each,
+# so a warp moves eight whole 128 B rows per instruction pair and no lane holds a row in
+# registers. A run goes as its two 16 B halves -- cp.async's widest -- split with
+# ``zipped_divide``, the form that stays correct on the swizzled operand.
+_CP_LANES_PER_ROW = HD // _KRUN
+_CP_ROWS_PER_PASS = M // _CP_LANES_PER_ROW
+_CP_HALF = _KRUN // 2
+
+
+def _cp_kv_rows(src, base, rowtab, tab, kvrm, dst, st, lane, v_major: bool):
+ """``cp.async`` one head's ``M`` key rows of a KV tile into a K or V operand stage.
+ ``rowtab[tab, key]`` is the source row of key ``key`` (see ``panel_key_row``), so a lane
+ copies rows other lanes own. ``v_major`` picks the MN-major V layout over K-major K.
+ The copies join the thread's open cp.async group; the caller commits and waits.
+ """
+ atom = cute.make_copy_atom(cpasync.CopyG2SOp(), IO, num_bits_per_copy=_CP_HALF * 16)
+ run = lane % _CP_LANES_PER_ROW
+ for p in range(M // _CP_ROWS_PER_PASS):
+ key = p * _CP_ROWS_PER_PASS + lane // _CP_LANES_PER_ROW
+ row = base + rowtab[tab, key] * kvrm
+ if v_major:
+ krow = dst[((None, key % 64 % 16), 0, key % 64 // 16, st + key // 64)]
+ drun = cute.zipped_divide(krow, cute.make_layout(_KRUN))[None, run]
+ else:
+ drun = dst[((key, None), 0, run, st)]
+ halves = cute.zipped_divide(drun, cute.make_layout(_CP_HALF))
+ for h in range(_KRUN // _CP_HALF):
+ src_half = cute.make_tensor(
+ _row_vec_ptr(src, row, run * _KRUN + h * _CP_HALF, HD), cute.make_layout(_CP_HALF)
+ )
+ cute.copy(atom, src_half, halves[None, h])
+
+
def _store_head_row(dst, row, vals, n=HD, c0: int = 0):
"""Write ``n`` accumulator values to ``dst[row, c0:c0+n]``, vectorised.
``dst`` rows are always ``HD`` wide; ``c0``/``n`` let a warpgroup that owns only
@@ -187,14 +224,18 @@ def _store_head_row(dst, row, vals, n=HD, c0: int = 0):
)
-def _alias(buf, layout, off: int = 0):
+def _alias(buf, layout, off=0):
"""A UMMA-operand view over ``buf``'s storage, ``off`` elements in.
tcgen05 wants the swizzle on the pointer rather than folded into the layout, so
the composed layout is split apart here the way ``SmemAllocator.allocate_tensor``
would have done it. Re-asserting the alignment matters: adding an offset drops a
pointer's static alignment and a swizzled operand needs its 128 bytes back.
+ ``buf`` must be raw (unswizzled) storage. ``off`` may be a runtime value, and must
+ then be a whole number of K-stages (``M * TILE_K`` elements): the swizzle only
+ permutes address bits below a stage, so the shifted view swizzles exactly like the
+ unshifted one and its stage ``s`` is the unshifted view's stage ``s + off / (M * TILE_K)``.
"""
- ptr = buf.iterator if off == 0 else (buf.iterator + off).align(128)
+ ptr = buf.iterator if isinstance(off, int) and off == 0 else (buf.iterator + off).align(128)
return cute.make_tensor(cute.recast_ptr(ptr, layout.inner, dtype=IO), layout.outer)
diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/fna_types.py b/packages/ltx-kernels/src/ltx_kernels/vae/fna_types.py
index 53b398e..72f2b75 100644
--- a/packages/ltx-kernels/src/ltx_kernels/vae/fna_types.py
+++ b/packages/ltx-kernels/src/ltx_kernels/vae/fna_types.py
@@ -28,6 +28,9 @@ TILE_K = TILER_N128[2]
assert TILE_N == M, "attention_tile indexes an N=TILE_N key tile by lane (0..M-1)"
EPS = 1e-6
HD = 64 # head_dim; fixed by the DiffVAE attention
+# A head is one K-stage of an A operand, so a runtime head index reaches its channels by
+# shifting the operand view whole stages (``fna_smem._alias``) with static slice coordinates.
+assert HD == TILE_K, "per-head operand views assume one head per K-stage"
# Warpgroups per CTA -- 128 threads, four warps, the tcgen05 scheduling unit. Both
# kernels hold all 512 TMEM columns, so they are pinned to one CTA per SM and the only
@@ -44,10 +47,15 @@ HEAD_COLS = HD // NUM_WARPGROUPS
HEADS_PER_TILE = TILE_N // HD # heads in one N=128 projection tile
# Warpgroups sharing a head. Two warpgroups gives one, so the Q/K RMSNorms and RoPE
# stay register-local; four gives two, and they then need a partial reduction over the
-# pair (``head_sum2``) and a channel-ranged RoPE. Both are written generically, so the
-# two-warpgroup build is the ``WARPGROUPS_PER_HEAD == 1`` special case of the same code.
+# pair (``head_sum2``) and a channel-ranged RoPE. ``head_sum2`` and the K/V pass are
+# written generically, so the two-warpgroup build is the ``WARPGROUPS_PER_HEAD == 1``
+# special case of the same code.
WARPGROUPS_PER_HEAD = NUM_WARPGROUPS // HEADS_PER_TILE
assert HEADS_PER_TILE * WARPGROUPS_PER_HEAD == NUM_WARPGROUPS, "NUM_WARPGROUPS must be a multiple of TILE_N // HD"
+# The other row passes -- the query pass, the residual and AdaLN passes, the P and MLP
+# hidden fills -- are one code copy for all warpgroups: each reaches its ``ACC_COLS``
+# chunk through an operand view shifted by whole K-stages, so a chunk must be one stage.
+assert ACC_COLS == TILE_K, "warpgroup-agnostic row passes need one K-stage per warpgroup chunk"
# One cooperative-copy step moves a whole ``_KRUN`` run per thread, so both ends of
# the copy vectorise: 32B from global (2x LDG.E.128) and one swizzled-slice store.
VEC = 16 # bf16 elements per thread per cooperative-copy step
diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/na_attn_dsl.py b/packages/ltx-kernels/src/ltx_kernels/vae/na_attn_dsl.py
index 6533fd2..fa05a3c 100644
--- a/packages/ltx-kernels/src/ltx_kernels/vae/na_attn_dsl.py
+++ b/packages/ltx-kernels/src/ltx_kernels/vae/na_attn_dsl.py
@@ -26,6 +26,7 @@ import cutlass.utils.blackwell_helpers as sm100_utils
import torch
from cutlass.utils import SmemAllocator
+from ltx_kernels.devices import upload_from_host
from ltx_kernels.vae.aot import KERNEL_CACHE, fake_dynamic_matrix, fake_dynamic_vector
from ltx_kernels.vae.availability import UNSUPPORTED_MESSAGE, gpu_supports_dsl_kernels
from ltx_kernels.vae.fna_attn_core import (
@@ -179,6 +180,8 @@ def _na_kernel(
pool = smem.allocate_tensor(IO, cute.make_layout((pool_elems,)), byte_alignment=128)
sO = _alias(pool, o_layout)
sP = _alias(pool, p_layout)
+ # This warpgroup's chunk of ``sP`` at stage 0, for ``attention_tile``'s P fill.
+ sP_own = _alias(pool, p_layout, wg * (M * TILE_K))
sK = _alias(pool, k_layout, 2 * M * TILE_K)
sV = _alias(pool, v_layout, 4 * M * TILE_K)
sQ = smem.allocate_tensor(IO, q_layout.outer, byte_alignment=128, swizzle=q_layout.inner)
@@ -323,7 +326,7 @@ def _na_kernel(
if deep:
attention_tile(
(frQ, frK, frP, frV),
- (sK, sV, sP),
+ (sK, sV, sP_own),
acc_qk,
acc_o,
(qkbar, pvbar),
@@ -390,7 +393,7 @@ def _na_kernel(
v_src = select_tensor(video_kv != 0, v, v_kf)
attention_tile(
(frQ, frK, frP, frV),
- (sK, sV, sP),
+ (sK, sV, sP_own),
acc_qk,
acc_o,
(qkbar, pvbar),
@@ -422,7 +425,7 @@ def _na_kernel(
pass_i += 1
cute.arch.sync_threads()
- attn_epilogue(acc_o, l_sm, oreg, sO, lane, wg, HG)
+ attn_epilogue(acc_o, l_sm, oreg, pool, o_layout, lane, wg, HG)
cute.arch.sync_threads()
_scatter_a(sO, y_dst, rowix_w, copy_slot, HG * HD, c_off)
cute.arch.sync_threads()
@@ -743,9 +746,9 @@ def _keyframe_operands(
enabled=True,
k=_dyn_act(rows(k_keyframes)),
v=_dyn_act(rows(v_keyframes)),
- slots=_dyn(slots_cpu.reshape(-1).to(device=device)),
- planes=_dyn(planes.reshape(-1).to(device=device)),
- counts=_dyn(counts.to(device=device)),
+ slots=_dyn(upload_from_host(slots_cpu.reshape(-1).contiguous(), device)),
+ planes=_dyn(upload_from_host(planes.reshape(-1).contiguous(), device)),
+ counts=_dyn(upload_from_host(counts, device)),
)
@@ -844,8 +847,8 @@ def run_na_attention_bound(
y_kf_buf = torch.empty((n_kf * H * W + M, NH * hd), device=q_.device, dtype=torch.bfloat16)
y_kf_dyn = _dyn_act(y_kf_buf)
q_kf_ = _dyn_act(q_kf_rows)
- kfv = _dyn(video_frames.reshape(-1).to(device=q_.device))
- kfv_counts = _dyn((video_frames >= 0).sum(dim=-1).to(dtype=torch.int32, device=q_.device))
+ kfv = _dyn(upload_from_host(video_frames.reshape(-1).contiguous(), q_.device))
+ kfv_counts = _dyn(upload_from_host((video_frames >= 0).sum(dim=-1).to(dtype=torch.int32), q_.device))
n_kfq_tiles = n_kf * grid_h * grid_w
else:
_, _, q_kf_, y_kf_dyn = _inert_operands(hd, NH * hd, q_.device)
diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/rope_dsl.py b/packages/ltx-kernels/src/ltx_kernels/vae/rope_dsl.py
new file mode 100644
index 0000000..f9ce454
--- /dev/null
+++ b/packages/ltx-kernels/src/ltx_kernels/vae/rope_dsl.py
@@ -0,0 +1,195 @@
+"""Absolute 3D RoPE over a whole ``(B, T, H, W, NH, HD)`` volume in one CuTe DSL launch.
+The rotation is :func:`ltx_kernels.vae.fna_attn_core.apply_rope`, the same code the fused
+DiffVAE block runs on its Q/K rows; here thread ``i`` of a block owns head row ``i``, rows
+are staged through SMEM so global traffic stays coalesced, and the cos/sin come from
+global tables. No tensor-core instructions are
+involved, so it builds for any CUDA architecture the DSL targets.
+The caller builds the tables, so they can hold exactly the angles a reference
+implementation uses -- integer or fractional positions alike.
+"""
+
+# NOTE: no `from __future__ import annotations` -- cute.jit needs real annotations.
+
+import cutlass
+import cutlass.cute as cute
+import torch
+from cutlass.cute.runtime import from_dlpack
+from cutlass.utils import SmemAllocator
+
+from ltx_kernels.vae.aot import KERNEL_CACHE, fake_dynamic_matrix
+from ltx_kernels.vae.fna_attn_core import apply_rope
+from ltx_kernels.vae.fna_smem import _row_vec_ptr
+from ltx_kernels.vae.fna_types import ACC, HD, IO
+
+_THREADS = 128
+# Global traffic moves in 16B slices, consecutive lanes on consecutive slices, so a warp's
+# loads and stores cover whole 128B lines. Rows are staged through SMEM, padded by one
+# slice so the per-thread row reads of an 8-lane phase land on distinct banks.
+_VEC = 8
+_LANES_PER_ROW = HD // _VEC
+_ROW_STRIDE = HD + _VEC
+
+
+def _rope_kernel(
+ x: cute.Tensor, # (rows, HD) bf16, rows ordered (B, T, H, W, NH)
+ y: cute.Tensor, # (rows, HD) bf16
+ cos_t: cute.Tensor, # (T, d_t / 2) f32
+ sin_t: cute.Tensor,
+ cos_h: cute.Tensor, # (H, d_h / 2) f32
+ sin_h: cute.Tensor,
+ cos_w: cute.Tensor, # (W, d_w / 2) f32
+ sin_w: cute.Tensor,
+ n_rows: cutlass.Int32,
+ T: cutlass.Int32,
+ H: cutlass.Int32,
+ W: cutlass.Int32,
+ NH: cutlass.Int32,
+ d_t: cutlass.Constexpr,
+ d_h: cutlass.Constexpr,
+ d_w: cutlass.Constexpr,
+) -> None:
+ tidx, _, _ = cute.arch.thread_idx()
+ bidx, _, _ = cute.arch.block_idx()
+ row0 = bidx * _THREADS
+ smem = SmemAllocator()
+ tile = smem.allocate_tensor(IO, cute.make_layout((_THREADS * _ROW_STRIDE,)), byte_alignment=16)
+ frag = cute.make_rmem_tensor(cute.make_layout(_VEC), IO)
+
+ # Coalesced fill: consecutive lanes take consecutive 16B slices of consecutive rows.
+ for k in range(_LANES_PER_ROW):
+ r = (k * _THREADS + tidx) // _LANES_PER_ROW
+ c = (k * _THREADS + tidx) % _LANES_PER_ROW * _VEC
+ if row0 + r < n_rows:
+ cute.autovec_copy(cute.make_tensor(_row_vec_ptr(x, row0 + r, c, HD), cute.make_layout(_VEC)), frag)
+ cute.autovec_copy(frag, _smem_slice(tile, r, c))
+ cute.arch.sync_threads()
+
+ # Thread ``i`` rotates row ``i`` in registers.
+ row = row0 + tidx
+ if row < n_rows:
+ token = row // NH
+ w = token % W
+ h = (token // W) % H
+ t = (token // (W * H)) % T
+ reg = cute.make_rmem_tensor(cute.make_layout(HD), ACC)
+ for k in range(_LANES_PER_ROW):
+ cute.autovec_copy(_smem_slice(tile, tidx, k * _VEC), frag)
+ for e in range(_VEC):
+ reg[k * _VEC + e] = ACC(frag[e])
+ apply_rope(reg, d_t, d_h, d_w, (cos_t, sin_t, cos_h, sin_h, cos_w, sin_w), t, h, w)
+ for k in range(_LANES_PER_ROW):
+ for e in range(_VEC):
+ frag[e] = IO(reg[k * _VEC + e])
+ cute.autovec_copy(frag, _smem_slice(tile, tidx, k * _VEC))
+ cute.arch.sync_threads()
+
+ for k in range(_LANES_PER_ROW):
+ r = (k * _THREADS + tidx) // _LANES_PER_ROW
+ c = (k * _THREADS + tidx) % _LANES_PER_ROW * _VEC
+ if row0 + r < n_rows:
+ cute.autovec_copy(_smem_slice(tile, r, c), frag)
+ cute.autovec_copy(frag, cute.make_tensor(_row_vec_ptr(y, row0 + r, c, HD), cute.make_layout(_VEC)))
+
+
+def _smem_slice(tile, row, col):
+ """The 16B slice at ``(row, col)`` of the padded SMEM row tile."""
+ return cute.make_tensor((tile.iterator + (row * _ROW_STRIDE + col)).align(16), cute.make_layout(_VEC))
+
+
+def _launch(
+ x: cute.Tensor,
+ y: cute.Tensor,
+ cos_t: cute.Tensor,
+ sin_t: cute.Tensor,
+ cos_h: cute.Tensor,
+ sin_h: cute.Tensor,
+ cos_w: cute.Tensor,
+ sin_w: cute.Tensor,
+ n_rows: cutlass.Int32,
+ T: cutlass.Int32,
+ H: cutlass.Int32,
+ W: cutlass.Int32,
+ NH: cutlass.Int32,
+ d_t: cutlass.Constexpr,
+ d_h: cutlass.Constexpr,
+ d_w: cutlass.Constexpr,
+) -> None:
+ _rope_kernel(x, y, cos_t, sin_t, cos_h, sin_h, cos_w, sin_w, n_rows, T, H, W, NH, d_t, d_h, d_w).launch(
+ grid=[(n_rows + _THREADS - 1) // _THREADS, 1, 1], block=[_THREADS, 1, 1]
+ )
+
+
+def _compile_kernel(*, rope_dims: "tuple[int, int, int]", options: str | None = None):
+ """Compile one ``(d_t, d_h, d_w)`` split from storage-free descriptors."""
+ rows = fake_dynamic_matrix(cutlass.BFloat16, assumed_align=16)
+ tables = [fake_dynamic_matrix(cutlass.Float32, assumed_align=4) for _ in range(6)]
+ return cute.compile(
+ _launch,
+ rows,
+ fake_dynamic_matrix(cutlass.BFloat16, assumed_align=16),
+ *tables,
+ 1,
+ 1,
+ 1,
+ 1,
+ 1,
+ *rope_dims,
+ **({"options": options} if options else {}),
+ )
+
+
+def abs_rope_supported(x: torch.Tensor, rope_dims: "tuple[int, int, int]") -> bool:
+ """Whether :func:`run_abs_rope` takes ``x``: a contiguous CUDA bf16 ``(B, T, H, W, NH, 64)`` volume."""
+ d_t, d_h, d_w = rope_dims
+ return (
+ x.is_cuda
+ and x.dtype == torch.bfloat16
+ and x.dim() == 6
+ and x.shape[-1] == HD
+ and x.is_contiguous()
+ and x.numel() // HD < 2**31
+ and d_t + d_h + d_w == HD
+ and d_t % 2 == 0
+ and d_h % 2 == 0
+ )
+
+
+def run_abs_rope(
+ x: torch.Tensor,
+ rope_dims: "tuple[int, int, int]",
+ tables: "tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]",
+ *,
+ out: torch.Tensor | None = None,
+) -> torch.Tensor:
+ """Rotate every head row of ``x`` by its (t, h, w) angles, into ``out`` (a new tensor by default).
+ ``tables`` is ``(cos_t, sin_t, cos_h, sin_h, cos_w, sin_w)``, fp32 ``(span, dim / 2)``
+ tables over ``T``, ``H`` and ``W`` of ``x``. Check :func:`abs_rope_supported` first.
+ ``out`` may be ``x`` itself: each CTA reads all of its rows into shared memory before it
+ writes any of them back. Otherwise it must not overlap ``x``. It must be on ``x``'s device.
+ """
+ _, t, h, w, nh, _ = x.shape
+ rows = x.view(-1, HD)
+ if out is None:
+ y = torch.empty_like(rows)
+ else:
+ if out.shape != x.shape or out.dtype != x.dtype or out.device != x.device or not out.is_contiguous():
+ raise ValueError(
+ f"out must be a contiguous {tuple(x.shape)} {x.dtype} tensor on {x.device}, "
+ f"got {tuple(out.shape)} {out.dtype} on {out.device}"
+ )
+ y = out.view(-1, HD)
+ with torch.cuda.device(x.device):
+ compiled = KERNEL_CACHE.get(_compile_kernel, x.device, rope_dims=tuple(rope_dims))
+ compiled(
+ from_dlpack(rows, assumed_align=16).mark_layout_dynamic(),
+ from_dlpack(y, assumed_align=16).mark_layout_dynamic(),
+ *(from_dlpack(table.contiguous(), assumed_align=4).mark_layout_dynamic() for table in tables),
+ rows.shape[0],
+ t,
+ h,
+ w,
+ nh,
+ )
+ return y.view(x.shape)
diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/workspace.py b/packages/ltx-kernels/src/ltx_kernels/vae/workspace.py
new file mode 100644
index 0000000..aa2c2df
--- /dev/null
+++ b/packages/ltx-kernels/src/ltx_kernels/vae/workspace.py
@@ -0,0 +1,164 @@
+"""Reusable device buffers for one tiled DiffVAE decode.
+Per-tile allocation of the large stage-5 buffers leaves the caching allocator 25-45 GiB above what is ever live;
+:class:`DecodeWorkspace` keeps them for the whole decode as activation volumes leased per tile, uninitialised like
+``torch.empty``; without a workspace every helper is a plain one.
+"""
+
+from __future__ import annotations
+
+import contextvars
+import math
+from collections.abc import Iterator
+from contextlib import contextmanager
+from dataclasses import dataclass
+
+import torch
+
+__all__ = [
+ "DecodeWorkspace",
+ "active_decode_workspace",
+ "decode_workspace",
+ "workspace_phase_empty",
+ "workspace_phase_reset",
+ "workspace_volume",
+]
+
+# Phase tensors start on a multiple of this many elements (keeps every carved buffer aligned).
+_PHASE_ALIGN_ELEMS = 128
+
+_ACTIVE: contextvars.ContextVar[DecodeWorkspace | None] = contextvars.ContextVar(
+ "ltx_kernels_decode_workspace", default=None
+)
+
+
+@dataclass
+class _VolumeSlot:
+ buffer: torch.Tensor # flat, one dtype / device
+ leased: bool = False
+ phase_offset: int = 0 # elements handed out by phase_take since the last phase_reset
+
+ @property
+ def capacity(self) -> int:
+ return self.buffer.numel()
+
+
+class DecodeWorkspace:
+ """Per-tile leased volumes and the phase arena carved from the idle ones; see the module docstring."""
+
+ def __init__(self, large_volume_numel: int = 0) -> None:
+ """``large_volume_numel``: capacity of every large slot, the decode's biggest volume, so later tiles reuse it.
+ A request counts as large from half that size; smaller ones (keyframe planes) get exactly what they ask for.
+ """
+ self._volumes: list[_VolumeSlot] = []
+ self._large_numel = max(0, int(large_volume_numel))
+
+ # -- leased volumes ------------------------------------------------------------------
+ def _is_large(self, numel: int) -> bool:
+ return bool(self._large_numel) and numel * 2 >= self._large_numel
+
+ def volume(self, numel: int, *, dtype: torch.dtype, device: torch.device) -> torch.Tensor:
+ """Lease the smallest free buffer that holds ``numel`` elements, or allocate a new one.
+ Large requests use large-capacity slots; small ones never lease a large slot, so they cannot displace one.
+ """
+ large = self._is_large(numel)
+ best: _VolumeSlot | None = None
+ for slot in self._volumes:
+ buf = slot.buffer
+ if slot.leased or buf.dtype != dtype or buf.device != device or slot.capacity < numel:
+ continue
+ if slot.phase_offset:
+ continue
+ if self._large_numel and (slot.capacity >= self._large_numel) != large:
+ continue
+ if best is None or slot.capacity < best.capacity:
+ best = slot
+ if best is None:
+ capacity = max(numel, 1)
+ if large:
+ capacity = max(capacity, self._large_numel)
+ best = _VolumeSlot(torch.empty(capacity, dtype=dtype, device=device))
+ self._volumes.append(best)
+ best.leased = True
+ return best.buffer[:numel]
+
+ def release_tile(self) -> None:
+ """End of a tile: every leased volume is free for the next one (nothing is deallocated)."""
+ for slot in self._volumes:
+ slot.leased = False
+ self.phase_reset()
+
+ # -- phase arena: short-lived tensors carved out of the idle volume slots --------------
+ def phase_take(
+ self,
+ numel: int,
+ *,
+ dtype: torch.dtype,
+ device: torch.device,
+ not_aliasing: tuple[torch.Tensor, ...] = (),
+ ) -> torch.Tensor | None:
+ """Bump-allocate ``numel`` elements from a free (unleased) volume slot, or ``None`` if none has room.
+ For a phase whose tensors all die before the volumes are next leased; :meth:`phase_reset` frees them all.
+ ``not_aliasing``: slots backing these tensors are skipped (a custom op's output may not alias its inputs).
+ """
+ avoid = {t.untyped_storage().data_ptr() for t in not_aliasing}
+ for slot in self._volumes:
+ buf = slot.buffer
+ if slot.leased or buf.dtype != dtype or buf.device != device:
+ continue
+ if avoid and buf.untyped_storage().data_ptr() in avoid:
+ continue
+ offset = slot.phase_offset
+ if offset + numel > slot.capacity:
+ continue
+ slot.phase_offset = offset + (-(-numel // _PHASE_ALIGN_ELEMS)) * _PHASE_ALIGN_ELEMS
+ return buf[offset : offset + numel]
+ return None
+
+ def phase_reset(self) -> None:
+ for slot in self._volumes:
+ slot.phase_offset = 0
+
+
+def active_decode_workspace() -> DecodeWorkspace | None:
+ return _ACTIVE.get()
+
+
+@contextmanager
+def decode_workspace(workspace: DecodeWorkspace | None) -> Iterator[DecodeWorkspace | None]:
+ """Make ``workspace`` the one :func:`workspace_volume` / :func:`workspace_phase_empty` draw from."""
+ token = _ACTIVE.set(workspace)
+ try:
+ yield workspace
+ finally:
+ _ACTIVE.reset(token)
+
+
+def workspace_phase_empty(
+ shape: tuple[int, ...],
+ *,
+ dtype: torch.dtype,
+ device: torch.device,
+ not_aliasing: tuple[torch.Tensor, ...] = (),
+) -> torch.Tensor:
+ """``torch.empty(shape)`` carved from the active workspace's idle volumes (see
+ :meth:`DecodeWorkspace.phase_take`); a plain ``empty`` without a workspace or when nothing fits."""
+ ws = _ACTIVE.get()
+ if ws is not None:
+ flat = ws.phase_take(math.prod(shape), dtype=dtype, device=device, not_aliasing=not_aliasing)
+ if flat is not None:
+ return flat.view(shape)
+ return torch.empty(shape, dtype=dtype, device=device)
+
+
+def workspace_phase_reset() -> None:
+ ws = _ACTIVE.get()
+ if ws is not None:
+ ws.phase_reset()
+
+
+def workspace_volume(shape: tuple[int, ...], *, dtype: torch.dtype, device: torch.device) -> torch.Tensor:
+ """``torch.empty(shape)`` leased from the active workspace until its tile ends (plain ``empty`` without one)."""
+ ws = _ACTIVE.get()
+ if ws is None:
+ return torch.empty(shape, dtype=dtype, device=device)
+ return ws.volume(math.prod(shape), dtype=dtype, device=device).view(shape)
diff --git a/packages/ltx-pipelines/CLAUDE.md b/packages/ltx-pipelines/CLAUDE.md
index 18f70d5..1fcc225 100644
--- a/packages/ltx-pipelines/CLAUDE.md
+++ b/packages/ltx-pipelines/CLAUDE.md
@@ -22,16 +22,18 @@ Pipelines stay at `ltx_pipelines/<name>.py`. Distilled, TI2Vid two-stage, A2Vid
CLI: on `python -m ltx_pipelines.distilled`, `python -m ltx_pipelines.ti2vid_two_stages` (and their MGPU runners), `python -m ltx_pipelines.a2vid_two_stage`, or `python -m ltx_pipelines.dubit`, pass `--chunk-pixel-frames` (default 97) and/or `--chunk-carry-frames` (default 25) via `add_chunk_layout_args` (`utils/args.py`); `--chunk-blend-frames` controls decoded video seam crossfade length (default: full carry, `0` for a hard cut). When any layout flag is set, `chunk_config_from_args` builds a `ChunkConfig` and the CLI/MGPU driver streams `stream_chunks` through `split_decoded_chunks` into `encode_video`; otherwise it encodes `PipelineOutput` from `__call__`. With no chunk flags, `stream_chunks` / `uniform_chunk_layouts` plans one chunk covering the snapped clip.
+TI2Vid two-stage freezes stage-1 audio during stage 2 by default. Pass `refine_audio=True` (CLI: `--refine-audio`)
+to denoise both modalities in stage 2, including when replacing the former keyframe-interpolation pipeline.
+
## Pipeline selection
| Pipeline | File | Stages | Model | Sampler | Use case |
|----------|------|--------|-------|---------|----------|
| `TI2VidOneStagePipeline` | `ti2vid_one_stage.py` | 1 | Full | Euler | Simple text/image-to-video |
| `T2AOneStagePipeline` | `t2a_one_stage.py` | 1 | Full | Euler | Text-to-audio (audio-only output, no video branch) |
-| `TI2VidTwoStagesPipeline` | `ti2vid_two_stages.py` | 2 | Full + distilled LoRA | Euler | Guided two-stage (CFG/STG) |
+| `TI2VidTwoStagesPipeline` | `ti2vid_two_stages.py` | 2 | Full + distilled LoRA | Euler | Guided two-stage (CFG/STG), including keyframe interpolation |
| `TI2VidTwoStagesHQPipeline` | `ti2vid_two_stages_hq.py` | 2 | Full + distilled LoRA (both stages) | Res2s | Guided two-stage, res_2s, fewer steps |
| `A2VidPipelineTwoStage` | `a2vid_two_stage.py` | 2 | Full + distilled LoRA | Euler | Audio-conditioned video |
-| `KeyframeInterpolationPipeline` | `keyframe_interpolation.py` | 2 | Full + distilled LoRA | Euler | Keyframe interpolation |
| `DFRPipeline` | `dfr_pipeline.py` | 2 (+ optional tiled temporal) | Distilled checkpoint + detailing IC-LoRA stage 2 | Euler ancestral (2.5+) | Production quality (keyframes + detailing + optional temporal x2) |
| `DistilledPipeline` | `distilled.py` | 2 | Distilled only | Euler ancestral (2.5+) | Fastest inference |
| `ICLoraPipeline` | `ic_lora.py` | 2 | Distilled only | Euler | Video-to-video with IC-LoRA control |
@@ -57,23 +59,23 @@ CLI: on `python -m ltx_pipelines.distilled`, `python -m ltx_pipelines.ti2vid_two
## LoRA conventions
- No default LoRAs. `loras` param defaults to empty list/tuple. `DEFAULT_LORA_STRENGTH = 1.0`.
-- Two-stage non-distilled pipelines require `distilled_lora` (applied to stage 2 only in TI2Vid/A2Vid/Keyframe).
+- Two-stage non-distilled pipelines require `distilled_lora` (applied to stage 2 only in TI2Vid/A2Vid).
- HQ applies distilled LoRA to **both** stages with separate `distilled_lora_strength_stage_1` / `_stage_2` params.
- DFR loads a distilled checkpoint (no distilled LoRA). Stage 2 adds a detailing IC-LoRA via `with_loras`.
## Shared building blocks (`utils/blocks.py`)
-- `DiffusionStage` -- owns transformer lifecycle; builds model on call, frees on exit via `gpu_model()` context manager (moves params to meta device to release GPU/CPU memory). Accepts an optional `loop` override. `__init__` takes a pre-built transformer builder; pipelines construct it via the `DiffusionStage.from_checkpoint(checkpoint_path, ..., loras=...)` classmethod, which builds the standard (and, when offloading, streaming) builders. `with_builder` / `with_loras` return a new stage with a swapped builder / LoRA set without re-specifying config. ``supports_generated_keyframes`` reads ``use_keyframes_abs_pos_embedding`` from the checkpoint config (no weights loaded). ``__call__`` re-checks slot conditionings via ``_assert_supports_conditionings`` as a backstop for callers that build items directly.
+- `DiffusionStage` -- owns transformer lifecycle; builds model on call, frees on exit via `gpu_model()` context manager (moves params to meta device to release GPU/CPU memory), except that a builder whose weights the registry keeps GPU-resident (`keeps_gpu_resident_weights`) under `AllocatorTrimStrategy.DEFER` leaves its build assigned: every build reassigns those resident weights, so a dispose would free nothing. Accepts an optional `loop` override. `__init__` takes a pre-built transformer builder; pipelines construct it via the `DiffusionStage.from_checkpoint(checkpoint_path, ..., loras=...)` classmethod, which builds the standard (and, when offloading, streaming) builders. `with_builder` / `with_loras` return a new stage with a swapped builder / LoRA set without re-specifying config. ``supports_generated_keyframes`` reads ``use_keyframes_abs_pos_embedding`` from the checkpoint config (no weights loaded). ``__call__`` re-checks slot conditionings via ``_assert_supports_conditionings`` as a backstop for callers that build items directly.
`__call__` takes `modalities: VideoAudio[ModalitySpec]` and is a latent entry point: each `ModalitySpec` brings the latent to denoise, and that latent is the stage's only source of shape -- pixel dimensions never reach it, and it never sizes a latent itself. A pipeline's *first* stage seeds the latent with `create_initial_video_latent` / `create_initial_audio_latent` / `create_initial_av_latents` (`utils/helpers.py`); every later stage passes the previous stage's output, upscaled or not. So latents are allocated once per run, at the pipeline head. Sizing audio off a playback rate that differs from the video RoPE time base is done by building the audio latent at that rate, not by a stage argument. Returns `VideoAudio`, which unpacks as `(video, audio)`.
- `PromptEncoder` -- Gemma text encoder + embeddings processor (video 4096-dim, audio 2048-dim).
-- `ImageConditioner` / `AudioConditioner` -- temporary encoder scope; builds encoder, passes to callable, frees. `ImageConditioner` is additionally the single owner of the image-conditioning CRF: `resolve_crf(images)` (called near the top of every pipeline's `__call__`) fills in the H.264 CRF of any `ImageConditioningInput` (`utils.types`) that left it unset, reading it from the checkpoint's `model_version` (`detect_params`, lazily and cached). So omitting `crf` means "use what matches this model" (33 through LTX-2.3, 18 from 2.4), while an explicit `crf` -- including `0` for no re-compression -- is always honoured.
+- `ImageConditioner` / `AudioConditioner` -- temporary encoder scope; builds encoder, passes to callable, frees. `ImageConditioner` is additionally the single owner of the image-conditioning CRF: `resolve_crf(images)` fills in the H.264 CRF of any `ImageConditioningInput` (`utils.types`) that left it unset, reading it from the checkpoint's `model_version` (`detect_params`, lazily and cached). So omitting `crf` means "use what matches this model" (33 through LTX-2.3, 18 from 2.4), while an explicit `crf` -- including `0` for no re-compression -- is always honoured. `load_conditionings(images, color_space=None)` (called near the top of every pipeline's `__call__`) resolves the CRF that way and decodes each distinct still once for the run (SDR re-compressed at its CRF; EXR with the run's `color_space`). It returns `LoadedImageConditioning`s (`utils.types`): the decoded still plus the input's `binding` and `strength`, with `frame_idx` / `assert_in_clip` / `map_frame_index` / `to_item`. Everything after it -- chunk and DFR rebasing, stage/tile/window plumbing, `image_conditionings()` -- takes only `LoadedImageConditioning`s; rebasing keeps the still, so encodes only resize per resolution. `ImageConditioningInput` is the user-facing request (path, strength, binding, CRF).
- `VideoUpsampler` -- 2x spatial upsampling via encoder + upsampler.
- `VideoDecoder` / `AudioDecoder` -- latent-to-pixel decoding (iterator for video, `Audio` for audio). `VideoDecoder` takes a single decoder `checkpoint_path` and `diffvae_optimization` (`DiffVAEMode`, default `CHUNKED_EAGER`); pipelines pass `model_paths.video_vae()`, the same file `ImageConditioner` and `VideoUpsampler` build their encoder from -- a video VAE checkpoint carries encoder and decoder together. Decoder kind (conv vs diffusion) is chosen from checkpoint metadata (`is_diffusion_video_vae`); distilled DiffVAE uses fixed 2 Euler steps. CLI: `--video-vae-path` is the video VAE slot in both monolith (optional override; defaults via `ModelPaths.from_monolith`) and split modes; `--diffvae-optimization` (see `docs/optimization.md#diffusion-vae-decoder` for mode meanings and relative compile/runtime/VRAM factors). Multi-GPU pipelines (`*_mgpu.py`) accept the same flags and wrap the loaded decoder in `DistributedVideoDecoder`.
- `DurationPredictor` -- predicts a frame count from `PromptEncoder`'s connector token outputs via `DurationHead` (`ltx_core.duration_head`), snapped to the VAE's temporal grid. Unlike other blocks, it holds the built model directly (`__init__` takes a `DurationHead`, not a builder) since the checkpoint is only a few MB -- no build-on-call/free-on-exit needed. Pipelines construct it via `DurationPredictor.from_checkpoint(checkpoint_path, dtype, device)`, which returns `None` instead of a predictor if the checkpoint has no `duration_head.*` weights (checkpoints predating LTX-2.5 / gemma4). Called with `(video_encoding, audio_encoding, frame_rate=...)` -- either may be `None` but not both. Wired into `TI2VidOneStagePipeline`, `TI2VidTwoStagesPipeline`, `TI2VidTwoStagesHQPipeline`, `DistilledPipeline`, `DFRPipeline`, and `T2AOneStagePipeline` (audio-only: `video_encoding=None`), plus their MGPU runners: when `num_frames` is omitted (`None`), `require_num_frames_source` (called at the top of `__call__`, before any work) raises immediately if no `DurationPredictor` is available, otherwise it's auto-predicted from the caption. Auto-duration resolves inside `__call__`; read the final count from ``PipelineOutput.num_frames`` (or the local ``num_frames`` variable before return).
### Memory management
-- **Model lifecycle**: All blocks build their model on call and free it on exit. `gpu_model()` moves params to `"meta"` device on exit, immediately releasing storage. No model persists between calls.
+- **Model lifecycle**: All blocks build their model on call and free it on exit. `gpu_model()` moves params to `"meta"` device on exit, immediately releasing storage. The one exception is a `DiffusionStage` transformer with registry-resident weights under `DEFER`, which stays assigned (see `DiffusionStage` above); otherwise no model persists between calls.
- **Block streaming**: When offloading is enabled, `DiffusionStage` wraps the transformer in `BlockStreamingWrapper`. Blocks live on pinned CPU memory; only 2 blocks are buffered on GPU at a time (one for compute, one for async H2D copy on a separate CUDA stream).
- **Batch splitting**: `BatchSplitAdapter` wraps the transformer and splits inputs exceeding `max_batch_size` into sequential chunks. If guidance needs B=4 but `max_batch_size=1`, it runs 4 sequential B=1 passes. Higher `max_batch_size` reduces layer-streaming PCIe transfers at the cost of peak memory.
@@ -81,7 +83,7 @@ CLI: on `python -m ltx_pipelines.distilled`, `python -m ltx_pipelines.ti2vid_two
- `SimpleDenoiser` -- single forward pass (B=1), no guidance. Used by distilled pipelines and all stage 2.
- `GuidedDenoiser` -- CFG/STG with static `MultiModalGuider` instances (HQ, A2Vid, Retake non-distilled).
-- `FactoryGuidedDenoiser` -- per-step guider creation via factory (OneStageTI2Vid, TwoStagesTI2Vid, Keyframe).
+- `FactoryGuidedDenoiser` -- per-step guider creation via factory (OneStageTI2Vid, TwoStagesTI2Vid).
All denoisers return `VideoAudio` of `DenoisedLatentResult` (defined in `utils/types.py`). It still unpacks as `(video_result, audio_result)`; either side may be `None` for absent modalities. `DenoisedLatentResult.denoised` is the final blended tensor. Guided denoisers additionally populate per-pass fields (`.cond`, `.uncond`, `.ptb`, `.mod`) on each result; `SimpleDenoiser` leaves these `None`.
@@ -97,9 +99,9 @@ Every video pipeline ``__call__`` returns ``PipelineOutput``: ``video``, ``audio
- **HQ**: Res2s second-order sampler for **both** stages, latent-dependent sigma schedule, distilled LoRA on both stages with separate strengths.
- **A2Vid**: Audio frozen in both stages (`frozen=True, noise_scale=0.0`). That zeros `denoise_mask` and forces audio `Modality.sigma=0` (prompt AdaLN / a2v gate). Returns original audio (not VAE-decoded); no `AudioDecoder`. Chunked mode seeds each chunk from the encoded source latent (`generate_uniform_chunks(audio_latent=)`) and muxes the original waveform after decode (`replace_chunks_audio`). CLI `--num-frames` and `--audio-max-duration` are mutually exclusive: with `--audio-max-duration`, `num_frames` is omitted (`None`) and derived from the decoded clip (capped by remaining audio after `--audio-start-time`); with `--num-frames` or neither, that frame count is used and audio is clipped to match. Returns ``PipelineOutput`` like the other two-stage pipelines.
-- **IC-LoRA**: `VideoConditionByReferenceLatent`, `reference_downscale_factor` / `reference_temporal_scale_factor` from LoRA metadata, `skip_stage_2`, attention mask downsampling. Default stage 2 is LoRA-free (`combined_image_conditionings` only); `apply_ic_lora` / `--stage-2-ic-lora` keeps the reference on a stage. CLI is two stages (`--tile` plus `--tile-height` / `--tile-width` for transformer windows). Longer ladders use `ICLoraPipeline(stages=...)`. Temporal chunking: `stream_chunks()` / `chunk_config` compose the shared chunk operators with per-stage LoRA and `diffusion_stage_with_tiling`; reference video uses `SequentialVideoFrameSource` and chunk-local mask/temporal scaling via `reference_video_conditionings_for_chunk`. CLI `--chunk-pixel-frames` / `--chunk-carry-frames` / `--chunk-blend-frames` stream encode like Distilled/Dub-It.
+- **IC-LoRA**: `VideoConditionByReferenceLatent`, `reference_downscale_factor` / `reference_temporal_scale_factor` from LoRA metadata, `skip_stage_2`, attention mask downsampling. Default stage 2 is LoRA-free (`image_conditionings` only); `apply_ic_lora` / `--stage-2-ic-lora` keeps the reference on a stage. CLI is two stages (`--tile` plus `--tile-height` / `--tile-width` for transformer windows). Longer ladders use `ICLoraPipeline(stages=...)`. Temporal chunking: `stream_chunks()` / `chunk_config` compose the shared chunk operators with per-stage LoRA and `diffusion_stage_with_tiling`; reference video uses `SequentialVideoFrameSource` and chunk-local mask/temporal scaling via `reference_video_conditionings_for_chunk`. CLI `--chunk-pixel-frames` / `--chunk-carry-frames` / `--chunk-blend-frames` stream encode like Distilled/Dub-It.
- **Dub-It**: Standalone pipeline; IC reference **video** helpers and audio-reference patchify (negative RoPE) in `iclora_utils.py`. Appends frozen audio-reference tokens via `AudioConditionByReferenceLatent` (ltx-core), matching video token order (`[target | ref]`) while keeping reference RoPE positions negative (training-compatible). Single IC-LoRA on both stages; full IC-LoRA video conditioning at stage 1 and 2; stage-2 audio is frozen with S1 latent as initial state and uses S1-derived ref. Final audio decoded from stage 1 latent. Chunked mode walks the IC-LoRA reference video once through `SequentialVideoFrameSource` (overlap lookback) and encodes audio-reference tokens in the chunk factories (stage 2 swaps the audio factory to self-reference via `replace_audio_conditionings`). The Dub-It CLI does not expose `--conditioning-attention-mask`; use `ic_lora.py` if you need spatial IC attention masking.
-- **Keyframe**: Uses `image_conditionings_by_adding_guiding_latent` in both stages (all frames as keyframe guidance, no replacement) -- unlike TI2Vid which uses `combined_image_conditionings` (frame_idx=0 replaces, others guide).
+- **TI2Vid two-stage (image conditioning)**: Preserves each caller-supplied binding through chunking and DFR rebasing. Use `ImageConditioningInput.replace()` / `.guide()` / `.at()` (or `--image` with optional trailing `replace` / `guide`) explicitly; the pipeline does not rewrite bindings. Keyframe interpolation uses `.guide()` at endpoint frames plus optional `refine_audio=True` (see `ti2vid_two_stages.py`).
- **Retake**: `TemporalRegionMask` for selective time-window regeneration. `regenerate_video`/`regenerate_audio` flags. Conditional distilled/full behavior.
- **Distilled**: Single `self.stage` reused for both stages (not `stage_1`/`stage_2`). The sampler is resolved in `__init__` from the checkpoint generation -- `should_use_ancestral_sampler(path)` (`detect_model_version(...) >= (2, 5)`) sets `self.use_ancestral_sampler`, so an LTX-2.5+ checkpoint samples every stage with `euler_ancestral_denoising_loop(..., eta=1.0, s_noise=1.0)` (ComfyUI's `sample_euler_ancestral_RF` defaults), and anything older samples with plain Euler. There is no per-call override; assign the attribute before calling to pin a sampler (e.g. to A/B both on one checkpoint). `distilled_mgpu.py` detects per rank from the same checkpoint, so ranks stay symmetric. Each pass has its own noise-seed offset (`ANCESTRAL_NOISE_SEED_OFFSET`, `ANCESTRAL_STAGE_2_NOISE_SEED_OFFSET`), so no two of them inject the same noise and stage 1's first draw is not bit-identical to the initial `GaussianNoiser` noise. Within a pass the seed advances by one on each denoise call, so sequential chunks do not reuse ancestral noise across a seam; a single `DiffusionStage` call is index 0, so one-shot output matches a fixed-seed partial.
- **DFR** (Diffusion Fidelity Rendering, `dfr_pipeline.py` orchestrator + `dfr_stages.py` +
@@ -203,8 +205,8 @@ Every video pipeline ``__call__`` returns ``PipelineOutput``: ``video``, ``audio
## Image conditioning helpers (`utils/helpers.py`)
-- `combined_image_conditionings()` -- images with `frame_idx==0` replace latent (`VideoConditionByLatentIndex`), others guide (`VideoConditionByKeyframeIndex`).
-- `image_conditionings_by_adding_guiding_latent()` -- all images become keyframe guidance regardless of `frame_idx`.
+- `image_conditionings()` -- encodes `LoadedImageConditioning`s using each `binding` (`ImageReplace` / `ImageGuide`). Both bindings carry pixel-frame indices; `ImageReplace` accepts causal latent starts (`0` or `1 + 8 * k`) and converts to a latent index only when lowered to `VideoConditionByLatentIndex`.
+- `ImageConditioningInput.at()` -- combined i2v default: frame 0 replaces latent, later frames guide.
- `evenly_spaced_keyframe_positions()` -- evenly spaced **interior** positions (both endpoints excluded).
- `generated_keyframe_conditionings()` / `resolve_generated_keyframes()` -- turn the pipeline-level ``generated_keyframes`` arg into ``VideoGeneratedKeyframeSlots`` or pixel indices.
- `decode_keyframes_from_slots()` -- build ``DecodeKeyframes`` from carried slot latents, dropping planes past a trimmed canvas.
diff --git a/packages/ltx-pipelines/README.md b/packages/ltx-pipelines/README.md
index a088df2..12bbb5b 100644
--- a/packages/ltx-pipelines/README.md
+++ b/packages/ltx-pipelines/README.md
@@ -43,6 +43,7 @@ See [Installation & Usage](docs/installation.md) for full setup, CLI modules, an
| [Available Pipelines](docs/pipelines.md) | Full reference for all 12 pipelines, including [Running DFR](docs/pipelines.md#running-dfr) |
| [Conditioning Types](docs/conditioning.md) | Image and video conditioning methods |
| [HDR Support](docs/hdr.md) | Native EXR/`--hdr` colour spaces, I/O, and examples |
+| [VideoSeal marking](docs/marking.md) | Opt-in post-hoc VideoSeal / AudioSeal (`--marker-root-path`) |
| [Multimodal Guidance](docs/multimodal-guidance.md) | CFG / STG / modality guidance parameters and tuning |
| [Optimization Tips](docs/optimization.md) | FP8 quantization, `torch.compile`, gradient estimation |
| [Multi-GPU Inference](docs/multigpu/README.md) | Run a single generation across GPUs for latency (SP, TDP, distributed VAE, distributed Gemma) |
diff --git a/packages/ltx-pipelines/docs/conditioning.md b/packages/ltx-pipelines/docs/conditioning.md
index 5cb68b2..fab7fb0 100644
--- a/packages/ltx-pipelines/docs/conditioning.md
+++ b/packages/ltx-pipelines/docs/conditioning.md
@@ -4,15 +4,27 @@ Pipelines use different conditioning methods from [`ltx-core`](../../ltx-core/)
## Image Conditioning
-All pipelines support image conditioning, but with different methods:
-
-- **Replacing Latents** ([`image_conditionings_by_replacing_latent`](../src/ltx_pipelines/utils/helpers.py)):
- - Replaces the latent at a specific frame with the encoded image
- - Strong control over specific frames
-
-- **Guiding Latents** ([`image_conditionings_by_adding_guiding_latent`](../src/ltx_pipelines/utils/helpers.py)):
- - Adds the image as a guiding signal rather than replacing
- - Better for smooth interpolation between keyframes
+Stills are passed as [`ImageConditioningInput`](../src/ltx_pipelines/utils/types.py) values with an
+[`ImageReplace`](../src/ltx_pipelines/utils/types.py) or [`ImageGuide`](../src/ltx_pipelines/utils/types.py)
+binding. [`image_conditionings()`](../src/ltx_pipelines/utils/helpers.py) encodes each still and
+emits the matching core conditioning item (`VideoConditionByLatentIndex` or
+`VideoConditionByKeyframeIndex`).
+
+`ImageConditioningInput` is a pixel-space API: every `frame_idx` is a zero-based index on the
+decoded output video's frame timeline, never an index into the VAE latent. This applies equally to
+`ImageConditioningInput.at()`, `.replace()`, `.guide()`, `ImageReplace`, `ImageGuide`, and the
+`--image` (optional trailing `replace` / `guide` binding keyword) CLI flags. Chunking and temporal rebasing also
+map these values in pixel space.
+
+`ImageGuide` accepts any frame in the clip. `ImageReplace` must point to the start of a causal VAE
+latent: pixel frame `0`, or `1 + 8 * k` (`1`, `9`, `17`, ...). The pipeline converts that
+pixel-frame index to a latent index only when it creates the core conditioning item:
+
+```python
+opening = ImageConditioningInput.replace("opening.png", frame_idx=0, strength=1.0)
+later_replacement = ImageConditioningInput.replace("later.png", frame_idx=9, strength=1.0)
+guide = ImageConditioningInput.guide("guide.png", frame_idx=80, strength=0.8)
+```
## Video Conditioning
diff --git a/packages/ltx-pipelines/docs/installation.md b/packages/ltx-pipelines/docs/installation.md
index 44e1005..cde19ef 100644
--- a/packages/ltx-pipelines/docs/installation.md
+++ b/packages/ltx-pipelines/docs/installation.md
@@ -77,7 +77,6 @@ what pipeline constructors take.
- `ltx_pipelines.ti2vid_one_stage` - Single-stage text/image-to-video. ([docs](pipelines.md#3-ti2vidonestagepipeline), [source](../src/ltx_pipelines/ti2vid_one_stage.py))
- `ltx_pipelines.t2a_one_stage` - Single-stage text-to-audio (audio-only output). ([docs](pipelines.md#11-t2aonestagepipeline), [source](../src/ltx_pipelines/t2a_one_stage.py))
- `ltx_pipelines.ic_lora` - Video-to-video with IC-LoRA. ([docs](pipelines.md#5-iclorapipeline), [source](../src/ltx_pipelines/ic_lora.py))
-- `ltx_pipelines.keyframe_interpolation` - Keyframe interpolation. ([docs](pipelines.md#6-keyframeinterpolationpipeline), [source](../src/ltx_pipelines/keyframe_interpolation.py))
- `ltx_pipelines.a2vid_two_stage` - Audio-to-video generation conditioned on an input audio. ([docs](pipelines.md#7-a2vidpipelinetwostage), [source](../src/ltx_pipelines/a2vid_two_stage.py))
- `ltx_pipelines.retake` - Regenerate a time region of an existing video. ([docs](pipelines.md#8-retakepipeline), [source](../src/ltx_pipelines/retake.py))
- `ltx_pipelines.hdr_ic_lora` - Video-to-video SDR→HDR: writes a BT.2020/HLG master plus an EXR sequence (`--exr-colorspace`, default ACEScg). ([docs](pipelines.md#9-hdriclorapipeline), [source](../src/ltx_pipelines/hdr_ic_lora.py))
@@ -99,6 +98,7 @@ These flags are shared across the pipeline CLIs (they come from a common base pa
- `--lora <path> [strength]` - apply a LoRA (repeatable; default strength 1.0).
- `--enhance-prompt` - rewrite the prompt with the built-in enhancer before generation.
- `--hdr {SRGB_LINEAR,ACESCG,ACESCCT}` - on pipelines that accept image / video conditioning (not Dub-It, HDR IC-LoRA, or T2A): declare the source colour space for EXR stills / EXR-frame folders. Required whenever any EXR input is passed; omit for SDR. With `--hdr` set, encode writes half EXR frames plus a BT.2020/HLG master. See [HDR Support](hdr.md).
+- `--marker-root-path <dir>` - optional local `path/to/marking` pack for post-hoc VideoSeal (SDR pixels) and AudioSeal (stereo PCM). Requires `uv sync --extra videoseal --extra audioseal`. HDR encode skips video stamping and still stamps audio. See [VideoSeal marking](marking.md).
- `--video-vae-path <path>` - video VAE `.safetensors` (encoder + decoder). Split: the `vae/` component. Monolith: optional override of the VAE bundled in `--checkpoint-path`/`--distilled-checkpoint-path`. When it is a diffusion VAE, decoding builds a `DiffusionVideoDecoder`, which for best performance requires the `natten` extra. See [Optimization Tips](optimization.md#diffusion-vae-decoder).
- `--diffvae-optimization {chunked_eager,chunked_compile,combined_compile,blackwell_dsl}` - DiffVAE decode preset (ignored for convolutional VAEs). `blackwell_dsl` is the fast path on datacenter Blackwell.
- `--chunk-pixel-frames` / `--chunk-carry-frames` - on `distilled`, `ti2vid_two_stages` (and their MGPU runners), `a2vid_two_stage`, and `dubit`: pass either flag to generate in windows instead of one full-clip chunk. Defaults are 97 pixel frames and 25-pixel carry when the other is set. Pixel frame counts must be on the causal grid (`8k+1`). Off-grid `--num-frames` is floored to the causal grid.
diff --git a/packages/ltx-pipelines/docs/marking.md b/packages/ltx-pipelines/docs/marking.md
new file mode 100644
index 0000000..cc422c4
--- /dev/null
+++ b/packages/ltx-pipelines/docs/marking.md
@@ -0,0 +1,95 @@
+# VideoSeal marking
+
+Opt-in post-hoc VideoSeal on SDR H.264 encode and AudioSeal on stereo PCM. These are **not** cryptographic signatures; `detected` is each marker's own call (VideoSeal: bit-test `log10(p) < -6`; AudioSeal: presence score). `bit_accuracy` / `p_value` are the binomial test of decoded bits vs the baked payload.
+
+## Install
+
+```bash
+uv sync --extra videoseal --extra audioseal
+# or
+pip install 'ltx-pipelines[videoseal,audioseal]'
+```
+
+PyPI `videoseal` 1.0.1 predates the PixelSeal card, so the extra installs videoseal from git. AudioSeal is PyPI `audioseal` 0.2+.
+
+The library does not download weights. Pass a local `path/to/marking` pack directory on `--marker-root-path`. Expected files (all required):
+
+- `pixelseal.pth`
+- `pixelseal.yaml`
+- `videoseal_attenuation.yaml`
+- `audioseal_generator.pth`
+- `audioseal_detector.pth`
+
+## CLI
+
+Video-generation and editing CLIs, HDR IC-LoRA, and text-to-audio (`t2a_one_stage`) accept `--marker-root-path DIR`. Omit the flag for unchanged pixels / PCM (`markers_from_args` returns `(None, None)`). HDR encode skips VideoSeal (logs that video marking is unavailable with HDR) and still stamps AudioSeal PCM before AAC/HLG mux. A2Vid two-stage stamps video only (source audio is muxed unmarked). Text-to-audio uses the same flag and only the audio marker (the pack still loads both).
+
+```bash
+python -m ltx_pipelines.distilled \
+ ... \
+ --marker-root-path path/to/marking \
+ --output-path out.mp4
+
+python -m ltx_pipelines.t2a_one_stage \
+ ... \
+ --marker-root-path path/to/marking \
+ --output-path out.wav
+```
+
+Detect from Python with a loaded marker (there is no inspect CLI):
+
+```python
+import torch
+from ltx_pipelines.marking import create_markers, detect_audio_mark, detect_video_mark
+
+video_marker, audio_marker = create_markers("path/to/marking", torch.device("cuda"))
+print(detect_video_mark(video_marker, "out.mp4"))
+print(detect_audio_mark(audio_marker, "out.mp4"))
+```
+
+## Library
+
+`encode_video(..., video_marker=None, audio_marker=None)` leaves pixels and PCM unchanged. Pass resident markers; encode does not load weights. HDR callers omit `video_marker`. A2Vid callers omit `audio_marker`. Each yielded `Audio` is stamped independently (encode does not stitch). `encode_audio(..., audio_marker=None)` is the WAV path.
+
+From a pack (`create` loads, `eval()`s, drops grad, and moves to `device`):
+
+```python
+import torch
+from ltx_pipelines.marking import create_markers
+from ltx_pipelines.utils.media_io import encode_video
+
+device = torch.device("cuda")
+video_marker, audio_marker = create_markers("path/to/marking", device)
+with torch.inference_mode():
+ encode_video(..., video_marker=video_marker, audio_marker=audio_marker)
+```
+
+Without a pack, construct the seal classes from file paths:
+
+```python
+import torch
+from ltx_pipelines.marking.audio_seal import AudioSealMarker
+from ltx_pipelines.marking.video_seal import VideoSealMarker
+
+device = torch.device("cuda")
+video_marker = VideoSealMarker.create(
+ "path/to/marking/pixelseal.pth",
+ card="path/to/marking/pixelseal.yaml",
+ attenuation_config="path/to/marking/videoseal_attenuation.yaml",
+ device=device,
+)
+audio_marker = AudioSealMarker.create(
+ "path/to/marking/audioseal_generator.pth",
+ "path/to/marking/audioseal_detector.pth",
+ device=device,
+)
+
+with torch.inference_mode():
+ encode_video(..., video_marker=video_marker, audio_marker=audio_marker)
+```
+
+`VideoMarker.embed` stamps one `[F, H, W, C]` RGB tensor in `[0, 1]`. `VideoMarker.detect` takes an iterator of `[H, W, C]` frames in the same domain. `AudioMarker.embed` / `AudioMarker.detect` take `Audio`. File I/O is `detect_video_mark` / `detect_audio_mark` in `ltx_pipelines.marking`.
+
+Pack load uses the local card YAML (not packaged `videoseal/cards/`). Stereo is stamped as a batch of two mono channels (no downmix). Native 24/48 kHz; AudioSeal 0.2 does not resample.
+
+Public API in `ltx_pipelines.marking` is the `VideoMarker` / `AudioMarker` protocols, `create_markers`, `MarkDetectResult`, and detect helpers. `VideoSealMarker.create` / `AudioSealMarker.create` live in `ltx_pipelines.marking.video_seal` and `ltx_pipelines.marking.audio_seal` (not re-exported, so importing `ltx_pipelines.marking` does not require extras) and return those protocols. Filename constants for the pack (`PIXELSEAL_CHECKPOINT_NAME`, …) live on the root helper.
diff --git a/packages/ltx-pipelines/docs/multigpu/controller.md b/packages/ltx-pipelines/docs/multigpu/controller.md
index b78dfd4..966dfe4 100644
--- a/packages/ltx-pipelines/docs/multigpu/controller.md
+++ b/packages/ltx-pipelines/docs/multigpu/controller.md
@@ -17,12 +17,17 @@ MGPUController(
num_gpus: int | None = None, # GPUs 0..num_gpus-1 (default: all visible)
devices: Sequence[int] | None = None, # place on specific physical GPUs, e.g. [2, 3]
logs_specs: LogsSpecs | None = None,
+ cpu_threads_per_rank: int | None = 8, # intra-op CPU threads per rank (None: torch's default)
)
```
`num_gpus` and `devices` are mutually exclusive. `devices=[2, 3]` puts rank r on
`cuda:devices[r]`, so two controllers can share one machine on disjoint GPU sets.
+`cpu_threads_per_rank` sets every rank's intra-op CPU threads with `torch.set_num_threads`;
+`None` leaves torch's default. Ranks run only small host tensor ops, which a pool as wide as
+the machine stalls (tens to hundreds of ms per op on a 176-CPU host).
+
Lifecycle:
| Method | What it does |
diff --git a/packages/ltx-pipelines/docs/multigpu/gemma.md b/packages/ltx-pipelines/docs/multigpu/gemma.md
index ac022bb..bca65aa 100644
--- a/packages/ltx-pipelines/docs/multigpu/gemma.md
+++ b/packages/ltx-pipelines/docs/multigpu/gemma.md
@@ -14,6 +14,18 @@ lightweight `AccelerateGemmaWrapper` stub that receives the encoded embeddings o
NCCL. The source rank fuses all prompts into one Gemma call, then broadcasts each
output.
+Gemma's shards share GPUs with the other ranks, so the non-source ranks must not
+post their NCCL receives while Gemma is still running: a receive kernel spins on
+its GPU, and the two processes then time-slice it. Each builder therefore owns a
+`BroadcastCoordinator` (`ltx_core.multigpu.broadcast`) on the default store and
+hands it to every wrapper it builds. The source rank synchronizes the devices
+holding Gemma's outputs and signals; the other ranks poll the store every 1 ms on
+the host and only then receive. `enhance_*` results are gated the same way.
+A waiting rank raises `TimeoutError` after 10 minutes (NCCL's default collective
+timeout), e.g. when the source rank failed before broadcasting. Construct the
+builders on every rank in `setup()`, in the same order and before any job runs:
+each builder's store key comes from its construction order.
+
The first `build()` loads Gemma via `load_gemma_with_device_map` (directory
`from_pretrained(device_map="auto")`, or single-file TE with the same balanced
placement) and caches the full state dict (including non-persistent buffers) in
diff --git a/packages/ltx-pipelines/docs/optimization.md b/packages/ltx-pipelines/docs/optimization.md
index c11891c..a857840 100644
--- a/packages/ltx-pipelines/docs/optimization.md
+++ b/packages/ltx-pipelines/docs/optimization.md
@@ -217,12 +217,12 @@ That `natten` extra pins `natten==0.21.7+torch2130cu132` with `torch==2.13.0` (c
| stage-5 VRAM coefficient | 5 | 7 | 11 | 2.5 |
| withheld reserve | 1 GiB | 2 GiB | 2 GiB | 2 GiB |
| `joint_na3d`, with keyframes | Triton > pure-torch | Triton > pure-torch | Triton > pure-torch | `na_attention_joint_dsl` |
-| stage-5 VRAM coefficient | 5 | **5** | **15** | 2.5 |
+| stage-5 VRAM coefficient | 5 | **5** | **15** | **4.0** |
| withheld reserve | 1 GiB (2 on MATH SDPA) | 1 GiB (2 on MATH SDPA) | 1 GiB (2 on MATH SDPA) | 2 GiB |
-A keyframe decode is **not** compiled -- the compiled entry points are the keyframe-free ones -- so its diff blocks run eager whatever the mode asked for, and the budget follows that: `chunked_compile` drops to the `chunked_eager` coefficient, and `combined_compile` needs **15** rather than 11 because it holds several live full-volume intermediates where the compiled path holds one fused buffer. On a host whose joint attention falls back to torch's MATH SDPA kernel (no CUDA, e.g. MPS) the reserve grows by a GiB, since that kernel materializes its score block.
+A keyframe decode is **not** compiled -- the compiled entry points are the keyframe-free ones -- so its diff blocks run eager whatever the mode asked for, and the budget follows that: `chunked_compile` drops to the `chunked_eager` coefficient, and `combined_compile` needs **15** rather than 11 because it holds several live full-volume intermediates where the compiled path holds one fused buffer. `blackwell_dsl` serves keyframes with its fused joint kernel, but the keyframe decode's eager det blocks and keyframe planes raise its measured working set to **4.0**. On a host whose joint attention falls back to torch's MATH SDPA kernel (no CUDA, e.g. MPS) the reserve grows by a GiB, since that kernel materializes its score block.
-Pipeline `AUTO_TILING` resolves via `tiling_config_for_vae`: for a **Conv VAE** that is aspect-coupled long-side `768/64` spatial tiles plus temporal `80/24` (`TileSizeConfig.from_long_side`). For **DiffVAE**, recommended decode tiling enumerates legal per-axis tile **sizes** on the VAE grid (via the same `split_by_size` path decode uses) and emits a `TileSizeConfig` with independent `frames` / `height` / `width` `DimensionSizeConfig` values. Tile sizes are floored at ``2xOverlap`` so blend masks stay complementary. Stage-5 coefficients and reserves come from the grid above; `tiling_config_for_vae(..., keyframes=True)` selects the keyframe column, and a `chunked_compile` remapped to the Triton/eager fallback uses the `chunked_eager` row.
+Pipeline `AUTO_TILING` resolves via `tiling_config_for_vae`: for a **Conv VAE** that is aspect-coupled long-side `768/64` spatial tiles plus temporal `80/24` (`TileSizeConfig.from_long_side`). For **DiffVAE**, recommended decode tiling enumerates legal per-axis tile **sizes** on the VAE grid (via the same `split_by_size` path decode uses) and emits a `TileSizeConfig` with independent `frames` / `height` / `width` `DimensionSizeConfig` values. Tile sizes are floored at ``2xOverlap`` so blend masks stay complementary. The resolved tile is logged at INFO. Stage-5 coefficients and reserves come from the grid above; `tiling_config_for_vae(..., keyframes=True)` selects the keyframe column, and a `chunked_compile` remapped to the Triton/eager fallback uses the `chunked_eager` row.
**DFR** always keyframe-decodes (`decode_video(keyframes=)`), so `AUTO_TILING` uses the **keyframe** VRAM column. At 4K with temporal rounds, decode is usually the memory cliff, not the denoise; there is no decode-tile CLI flag. `--spatial-upscalings 2` keeps stage-2 denoise at half resolution. Each temporal tile reloads the transformer. See [Running DFR](pipelines.md#running-dfr).
diff --git a/packages/ltx-pipelines/docs/pipeline-selection.md b/packages/ltx-pipelines/docs/pipeline-selection.md
index 9ccd3b1..8455785 100644
--- a/packages/ltx-pipelines/docs/pipeline-selection.md
+++ b/packages/ltx-pipelines/docs/pipeline-selection.md
@@ -19,7 +19,7 @@ Do you need to condition on existing images/videos?
├─ YES → Do you have reference videos for video-to-video?
│ ├─ YES → Use ICLoraPipeline
│ └─ NO → Do you have multiple keyframe images to interpolate?
-│ ├─ YES → Use KeyframeInterpolationPipeline
+│ ├─ YES → Use TI2VidTwoStagesPipeline with `--image … guide` bindings
│ └─ NO → Image-to-video (one still)
│ ├─ Production quality / higher fps → DFRPipeline
│ └─ Guided CFG/STG two-stage → TI2VidTwoStagesPipeline
@@ -43,11 +43,10 @@ Do you need to condition on existing images/videos?
| -------- | ------ | --- | ---------- | ------------- | -------- |
| [**DistilledPipeline**](pipelines.md#4-distilledpipeline) | 2 | ❌ | ✅ | Image | Fastest inference (starting point) |
| [**DFRPipeline**](pipelines.md#12-dfrpipeline) | 2 (+ optional temporal rounds) | ❌ | ✅ spatial + optional temporal 2x | Image + [generated keyframes](conditioning.md#generated-keyframe-slots) | **Production quality**; higher effective frame rate |
-| [**TI2VidTwoStagesPipeline**](pipelines.md#1-ti2vidtwostagespipeline) | 2 | ✅ | ✅ | Image | Guided two-stage with CFG/STG |
+| [**TI2VidTwoStagesPipeline**](pipelines.md#1-ti2vidtwostagespipeline) | 2 | ✅ | ✅ | Image + keyframes | Guided two-stage with CFG/STG; animation and interpolation |
| [**TI2VidTwoStagesHQPipeline**](pipelines.md#2-ti2vidtwostageshqpipeline) | 2 | ✅ | ✅ | Image | Same guided flow, res_2s sampler |
| [**TI2VidOneStagePipeline**](pipelines.md#3-ti2vidonestagepipeline) | 1 | ✅ | ❌ | Image | Educational, prototyping |
| [**ICLoraPipeline**](pipelines.md#5-iclorapipeline) | 2 | ✅ | ✅ | Image + Video | Video-to-video transformations |
-| [**KeyframeInterpolationPipeline**](pipelines.md#6-keyframeinterpolationpipeline) | 2 | ✅ | ✅ | Keyframes | Animation, interpolation |
| [**A2VidPipelineTwoStage**](pipelines.md#7-a2vidpipelinetwostage) | 2 | ✅ | ✅ | Audio + Image | Audio-driven video generation |
| [**RetakePipeline**](pipelines.md#8-retakepipeline) | 1 | ✅ | ❌ | Source Video | Regenerating a time region of a video |
| [**HDRICLoraPipeline**](pipelines.md#9-hdriclorapipeline) | 1 | ❌ | ❌ | Video | SDR-to-HDR with an HDR IC-LoRA (HLG master + EXR via `--exr-colorspace`); see also native [`--hdr`](hdr.md) |
diff --git a/packages/ltx-pipelines/docs/pipelines.md b/packages/ltx-pipelines/docs/pipelines.md
index 5ca9213..9b39bd1 100644
--- a/packages/ltx-pipelines/docs/pipelines.md
+++ b/packages/ltx-pipelines/docs/pipelines.md
@@ -6,15 +6,22 @@ Full reference for each pipeline. See the [Pipeline Selection Guide](pipeline-se
## 1. TI2VidTwoStagesPipeline
-**Best for:** Guided two-stage text/image-to-video with CFG/STG and 2x upsampling.
+**Best for:** Guided two-stage text/image-to-video and keyframe interpolation with CFG/STG and 2x upsampling.
**Source**: [`src/ltx_pipelines/ti2vid_two_stages.py`](../src/ltx_pipelines/ti2vid_two_stages.py)
-Two-stage generation: Stage 1 generates low-resolution video with [multimodal guidance](multimodal-guidance.md), Stage 2 upsamples to 2x resolution with distilled LoRA refinement. Supports image conditioning. Slower than DistilledPipeline; for production quality use [`DFRPipeline`](#12-dfrpipeline).
+Two-stage generation: Stage 1 generates low-resolution video with [multimodal guidance](multimodal-guidance.md),
+Stage 2 upsamples to 2x resolution with distilled LoRA refinement. Supports image conditioning, including
+[keyframe interpolation](#6-keyframe-interpolation) through explicit keyframe-guide bindings. Slower than
+DistilledPipeline; for production quality use [`DFRPipeline`](#12-dfrpipeline).
-**Extra CLI arguments:** `--chunk-pixel-frames` (default 97) and/or `--chunk-carry-frames` (default 25) enable long-video windowing; `--chunk-blend-frames` controls the decoded seam crossfade (default: full carry, `0` for a hard cut). Also on `ltx_pipelines.ti2vid_two_stages_mgpu`.
+**Extra CLI arguments:** `--chunk-pixel-frames` (default 97) and/or `--chunk-carry-frames` (default 25) enable
+long-video windowing; `--chunk-blend-frames` controls the decoded seam crossfade (default: full carry, `0` for a hard cut).
+`--refine-audio` denoises audio again in stage 2 instead of preserving stage-1 audio. These flags are also available on
+`ltx_pipelines.ti2vid_two_stages_mgpu`.
-**Use when:** You want CFG/STG-guided two-stage text/image-to-video, not the DFR detailing path.
+**Use when:** You want CFG/STG-guided two-stage text/image-to-video or keyframe interpolation, not the DFR detailing
+path.
---
@@ -115,13 +122,19 @@ python -m ltx_pipelines.ic_lora \
---
-## 6. KeyframeInterpolationPipeline
+## 6. Keyframe interpolation
**Best for:** Generating videos by interpolating between keyframe images.
-**Source**: [`src/ltx_pipelines/keyframe_interpolation.py`](../src/ltx_pipelines/keyframe_interpolation.py)
+Use [`TI2VidTwoStagesPipeline`](#1-ti2vidtwostagespipeline) with explicit keyframe-guide image bindings. In Python,
+construct each image with `ImageConditioningInput.guide(path, frame_idx, strength)`, where `frame_idx` is a
+zero-based pixel-frame index on the decoded output timeline; on the CLI, pass each image as
+`--image PATH FRAME_IDX STRENGTH guide [CRF]` and add `--refine-audio`. The CLI index uses the same pixel-frame
+unit.
-Two-stage generation with keyframe interpolation. Uses guiding latents (additive conditioning) instead of replacing latents for smoother transitions. [Multimodal guidance](multimodal-guidance.md) in stage 1, upsampling in stage 2.
+The two-stage pipeline uses guiding latents (additive conditioning) for these inputs, with
+[multimodal guidance](multimodal-guidance.md) in stage 1 and upsampling in stage 2. Pass `refine_audio=True` to
+also refine audio in stage 2, preserving the former keyframe-interpolation behavior.
**Use when:** You have keyframe images and want to interpolate between them, creating smooth transitions, or animation/motion interpolation tasks.
diff --git a/packages/ltx-pipelines/pyproject.toml b/packages/ltx-pipelines/pyproject.toml
index 767bd72..618a46d 100644
--- a/packages/ltx-pipelines/pyproject.toml
+++ b/packages/ltx-pipelines/pyproject.toml
@@ -1,11 +1,20 @@
[project]
name = "ltx-pipelines"
-version = "1.4.2"
+version = "1.5.0"
description = "Pipelines implementation for Lightricks' LTX-2 model"
readme = "README.md"
requires-python = ">=3.10"
dependencies = ["ltx-core", "av", "tqdm", "pillow", "openimageio", "cloudpickle>=3.1"]
+[project.optional-dependencies]
+# Videoseal 1.0.1 on PyPI predates PixelSeal; pin a commit that includes pixelseal.yaml.
+videoseal = [
+ "videoseal @ git+https://github.com/facebookresearch/videoseal.git@870ca7fb33578b90f14c602016b6c2788096226e",
+]
+audioseal = [
+ "audioseal>=0.2,<1",
+]
+
[build-system]
requires = ["uv_build>=0.9.8,<0.10.0"]
build-backend = "uv_build"
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/__init__.py b/packages/ltx-pipelines/src/ltx_pipelines/__init__.py
index 4a6af55..9ecb4f1 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/__init__.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/__init__.py
@@ -7,7 +7,6 @@ This package provides ready-to-use pipelines for video generation:
- DistilledPipeline: Fast distilled two-stage generation
- DubItPipeline: Dub-It with IC-LoRA and audio conditioning
- ICLoraPipeline: Image/video conditioning with distilled LoRA
-- KeyframeInterpolationPipeline: Keyframe-based video interpolation
- DFRPipeline: keyframe slots → spatial detailing → optional tiled temporal x2 (Diffusion Fidelity Rendering)
- RetakePipeline: Regenerate a time region (retake) of an existing video
For more detailed components and utilities, import from specific submodules
@@ -27,7 +26,6 @@ if TYPE_CHECKING:
from ltx_pipelines.distilled import DistilledPipeline
from ltx_pipelines.dubit import DubItPipeline
from ltx_pipelines.ic_lora import ICLoraPipeline
- from ltx_pipelines.keyframe_interpolation import KeyframeInterpolationPipeline
from ltx_pipelines.retake import RetakePipeline
from ltx_pipelines.t2a_one_stage import T2AOneStagePipeline
from ltx_pipelines.ti2vid_one_stage import TI2VidOneStagePipeline
@@ -42,7 +40,6 @@ _EXPORTS = {
"DistilledPipeline": "ltx_pipelines.distilled",
"DubItPipeline": "ltx_pipelines.dubit",
"ICLoraPipeline": "ltx_pipelines.ic_lora",
- "KeyframeInterpolationPipeline": "ltx_pipelines.keyframe_interpolation",
"RetakePipeline": "ltx_pipelines.retake",
"T2AOneStagePipeline": "ltx_pipelines.t2a_one_stage",
"TI2VidOneStagePipeline": "ltx_pipelines.ti2vid_one_stage",
@@ -55,7 +52,6 @@ __all__ = [
"DistilledPipeline",
"DubItPipeline",
"ICLoraPipeline",
- "KeyframeInterpolationPipeline",
"PipelineOutput",
"RetakePipeline",
"T2AOneStagePipeline",
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/a2vid_two_stage.py b/packages/ltx-pipelines/src/ltx_pipelines/a2vid_two_stage.py
index 744adb7..3f1b9cf 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/a2vid_two_stage.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/a2vid_two_stage.py
@@ -42,6 +42,7 @@ from ltx_pipelines.utils.args import (
add_keyframe_decode_arg,
chunk_config_from_args,
default_2_stage_arg_parser,
+ markers_from_args,
resolve_cli_params,
)
from ltx_pipelines.utils.blocks import (
@@ -75,7 +76,13 @@ from ltx_pipelines.utils.media_io import (
vae_dtype_for_hdr,
)
from ltx_pipelines.utils.model_paths import ModelPaths
-from ltx_pipelines.utils.types import ImageConditioningInput, OffloadMode, PipelineOutput, VideoAudio
+from ltx_pipelines.utils.types import (
+ ImageConditioningInput,
+ LoadedImageConditioning,
+ OffloadMode,
+ PipelineOutput,
+ VideoAudio,
+)
logger = logging.getLogger(__name__)
@@ -90,7 +97,7 @@ class _A2VidRunContext:
noiser: GaussianNoiser
dtype: torch.dtype
vae_dtype: torch.dtype
- images: list[ImageConditioningInput]
+ images: list[LoadedImageConditioning]
v_context_p: torch.Tensor
a_context_p: torch.Tensor
v_context_n: torch.Tensor
@@ -207,6 +214,7 @@ class A2VidPipelineTwoStage:
num_frames: int | None,
frame_rate: float,
images: list[ImageConditioningInput],
+ color_space: EXRColorSpace | None,
audio_path: str,
audio_start_time: float,
audio_max_duration: float | None,
@@ -219,7 +227,8 @@ class A2VidPipelineTwoStage:
chunk_config: ChunkConfig | None = None,
) -> _A2VidRunContext:
"""Validate inputs, decode/encode audio, and resolve frame count / tiling before chunk planning."""
- images = self.image_conditioner.resolve_crf(images)
+ enhance_prompt_image = images[0].path if images else None
+ images = self.image_conditioner.load_conditionings(images, color_space)
assert_resolution(height=height, width=width, is_two_stage=True)
assert_generated_keyframes_request(decode_with_keyframes, generated_keyframes, self.stage_1)
@@ -250,7 +259,7 @@ class A2VidPipelineTwoStage:
[prompt, negative_prompt],
enhance_first_prompt=enhance_prompt,
enhance_static_cache=enhance_static_cache,
- enhance_prompt_image=images[0][0] if len(images) > 0 else None,
+ enhance_prompt_image=enhance_prompt_image,
)
v_context_p, a_context_p = ctx_p.video_encoding, ctx_p.audio_encoding
v_context_n, _ = ctx_n.video_encoding, ctx_n.audio_encoding
@@ -300,9 +309,8 @@ class A2VidPipelineTwoStage:
def _video_conditionings(
self,
- images: list[ImageConditioningInput],
+ images: list[LoadedImageConditioning],
num_frames: int,
- color_space: EXRColorSpace | None,
) -> MakeVideoConditionings:
assert_image_frames_in_clip(images, num_frames)
@@ -314,7 +322,6 @@ class A2VidPipelineTwoStage:
chunk,
images=images,
video_encoder=enc,
- color_space=color_space,
)
)
@@ -415,6 +422,7 @@ class A2VidPipelineTwoStage:
num_frames=num_frames,
frame_rate=frame_rate,
images=images,
+ color_space=color_space,
audio_path=audio_path,
audio_start_time=audio_start_time,
audio_max_duration=audio_max_duration,
@@ -459,7 +467,7 @@ class A2VidPipelineTwoStage:
config=ctx.chunk_config,
video_scale_factors=self.stage_1.video_scale_factors,
audio_latent=ctx.encoded_audio_latent,
- make_video_conditionings=self._video_conditionings(ctx.images, ctx.num_frames, color_space),
+ make_video_conditionings=self._video_conditionings(ctx.images, ctx.num_frames),
generated_keyframes=generated_keyframes,
)
base = denoise_chunks(
@@ -590,6 +598,8 @@ def main() -> None:
params = resolve_cli_params()
parser = add_chunk_layout_args(add_keyframe_decode_arg(add_generated_keyframes_arg(build_a2vid_arg_parser(params))))
args = parser.parse_args()
+ hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)
+ video_marker, _ = markers_from_args(args, device=get_device(), color_space=hdr)
num_frames, audio_max_duration = resolve_a2vid_duration_or_exit(parser, args, default_num_frames=params.num_frames)
pipeline = A2VidPipelineTwoStage(
model_paths=args.model_paths,
@@ -602,7 +612,6 @@ def main() -> None:
prompt_enhancer_gemma_root=args.prompt_enhancer_gemma_root,
diffvae_optimization=args.diffvae_optimization,
)
- hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)
vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)
chunk_config = chunk_config_from_args(args)
video_guider_params = MultiModalGuiderParams(
@@ -656,6 +665,7 @@ def main() -> None:
video_chunks_number=video_chunks_number,
color_space=hdr,
audio_sampling_rate=audio_sampling_rate,
+ video_marker=video_marker,
)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/alpha_gen.py b/packages/ltx-pipelines/src/ltx_pipelines/alpha_gen.py
index dc3e0c5..6fe05fe 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/alpha_gen.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/alpha_gen.py
@@ -37,10 +37,10 @@ from ltx_pipelines.utils.blocks import (
from ltx_pipelines.utils.denoisers import FactoryGuidedDenoiser
from ltx_pipelines.utils.helpers import (
assert_resolution,
- combined_image_conditionings,
create_initial_video_latent,
ensure_tiling_config,
get_device,
+ image_conditionings,
tiling_scale_factors_for_vae,
)
from ltx_pipelines.utils.media_io import EXRColorSpace, encode_video, resolve_hdr_color_space, vae_dtype_for_hdr
@@ -49,6 +49,7 @@ from ltx_pipelines.utils.types import (
DEFAULT_AUTO_DURATION,
AutoDuration,
ImageConditioningInput,
+ LoadedImageConditioning,
ModalitySpec,
OffloadMode,
PipelineOutput,
@@ -128,7 +129,7 @@ class AlphaGenPipeline:
if not video_conditioning:
raise ValueError("Alpha-gen requires at least one reference video")
require_num_frames_source(num_frames, self.duration_predictor)
- images = self.image_conditioner.resolve_crf(images)
+ images = self.image_conditioner.load_conditionings(images, color_space)
assert_resolution(height=height, width=width, is_two_stage=False)
generator = torch.Generator(device=self.device).manual_seed(seed)
@@ -189,21 +190,20 @@ class AlphaGenPipeline:
def _conditionings(
self,
video_encoder: VideoEncoder,
- images: list[ImageConditioningInput],
+ images: list[LoadedImageConditioning],
video_conditioning: list[tuple[str, float]],
height: int,
width: int,
num_frames: int,
color_space: EXRColorSpace | None,
) -> list[ConditioningItem]:
- conditionings = combined_image_conditionings(
+ conditionings = image_conditionings(
images=images,
height=height,
width=width,
video_encoder=video_encoder,
dtype=self.dtype,
device=self.device,
- color_space=color_space,
)
append_ic_lora_reference_video_conditionings(
conditionings,
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/chunks/conditionings.py b/packages/ltx-pipelines/src/ltx_pipelines/chunks/conditionings.py
index 883be7d..6dbc661 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/chunks/conditionings.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/chunks/conditionings.py
@@ -26,48 +26,46 @@ from ltx_pipelines.iclora_utils import (
append_ic_lora_reference_video_conditionings,
patchify_audio_reference_latent,
)
-from ltx_pipelines.utils.helpers import combined_image_conditionings
+from ltx_pipelines.utils.helpers import image_conditionings
from ltx_pipelines.utils.media_io import is_exr_dir
from ltx_pipelines.utils.media_io.color_config import EXRColorSpace
-from ltx_pipelines.utils.types import ImageConditioningInput
+from ltx_pipelines.utils.types import LoadedImageConditioning
-def assert_image_frames_in_clip(images: Sequence[ImageConditioningInput], num_frames: int) -> None:
- """Raise if any image ``frame_idx`` lies outside the planned stitched clip."""
+def assert_image_frames_in_clip(images: Sequence[LoadedImageConditioning], num_frames: int) -> None:
+ """Raise if any image binding index lies outside the planned stitched clip."""
for image in images:
- if image.frame_idx < 0 or image.frame_idx >= num_frames:
- raise ValueError(f"frame_idx={image.frame_idx} is outside the planned stitched range [0, {num_frames})")
+ image.assert_in_clip(num_frames)
-def images_for_chunk(chunk: Chunk, images: Sequence[ImageConditioningInput]) -> list[ImageConditioningInput]:
+def images_for_chunk(chunk: Chunk, images: Sequence[LoadedImageConditioning]) -> list[LoadedImageConditioning]:
"""Remap clip-level image inputs onto ``chunk``'s stitched kept window."""
- return [
- image._replace(frame_idx=local)
- for image in images
- if (local := chunk.layout.local_frame_index(image.frame_idx)) is not None
- ]
+ local_images: list[LoadedImageConditioning] = []
+ for image in images:
+ local_image = image.map_frame_index(chunk.layout.local_frame_index)
+ if local_image is not None:
+ local_images.append(local_image)
+ return local_images
def image_conditionings_for_chunk(
chunk: Chunk,
*,
- images: Sequence[ImageConditioningInput],
+ images: Sequence[LoadedImageConditioning],
video_encoder: VideoEncoder,
- color_space: EXRColorSpace | None = None,
) -> list[ConditioningItem]:
"""Encode image conditionings that land in ``chunk``, at its current latent spatial size."""
local = images_for_chunk(chunk, images)
if not local or chunk.video is None:
return []
height, width = _pixel_hw(chunk)
- return combined_image_conditionings(
- images=local,
+ return image_conditionings(
+ images=list(local),
height=height,
width=width,
video_encoder=video_encoder,
dtype=chunk.video.dtype,
device=chunk.video.device,
- color_space=color_space,
)
@@ -155,10 +153,7 @@ def audio_reference_conditionings_for_chunk(
strength: float = 1.0,
negative_positions: bool = True,
) -> list[ConditioningItem]:
- """Build audio-reference tokens for ``chunk``, sliced to its layout window.
- ``latent`` is the full-clip VAE latent. ``latent is None`` means use the chunk's own
- audio latent (stage-2 self-reference).
- """
+ """Build audio-reference tokens for ``chunk``, sliced to its layout window."""
source = chunk.audio if latent is None else audio_latent_for_layout(latent, chunk.layout, fps)
if source is None:
raise ValueError("audio reference with latent=None requires chunk.audio")
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/chunks/operators.py b/packages/ltx-pipelines/src/ltx_pipelines/chunks/operators.py
index cd3cdcc..cd43294 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/chunks/operators.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/chunks/operators.py
@@ -270,6 +270,8 @@ def _decode_chunk_video(
decoded_batches = list(video_decoder(chunk.video, tiling_config, generator, **decode_kwargs))
if not decoded_batches:
return None
+ if len(decoded_batches) == 1:
+ return decoded_batches[0]
return torch.cat(decoded_batches, dim=0)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/dfr_helpers/lanczos.py b/packages/ltx-pipelines/src/ltx_pipelines/dfr_helpers/lanczos.py
new file mode 100644
index 0000000..abcb983
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/dfr_helpers/lanczos.py
@@ -0,0 +1,95 @@
+"""Pillow's 8-bit Lanczos-x2 resample as torch ops, so it runs on the frames' own device.
+Pillow resizes in two separable passes of fixed-point Lanczos-3 weights (22 fractional bits),
+rounding and clamping to a byte after each pass. The same integer arithmetic in float64 is exact
+(every product and partial sum stays below 2**53), so the bytes match Pillow's on any device.
+"""
+
+from __future__ import annotations
+
+import math
+from functools import lru_cache
+
+import torch
+
+from ltx_core.devices import upload_from_host
+
+FIXED_POINT_BITS = 22
+LANCZOS_SUPPORT = 3.0
+
+
+def rgb_to_uint8_fhwc(frames: torch.Tensor) -> torch.Tensor:
+ """``(F, H, W, C)`` RGB in ``[0, 1]`` to bytes, rounding the way the Pillow path does on the host."""
+ return (frames.detach().float().clamp(0, 1) * 255.0).round().to(torch.uint8)
+
+
+def _lanczos3(x: float) -> float:
+ if not -LANCZOS_SUPPORT <= x < LANCZOS_SUPPORT:
+ return 0.0
+
+ def sinc(v: float) -> float:
+ if v == 0.0:
+ return 1.0
+ v *= math.pi
+ return math.sin(v) / v
+
+ return sinc(x) * sinc(x / LANCZOS_SUPPORT)
+
+
+@lru_cache(maxsize=8)
+def _lanczos_weights(in_size: int, out_size: int) -> torch.Tensor:
+ """Pillow's fixed-point Lanczos-3 taps as a ``(2, out_size, taps)`` float64 tensor.
+ ``[0]`` is each tap's source index and ``[1]`` its weight, normalised per output row and
+ rounded to ``FIXED_POINT_BITS``. A row with fewer taps than the widest pads with weight 0
+ on index 0. Indices are small integers, exact in float64, so one tensor carries both.
+ """
+ scale = in_size / out_size
+ filterscale = max(scale, 1.0)
+ support = LANCZOS_SUPPORT * filterscale
+ ss = 1.0 / filterscale
+ rows: list[tuple[int, list[float]]] = []
+ for xx in range(out_size):
+ center = (xx + 0.5) * scale
+ xmin = max(int(center - support + 0.5), 0)
+ xmax = min(int(center + support + 0.5), in_size) - xmin
+ k = [_lanczos3((x + xmin - center + 0.5) * ss) for x in range(xmax)]
+ total = sum(k)
+ fixed = []
+ for w in k:
+ weight = w / total if total != 0.0 else w
+ v = weight * (1 << FIXED_POINT_BITS)
+ fixed.append(float(int(-0.5 + v) if weight < 0 else int(0.5 + v)))
+ rows.append((xmin, fixed))
+ taps = torch.zeros(2, out_size, max(len(fixed) for _, fixed in rows), dtype=torch.float64)
+ for xx, (xmin, fixed) in enumerate(rows):
+ taps[0, xx, : len(fixed)] = torch.arange(xmin, xmin + len(fixed), dtype=torch.float64)
+ taps[1, xx, : len(fixed)] = torch.tensor(fixed, dtype=torch.float64)
+ return taps
+
+
+def _fixed_point_pass(v: torch.Tensor, k: torch.Tensor) -> torch.Tensor:
+ """Resample the last axis of ``v`` with taps ``k``: fixed-point accumulate, rounding shift, byte clamp.
+ One gather and multiply-add per tap; every product and partial sum is an integer below
+ 2**53, so the float64 sum is exact in any order.
+ """
+ index = k[0].long()
+ acc = torch.full((*v.shape[:-1], k.shape[1]), float(1 << (FIXED_POINT_BITS - 1)), dtype=v.dtype, device=v.device)
+ for tap in range(k.shape[2]):
+ acc.addcmul_(v[..., index[:, tap]], k[1, :, tap])
+ return torch.floor(acc / float(1 << FIXED_POINT_BITS)).clamp_(0.0, 255.0)
+
+
+@lru_cache(maxsize=8)
+def _device_lanczos_weights(in_size: int, out_size: int, device: torch.device) -> torch.Tensor:
+ """:func:`_lanczos_weights` on ``device``, uploaded once without waiting for the device."""
+ return upload_from_host(_lanczos_weights(in_size, out_size), device)
+
+
+def lanczos_x2_u8_hwc(frame: torch.Tensor) -> torch.Tensor:
+ """``(H, W, C)`` uint8 to ``(2H, 2W, C)`` uint8, byte-identical to ``Image.resize((2W, 2H), LANCZOS)``."""
+ height, width, _ = frame.shape
+ kx = _device_lanczos_weights(width, 2 * width, frame.device)
+ ky = _device_lanczos_weights(height, 2 * height, frame.device)
+ x = frame.to(torch.float64).permute(2, 0, 1)
+ x = _fixed_point_pass(x, kx)
+ x = _fixed_point_pass(x.transpose(1, 2), ky).transpose(1, 2)
+ return x.permute(1, 2, 0).to(torch.uint8)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/dfr_helpers/ops.py b/packages/ltx-pipelines/src/ltx_pipelines/dfr_helpers/ops.py
index cdc9fca..4cbdfc5 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/dfr_helpers/ops.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/dfr_helpers/ops.py
@@ -5,6 +5,7 @@ No ``DiffusionStage`` / Gemma / ``ModelPaths``. Imported by ``dfr_stages`` and `
from __future__ import annotations
import logging
+import math
from collections.abc import Iterable, Sequence
from dataclasses import dataclass, replace
@@ -15,7 +16,8 @@ from PIL import Image
from ltx_core.conditioning import ConditioningItem, VideoConditionByKeyframeIndex, VideoConditionByLatentIndex
from ltx_core.tiling import DimensionTilingConfig, TileCountConfig
from ltx_core.types import Audio, AudioLatentShape, VideoPixelShape
-from ltx_pipelines.utils.types import ImageConditioningInput
+from ltx_pipelines.dfr_helpers.lanczos import lanczos_x2_u8_hwc, rgb_to_uint8_fhwc
+from ltx_pipelines.utils.types import LoadedImageConditioning
logger = logging.getLogger(__name__)
@@ -36,32 +38,43 @@ def conditioning_fps(playback_fps: float) -> float:
return MAX_CONDITIONING_FPS if playback_fps > SNAP_CONDITIONING_FPS_ABOVE else playback_fps
-# HDR stage-2 spatial overlap. Temporal windows are sequential and pinned; spatial tiles inside
-# a time window blend after every Euler step; time windows are not blended. The first detailing
-# step is 2x2; remaining steps retile to 4x4. A two-value schedule stays 2x2.
-EPILOGUE_SPATIAL_OVERLAP = 10
-EPILOGUE_SPATIAL_COARSE_TILES = 2
-EPILOGUE_SPATIAL_TILES = 4
+# Spatial epilogue tiling. Temporal windows are sequential and pinned and are not blended; spatial
+# tiles inside a time window blend after every Euler step. Every step uses one grid of tiles at most
+# EPILOGUE_TILE_CELLS latent cells per side, overlapping by EPILOGUE_SPATIAL_OVERLAP cells: the blend
+# only has to hide two tiles' per-step disagreement on the shared cells, which 2 cells (64 px) keep
+# within ~0.5 dB of the rest of the frame.
+EPILOGUE_SPATIAL_OVERLAP = 2
+EPILOGUE_TILE_CELLS = 32
EPILOGUE_KEYFRAME_STRENGTH = 1.0
DETAILING_LORA_STRENGTH = 0.5
+def epilogue_tile_count(cells: int) -> int:
+ """Tiles along a latent axis of ``cells`` cells: the fewest whose overlapping tiles fit ``EPILOGUE_TILE_CELLS``."""
+ stride = EPILOGUE_TILE_CELLS - EPILOGUE_SPATIAL_OVERLAP
+ return max(1, math.ceil((cells - EPILOGUE_SPATIAL_OVERLAP) / stride))
+
+
def lanczos_x2_fhwc(frames: torch.Tensor) -> torch.Tensor:
- """Stretch each RGB frame 2x with Lanczos. ``frames`` is ``(F, H, W, C)`` in ``[0, 1]``."""
+ """Stretch each RGB frame 2x with Lanczos. ``frames`` is ``(F, H, W, C)`` in ``[0, 1]``.
+ CUDA frames are resampled on their device by :mod:`ltx_pipelines.dfr_helpers.lanczos`, CPU
+ frames by Pillow; both quantize to bytes first and return the same pixels.
+ """
if frames.ndim != 4:
raise ValueError(f"Expected (F, H, W, C) RGB, got shape {tuple(frames.shape)}")
if frames.shape[0] < 1:
raise ValueError("Need at least one frame to Lanczos-upsample")
+ if frames.shape[-1] not in (1, 3):
+ raise ValueError(f"Lanczos x2 expects 1 or 3 channels, got {frames.shape[-1]}")
+ if frames.is_cuda:
+ resized = torch.stack([lanczos_x2_u8_hwc(frame) for frame in rgb_to_uint8_fhwc(frames)]).float()
+ # A tensor divisor keeps this an IEEE division, as on the host (a Python scalar becomes a reciprocal multiply).
+ return resized / torch.full((), 255.0, device=resized.device)
out: list[torch.Tensor] = []
for frame in frames:
height, width, channels = frame.shape
- array = (frame.detach().float().clamp(0, 1).cpu().numpy() * 255.0).round().astype(np.uint8)
- if channels == 1:
- image = Image.fromarray(array[..., 0], mode="L")
- elif channels == 3:
- image = Image.fromarray(array, mode="RGB")
- else:
- raise ValueError(f"Lanczos x2 expects 1 or 3 channels, got {channels}")
+ array = rgb_to_uint8_fhwc(frame).numpy()
+ image = Image.fromarray(array[..., 0], mode="L") if channels == 1 else Image.fromarray(array, mode="RGB")
image = image.resize((width * 2, height * 2), resample=Image.Resampling.LANCZOS)
resized = torch.from_numpy(np.asarray(image, dtype=np.float32) / 255.0)
if resized.ndim == 2:
@@ -279,22 +292,28 @@ def lead_in_carryover(
def rebase_image_conditionings(
- images: Sequence[ImageConditioningInput],
+ images: Sequence[LoadedImageConditioning],
*,
pixel_scale: int,
pixel_start: int = 0,
pixel_end: int | None = None,
-) -> list[ImageConditioningInput]:
+) -> list[LoadedImageConditioning]:
"""Map user ``frame_idx`` values from the requested canvas onto a temporally-upsampled grid.
After ``r`` temporal rounds the same moment sits at ``frame_idx * 2**r``. When ``pixel_end`` is
set, only images inside ``[pixel_start, pixel_end]`` are kept and indices become tile-local.
"""
- rebased: list[ImageConditioningInput] = []
- for image in images:
- scaled = image.frame_idx * pixel_scale
+
+ def map_frame(frame_idx: int) -> int | None:
+ scaled = frame_idx * pixel_scale
if pixel_end is not None and not (pixel_start <= scaled <= pixel_end):
- continue
- rebased.append(ImageConditioningInput(image.path, scaled - pixel_start, image.strength, image.crf))
+ return None
+ return scaled - pixel_start
+
+ rebased: list[LoadedImageConditioning] = []
+ for image in images:
+ rebased_image = image.map_frame_index(map_frame)
+ if rebased_image is not None:
+ rebased.append(rebased_image)
return rebased
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/dfr_mgpu.py b/packages/ltx-pipelines/src/ltx_pipelines/dfr_mgpu.py
index 785041d..963b0e9 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/dfr_mgpu.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/dfr_mgpu.py
@@ -13,8 +13,8 @@ from __future__ import annotations
import logging
from collections.abc import Callable, Iterator, Sequence
from contextlib import contextmanager, nullcontext
-from functools import wraps
from multiprocessing import SimpleQueue
+from types import SimpleNamespace
from typing import Any
import torch
@@ -24,21 +24,23 @@ from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy
from ltx_core.loader import LoraPathStrengthAndSDOps
from ltx_core.loader.registry import ModelRegistry
from ltx_core.model.transformer.compiling import CompilationConfig
-from ltx_core.model.video_vae import AUTO_TILING, AutoTiling, TilingConfig, get_video_chunks_number
+from ltx_core.model.video_vae import AUTO_TILING, get_video_chunks_number
from ltx_core.model.video_vae.transformer import DiffVAEMode
from ltx_core.multigpu.transformer.attention import AttentionManager
-from ltx_core.multigpu.vae.distributed_decoder import dist_rank_tile_pixel_shape
from ltx_core.quantization import QuantizationPolicy
from ltx_core.quantization.fp8_cast import build_policy as _build_fp8_cast_policy
from ltx_core.tiling import DimensionTilingConfig, TileCountConfig, balanced_tile_split
-from ltx_core.types import VIDEO_SCALE_FACTORS, SpatioTemporalScaleFactors, VideoLatentShape, VideoPixelShape
+from ltx_core.types import VIDEO_SCALE_FACTORS, SpatioTemporalScaleFactors
from ltx_pipelines.dfr_pipeline import DFRPipeline, add_dfr_cli_args
from ltx_pipelines.multigpu.controller import MGPUController
+from ltx_pipelines.multigpu.decode_tiling import dist_auto_tiling
from ltx_pipelines.multigpu.gemma_builders import AccelerateGemmaBuilder
from ltx_pipelines.multigpu.runner import MGPURunner
from ltx_pipelines.multigpu.sp_builder import SequenceParallelBuilder
from ltx_pipelines.multigpu.vae_builders import DistributedDecoderBuilder
from ltx_pipelines.multigpu.weight_tracker import TransformerWeightTracker
+from ltx_pipelines.utils.args import markers_from_args
+from ltx_pipelines.utils.helpers import get_device
from ltx_pipelines.utils.media_io import encode_video
from ltx_pipelines.utils.model_paths import ModelPaths
from ltx_pipelines.utils.types import DEFAULT_AUTO_DURATION, AutoDuration
@@ -62,62 +64,20 @@ _DEFAULT_VAE_TEMPORAL_BATCHES = 16
def _dfr_dist_tile_auto_tiling(
vae_tiling: TileCountConfig,
*,
- rank: int,
world_size: int,
+ group: dist.ProcessGroup | None,
scale: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,
) -> Iterator[None]:
- """Rewrite ``DFRPipeline`` AUTO recommend to this rank's Dist-tile pixel shape.
+ """Plan ``DFRPipeline``'s AUTO decode tiling for the group's Dist tiles.
Patches ``ltx_pipelines.dfr_pipeline.ensure_tiling_config`` (the name
- ``DFRPipeline.__call__`` looks up) for the duration of the ``with`` block.
- Dist stays a pass-through; the pipeline still returns the resolved
- ``TilingConfig``. Explicit configs and ``None`` are unchanged.
+ ``DFRPipeline.__call__`` looks up) with :func:`dist_auto_tiling` for the duration of the
+ ``with`` block, so every rank resolves the same ``TilingConfig``. Explicit configs and
+ ``None`` are unchanged.
"""
import ltx_pipelines.dfr_pipeline as dfr_mod # noqa: PLC0415
orig = dfr_mod.ensure_tiling_config
-
- @wraps(orig)
- def ensure_tiling_config_dist_tile(
- tiling_config: TilingConfig | AutoTiling | None,
- *,
- video_shape: VideoPixelShape,
- scale_factors: SpatioTemporalScaleFactors,
- vae_checkpoint_path: str,
- diffvae_optimization: DiffVAEMode = DiffVAEMode.CHUNKED_EAGER,
- device: torch.device | None = None,
- free_bytes: int | None = None,
- keyframes: bool = False,
- ) -> TilingConfig | None:
- if isinstance(tiling_config, AutoTiling):
- latent_shape = VideoLatentShape.from_pixel_shape(video_shape, scale_factors=scale)
- tile_shape = dist_rank_tile_pixel_shape(
- vae_tiling,
- latent_shape,
- scale,
- rank,
- world_size,
- fps=video_shape.fps,
- batch=video_shape.batch,
- )
- logger.info(
- "rank=%s AUTO_TILING Dist tile video_shape=%s (full canvas %s)",
- rank,
- tile_shape,
- video_shape,
- )
- video_shape = tile_shape
- return orig(
- tiling_config,
- video_shape=video_shape,
- scale_factors=scale_factors,
- vae_checkpoint_path=vae_checkpoint_path,
- diffvae_optimization=diffvae_optimization,
- device=device,
- free_bytes=free_bytes,
- keyframes=keyframes,
- )
-
- dfr_mod.ensure_tiling_config = ensure_tiling_config_dist_tile
+ dfr_mod.ensure_tiling_config = dist_auto_tiling(orig, vae_tiling, world_size=world_size, group=group, scale=scale)
try:
yield
finally:
@@ -244,15 +204,20 @@ class DFRRunner(MGPURunner):
enhance_static_cache: bool = False,
temporal_upscalings: int = 0,
spatial_upscalings: int = 1,
+ marker_root_path: str | None = None,
) -> Iterator[str | None]:
# The pipeline raises ValueError on invalid input (symmetric across ranks); the controller
# catches that and turns it into a recoverable RunnerError. Anything else is fatal.
+ video_marker, audio_marker = markers_from_args(
+ SimpleNamespace(marker_root_path=marker_root_path),
+ device=get_device(),
+ )
vae_tiling = self._vae_tiling
auto_tile = (
_dfr_dist_tile_auto_tiling(
vae_tiling,
- rank=dist.get_rank(self.groups.vae_group),
world_size=dist.get_world_size(self.groups.vae_group),
+ group=self.groups.vae_group,
)
if vae_tiling is not None
else nullcontext()
@@ -282,12 +247,19 @@ class DFRRunner(MGPURunner):
audio=result.audio,
output_path=output_path,
video_chunks_number=get_video_chunks_number(result.num_frames, result.tiling_config),
+ video_marker=video_marker,
+ audio_marker=audio_marker,
+ # The distributed decoder's batches are complete on every stream when handed out.
+ chunks_complete=True,
)
yield output_path
if __name__ == "__main__":
- from ltx_pipelines.utils.args import default_2_stage_distilled_arg_parser, resolve_cli_params
+ from ltx_pipelines.utils.args import (
+ default_2_stage_distilled_arg_parser,
+ resolve_cli_params,
+ )
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
params = resolve_cli_params(distilled=True)
@@ -321,6 +293,7 @@ if __name__ == "__main__":
enhance_static_cache=args.enhance_static_cache,
temporal_upscalings=args.temporal_upscalings,
spatial_upscalings=args.spatial_upscalings,
+ marker_root_path=args.marker_root_path,
):
pass # drive the job to completion; the runner writes the file as a side effect
finally:
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/dfr_pipeline.py b/packages/ltx-pipelines/src/ltx_pipelines/dfr_pipeline.py
index 146b3c8..60c022a 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/dfr_pipeline.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/dfr_pipeline.py
@@ -22,7 +22,8 @@ from ltx_core.model.video_vae import (
get_video_chunks_number,
)
from ltx_core.model.video_vae.transformer import DiffVAEMode
-from ltx_core.quantization import QuantizationPolicy
+from ltx_core.quantization import QuantizationPolicy, fp8_cast_fuse_rule
+from ltx_core.quantization.fp8_cast import supports_fp8_gemm, with_rowwise_fp8_gemm
from ltx_core.types import VIDEO_SCALE_FACTORS, VideoPixelShape
from ltx_pipelines.dfr_helpers.layout import resolve_canvas
from ltx_pipelines.dfr_helpers.ops import (
@@ -47,6 +48,7 @@ from ltx_pipelines.iclora_utils import read_lora_reference_downscale_factor
from ltx_pipelines.utils.args import (
LoraAction,
default_2_stage_distilled_arg_parser,
+ markers_from_args,
resolve_cli_params,
resolve_existing_path,
)
@@ -241,6 +243,19 @@ class DFRPipeline:
fps=fps,
)
+ def _epilogue_stage(self) -> DiffusionStage:
+ """The detailing stage the spatial epilogue denoises with.
+ On an fp8-cast policy and a device with fp8 GEMMs, its long video projections multiply
+ e4m3 activations (:func:`~ltx_core.quantization.fp8_cast.with_rowwise_fp8_gemm`): the
+ epilogue is the last pass, where the precision costs least. Derived per call so it
+ follows whatever builder the stage carries (e.g. a multi-GPU runner's).
+ """
+ stage = self.stage_detailing
+ policy = stage.quantization
+ if policy is None or policy.fuse_rule is not fp8_cast_fuse_rule or not supports_fp8_gemm(self.device):
+ return stage
+ return stage.with_quantization(with_rowwise_fp8_gemm(policy))
+
def __call__( # noqa: PLR0913, PLR0915
self,
prompt: str,
@@ -267,7 +282,8 @@ class DFRPipeline:
raise ValueError(f"spatial_upscalings must be 1 or 2, got {spatial_upscalings}")
require_num_frames_source(num_frames, self.duration_predictor)
- images = self.image_conditioner.resolve_crf(images)
+ enhance_prompt_image = images[0].path if images else None
+ images = self.image_conditioner.load_conditionings(images)
assert_resolution(
height=height,
width=width,
@@ -284,7 +300,7 @@ class DFRPipeline:
[prompt],
enhance_first_prompt=enhance_prompt,
enhance_static_cache=enhance_static_cache,
- enhance_prompt_image=images[0][0] if len(images) > 0 else None,
+ enhance_prompt_image=enhance_prompt_image,
)
video_context, audio_context = ctx_p.video_encoding, ctx_p.audio_encoding
@@ -355,47 +371,49 @@ class DFRPipeline:
)
last_window_seams: list[int] = []
- for round_idx in range(1, temporal_upscalings + 1):
- assert self.temporal_upsampler is not None
- if not keyframes:
- raise RuntimeError(f"Temporal round {round_idx}: missing carry-forward keyframes")
- video_shape = video_shape._replace(frames=2 * (video_shape.frames - 1) + 1, fps=2 * video_shape.fps)
- video_state, keyframes, last_window_seams = run_one_temporal_round(
- round_idx=round_idx,
- stage=self.stage,
- temporal_upsampler=self.temporal_upsampler,
- image_conditioner=self.image_conditioner,
- video_state=video_state,
- keyframes=keyframes,
- video_shape=video_shape,
- images=images,
- video_context=video_context,
- audio_context=audio_context,
- audio_latent=audio_latent,
- source_duration=source_duration,
- seed=seed,
- noiser=noiser,
- sigmas=DISTILLED_SIGMAS[4:].to(dtype=torch.float32, device=self.device),
- device=self.device,
- dtype=dtype,
- temporal_scale=temporal_scale,
- )
- self._emit_stage(
- f"t{round_idx}",
- latent=video_state.latent,
- keyframes=keyframes,
- height=video_shape.height,
- width=video_shape.width,
- num_frames=video_shape.frames,
- fps=video_shape.fps,
- )
+ # One transformer for every temporal tile of every round instead of a build per tile.
+ with self.stage.resident():
+ for round_idx in range(1, temporal_upscalings + 1):
+ assert self.temporal_upsampler is not None
+ if not keyframes:
+ raise RuntimeError(f"Temporal round {round_idx}: missing carry-forward keyframes")
+ video_shape = video_shape._replace(frames=2 * (video_shape.frames - 1) + 1, fps=2 * video_shape.fps)
+ video_state, keyframes, last_window_seams = run_one_temporal_round(
+ round_idx=round_idx,
+ stage=self.stage,
+ temporal_upsampler=self.temporal_upsampler,
+ image_conditioner=self.image_conditioner,
+ video_state=video_state,
+ keyframes=keyframes,
+ video_shape=video_shape,
+ images=images,
+ video_context=video_context,
+ audio_context=audio_context,
+ audio_latent=audio_latent,
+ source_duration=source_duration,
+ seed=seed,
+ noiser=noiser,
+ sigmas=DISTILLED_SIGMAS[4:].to(dtype=torch.float32),
+ device=self.device,
+ dtype=dtype,
+ temporal_scale=temporal_scale,
+ )
+ self._emit_stage(
+ f"t{round_idx}",
+ latent=video_state.latent,
+ keyframes=keyframes,
+ height=video_shape.height,
+ width=video_shape.width,
+ num_frames=video_shape.frames,
+ fps=video_shape.fps,
+ )
num_frames = video_shape.frames
current_fps = video_shape.fps
# --- Spatial epilogue (spatial_upscalings == 2): s2 is h/2; this pass is final H x W ---
if spatial_upscalings == 2:
epi = run_spatial_epilogue(
- stage=self.stage_detailing,
+ stage=self._epilogue_stage(),
upsampler=self.upsampler,
image_conditioner=self.image_conditioner,
video_decoder=self.video_decoder,
@@ -498,6 +516,7 @@ def main() -> None:
params = resolve_cli_params(distilled=True)
parser = add_dfr_cli_args(default_2_stage_distilled_arg_parser(params=params, supports_auto_duration=True))
args = parser.parse_args()
+ video_marker, audio_marker = markers_from_args(args, device=get_device())
pipeline = DFRPipeline(
model_paths=args.model_paths,
@@ -532,6 +551,8 @@ def main() -> None:
audio=result.audio,
output_path=args.output_path,
video_chunks_number=get_video_chunks_number(result.num_frames, result.tiling_config),
+ video_marker=video_marker,
+ audio_marker=audio_marker,
)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/dfr_stages.py b/packages/ltx-pipelines/src/ltx_pipelines/dfr_stages.py
index f6aa71e..6dc32bf 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/dfr_stages.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/dfr_stages.py
@@ -21,6 +21,7 @@ import torch
from ltx_core.components.noisers import GaussianNoiser, Noiser
from ltx_core.conditioning import ConditioningItem, VideoConditionByReferenceLatent, VideoGeneratedKeyframeSlots
+from ltx_core.devices import upload_from_host
from ltx_core.model.video_vae import VideoEncoder
from ltx_core.tiling import DimensionTilingConfig, TileCountConfig
from ltx_core.tools import LatentTools, VideoLatentTools
@@ -29,14 +30,13 @@ from ltx_pipelines.dfr_helpers.layout import TemporalTile, TemporalTilePlan
from ltx_pipelines.dfr_helpers.ops import (
ANCHOR_KEYFRAME_STRENGTH,
EPILOGUE_KEYFRAME_STRENGTH,
- EPILOGUE_SPATIAL_COARSE_TILES,
EPILOGUE_SPATIAL_OVERLAP,
- EPILOGUE_SPATIAL_TILES,
TEMPORAL_ANCESTRAL_ETA,
TilePrefix,
audio_latent_for_tile,
clamp_tile_to_latent,
conditioning_fps,
+ epilogue_tile_count,
keyframe_conditionings_from_latents,
lanczos_x2_fhwc,
lead_in_carryover,
@@ -47,14 +47,14 @@ from ltx_pipelines.dfr_helpers.ops import (
from ltx_pipelines.utils.blocks import DiffusionStage, ImageConditioner, VideoDecoder, VideoUpsampler
from ltx_pipelines.utils.denoisers import SimpleDenoiser
from ltx_pipelines.utils.helpers import (
- combined_image_conditionings,
create_initial_audio_latent,
create_initial_video_latent,
+ image_conditionings,
)
from ltx_pipelines.utils.media_io import to_vae_range
from ltx_pipelines.utils.samplers import euler_ancestral_denoising_loop
from ltx_pipelines.utils.tiled_diffusion import TiledDiffusionModel
-from ltx_pipelines.utils.types import ImageConditioningInput, ModalitySpec, VideoAudio
+from ltx_pipelines.utils.types import LoadedImageConditioning, ModalitySpec, VideoAudio
logger = logging.getLogger(__name__)
@@ -152,7 +152,7 @@ def _stack_keyframes(keyframes: KeyframeCarry) -> tuple[list[int], torch.Tensor]
def _image_conditionings(
image_conditioner: ImageConditioner,
- images: Sequence[ImageConditioningInput],
+ images: Sequence[LoadedImageConditioning],
*,
height: int,
width: int,
@@ -160,7 +160,7 @@ def _image_conditionings(
device: torch.device,
) -> list[ConditioningItem]:
return image_conditioner(
- lambda enc: combined_image_conditionings(
+ lambda enc: image_conditionings(
images=list(images),
height=height,
width=width,
@@ -174,7 +174,7 @@ def _image_conditionings(
def _prepare_stage1_specs(
*,
image_conditioner: ImageConditioner,
- images: list[ImageConditioningInput],
+ images: list[LoadedImageConditioning],
video_shape: VideoPixelShape,
positions: Sequence[int],
video_context: torch.Tensor,
@@ -229,7 +229,7 @@ def _prepare_stage2_specs( # noqa: PLR0913
*,
upsampler: VideoUpsampler,
image_conditioner: ImageConditioner,
- images: list[ImageConditioningInput],
+ images: list[LoadedImageConditioning],
video_shape: VideoPixelShape,
keyframes: KeyframeCarry,
half_res_reference: torch.Tensor,
@@ -287,7 +287,7 @@ def denoise_stage1( # noqa: PLR0913
*,
stage: DiffusionStage,
image_conditioner: ImageConditioner,
- images: list[ImageConditioningInput],
+ images: list[LoadedImageConditioning],
video_shape: VideoPixelShape,
positions: Sequence[int],
video_context: torch.Tensor,
@@ -306,7 +306,8 @@ def denoise_stage1( # noqa: PLR0913
``1.0``, which would noise the latent before freezing it).
Transformer fps snaps above 30; audio duration stays at playback ``video_shape.fps``.
"""
- sigmas = sigmas.to(dtype=torch.float32, device=device)
+ # Kept on the host: the loop uploads the schedule, and host reads of it stay free.
+ sigmas = sigmas.to(dtype=torch.float32)
prepared = _prepare_stage1_specs(
image_conditioner=image_conditioner,
images=images,
@@ -344,7 +345,7 @@ def denoise_stage2( # noqa: PLR0913
stage: DiffusionStage,
image_conditioner: ImageConditioner,
upsampler: VideoUpsampler,
- images: list[ImageConditioningInput],
+ images: list[LoadedImageConditioning],
video_shape: VideoPixelShape,
keyframes: KeyframeCarry,
half_res_reference: torch.Tensor,
@@ -362,7 +363,8 @@ def denoise_stage2( # noqa: PLR0913
Upsamples ``half_res_reference`` and ``keyframes`` as its first step, matching temporal
rounds and the epilogue. Audio is re-noised for cross-modal attention only.
"""
- sigmas = sigmas.to(dtype=torch.float32, device=device)
+ # Kept on the host: the loop uploads the schedule, and host reads of it stay free.
+ sigmas = sigmas.to(dtype=torch.float32)
prepared = _prepare_stage2_specs(
upsampler=upsampler,
image_conditioner=image_conditioner,
@@ -425,7 +427,7 @@ def _temporal_tile_video_input(
def _temporal_tile_conditionings( # noqa: PLR0913
*,
image_conditioner: ImageConditioner,
- images: list[ImageConditioningInput],
+ images: list[LoadedImageConditioning],
canvas: VideoPixelShape,
tile_video: TemporalTileVideo,
tile: TemporalTile,
@@ -499,7 +501,7 @@ def _temporal_tile_conditionings( # noqa: PLR0913
def _prepare_temporal_tile_specs( # noqa: PLR0913
*,
image_conditioner: ImageConditioner,
- images: list[ImageConditioningInput],
+ images: list[LoadedImageConditioning],
canvas: VideoPixelShape,
video_latent: torch.Tensor,
tile_index: int,
@@ -606,7 +608,7 @@ def _denoise_temporal_tile( # noqa: PLR0913
*,
stage: DiffusionStage,
image_conditioner: ImageConditioner,
- images: list[ImageConditioningInput],
+ images: list[LoadedImageConditioning],
canvas: VideoPixelShape,
video_latent: torch.Tensor,
tile_index: int,
@@ -683,7 +685,7 @@ def run_one_temporal_round( # noqa: PLR0913
video_state: LatentState,
keyframes: KeyframeCarry,
video_shape: VideoPixelShape,
- images: list[ImageConditioningInput],
+ images: list[LoadedImageConditioning],
video_context: torch.Tensor,
audio_context: torch.Tensor,
audio_latent: torch.Tensor,
@@ -751,51 +753,47 @@ def run_one_temporal_round( # noqa: PLR0913
return next_video_state, dict(sorted(plane_at.items())), seam_positions
-def _decode_carry_keyframes(
+def _rebuild_carry_keyframes(
video_decoder: VideoDecoder,
+ image_conditioner: ImageConditioner,
keyframes: KeyframeCarry,
*,
seed: int,
device: torch.device,
-) -> dict[int, torch.Tensor]:
- """Decode each carry keyframe as its own 1-frame RGB clip.
- The VAE is causal, so a stacked encode/decode would bleed neighbouring planes. Each
- plane is a standalone one-pixel-frame latent, matching ``DecodeKeyframes``. Dist
- ``decode_single_frames`` splits the planes across the ranks and gathers them back, so
- every rank gets every plane instead of the empty worker yield from ``decode_video``.
- Generators are per plane, which keeps the pixels independent of the world size.
+ dtype: torch.dtype,
+) -> KeyframeCarry:
+ """Decode, Lanczos-x2 and re-encode each carry plane on the rank that owns it.
+ Each plane is a standalone one-pixel-frame clip with its own generator, so its pixels do
+ not depend on the world size. A Dist decoder splits the planes round-robin over the group;
+ each rank resamples and encodes its own planes and only the small encoded latents cross the
+ group. The Lanczos runs on the decode device right after each plane; every encode is queued
+ before any read-back. Returns every plane on every rank.
"""
for position, plane in keyframes.items():
if plane.ndim != 5 or plane.shape[2] != 1:
raise ValueError(f"Expected one keyframe plane at {position}, got {tuple(plane.shape)}")
- n_planes = len(keyframes)
- logger.info("Epilogue: decoding %d carry keyframes separately", n_planes)
- planes = list(keyframes.values())
- generators = [torch.Generator(device=device).manual_seed(seed + 4000 + index) for index in range(n_planes)]
- rgb_planes: list[torch.Tensor] = []
- for index, rgb in enumerate(video_decoder.decode_single_frames(planes, generators)):
+ positions = list(keyframes)
+ logger.info("Epilogue: decoding %d carry keyframes separately", len(positions))
+ generators = [torch.Generator(device=device).manual_seed(seed + 4000 + index) for index in range(len(positions))]
+ owned = video_decoder.decode_owned_single_frames(list(keyframes.values()), generators)
+
+ resampled: dict[int, torch.Tensor] = {}
+ for index, rgb in zip(owned.indices, owned.planes, strict=True):
if rgb.shape[0] < 1:
raise RuntimeError(f"Decoder returned no pixels for carry keyframe {index}")
- rgb_planes.append(rgb.detach().cpu())
- if len(rgb_planes) != n_planes:
- raise RuntimeError(f"Expected {n_planes} decoded carry keyframes, got {len(rgb_planes)}")
- return dict(zip(keyframes, rgb_planes, strict=True))
-
-
-def _encode_keyframe_pixels(
- enc: VideoEncoder,
- pixel_keyframes: dict[int, torch.Tensor],
- *,
- dtype: torch.dtype,
- device: torch.device,
-) -> KeyframeCarry:
- """Encode each Lanczos RGB clip as a one-frame latent plane."""
+ resampled[index] = lanczos_x2_fhwc(rgb.detach())
- def encode(rgb: torch.Tensor) -> torch.Tensor:
- sample = to_vae_range(rgb.to(device=device, dtype=torch.float32))
- return enc(sample.permute(3, 0, 1, 2).unsqueeze(0).contiguous().to(dtype=dtype))
+ def encode(enc: VideoEncoder) -> dict[int, torch.Tensor]:
+ on_device = {}
+ for index, rgb in resampled.items():
+ sample = to_vae_range(rgb.to(device=device, dtype=torch.float32))
+ on_device[index] = enc(sample.permute(3, 0, 1, 2).unsqueeze(0).contiguous().to(dtype=dtype))
+ return {index: plane.cpu() for index, plane in on_device.items()}
- return {position: encode(rgb) for position, rgb in pixel_keyframes.items()}
+ encoded = owned.gather(image_conditioner(encode))
+ if len(encoded) != len(positions):
+ raise RuntimeError(f"Expected {len(positions)} encoded carry keyframes, got {len(encoded)}")
+ return {position: upload_from_host(encoded[index], device) for index, position in enumerate(positions)}
def _rebuild_epilogue_keyframes(
@@ -804,24 +802,21 @@ def _rebuild_epilogue_keyframes(
image_conditioner: ImageConditioner,
keyframes: KeyframeCarry,
video_state: LatentState,
- images: Sequence[ImageConditioningInput],
+ images: Sequence[LoadedImageConditioning],
temporal_upscalings: int,
seed: int,
device: torch.device,
dtype: torch.dtype,
) -> tuple[KeyframeCarry, torch.Tensor | None]:
- """Decode carry planes, Lanczos-x2, and encode. Opening plane if no frame-0 image.
- Decoder is freed before the encoder is built: decode completes, then
- ``image_conditioner`` loads the encoder for the RGB planes only.
- """
+ """Rebuild the carry planes at the epilogue resolution, plus the opening plane if no image conditions frame 0."""
opening_position = -1
to_decode = dict(keyframes)
epi_images = rebase_image_conditionings(images, pixel_scale=2**temporal_upscalings)
if not any(image.frame_idx == 0 for image in epi_images):
to_decode[opening_position] = video_state.latent[:1, :, :1]
- pixel_keyframes = _decode_carry_keyframes(video_decoder, to_decode, seed=seed, device=device)
- pixel_keyframes = {position: lanczos_x2_fhwc(rgb) for position, rgb in pixel_keyframes.items()}
- encoded = image_conditioner(lambda enc: _encode_keyframe_pixels(enc, pixel_keyframes, dtype=dtype, device=device))
+ encoded = _rebuild_carry_keyframes(
+ video_decoder, image_conditioner, to_decode, seed=seed, device=device, dtype=dtype
+ )
opening = encoded.pop(opening_position, None)
return encoded, opening
@@ -1004,7 +999,7 @@ def _execute_tiled_spatial_denoise(
)
video_state, _ = tiled_stage(
denoiser=SimpleDenoiser(video_context, audio_context),
- sigmas=sigmas.to(dtype=torch.float32, device=device),
+ sigmas=sigmas.to(dtype=torch.float32),
noiser=GaussianNoiser(generator=torch.Generator(device=device).manual_seed(seed + 2000)),
modalities=VideoAudio(video=prepared.video, audio=prepared.audio),
loop=loop,
@@ -1046,92 +1041,89 @@ def _run_epilogue_windows( # noqa: PLR0913
temporal_scale: int,
loop: Callable[..., tuple[LatentState | None, LatentState | None]] | None = None,
) -> torch.Tensor:
- """Detail each temporal window with one coarse-grid step, then any remaining steps on the fine grid."""
+ """Detail each temporal window with one step, then any remaining steps, on one spatial tile grid."""
stitched: list[torch.Tensor] = []
previous: tuple[int, torch.Tensor] | None = None
coarse_sigmas, fine_sigmas = _epilogue_sigma_phases(sigmas)
- coarse_tiling = TileCountConfig(
+ tiling = TileCountConfig(
frames=DimensionTilingConfig(1, overlap=0),
- height=DimensionTilingConfig(EPILOGUE_SPATIAL_COARSE_TILES, EPILOGUE_SPATIAL_OVERLAP),
- width=DimensionTilingConfig(EPILOGUE_SPATIAL_COARSE_TILES, EPILOGUE_SPATIAL_OVERLAP),
+ height=DimensionTilingConfig(epilogue_tile_count(int(latent.shape[3])), EPILOGUE_SPATIAL_OVERLAP),
+ width=DimensionTilingConfig(epilogue_tile_count(int(latent.shape[4])), EPILOGUE_SPATIAL_OVERLAP),
)
scale_factors = stage.video_scale_factors
- for index, window in enumerate(windows):
- prepared_window = _prepare_epilogue_window_specs(
- index=index,
- num_windows=len(windows),
- window=window,
- prefix=prefixes[index],
- latent=latent,
- guide=guide,
- plane_at=keyframes,
- window_images=window_images[index],
- keyframes=keyframes,
- opening_plane=opening_plane,
- previous=previous,
- downscale_factor=downscale_factor,
- canvas=canvas,
- audio_latent=audio_latent,
- source_duration=source_duration,
- video_context=video_context,
- audio_context=audio_context,
- noise_scale=float(coarse_sigmas[0].item()),
- temporal_scale=temporal_scale,
- scale_factors=scale_factors,
- device=device,
- dtype=dtype,
- )
- window_latent = _execute_tiled_spatial_denoise(
- stage,
- prepared=prepared_window.denoise,
- latent=prepared_window.window_latent,
- tiling=coarse_tiling,
- sigmas=coarse_sigmas,
- video_context=video_context,
- audio_context=audio_context,
- seed=seed + 100 * index,
- device=device,
- loop=loop,
- )
- if fine_sigmas is not None:
- fine_tiling = TileCountConfig(
- frames=DimensionTilingConfig(1, overlap=0),
- height=DimensionTilingConfig(EPILOGUE_SPATIAL_TILES, EPILOGUE_SPATIAL_OVERLAP),
- width=DimensionTilingConfig(EPILOGUE_SPATIAL_TILES, EPILOGUE_SPATIAL_OVERLAP),
- )
- audio_latent_in = prepared_window.denoise.audio.latent
- if audio_latent_in is None:
- raise RuntimeError("Epilogue window prepared no audio latent")
- fine_prepared = _prepare_tiled_spatial_denoise_specs(
- latent=window_latent,
- conditionings=list(prepared_window.denoise.video.conditionings),
+ # One transformer for every window and tile instead of a build per tile.
+ with stage.resident():
+ for index, window in enumerate(windows):
+ prepared_window = _prepare_epilogue_window_specs(
+ index=index,
+ num_windows=len(windows),
+ window=window,
+ prefix=prefixes[index],
+ latent=latent,
+ guide=guide,
+ plane_at=keyframes,
+ window_images=window_images[index],
+ keyframes=keyframes,
+ opening_plane=opening_plane,
+ previous=previous,
+ downscale_factor=downscale_factor,
+ canvas=canvas,
+ audio_latent=audio_latent,
+ source_duration=source_duration,
video_context=video_context,
audio_context=audio_context,
- audio_latent=audio_latent_in,
- fps=prepared_window.denoise.canvas.fps,
+ noise_scale=float(coarse_sigmas[0].item()),
+ temporal_scale=temporal_scale,
scale_factors=scale_factors,
device=device,
dtype=dtype,
- noise_scale=0.0,
)
window_latent = _execute_tiled_spatial_denoise(
stage,
- prepared=fine_prepared,
- latent=window_latent,
- tiling=fine_tiling,
- sigmas=fine_sigmas,
+ prepared=prepared_window.denoise,
+ latent=prepared_window.window_latent,
+ tiling=tiling,
+ sigmas=coarse_sigmas,
video_context=video_context,
audio_context=audio_context,
seed=seed + 100 * index,
device=device,
loop=loop,
)
- prefix = prepared_window.prefix
- previous = (
- prefix.video_start_cell - 1 if prefix else prepared_window.interval_start,
- window_latent.detach(),
- )
- stitched.append(window_latent[:, :, prepared_window.pinned_cells :])
+ if fine_sigmas is not None:
+ audio_latent_in = prepared_window.denoise.audio.latent
+ if audio_latent_in is None:
+ raise RuntimeError("Epilogue window prepared no audio latent")
+ fine_prepared = _prepare_tiled_spatial_denoise_specs(
+ latent=window_latent,
+ conditionings=list(prepared_window.denoise.video.conditionings),
+ video_context=video_context,
+ audio_context=audio_context,
+ audio_latent=audio_latent_in,
+ fps=prepared_window.denoise.canvas.fps,
+ scale_factors=scale_factors,
+ device=device,
+ dtype=dtype,
+ noise_scale=0.0,
+ )
+ window_latent = _execute_tiled_spatial_denoise(
+ stage,
+ prepared=fine_prepared,
+ latent=window_latent,
+ tiling=tiling,
+ sigmas=fine_sigmas,
+ video_context=video_context,
+ audio_context=audio_context,
+ seed=seed + 100 * index,
+ device=device,
+ loop=loop,
+ )
+ prefix = prepared_window.prefix
+ previous = (
+ prefix.video_start_cell - 1 if prefix else prepared_window.interval_start,
+ window_latent.detach(),
+ )
+ stitched.append(window_latent[:, :, prepared_window.pinned_cells :])
return torch.cat(stitched, dim=2)
@@ -1165,7 +1157,7 @@ def _encode_epilogue_window_images(
*,
windows: TemporalTilePlan,
prefixes: Sequence[TilePrefix | None],
- images: Sequence[ImageConditioningInput],
+ images: Sequence[LoadedImageConditioning],
temporal_upscalings: int,
video_shape: VideoPixelShape,
dtype: torch.dtype,
@@ -1188,7 +1180,7 @@ def _encode_epilogue_window_images(
if image.frame_idx + origin >= resume
]
per_window.append(
- combined_image_conditionings(
+ image_conditionings(
images=inside,
height=video_shape.height,
width=video_shape.width,
@@ -1210,7 +1202,7 @@ def run_spatial_epilogue( # noqa: PLR0913
video_decoder: VideoDecoder,
video_state: LatentState,
keyframes: KeyframeCarry,
- images: Sequence[ImageConditioningInput],
+ images: Sequence[LoadedImageConditioning],
video_shape: VideoPixelShape,
temporal_upscalings: int,
last_window_seams: Sequence[int],
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/distilled.py b/packages/ltx-pipelines/src/ltx_pipelines/distilled.py
index b80b0ee..0db6c5a 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/distilled.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/distilled.py
@@ -39,6 +39,7 @@ from ltx_pipelines.utils.args import (
add_keyframe_decode_arg,
chunk_config_from_args,
default_2_stage_distilled_arg_parser,
+ markers_from_args,
resolve_cli_params,
)
from ltx_pipelines.utils.blocks import (
@@ -69,6 +70,7 @@ from ltx_pipelines.utils.types import (
AutoDuration,
Denoiser,
ImageConditioningInput,
+ LoadedImageConditioning,
OffloadMode,
PipelineOutput,
VideoAudio,
@@ -145,8 +147,8 @@ class _DistilledRunContext:
noiser: GaussianNoiser
dtype: torch.dtype
vae_dtype: torch.dtype
- #: ``images`` with every unset CRF filled in, so callers do not re-resolve it.
- images: list[ImageConditioningInput]
+ #: ``images`` loaded once for the run (CRF resolved, stills decoded), so callers do not reload them.
+ images: list[LoadedImageConditioning]
video_context: torch.Tensor
audio_context: torch.Tensor
num_frames: int
@@ -248,6 +250,7 @@ class DistilledPipeline:
width: int,
frame_rate: float,
images: list[ImageConditioningInput],
+ color_space: EXRColorSpace | None,
num_frames: int | AutoDuration,
vae_dtype: torch.dtype | None,
tiling_config: TilingConfig | AutoTiling | None,
@@ -259,7 +262,8 @@ class DistilledPipeline:
) -> _DistilledRunContext:
"""Validate inputs, encode prompts, and resolve frame count / tiling before chunk planning."""
require_num_frames_source(num_frames, self.duration_predictor)
- images = self.image_conditioner.resolve_crf(images)
+ enhance_prompt_image = images[0].path if images else None
+ images = self.image_conditioner.load_conditionings(images, color_space)
assert_resolution(height=height, width=width, is_two_stage=True)
assert_generated_keyframes_request(decode_with_keyframes, generated_keyframes, self.stage)
@@ -273,7 +277,7 @@ class DistilledPipeline:
[prompt],
enhance_first_prompt=enhance_prompt,
enhance_static_cache=enhance_static_cache,
- enhance_prompt_image=images[0][0] if len(images) > 0 else None,
+ enhance_prompt_image=enhance_prompt_image,
)
video_context, audio_context = ctx_p.video_encoding, ctx_p.audio_encoding
@@ -334,9 +338,8 @@ class DistilledPipeline:
def _video_conditionings(
self,
- images: list[ImageConditioningInput],
+ images: list[LoadedImageConditioning],
num_frames: int,
- color_space: EXRColorSpace | None,
) -> MakeVideoConditionings:
assert_image_frames_in_clip(images, num_frames)
@@ -348,7 +351,6 @@ class DistilledPipeline:
chunk,
images=images,
video_encoder=enc,
- color_space=color_space,
)
)
@@ -438,6 +440,7 @@ class DistilledPipeline:
width=width,
frame_rate=frame_rate,
images=images,
+ color_space=color_space,
num_frames=num_frames,
vae_dtype=vae_dtype,
tiling_config=tiling_config,
@@ -465,7 +468,7 @@ class DistilledPipeline:
dtype=ctx.dtype,
config=ctx.chunk_config,
video_scale_factors=self.stage.video_scale_factors,
- make_video_conditionings=self._video_conditionings(ctx.images, ctx.num_frames, color_space),
+ make_video_conditionings=self._video_conditionings(ctx.images, ctx.num_frames),
generated_keyframes=generated_keyframes,
)
@@ -515,6 +518,8 @@ def main() -> None:
)
)
args = parser.parse_args()
+ hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)
+ video_marker, audio_marker = markers_from_args(args, device=get_device(), color_space=hdr)
pipeline = DistilledPipeline(
model_paths=args.model_paths,
spatial_upsampler_path=args.spatial_upsampler_path,
@@ -525,7 +530,6 @@ def main() -> None:
prompt_enhancer_gemma_root=args.prompt_enhancer_gemma_root,
diffvae_optimization=args.diffvae_optimization,
)
- hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)
vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)
chunk_config = chunk_config_from_args(args)
chunks, num_frames, tiling_config = pipeline.stream_chunks(
@@ -564,6 +568,8 @@ def main() -> None:
video_chunks_number=video_chunks_number,
color_space=hdr,
audio_sampling_rate=audio_sampling_rate,
+ video_marker=video_marker,
+ audio_marker=audio_marker,
)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/distilled_mgpu.py b/packages/ltx-pipelines/src/ltx_pipelines/distilled_mgpu.py
index fad2d63..42381ce 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/distilled_mgpu.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/distilled_mgpu.py
@@ -13,6 +13,7 @@ from __future__ import annotations
import logging
from collections.abc import Callable, Iterator, Sequence
from multiprocessing import SimpleQueue
+from types import SimpleNamespace
from typing import Any
import torch
@@ -34,6 +35,8 @@ from ltx_pipelines.multigpu.runner import MGPURunner
from ltx_pipelines.multigpu.sp_builder import SequenceParallelBuilder
from ltx_pipelines.multigpu.vae_builders import DistributedDecoderBuilder
from ltx_pipelines.multigpu.weight_tracker import TransformerWeightTracker
+from ltx_pipelines.utils.args import markers_from_args
+from ltx_pipelines.utils.helpers import get_device
from ltx_pipelines.utils.media_io import EXRColorSpace, encode_video, resolve_hdr_color_space, vae_dtype_for_hdr
from ltx_pipelines.utils.model_paths import ModelPaths
from ltx_pipelines.utils.types import DEFAULT_AUTO_DURATION, AutoDuration
@@ -135,6 +138,9 @@ class DistilledRunner(MGPURunner):
vae_tiling=vae_tiling,
driver_rank=_DRIVER_RANK,
registry=registry,
+ # The driver concatenates the decoded clip on one device anyway, so assembling it
+ # in more temporal batches only adds a copy of the canvas.
+ num_temporal_batches=1,
)
self._pipeline = pipeline
@@ -157,44 +163,55 @@ class DistilledRunner(MGPURunner):
generated_keyframes: int | Sequence[int] = 0,
decode_with_keyframes: bool = False,
chunk_config: ChunkConfig | None = None,
+ marker_root_path: str | None = None,
) -> Iterator[str | None]:
# The pipeline raises ValueError on invalid input (symmetric across ranks); the controller
# catches that and turns it into a recoverable RunnerError. Anything else is fatal.
hdr = resolve_hdr_color_space(images=images or [], hdr=hdr)
- vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)
- chunks, _, _ = self._pipeline.stream_chunks(
- prompt=prompt,
- seed=seed,
- height=height,
- width=width,
- frame_rate=frame_rate,
- images=images or [],
- num_frames=num_frames,
- vae_dtype=vae_dtype,
- tiling_config=None,
- enhance_prompt=enhance_prompt,
- enhance_static_cache=enhance_static_cache,
- color_space=hdr,
- generated_keyframes=generated_keyframes,
- decode_with_keyframes=decode_with_keyframes,
- chunk_config=chunk_config,
- )
- if dist.get_rank() != _DRIVER_RANK:
- for _ in chunks:
- pass
- yield None
- return
- video, audio, audio_sampling_rate = split_decoded_chunks(chunks)
- encode_video(
- video=video,
- fps=frame_rate,
- audio=audio,
- output_path=output_path,
- video_chunks_number=1,
+ video_marker, audio_marker = markers_from_args(
+ SimpleNamespace(marker_root_path=marker_root_path),
+ device=get_device(),
color_space=hdr,
- audio_sampling_rate=audio_sampling_rate,
)
- yield output_path
+ vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)
+ # One transformer for both passes of every chunk instead of a build per call. Under DEFER the SP
+ # builder's builds stay assigned anyway, so holding one for the whole job costs no memory.
+ with self._pipeline.stage.resident():
+ chunks, _, _ = self._pipeline.stream_chunks(
+ prompt=prompt,
+ seed=seed,
+ height=height,
+ width=width,
+ frame_rate=frame_rate,
+ images=images or [],
+ num_frames=num_frames,
+ vae_dtype=vae_dtype,
+ tiling_config=None,
+ enhance_prompt=enhance_prompt,
+ enhance_static_cache=enhance_static_cache,
+ color_space=hdr,
+ generated_keyframes=generated_keyframes,
+ decode_with_keyframes=decode_with_keyframes,
+ chunk_config=chunk_config,
+ )
+ is_driver = dist.get_rank() == _DRIVER_RANK
+ if not is_driver:
+ for _ in chunks:
+ pass
+ else:
+ video, audio, audio_sampling_rate = split_decoded_chunks(chunks)
+ encode_video(
+ video=video,
+ fps=frame_rate,
+ audio=audio,
+ output_path=output_path,
+ video_chunks_number=1,
+ color_space=hdr,
+ audio_sampling_rate=audio_sampling_rate,
+ video_marker=video_marker,
+ audio_marker=audio_marker,
+ )
+ yield output_path if is_driver else None
if __name__ == "__main__":
@@ -244,6 +261,7 @@ if __name__ == "__main__":
generated_keyframes=args.num_generated_keyframes,
decode_with_keyframes=args.decode_with_keyframes,
chunk_config=chunk_config_from_args(args),
+ marker_root_path=args.marker_root_path,
):
pass # drive the job to completion; the runner writes the file as a side effect
finally:
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/dubit.py b/packages/ltx-pipelines/src/ltx_pipelines/dubit.py
index 0f00d5e..5e3021d 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/dubit.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/dubit.py
@@ -57,6 +57,7 @@ from ltx_pipelines.utils.args import (
add_keyframe_decode_arg,
chunk_config_from_args,
dubit_arg_parser,
+ markers_from_args,
resolve_cli_params,
)
from ltx_pipelines.utils.blocks import (
@@ -79,7 +80,13 @@ from ltx_pipelines.utils.helpers import (
)
from ltx_pipelines.utils.media_io import EXRColorSpace, decode_audio_from_file, encode_video, get_videostream_metadata
from ltx_pipelines.utils.model_paths import ModelPaths
-from ltx_pipelines.utils.types import ImageConditioningInput, OffloadMode, PipelineOutput, VideoAudio
+from ltx_pipelines.utils.types import (
+ ImageConditioningInput,
+ LoadedImageConditioning,
+ OffloadMode,
+ PipelineOutput,
+ VideoAudio,
+)
__all__ = ["DubItPipeline", "main", "patchify_dubit_audio_reference_latent"]
@@ -94,7 +101,7 @@ class _DubItRunContext:
noiser: GaussianNoiser
dtype: torch.dtype
vae_dtype: torch.dtype
- images: list[ImageConditioningInput]
+ images: list[LoadedImageConditioning]
video_context: torch.Tensor
audio_context: torch.Tensor
num_frames: int
@@ -204,6 +211,7 @@ class DubItPipeline:
height: int,
width: int,
images: list[ImageConditioningInput],
+ color_space: EXRColorSpace | None,
reference_video_path: str,
reference_strength: float,
enhance_prompt: bool,
@@ -215,7 +223,8 @@ class DubItPipeline:
chunk_config: ChunkConfig | None = None,
) -> _DubItRunContext:
"""Validate inputs, encode prompts/reference, and resolve frame count / tiling before chunk planning."""
- images = self.image_conditioner.resolve_crf(images)
+ enhance_prompt_image = images[0].path if images else None
+ images = self.image_conditioner.load_conditionings(images, color_space)
assert_resolution(height=height, width=width, is_two_stage=True)
assert_generated_keyframes_request(decode_with_keyframes, generated_keyframes, self.stage)
@@ -234,7 +243,7 @@ class DubItPipeline:
[prompt],
enhance_first_prompt=enhance_prompt,
enhance_static_cache=enhance_static_cache,
- enhance_prompt_image=images[0][0] if len(images) > 0 else None,
+ enhance_prompt_image=enhance_prompt_image,
)
video_context, audio_context = ctx_p.video_encoding, ctx_p.audio_encoding
@@ -276,7 +285,7 @@ class DubItPipeline:
def _video_conditionings(
self,
- images: list[ImageConditioningInput],
+ images: list[LoadedImageConditioning],
num_frames: int,
reference_video_path: str,
reference_strength: float,
@@ -291,7 +300,7 @@ class DubItPipeline:
def encode(enc: VideoEncoder) -> list[ConditioningItem]:
items = [
- *image_conditionings_for_chunk(chunk, images=images, video_encoder=enc, color_space=color_space),
+ *image_conditionings_for_chunk(chunk, images=images, video_encoder=enc),
*reference_video_conditionings_for_chunk(
chunk,
video_conditioning=[(reference_video_path, reference_strength)],
@@ -386,6 +395,7 @@ class DubItPipeline:
height=height,
width=width,
images=images,
+ color_space=color_space,
reference_video_path=reference_video_path,
reference_strength=reference_strength,
enhance_prompt=enhance_prompt,
@@ -476,6 +486,7 @@ def main() -> None:
add_keyframe_decode_arg(add_generated_keyframes_arg(dubit_arg_parser(params=params)))
)
args = parser.parse_args()
+ video_marker, audio_marker = markers_from_args(args, device=get_device())
if not args.lora or len(args.lora) != 1:
raise ValueError("Dub-It requires exactly one --lora (the Dub-It IC-LoRA).")
@@ -526,6 +537,8 @@ def main() -> None:
video_chunks_number=video_chunks_number,
color_space=None,
audio_sampling_rate=audio_sampling_rate,
+ video_marker=video_marker,
+ audio_marker=audio_marker,
)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/hdr_ic_lora.py b/packages/ltx-pipelines/src/ltx_pipelines/hdr_ic_lora.py
index 764af40..540d9bf 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/hdr_ic_lora.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/hdr_ic_lora.py
@@ -25,7 +25,10 @@ from ltx_core.model.video_vae.transformer import DiffVAEMode
from ltx_core.quantization import QuantizationPolicy
from ltx_core.types import LatentState, SpatioTemporalScaleFactors, VideoPixelShape
from ltx_pipelines.dfr_helpers.layout import resolve_canvas
-from ltx_pipelines.utils.args import hdr_ic_lora_arg_parser
+from ltx_pipelines.utils.args import (
+ hdr_ic_lora_arg_parser,
+ markers_from_args,
+)
from ltx_pipelines.utils.blocks import DiffusionStage, ImageConditioner, VideoDecoder
from ltx_pipelines.utils.constants import DISTILLED_SIGMA_VALUES
from ltx_pipelines.utils.denoisers import SimpleDenoiser
@@ -527,6 +530,7 @@ class HDRICLoraPipeline:
def main() -> None:
logging.basicConfig(level=logging.INFO)
args = hdr_ic_lora_arg_parser().parse_args()
+ video_marker, audio_marker = markers_from_args(args, device=get_device(), color_space=args.exr_colorspace)
logger.info("Loading HDRICLoraPipeline...")
pipeline = HDRICLoraPipeline(
@@ -554,6 +558,8 @@ def main() -> None:
output_path=str(output_path),
video_chunks_number=1,
color_space=args.exr_colorspace,
+ video_marker=video_marker,
+ audio_marker=audio_marker,
)
logger.info("Done → %s (+ EXR)", args.output_path)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/ic_lora.py b/packages/ltx-pipelines/src/ltx_pipelines/ic_lora.py
index 46baa6a..3008a51 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/ic_lora.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/ic_lora.py
@@ -47,6 +47,7 @@ from ltx_pipelines.utils.args import (
add_keyframe_decode_arg,
chunk_config_from_args,
default_2_stage_distilled_arg_parser,
+ markers_from_args,
resolve_cli_params,
)
from ltx_pipelines.utils.blocks import (
@@ -89,6 +90,7 @@ from ltx_pipelines.utils.tiled_diffusion import (
)
from ltx_pipelines.utils.types import (
ImageConditioningInput,
+ LoadedImageConditioning,
OffloadMode,
PipelineOutput,
VideoAudio,
@@ -265,7 +267,7 @@ class _ICLoraRunContext:
noiser: GaussianNoiser
dtype: torch.dtype
vae_dtype: torch.dtype
- images: list[ImageConditioningInput]
+ images: list[LoadedImageConditioning]
video_context: torch.Tensor
audio_context: torch.Tensor
num_frames: int
@@ -442,7 +444,8 @@ class ICLoraPipeline:
decode_with_keyframes: bool = False,
chunk_config: ChunkConfig | None = None,
) -> _ICLoraRunContext:
- images = self.image_conditioner.resolve_crf(images)
+ enhance_prompt_image = images[0].path if images else None
+ images = self.image_conditioner.load_conditionings(images, color_space)
assert_generated_keyframes_request(
decode_with_keyframes,
generated_keyframes,
@@ -478,7 +481,7 @@ class ICLoraPipeline:
[prompt],
enhance_first_prompt=enhance_prompt,
enhance_static_cache=enhance_static_cache,
- enhance_prompt_image=images[0][0] if len(images) > 0 else None,
+ enhance_prompt_image=enhance_prompt_image,
)
video_context, audio_context = ctx_p.video_encoding, ctx_p.audio_encoding
@@ -540,7 +543,6 @@ class ICLoraPipeline:
chunk,
images=ctx.images,
video_encoder=enc,
- color_space=ctx.color_space,
),
*reference_video_conditionings_for_chunk(
chunk,
@@ -887,6 +889,12 @@ def main() -> None:
),
)
args = parser.parse_args()
+ hdr = resolve_hdr_color_space(
+ images=args.images,
+ video_paths=[path for path, _ in args.video_conditioning],
+ hdr=args.hdr,
+ )
+ video_marker, audio_marker = markers_from_args(args, device=get_device(), color_space=hdr)
# Load mask video if provided via --conditioning-attention-mask
conditioning_attention_mask = None
@@ -921,11 +929,6 @@ def main() -> None:
skip_stage_2=args.skip_stage_2,
),
)
- hdr = resolve_hdr_color_space(
- images=args.images,
- video_paths=[path for path, _ in args.video_conditioning],
- hdr=args.hdr,
- )
vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)
chunk_config = chunk_config_from_args(args)
chunks, num_frames, tiling_config = pipeline.stream_chunks(
@@ -968,6 +971,8 @@ def main() -> None:
video_chunks_number=video_chunks_number,
color_space=hdr,
audio_sampling_rate=audio_sampling_rate,
+ video_marker=video_marker,
+ audio_marker=audio_marker,
)
except _IC_LORA_OOM as exc:
reraise_ic_lora_oom(exc, tile=args.tile)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/keyframe_interpolation.py b/packages/ltx-pipelines/src/ltx_pipelines/keyframe_interpolation.py
deleted file mode 100644
index ca96a77..0000000
--- a/packages/ltx-pipelines/src/ltx_pipelines/keyframe_interpolation.py
+++ /dev/null
@@ -1,375 +0,0 @@
-import logging
-
-import torch
-
-from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy
-from ltx_core.components.guiders import (
- MultiModalGuiderFactory,
- MultiModalGuiderParams,
- create_multimodal_guider_factory,
-)
-from ltx_core.components.noisers import GaussianNoiser
-from ltx_core.components.schedulers import LTX2Scheduler
-from ltx_core.loader import LoraPathStrengthAndSDOps
-from ltx_core.loader.registry import Registry
-from ltx_core.model.transformer.compiling import CompilationConfig
-from ltx_core.model.video_vae import AUTO_TILING, AutoTiling, TilingConfig, get_video_chunks_number
-from ltx_core.model.video_vae.transformer import DiffVAEMode
-from ltx_core.quantization import QuantizationPolicy
-from ltx_core.types import VideoPixelShape
-from ltx_pipelines.utils.args import (
- default_2_stage_arg_parser,
- resolve_cli_params,
-)
-from ltx_pipelines.utils.blocks import (
- AudioDecoder,
- DiffusionStage,
- ImageConditioner,
- PromptEncoder,
- VideoDecoder,
- VideoUpsampler,
-)
-from ltx_pipelines.utils.constants import (
- STAGE_2_DISTILLED_SIGMAS,
-)
-from ltx_pipelines.utils.denoisers import FactoryGuidedDenoiser, SimpleDenoiser
-from ltx_pipelines.utils.helpers import (
- assert_resolution,
- create_initial_av_latents,
- ensure_tiling_config,
- get_device,
- image_conditionings_by_adding_guiding_latent,
- tiling_scale_factors_for_vae,
-)
-from ltx_pipelines.utils.media_io import (
- EXRColorSpace,
- encode_video,
- resolve_hdr_color_space,
- vae_dtype_for_hdr,
-)
-from ltx_pipelines.utils.model_paths import ModelPaths
-from ltx_pipelines.utils.types import (
- ImageConditioningInput,
- ModalitySpec,
- OffloadMode,
- PipelineOutput,
- VideoAudio,
-)
-
-
-class KeyframeInterpolationPipeline:
- """
- Keyframe-based Two-stage video interpolation pipeline.
- Interpolates between keyframes to generate a video with smoother transitions.
- Stage 1 generates video at half of the target resolution, then Stage 2 upsamples
- by 2x and refines with additional denoising steps for higher quality output.
- Stage 1 uses full model while Stage 2 uses distilled LORA for efficiency,
- as the upsampled video already has good quality and just needs refinement.
- """
-
- def __init__( # noqa: PLR0913
- self,
- model_paths: ModelPaths,
- distilled_lora: list[LoraPathStrengthAndSDOps],
- spatial_upsampler_path: str,
- loras: list[LoraPathStrengthAndSDOps],
- device: torch.device | None = None,
- quantization: QuantizationPolicy | None = None,
- registry: Registry | None = None,
- compilation_config: CompilationConfig | None = None,
- offload_mode: OffloadMode = OffloadMode.NONE,
- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
- prompt_enhancer_gemma_root: str | None = None,
- diffvae_optimization: DiffVAEMode = DiffVAEMode.CHUNKED_EAGER,
- ):
- self.device = device or get_device()
- self.dtype = torch.bfloat16
- self._scheduler = LTX2Scheduler()
-
- self.prompt_encoder = PromptEncoder(
- model_paths,
- self.dtype,
- self.device,
- registry=registry,
- offload_mode=offload_mode,
- alloc_trim_strategy=alloc_trim_strategy,
- prompt_enhancer_gemma_root=prompt_enhancer_gemma_root,
- )
- self.image_conditioner = ImageConditioner(
- model_paths.video_vae(),
- self.dtype,
- self.device,
- registry=registry,
- alloc_trim_strategy=alloc_trim_strategy,
- )
- self.stage_1 = DiffusionStage.from_checkpoint(
- model_paths.transformer(),
- self.dtype,
- self.device,
- loras=tuple(loras),
- quantization=quantization,
- registry=registry,
- compilation_config=compilation_config,
- offload_mode=offload_mode,
- alloc_trim_strategy=alloc_trim_strategy,
- )
- stage_2_loras = (*tuple(loras), *tuple(distilled_lora))
- self.stage_2 = DiffusionStage.from_checkpoint(
- model_paths.transformer(),
- self.dtype,
- self.device,
- loras=stage_2_loras,
- quantization=quantization,
- registry=registry,
- compilation_config=compilation_config,
- offload_mode=offload_mode,
- alloc_trim_strategy=alloc_trim_strategy,
- )
- self.upsampler = VideoUpsampler(
- model_paths.video_vae(),
- spatial_upsampler_path,
- self.dtype,
- self.device,
- registry=registry,
- alloc_trim_strategy=alloc_trim_strategy,
- )
- self.video_decoder = VideoDecoder(
- model_paths.video_vae(),
- self.dtype,
- self.device,
- registry=registry,
- alloc_trim_strategy=alloc_trim_strategy,
- diffvae_optimization=diffvae_optimization,
- )
- self.audio_decoder = AudioDecoder(
- model_paths.audio_vae(),
- self.dtype,
- self.device,
- registry=registry,
- alloc_trim_strategy=alloc_trim_strategy,
- )
-
- def __call__( # noqa: PLR0913
- self,
- prompt: str,
- negative_prompt: str,
- seed: int,
- height: int,
- width: int,
- num_frames: int,
- frame_rate: float,
- num_inference_steps: int,
- video_guider_params: MultiModalGuiderParams | MultiModalGuiderFactory,
- audio_guider_params: MultiModalGuiderParams | MultiModalGuiderFactory,
- images: list[ImageConditioningInput],
- vae_dtype: torch.dtype | None = None,
- tiling_config: TilingConfig | AutoTiling | None = AUTO_TILING,
- enhance_prompt: bool = False,
- enhance_static_cache: bool = False,
- max_batch_size: int = 1,
- stage_1_sigmas: torch.Tensor | None = None,
- stage_2_sigmas: torch.Tensor = STAGE_2_DISTILLED_SIGMAS,
- color_space: EXRColorSpace | None = None,
- ) -> PipelineOutput:
- images = self.image_conditioner.resolve_crf(images)
- assert_resolution(height=height, width=width, is_two_stage=True)
-
- generator = torch.Generator(device=self.device).manual_seed(seed)
- noiser = GaussianNoiser(generator=generator)
- dtype = torch.bfloat16
- if vae_dtype is None:
- vae_dtype = dtype
-
- ctx_p, ctx_n = self.prompt_encoder(
- [prompt, negative_prompt],
- enhance_first_prompt=enhance_prompt,
- enhance_static_cache=enhance_static_cache,
- enhance_prompt_image=images[0][0] if len(images) > 0 else None,
- )
- v_context_p, a_context_p = ctx_p.video_encoding, ctx_p.audio_encoding
- v_context_n, a_context_n = ctx_n.video_encoding, ctx_n.audio_encoding
-
- scale_factors = tiling_scale_factors_for_vae(self.video_decoder.checkpoint_path)
- tiling_config = ensure_tiling_config(
- tiling_config,
- scale_factors=scale_factors,
- vae_checkpoint_path=self.video_decoder.checkpoint_path,
- video_shape=VideoPixelShape(batch=1, frames=num_frames, height=height, width=width, fps=frame_rate),
- diffvae_optimization=self.video_decoder.diffvae_optimization,
- device=self.device,
- )
-
- # Stage 1: Initial low resolution video generation.
- sigmas = (
- stage_1_sigmas if stage_1_sigmas is not None else self._scheduler.execute(steps=num_inference_steps)
- ).to(dtype=torch.float32, device=self.device)
-
- stage_1_output_shape = VideoPixelShape(
- batch=1,
- frames=num_frames,
- width=width // 2,
- height=height // 2,
- fps=frame_rate,
- )
- stage_1_conditionings = self.image_conditioner(
- lambda enc: image_conditionings_by_adding_guiding_latent(
- images=images,
- height=stage_1_output_shape.height,
- width=stage_1_output_shape.width,
- video_encoder=enc,
- dtype=dtype,
- device=self.device,
- color_space=color_space,
- )
- )
-
- video_guider_factory = create_multimodal_guider_factory(
- params=video_guider_params,
- negative_context=v_context_n,
- )
- audio_guider_factory = create_multimodal_guider_factory(
- params=audio_guider_params,
- negative_context=a_context_n,
- )
-
- video_latent, audio_latent = create_initial_av_latents(
- width=stage_1_output_shape.width,
- height=stage_1_output_shape.height,
- frames=num_frames,
- fps=frame_rate,
- device=self.device,
- dtype=self.dtype,
- video_scale_factors=self.stage_1.video_scale_factors,
- )
- video_state, audio_state = self.stage_1(
- denoiser=FactoryGuidedDenoiser(
- v_context=v_context_p,
- a_context=a_context_p,
- video_guider_factory=video_guider_factory,
- audio_guider_factory=audio_guider_factory,
- ),
- sigmas=sigmas,
- noiser=noiser,
- modalities=VideoAudio(
- video=ModalitySpec(
- latent=video_latent,
- conditioning_fps=frame_rate,
- context=v_context_p,
- conditionings=stage_1_conditionings,
- ),
- audio=ModalitySpec(
- latent=audio_latent,
- conditioning_fps=frame_rate,
- context=a_context_p,
- ),
- ),
- max_batch_size=max_batch_size,
- )
-
- # Stage 2: Upsample and refine the video at higher resolution with distilled LORA.
- upscaled_video_latent = self.upsampler(video_state.latent[:1])
-
- stage_2_sigmas = stage_2_sigmas.to(dtype=torch.float32, device=self.device)
- stage_2_output_shape = VideoPixelShape(batch=1, frames=num_frames, width=width, height=height, fps=frame_rate)
- stage_2_conditionings = self.image_conditioner(
- lambda enc: image_conditionings_by_adding_guiding_latent(
- images=images,
- height=stage_2_output_shape.height,
- width=stage_2_output_shape.width,
- video_encoder=enc,
- dtype=dtype,
- device=self.device,
- color_space=color_space,
- )
- )
-
- video_state, audio_state = self.stage_2(
- denoiser=SimpleDenoiser(v_context_p, a_context_p),
- sigmas=stage_2_sigmas,
- noiser=noiser,
- modalities=VideoAudio(
- video=ModalitySpec(
- latent=upscaled_video_latent,
- conditioning_fps=frame_rate,
- context=v_context_p,
- conditionings=stage_2_conditionings,
- noise_scale=stage_2_sigmas[0].item(),
- ),
- audio=ModalitySpec(
- latent=audio_state.latent,
- conditioning_fps=frame_rate,
- context=a_context_p,
- noise_scale=stage_2_sigmas[0].item(),
- ),
- ),
- )
-
- decoded_video = self.video_decoder(video_state.latent, tiling_config, generator, dtype=vae_dtype)
- decoded_audio = self.audio_decoder(audio_state.latent)
- return PipelineOutput(decoded_video, decoded_audio, num_frames, tiling_config)
-
-
-def main() -> None:
- logging.basicConfig(level=logging.INFO)
- params = resolve_cli_params()
- parser = default_2_stage_arg_parser(params=params)
- args = parser.parse_args()
- pipeline = KeyframeInterpolationPipeline(
- model_paths=args.model_paths,
- distilled_lora=args.distilled_lora,
- spatial_upsampler_path=args.spatial_upsampler_path,
- loras=tuple(args.lora) if args.lora else (),
- quantization=args.quantization,
- compilation_config=args.compile,
- offload_mode=args.offload_mode,
- prompt_enhancer_gemma_root=args.prompt_enhancer_gemma_root,
- diffvae_optimization=args.diffvae_optimization,
- )
- hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)
- vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)
- result = pipeline(
- prompt=args.prompt,
- negative_prompt=args.negative_prompt,
- seed=args.seed,
- height=args.height,
- width=args.width,
- num_frames=args.num_frames,
- frame_rate=args.frame_rate,
- num_inference_steps=args.num_inference_steps,
- video_guider_params=MultiModalGuiderParams(
- cfg_scale=args.video_cfg_guidance_scale,
- stg_scale=args.video_stg_guidance_scale,
- rescale_scale=args.video_rescale_scale,
- modality_scale=args.a2v_guidance_scale,
- skip_step=args.video_skip_step,
- stg_blocks=args.video_stg_blocks,
- ),
- audio_guider_params=MultiModalGuiderParams(
- cfg_scale=args.audio_cfg_guidance_scale,
- stg_scale=args.audio_stg_guidance_scale,
- rescale_scale=args.audio_rescale_scale,
- modality_scale=args.v2a_guidance_scale,
- skip_step=args.audio_skip_step,
- stg_blocks=args.audio_stg_blocks,
- ),
- images=args.images,
- vae_dtype=vae_dtype,
- color_space=hdr,
- enhance_prompt=args.enhance_prompt,
- enhance_static_cache=args.enhance_static_cache,
- tiling_config=AUTO_TILING,
- max_batch_size=args.max_batch_size,
- )
- encode_video(
- video=result.video,
- fps=args.frame_rate,
- audio=result.audio,
- output_path=args.output_path,
- video_chunks_number=get_video_chunks_number(result.num_frames, result.tiling_config),
- color_space=hdr,
- )
-
-
-if __name__ == "__main__":
- main()
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/marking/__init__.py b/packages/ltx-pipelines/src/ltx_pipelines/marking/__init__.py
new file mode 100644
index 0000000..3df6254
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/marking/__init__.py
@@ -0,0 +1,76 @@
+"""Post-hoc video and audio marking (Videoseal / AudioSeal opt-in).
+Importing this package does not require the ``videoseal`` or ``audioseal`` extras.
+A pack directory is loaded with :func:`create_markers`. Seal classes live in
+``ltx_pipelines.marking.video_seal`` and ``ltx_pipelines.marking.audio_seal``.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import torch
+
+from ltx_pipelines.marking.mark_detection import detect_audio_mark, detect_video_mark
+from ltx_pipelines.marking.types import AudioMarker, MarkDetectResult, VideoMarker
+
+PIXELSEAL_CHECKPOINT_NAME = "pixelseal.pth"
+PIXELSEAL_CARD_NAME = "pixelseal.yaml"
+PIXELSEAL_ATTENUATION_NAME = "videoseal_attenuation.yaml"
+AUDIOSEAL_GENERATOR_NAME = "audioseal_generator.pth"
+AUDIOSEAL_DETECTOR_NAME = "audioseal_detector.pth"
+
+_PACK_FILES = (
+ PIXELSEAL_CHECKPOINT_NAME,
+ PIXELSEAL_CARD_NAME,
+ PIXELSEAL_ATTENUATION_NAME,
+ AUDIOSEAL_GENERATOR_NAME,
+ AUDIOSEAL_DETECTOR_NAME,
+)
+
+
+def _require_marking_pack(root: Path | str) -> Path:
+ pack = Path(root)
+ missing = [name for name in _PACK_FILES if not (pack / name).is_file()]
+ if missing:
+ raise FileNotFoundError(f"marking pack {pack} requires {', '.join(missing)}")
+ return pack
+
+
+def create_markers(root: Path | str, device: torch.device) -> tuple[VideoMarker, AudioMarker]:
+ """Load VideoSeal and AudioSeal markers from a marking pack directory.
+ All pack files must be present. Both extras are imported (fail fast).
+ """
+ pack = _require_marking_pack(root)
+ try:
+ from ltx_pipelines.marking.audio_seal import AudioSealMarker # noqa: PLC0415
+ from ltx_pipelines.marking.video_seal import VideoSealMarker # noqa: PLC0415
+ except ModuleNotFoundError as exc:
+ raise ImportError("videoseal and audioseal packages are required") from exc
+ return (
+ VideoSealMarker.create(
+ pack / PIXELSEAL_CHECKPOINT_NAME,
+ card=pack / PIXELSEAL_CARD_NAME,
+ attenuation_config=pack / PIXELSEAL_ATTENUATION_NAME,
+ device=device,
+ ),
+ AudioSealMarker.create(
+ pack / AUDIOSEAL_GENERATOR_NAME,
+ pack / AUDIOSEAL_DETECTOR_NAME,
+ device=device,
+ ),
+ )
+
+
+__all__ = [
+ "AUDIOSEAL_DETECTOR_NAME",
+ "AUDIOSEAL_GENERATOR_NAME",
+ "PIXELSEAL_ATTENUATION_NAME",
+ "PIXELSEAL_CARD_NAME",
+ "PIXELSEAL_CHECKPOINT_NAME",
+ "AudioMarker",
+ "MarkDetectResult",
+ "VideoMarker",
+ "create_markers",
+ "detect_audio_mark",
+ "detect_video_mark",
+]
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/marking/audio_seal.py b/packages/ltx-pipelines/src/ltx_pipelines/marking/audio_seal.py
new file mode 100644
index 0000000..b964847
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/marking/audio_seal.py
@@ -0,0 +1,126 @@
+"""AudioSeal marker wrapping Meta's generator and detector."""
+
+from __future__ import annotations
+
+import logging
+from pathlib import Path
+
+import torch
+from audioseal import AudioSeal
+from einops import rearrange
+from torch import nn
+
+from ltx_core.types import Audio
+from ltx_pipelines.marking.score import score_bits
+from ltx_pipelines.marking.types import AudioMarker, MarkDetectResult
+
+logger = logging.getLogger(__name__)
+
+# Baked payload seed shared by embed and detect. Not a signature and not a model version.
+AUDIOSEAL_PAYLOAD_SEED = 0x4C5458 # ASCII "LTX"
+AUDIOSEAL_NBITS = 16
+
+
+class AudioSealMarker(AudioMarker):
+ """Adapter around loaded AudioSeal generator and detector modules."""
+
+ def __init__(self, generator: nn.Module, detector: nn.Module, *, nbits: int) -> None:
+ self._generator = generator
+ self._detector = detector
+ self._nbits = nbits
+ self._payload = baked_audioseal_payload(nbits)
+
+ @classmethod
+ def create(
+ cls,
+ generator: Path | str,
+ detector: Path | str,
+ *,
+ device: torch.device,
+ nbits: int = AUDIOSEAL_NBITS,
+ ) -> AudioSealMarker:
+ """Load local generator and detector; ``eval``, drop grad, move to ``device``."""
+ logger.info("Loading AudioSeal from %s and %s", generator, detector)
+ gen, det = load_audioseal(generator, detector, nbits=nbits)
+ for module in (gen, det):
+ module.eval()
+ module.requires_grad_(False)
+ module.to(device)
+ return cls(gen, det, nbits=nbits)
+
+ @property
+ def name(self) -> str:
+ return "audioseal"
+
+ def embed(self, audio: Audio) -> Audio:
+ """Stamp one stereo ``Audio`` as a batch of two mono channels (no downmix)."""
+ device = next(self._generator.parameters()).device
+ wav, pattern = waveform_to_channel_batch(audio.waveform)
+ wav = wav.to(device=device, dtype=torch.float32)
+ message = self._payload.to(device=device, dtype=torch.long).unsqueeze(0).expand(wav.shape[0], -1)
+ stamped = self._generator(wav, message=message)
+ return Audio(
+ waveform=rearrange(stamped, f"b 1 t -> {pattern}").to(
+ device=audio.waveform.device, dtype=audio.waveform.dtype
+ ),
+ sampling_rate=audio.sampling_rate,
+ )
+
+ def detect(self, audio: Audio) -> MarkDetectResult:
+ """Run the detector; ``detected`` is the clip-level presence score (mean >= 0.5)."""
+ device = next(self._detector.parameters()).device
+ wav, _ = waveform_to_channel_batch(audio.waveform)
+ wav = wav.to(device=device, dtype=torch.float32)
+ detection, message = self._detector(wav)
+ detect_score = float((detection[:, 1, :] > 0.5).to(dtype=torch.float32).mean().item())
+ decoded_bits = (message.mean(dim=0) > 0.5).to(dtype=torch.float32)
+ bit_accuracy, p_value = score_bits(decoded_bits, self._payload)
+ return MarkDetectResult(
+ detected=detect_score >= 0.5,
+ bit_accuracy=bit_accuracy,
+ p_value=p_value,
+ payload=self._payload.detach().cpu(),
+ )
+
+
+def baked_audioseal_payload(nbits: int = AUDIOSEAL_NBITS) -> torch.Tensor:
+ """Return the baked payload as a ``[nbits]`` integer ``{0, 1}`` tensor."""
+ generator = torch.Generator(device="cpu")
+ generator.manual_seed(AUDIOSEAL_PAYLOAD_SEED)
+ return torch.randint(0, 2, (nbits,), generator=generator)
+
+
+def waveform_to_channel_batch(waveform: torch.Tensor) -> tuple[torch.Tensor, str]:
+ """Convert PCM to AudioSeal ``(b, 1, t)`` (stereo as batch-2, no downmix).
+ The returned string is the einops source pattern (``1 b t``, ``t b``, or ``b t``).
+ """
+ if waveform.ndim == 3:
+ if waveform.shape[0] != 1:
+ raise ValueError(f"expected batched waveform (1, C, T); got {tuple(waveform.shape)}")
+ pattern = "1 b t"
+ elif waveform.ndim != 2:
+ raise ValueError(f"audio.waveform must be (C, T), (T, C), or (1, C, T); got {tuple(waveform.shape)}")
+ elif waveform.shape[1] == 2:
+ pattern = "t b"
+ elif waveform.shape[0] == 2:
+ pattern = "b t"
+ else:
+ raise ValueError(f"audio.waveform must be stereo (2, N) or (N, 2); got {tuple(waveform.shape)}")
+ return rearrange(waveform, f"{pattern} -> b 1 t"), pattern
+
+
+def load_audioseal(
+ generator: Path | str,
+ detector: Path | str,
+ *,
+ nbits: int = AUDIOSEAL_NBITS,
+) -> tuple[nn.Module, nn.Module]:
+ """Load generator and detector from local file paths (no Hub download)."""
+ generator_path = Path(generator)
+ detector_path = Path(detector)
+ missing = [str(path) for path in (generator_path, detector_path) if not path.is_file()]
+ if missing:
+ raise FileNotFoundError(f"AudioSeal checkpoints missing: {', '.join(missing)}")
+ generator = AudioSeal.load_generator(str(generator_path), nbits=nbits)
+ detector = AudioSeal.load_detector(str(detector_path), nbits=nbits)
+ return generator, detector
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/marking/mark_detection.py b/packages/ltx-pipelines/src/ltx_pipelines/marking/mark_detection.py
new file mode 100644
index 0000000..6651113
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/marking/mark_detection.py
@@ -0,0 +1,32 @@
+"""File-level mark detect: decode stamp-domain media, then call the marker."""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from pathlib import Path
+
+import torch
+from torch._prims_common import DeviceLikeType
+
+from ltx_core.devices import get_preferred_device
+from ltx_pipelines.marking.types import AudioMarker, MarkDetectResult, VideoMarker
+from ltx_pipelines.utils.media_io.decode import decode_audio_from_file, decode_video_by_frame
+
+
+def detect_video_mark(marker: VideoMarker, path: str | Path) -> MarkDetectResult:
+ """Detects a marked file at ``path`` if it was marked by marker."""
+ return marker.detect(_sdr_detect_frames(path, get_preferred_device()))
+
+
+def detect_audio_mark(marker: AudioMarker, path: str | Path) -> MarkDetectResult | None:
+ """Detects a marked file at ``path`` if it was marked by marker."""
+ decoded = decode_audio_from_file(str(path), get_preferred_device())
+ if decoded is None:
+ return None
+ return marker.detect(decoded)
+
+
+def _sdr_detect_frames(path: str | Path, device: DeviceLikeType) -> Iterator[torch.Tensor]:
+ """Yield SDR RGB frames ``[H, W, C]`` in ``[0, 1]``."""
+ for frame in decode_video_by_frame(str(path), device):
+ yield frame.squeeze(0).to(dtype=torch.float32).div_(255.0)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/marking/score.py b/packages/ltx-pipelines/src/ltx_pipelines/marking/score.py
new file mode 100644
index 0000000..d48df92
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/marking/score.py
@@ -0,0 +1,30 @@
+"""Binomial bit match of decoded bits against a baked payload.
+Does not decide ``detected`` — each marker applies its own call.
+"""
+
+from __future__ import annotations
+
+import torch
+from scipy import stats
+
+
+def score_bits(decoded: torch.Tensor, payload: torch.Tensor) -> tuple[float, float]:
+ """Binomial bit match of ``decoded`` against ``payload``.
+ Args:
+ decoded: ``{0, 1}`` bits (any shape). Flattened before the test.
+ payload: ``{0, 1}`` bits (any shape). Flattened length must match ``decoded``.
+ Returns:
+ ``(bit_accuracy, p_value)``. ``p_value`` is a one-tailed binomial test
+ (``alternative="greater"``) against a chance match of ``p=0.5``.
+ Raises:
+ ValueError: If the flattened lengths differ.
+ """
+ decoded = decoded.reshape(-1).to(dtype=torch.float32)
+ expected = payload.reshape(-1).to(device=decoded.device, dtype=torch.float32)
+ if decoded.numel() != expected.numel():
+ raise ValueError(f"detector bits {decoded.numel()} != payload bits {expected.numel()}")
+ matches = int((decoded == expected).sum().item())
+ nbits = int(expected.numel())
+ bit_accuracy = matches / nbits
+ p_value = float(stats.binomtest(matches, nbits, 0.5, alternative="greater").pvalue)
+ return bit_accuracy, p_value
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/marking/types.py b/packages/ltx-pipelines/src/ltx_pipelines/marking/types.py
new file mode 100644
index 0000000..18ee44a
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/marking/types.py
@@ -0,0 +1,59 @@
+"""Public video and audio marker protocols and detect result."""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from typing import NamedTuple, Protocol
+
+import torch
+
+from ltx_core.types import Audio
+
+
+class MarkDetectResult(NamedTuple):
+ """Clip-level detector output.
+ ``bit_accuracy`` and ``p_value`` are the binomial test of decoded bits vs the baked
+ payload. ``detected`` is this marker's own call (not that test).
+ """
+
+ detected: bool
+ bit_accuracy: float
+ p_value: float
+ payload: torch.Tensor | None
+
+
+class VideoMarker(Protocol):
+ """Embeds a watermark into RGB frames and detects it in a frame stream."""
+
+ @property
+ def name(self) -> str:
+ """Short identifier."""
+ ...
+
+ def embed(self, frames: torch.Tensor) -> torch.Tensor:
+ """Stamp one ``[F, H, W, C]`` RGB tensor in ``[0, 1]``; return the same layout."""
+ ...
+
+ def detect(self, frames: Iterator[torch.Tensor]) -> MarkDetectResult:
+ """Detect the watermark in a stream of ``[H, W, C]`` RGB frames in ``[0, 1]``."""
+ ...
+
+
+class AudioMarker(Protocol):
+ """Embeds a watermark into stereo PCM and detects it in an ``Audio`` track."""
+
+ @property
+ def name(self) -> str:
+ """Short identifier."""
+ ...
+
+ def embed(self, audio: Audio) -> Audio:
+ """Stamp one yielded stereo ``Audio`` track."""
+ ...
+
+ def detect(self, audio: Audio) -> MarkDetectResult:
+ """Detect the audio watermark in one stereo ``Audio`` track."""
+ ...
+
+
+__all__ = ["AudioMarker", "MarkDetectResult", "VideoMarker"]
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/marking/video_seal.py b/packages/ltx-pipelines/src/ltx_pipelines/marking/video_seal.py
new file mode 100644
index 0000000..6f001a9
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/marking/video_seal.py
@@ -0,0 +1,141 @@
+"""Videoseal video marker wrapping Meta's ``Videoseal`` module."""
+
+from __future__ import annotations
+
+import logging
+import math
+from collections.abc import Iterator
+from itertools import islice
+from pathlib import Path
+
+import torch
+from einops import rearrange
+from omegaconf import OmegaConf
+from videoseal.utils.cfg import SubModelConfig, VideosealConfig, setup_model
+
+from ltx_pipelines.marking.score import score_bits
+from ltx_pipelines.marking.types import MarkDetectResult, VideoMarker
+
+logger = logging.getLogger(__name__)
+
+# Baked payload seed shared by embed and detect. Not a signature and not a model version.
+VIDEOSEAL_PAYLOAD_SEED = 0x4C5458 # ASCII "LTX"
+
+# Max sampled frames held on GPU for one ``detect`` call. Logits are accumulated.
+_DETECT_FRAME_BATCH = 8
+
+
+class VideoSealMarker(VideoMarker):
+ """Adapter around a loaded Meta ``Videoseal`` module (PixelSeal, ChunkySeal, …)."""
+
+ def __init__(self, model: torch.nn.Module, *, name: str, nbits: int) -> None:
+ self._model = model
+ self._name = name
+ self._payload = baked_videoseal_payload(nbits)
+
+ @classmethod
+ def create(
+ cls,
+ checkpoint: Path | str,
+ *,
+ card: Path | str,
+ attenuation_config: Path | str,
+ device: torch.device,
+ ) -> VideoSealMarker:
+ """Load local checkpoint, card, and attenuation; ``eval``, drop grad, move to ``device``."""
+ card_path = Path(card)
+ logger.info("Loading Videoseal card %s from %s", card_path, checkpoint)
+ model, nbits = load_videoseal(
+ checkpoint,
+ card=card_path,
+ attenuation_config=attenuation_config,
+ )
+ model.eval()
+ model.requires_grad_(False)
+ model.to(device)
+ return cls(model, name=card_path.stem, nbits=nbits)
+
+ @property
+ def name(self) -> str:
+ return self._name
+
+ def embed(self, frames: torch.Tensor) -> torch.Tensor:
+ """Stamp one ``[F, H, W, C]`` RGB tensor; Videoseal resizes internally to 256 and back."""
+ device = next(self._model.parameters()).device
+ nchw = rearrange(frames.to(device=device, dtype=torch.float32), "f h w c -> f c h w")
+ payload = self._payload.to(device=device, dtype=torch.float32)
+ stamped = self._model.embed(nchw, msgs=payload, is_video=True, lowres_attenuation=True)["imgs_w"]
+ return rearrange(stamped, "f c h w -> f h w c").clamp(0.0, 1.0).to(device=frames.device)
+
+ def detect(self, frames: Iterator[torch.Tensor]) -> MarkDetectResult:
+ """Sample at ``step_size``, average logits; ``detected`` is ``log10(p_value) < -6``."""
+ device = next(self._model.parameters()).device
+ step_size = int(getattr(self._model, "step_size", 8))
+ logits: list[torch.Tensor] = []
+ sampled = _nchw_sampled_frames(frames, step_size=step_size, device=device)
+ for video in _nchw_batches(sampled, _DETECT_FRAME_BATCH):
+ bit_logits = self._model.detect(video, is_video=True)["preds"][:, 1:]
+ while bit_logits.ndim > 2:
+ bit_logits = bit_logits.mean(dim=-1)
+ logits.append(bit_logits)
+ if not logits:
+ raise ValueError("no frames to detect")
+ mean_logits = torch.cat(logits, dim=0).mean(dim=0)
+ decoded = (mean_logits > 0).to(dtype=torch.float32)
+ bit_accuracy, p_value = score_bits(decoded, self._payload)
+ detected = p_value == 0.0 or math.log10(p_value) < -6
+ return MarkDetectResult(
+ detected=detected,
+ bit_accuracy=bit_accuracy,
+ p_value=p_value,
+ payload=self._payload.detach().cpu(),
+ )
+
+
+def _nchw_sampled_frames(
+ frames: Iterator[torch.Tensor], *, step_size: int, device: torch.device
+) -> Iterator[torch.Tensor]:
+ for index, frame in enumerate(frames):
+ if index % step_size != 0:
+ continue
+ rgb = frame.to(device=device, dtype=torch.float32)
+ if rgb.ndim != 3 or rgb.shape[-1] != 3:
+ raise ValueError(f"expected frame [H, W, C]; got {tuple(rgb.shape)}")
+ yield rearrange(rgb, "h w c -> c h w")
+
+
+def _nchw_batches(frames: Iterator[torch.Tensor], batch_size: int) -> Iterator[torch.Tensor]:
+ while batch := list(islice(frames, batch_size)):
+ yield torch.stack(batch, dim=0)
+
+
+def baked_videoseal_payload(nbits: int) -> torch.Tensor:
+ """Return the baked payload as a ``[1, nbits]`` float ``{0, 1}`` tensor."""
+ generator = torch.Generator(device="cpu")
+ generator.manual_seed(VIDEOSEAL_PAYLOAD_SEED)
+ return torch.randint(0, 2, (1, nbits), generator=generator).float()
+
+
+def load_videoseal(
+ checkpoint: Path | str,
+ *,
+ card: Path | str,
+ attenuation_config: Path | str,
+) -> tuple[torch.nn.Module, int]:
+ """Build a Videoseal module from local checkpoint, card YAML, and attenuation YAML paths.
+ Videoseal's ``setup_model_from_checkpoint`` looks up repo-root ``configs/*.yaml``
+ that the installed package does not ship. Pass the JND attenuation YAML from the pack
+ (``videoseal_attenuation.yaml``).
+ """
+ card_path = Path(card)
+ if not card_path.is_file():
+ raise FileNotFoundError(f"Videoseal model card not found: {card_path}")
+ raw = OmegaConf.load(card_path)
+ raw.args.attenuation_config = str(Path(attenuation_config))
+ nbits = int(raw.args.nbits)
+ config = VideosealConfig(
+ args=raw.args,
+ embedder=SubModelConfig(model=str(raw.embedder.model), params=raw.embedder.params),
+ extractor=SubModelConfig(model=str(raw.extractor.model), params=raw.extractor.params),
+ )
+ return setup_model(config, Path(checkpoint)), nbits
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/controller.py b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/controller.py
index d90122d..b340e24 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/controller.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/controller.py
@@ -161,12 +161,18 @@ class MGPUController:
num_gpus: int | None = None,
devices: Sequence[int] | None = None,
logs_specs: LogsSpecs | None = None,
+ cpu_threads_per_rank: int | None = 8,
) -> None:
"""`num_gpus` uses GPUs 0..num_gpus-1 (default: all). `devices` places the fleet on specific
physical GPUs (e.g. `[2, 3]` -> rank r on cuda:devices[r]), so several controllers can share a
box on disjoint GPU sets; the two are mutually exclusive. Indices are as the controller sees
them; every GPU stays visible to each worker, which simply binds to its assigned one.
+ `cpu_threads_per_rank` sets each rank's intra-op CPU threads (`torch.set_num_threads`); None
+ keeps torch's default. Ranks only run small host tensor ops, which a pool as wide as the machine
+ stalls.
"""
+ if cpu_threads_per_rank is not None and cpu_threads_per_rank < 1:
+ raise ValueError(f"cpu_threads_per_rank must be >= 1: {cpu_threads_per_rank}.")
if devices is not None:
if num_gpus is not None:
raise ValueError("Pass either num_gpus or devices, not both.")
@@ -176,6 +182,7 @@ class MGPUController:
self._devices = list(devices) if devices is not None else None
self._num_gpus = num_gpus
self._logs_specs = logs_specs
+ self._cpu_threads_per_rank = cpu_threads_per_rank
self._spawn_ctx = torch_mp.get_context("spawn")
self._fleet: _RunnersFleet | None = None
@@ -232,6 +239,7 @@ class MGPUController:
num_gpus=num_gpus,
logs_specs=self._logs_specs or DefaultLogsSpecs(tee=Std.ALL),
device_ids=device_ids,
+ cpu_threads=self._cpu_threads_per_rank,
)
try:
self._await_ready(num_gpus, timeout.total_seconds())
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/decode_tiling.py b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/decode_tiling.py
new file mode 100644
index 0000000..1ec8c74
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/decode_tiling.py
@@ -0,0 +1,143 @@
+"""Distributed-decode tile planning shared by the multi-GPU runners.
+A Dist decode splits the latent over the ranks of a VAE group, and each rank decodes its share
+with a :class:`TilingConfig` of its own. :func:`dist_auto_tiling` resolves ``AUTO_TILING`` to one
+plan for the whole group: every rank plans the group's largest Dist tile from the group's
+smallest budgets, so the plans, and whether the fallback budget is needed, agree on every rank,
+cold or warm, without a further collective.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from functools import wraps
+from typing import Protocol
+
+import torch
+import torch.distributed as dist
+
+from ltx_core.devices import cuda_allocatable_bytes, resolve_device
+from ltx_core.model.video_vae import AutoTiling, TilingConfig
+from ltx_core.model.video_vae.transformer import DiffVAEMode
+from ltx_core.multigpu.vae.distributed_decoder import dist_rank_tile_pixel_shape, dist_rank_tiles
+from ltx_core.tiling import TileCountConfig
+from ltx_core.types import VIDEO_SCALE_FACTORS, SpatioTemporalScaleFactors, VideoLatentShape, VideoPixelShape
+
+# Share of device memory the decode plan may fill. The rest serves what runs beside the decode:
+# streamed chunks waiting for their temporal batch, the batch being blended, and the encoder.
+DECODE_LIVE_SHARE = 0.858
+# How ensure_tiling_config's ValueError starts when no tile fits the budget.
+_NO_TILE_FITS = "Cannot fit a DiffVAE decode tile"
+
+
+class EnsureTilingConfig(Protocol):
+ """The signature of :func:`ltx_pipelines.utils.helpers.ensure_tiling_config`."""
+
+ def __call__(
+ self,
+ tiling_config: TilingConfig | AutoTiling | None,
+ *,
+ video_shape: VideoPixelShape,
+ scale_factors: SpatioTemporalScaleFactors,
+ vae_checkpoint_path: str,
+ diffvae_optimization: DiffVAEMode = DiffVAEMode.CHUNKED_EAGER,
+ device: torch.device | None = None,
+ free_bytes: int | None = None,
+ keyframes: bool = False,
+ ) -> TilingConfig | None: ...
+
+
+@dataclass(frozen=True)
+class GroupDecodeBudget:
+ """The smallest decode budgets over a VAE group, in bytes.
+ Attributes:
+ planned: ``DECODE_LIVE_SHARE`` of device memory minus allocated bytes. Allocated bytes do
+ not depend on what the caching allocator holds, so this is the same cold or warm.
+ fallback: Everything a new allocation can still get (:func:`cuda_allocatable_bytes`), for
+ a clip the planned budget fits no tile for.
+ """
+
+ planned: int
+ fallback: int
+
+
+def group_decode_budget(device: torch.device | None, group: dist.ProcessGroup | None) -> GroupDecodeBudget:
+ """Each rank's complete budgets, MIN-reduced over ``group``; one collective and one host sync."""
+ resolved = resolve_device(device)
+ total = torch.cuda.get_device_properties(resolved).total_memory
+ planned = max(0, int(DECODE_LIVE_SHARE * total) - int(torch.cuda.memory_allocated(resolved)))
+ budgets = torch.tensor([planned, cuda_allocatable_bytes(resolved)], dtype=torch.int64, device=resolved)
+ dist.all_reduce(budgets, op=dist.ReduceOp.MIN, group=group)
+ planned, fallback = budgets.tolist()
+ return GroupDecodeBudget(planned=planned, fallback=fallback)
+
+
+def dist_max_tile_pixel_shape(
+ vae_tiling: TileCountConfig,
+ video_shape: VideoPixelShape,
+ world_size: int,
+ scale: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,
+) -> VideoPixelShape:
+ """:func:`dist_rank_tile_pixel_shape` of the rank with the largest Dist tile, the same on every rank."""
+ latent_shape = VideoLatentShape.from_pixel_shape(video_shape, scale_factors=scale)
+ shapes = [
+ dist_rank_tile_pixel_shape(
+ vae_tiling, latent_shape, scale, rank, world_size, fps=video_shape.fps, batch=video_shape.batch
+ )
+ for rank in range(world_size)
+ if dist_rank_tiles(vae_tiling, latent_shape, scale, rank, world_size)
+ ]
+ return max(shapes, key=lambda shape: shape.frames * shape.height * shape.width)
+
+
+def dist_auto_tiling(
+ ensure: EnsureTilingConfig,
+ vae_tiling: TileCountConfig,
+ *,
+ world_size: int,
+ group: dist.ProcessGroup | None,
+ scale: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,
+) -> EnsureTilingConfig:
+ """Wrap ``ensure`` so ``AUTO_TILING`` without ``free_bytes`` plans one Dist tile for the group.
+ The wrapper plans :func:`dist_max_tile_pixel_shape` from :func:`group_decode_budget`'s
+ planned budget, and from its fallback budget if no tile fits that. Every rank joins the one
+ collective and plans the same shape from the same numbers, so every rank resolves the same
+ plan or raises the same error. Explicit configs, ``None`` and calls with ``free_bytes`` pass
+ through unchanged.
+ """
+
+ @wraps(ensure)
+ def ensure_dist_tile(
+ tiling_config: TilingConfig | AutoTiling | None,
+ *,
+ video_shape: VideoPixelShape,
+ scale_factors: SpatioTemporalScaleFactors,
+ vae_checkpoint_path: str,
+ diffvae_optimization: DiffVAEMode = DiffVAEMode.CHUNKED_EAGER,
+ device: torch.device | None = None,
+ free_bytes: int | None = None,
+ keyframes: bool = False,
+ ) -> TilingConfig | None:
+ def plan(shape: VideoPixelShape, budget: int | None) -> TilingConfig | None:
+ return ensure(
+ tiling_config,
+ video_shape=shape,
+ scale_factors=scale_factors,
+ vae_checkpoint_path=vae_checkpoint_path,
+ diffvae_optimization=diffvae_optimization,
+ device=device,
+ free_bytes=budget,
+ keyframes=keyframes,
+ )
+
+ if not isinstance(tiling_config, AutoTiling) or free_bytes is not None:
+ return plan(video_shape, free_bytes)
+ tile_shape = dist_max_tile_pixel_shape(vae_tiling, video_shape, world_size, scale)
+ budget = group_decode_budget(device, group)
+ try:
+ return plan(tile_shape, budget.planned)
+ except ValueError as exc:
+ if not str(exc).startswith(_NO_TILE_FITS):
+ raise
+ return plan(tile_shape, budget.fallback)
+
+ return ensure_dist_tile
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/delegating_builder.py b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/delegating_builder.py
index 0be95cb..6061171 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/delegating_builder.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/delegating_builder.py
@@ -72,6 +72,10 @@ class DelegatingBuilder(Generic[InnerModelT]):
def keeps_gpu_resident_weights(self) -> bool:
return self._inner.keeps_gpu_resident_weights
+ @property
+ def builds_per_shape(self) -> bool:
+ return self._inner.builds_per_shape
+
def with_registry(self, registry: Registry) -> Self:
clone = copy.copy(self)
clone._inner = self._inner.with_registry(registry)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/fleet.py b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/fleet.py
index 420a191..fe1e45d 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/fleet.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/fleet.py
@@ -9,6 +9,7 @@ lives in ``runner.py``.
from __future__ import annotations
import contextlib
+import gc
import logging
import os
import socket
@@ -26,7 +27,7 @@ from torch.distributed.elastic.multiprocessing import start_processes
from torch.distributed.elastic.multiprocessing.api import LogsSpecs
from torch.distributed.elastic.multiprocessing.errors import ProcessFailure, record
-from ltx_pipelines.multigpu._broadcast import BroadcastCoordinator
+from ltx_core.multigpu.broadcast import BroadcastCoordinator
from ltx_pipelines.multigpu.nccl_groups import create_local_nccl_groups
from ltx_pipelines.multigpu.runner import MGPURunner, RunnerError
@@ -160,6 +161,18 @@ class _RankLink:
return shell
+def _freeze_long_lived_heap() -> None:
+ """After a rank's first job, move everything that survived it out of the cyclic GC's reach.
+ The first job builds the models, compiled graphs and caches the later jobs reuse; full
+ collections otherwise rescan those objects every time (3.3 s of pauses in a cold 4K DFR job).
+ Collected first, so only survivors are frozen; anything a later job leaves in a cycle is
+ still collected.
+ """
+ if gc.get_freeze_count() == 0:
+ gc.collect()
+ gc.freeze()
+
+
def _run_job(runner: MGPURunner, kwargs: dict[str, Any], job_id: int, rank: int, result: Queue[_JobResult]) -> None:
"""Run one job on this rank and put each item on the result queue: every yield is a `_JobResult`
(tagged by rank), forwarded straight to the caller; the rank's end -- a clean StopIteration or a
@@ -194,6 +207,7 @@ def _relay_loop(runner: MGPURunner, link: _RankLink, channels: _Channels) -> Non
if job is None: # shutdown sentinel (already broadcast to the others)
return
_run_job(runner, job.kwargs, job.job_id, _RELAY_RANK, channels.results)
+ _freeze_long_lived_heap()
def _worker_loop(runner: MGPURunner, link: _RankLink, channels: _Channels, rank: int) -> None:
@@ -203,6 +217,7 @@ def _worker_loop(runner: MGPURunner, link: _RankLink, channels: _Channels, rank:
if job is None: # shutdown sentinel
return
_run_job(runner, job.kwargs, job.job_id, rank, channels.results)
+ _freeze_long_lived_heap()
def _shutdown_distributed() -> None:
@@ -223,9 +238,15 @@ def _worker_entrypoint(
init_timeout: timedelta,
channels: _Channels,
device_ids: list[int] | None = None,
+ cpu_threads: int | None = 8,
) -> None:
"""Per-worker entry. @record turns a crash into a readable failure on the controller."""
local_rank = int(os.environ["LOCAL_RANK"])
+ # Torch's CPU pool defaults to one thread per core. A rank runs only small host tensor ops (conditioning
+ # masks, tile plans), which a pool that wide stalls -- tens to hundreds of ms per op on a 176-CPU host --
+ # while the GPU waits for the work they gate. None keeps torch's default.
+ if cpu_threads is not None:
+ torch.set_num_threads(cpu_threads)
# All GPUs stay visible; we bind this rank to its physical GPU by index. (Setting
# CUDA_VISIBLE_DEVICES here is unreliable -- the spawn bootstrap can touch CUDA, freezing
# the device list, before a late env override would apply.)
@@ -320,6 +341,7 @@ class _RunnersFleet:
num_gpus: int,
logs_specs: LogsSpecs,
device_ids: list[int] | None = None,
+ cpu_threads: int | None = 8,
) -> _RunnersFleet:
port = _find_free_port()
envs = {
@@ -334,7 +356,7 @@ class _RunnersFleet:
}
# device_ids maps rank -> physical GPU (None = identity); the worker binds to it. The runner
# class ships by value so a __main__/test-module runner survives the spawn (see _RunnerShipper).
- packed = (_RunnerShipper(runner_cls), setup_kwargs, init_timeout, channels, device_ids)
+ packed = (_RunnerShipper(runner_cls), setup_kwargs, init_timeout, channels, device_ids, cpu_threads)
args = dict.fromkeys(range(num_gpus), packed)
logger.info("Spawning %d MGPU workers...", num_gpus)
return cls(
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/gemma_builders.py b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/gemma_builders.py
index 7822dcb..d02152f 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/gemma_builders.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/gemma_builders.py
@@ -15,6 +15,7 @@ dispatch hooks. The outer wrapper is not cached.
from __future__ import annotations
import copy
+import itertools
import logging
from pathlib import Path
from typing import TYPE_CHECKING, Any
@@ -24,8 +25,10 @@ import torch.distributed as dist
from accelerate import dispatch_model
from transformers import AutoModelForImageTextToText
+from ltx_core.loader.helpers import assign_state_dict
from ltx_core.loader.primitives import BuilderProtocol, StateDict
from ltx_core.loader.registry import Registry, module_registry_key
+from ltx_core.multigpu.broadcast import BroadcastCoordinator
from ltx_core.multigpu.gemma.accelerate_wrapper import AccelerateGemmaWrapper
from ltx_core.multigpu.gemma.loader import load_gemma_with_device_map
from ltx_core.text_encoders.gemma import gemma_model_config
@@ -36,6 +39,9 @@ if TYPE_CHECKING:
logger = logging.getLogger(__name__)
+# Numbers this process's builders in construction order, which every rank shares, to give each its own store key.
+_builder_ids = itertools.count()
+
def _assign_non_persistent(model: torch.nn.Module, non_persistent_sd: dict[str, torch.Tensor]) -> None:
for name, tensor in non_persistent_sd.items():
@@ -69,6 +75,9 @@ class AccelerateGemmaBuilder(BuilderProtocol):
Conforms to the builder interface expected by ``PromptEncoder``:
``build(device, dtype) -> model``. Non-source ranks get a lightweight
:class:`AccelerateGemmaWrapper` that receives embeddings via broadcast.
+ Every rank must construct its builders in the same order (in ``setup``, before any job): each
+ gets its own :class:`BroadcastCoordinator`, keyed by construction order, which it hands to every
+ wrapper it builds.
"""
def __init__(
@@ -88,6 +97,10 @@ class AccelerateGemmaBuilder(BuilderProtocol):
self._src_rank = src_rank
self._is_src = dist.get_rank() == src_rank
self._dtype = dtype
+ store = dist.distributed_c10d.PrefixStore(
+ f"ltx_gemma_broadcast/{next(_builder_ids)}/", dist.distributed_c10d._get_default_store()
+ )
+ self._coordinator = BroadcastCoordinator(store=store, is_driver=self._is_src)
# Cached on the src rank after first build (non-tensor objects).
self._config: object | None = None
self._hf_device_map: dict[str, int | str] | None = None
@@ -133,6 +146,7 @@ class AccelerateGemmaBuilder(BuilderProtocol):
src_rank=dist.get_group_rank(self._broadcast_group, self._src_rank),
dtype=dtype,
device=device,
+ coordinator=self._coordinator,
)
def _build_encoder(self, dtype: torch.dtype) -> LTXGemmaTextEncoder:
@@ -147,7 +161,7 @@ class AccelerateGemmaBuilder(BuilderProtocol):
"registry must outlive the builder."
)
encoder = cached_model # type: ignore[assignment]
- encoder.model.load_state_dict(cached_sd.sd, strict=False, assign=True)
+ assign_state_dict(encoder.model, cached_sd.sd)
encoder.dtype = dtype
return encoder
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/tdp_builder.py b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/tdp_builder.py
index 71d586c..7792c1d 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/tdp_builder.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/tdp_builder.py
@@ -50,6 +50,11 @@ class TiledDataParallelBuilder(DelegatingBuilder[InnerModelT], Generic[InnerMode
self._tiling = tiling
self._normalize_positions = normalize_positions
+ @property
+ def builds_per_shape(self) -> bool:
+ # Each build wraps the model for one call's video tools, so it cannot serve another shape.
+ return True
+
@property
def keeps_gpu_resident_weights(self) -> bool:
# Weight tracker keeps registry tensors GPU-resident and rebinds them across builds.
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/weight_tracker.py b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/weight_tracker.py
index 7d5d314..d7db6f9 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/weight_tracker.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/weight_tracker.py
@@ -4,11 +4,16 @@ Shared infrastructure used by both TDP and SP builders.
from __future__ import annotations
+import functools
+from collections.abc import Iterable
+
+import safetensors
import torch
import torch.distributed as dist
from ltx_core.loader.fuse_loras import fuse_lora_weights
from ltx_core.loader.primitives import LoraPathStrengthAndSDOps, LoraStateDictWithStrength, StateDict
+from ltx_core.loader.sd_ops import SDOps
from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder
from ltx_core.model.model_protocol import ModelType
from ltx_core.multigpu.sharded_sd import ShardedSD
@@ -20,8 +25,10 @@ def _apply_loras_inplace(
loras: tuple[LoraPathStrengthAndSDOps, ...],
builder: Builder, # type: ignore[type-arg]
lora_keys: frozenset[str],
-) -> None:
- """Reset *target* to clean weights from *source*, then fuse all LoRAs in one pass."""
+) -> list[str]:
+ """Reset *target* to clean weights from *source*, then fuse all LoRAs in one pass.
+ Returns the keys the fusion wrote.
+ """
for key, clean_weight in source.items():
target[key].copy_(clean_weight)
@@ -41,8 +48,32 @@ def _apply_loras_inplace(
target_sd = StateDict(
sd=target, device=next(iter(target.values())).device, size=0, dtype={next(iter(target.values())).dtype}
)
+ written: list[str] = []
for key, fused in fuse_lora_weights(target_sd, lora_sds, fuse_rule=builder.fuse_rule):
target[key].copy_(fused)
+ written.append(key)
+ return written
+
+
+def _lora_module_prefixes(path: str, sd_ops: SDOps) -> frozenset[str]:
+ """The model modules a LoRA file targets, from its header's ``lora_A`` keys mapped through ``sd_ops``."""
+ suffix = ".lora_A.weight"
+ with safetensors.safe_open(path, framework="pt") as f:
+ names = [sd_ops.apply_to_key(key) for key in f.keys()] # noqa: SIM118 -- safe_open is not a mapping
+ return frozenset(name[: -len(suffix)] for name in names if name is not None and name.endswith(suffix))
+
+
+def lora_footprint(keys: Iterable[str], loras: Iterable[LoraPathStrengthAndSDOps]) -> frozenset[str]:
+ """The keys of ``keys`` a LoRA set can write: every tensor of a module one of its non-zero LoRAs targets.
+ That covers the fused ``.weight`` and any companion a fuse rule rewrites with it (``.weight_scale``, ...).
+ Read from the LoRA files' headers, so every rank derives the same set without a collective.
+ """
+ prefixes: set[str] = set()
+ for lora in loras:
+ if lora.strength != 0:
+ prefixes |= _lora_module_prefixes(lora.path, lora.sd_ops)
+ return frozenset(key for key in keys if key.rpartition(".")[0] in prefixes)
class TransformerWeightTracker:
@@ -57,6 +88,9 @@ class TransformerWeightTracker:
After in-place LoRA fusion on the owning rank, this broadcasts the
fused results to all other ranks so every rank sees the same weights.
Both are created together and are always either both ``None`` or both set.
+ A swap touches only the LoRA footprints (:func:`lora_footprint`): the keys the outgoing set
+ wrote (``dirty``) are restored and the incoming set's are re-fused, and only their union is
+ broadcast. Weights neither set targets stay as they are, clean.
With ``no_lora_swap``, the LoRA set is assumed fixed (none, or one set):
the backup clone is skipped and any swap or reset raises.
"""
@@ -71,6 +105,7 @@ class TransformerWeightTracker:
self.stored_sd: ShardedSD | None = None
self.broadcast_sd: ShardedSD | None = None
self.loras: list[tuple[str, float, str]] = []
+ self.dirty: frozenset[str] = frozenset() # keys holding LoRA-fused values, the same on every rank
@property
def staging(self) -> torch.Tensor:
@@ -86,7 +121,7 @@ class TransformerWeightTracker:
return sorted(lora_list) == sorted(self.loras)
def reset_loras(self, target_sd: dict[str, torch.Tensor]) -> None:
- """Restore *target_sd* to original (pre-LoRA) weights.
+ """Restore *target_sd* to original (pre-LoRA) weights: the keys the applied LoRAs wrote.
No-op if no LoRAs are currently applied. This is a cooperative
operation — all ranks must call it simultaneously.
"""
@@ -96,8 +131,8 @@ class TransformerWeightTracker:
raise RuntimeError("no_lora_swap tracker has no backup to reset from")
if self.stored_sd is None:
raise RuntimeError("stored_sd must be initialised before reset_loras")
- self.loras = []
- self.stored_sd.broadcast_shards_into(target_sd, self.staging)
+ dirty, self.loras, self.dirty = self.dirty, [], frozenset()
+ self.stored_sd.subset(dirty).broadcast_shards_into(target_sd, self.staging)
def _local_lora_keys(self) -> frozenset[str]:
"""Derive LoRA key names from the locally owned model keys."""
@@ -119,9 +154,10 @@ class TransformerWeightTracker:
builder: Builder, # type: ignore[type-arg]
) -> None:
"""Fuse *loras* into *target_sd* in-place (trailing ``_`` denotes in-place).
- Skips work when the requested LoRAs already match. Restores stored
- weights before applying new LoRAs. This is a cooperative operation —
- all ranks must call it simultaneously.
+ Skips work when the requested LoRAs already match. Restores the stored
+ weights of the outgoing set's footprint, fuses the new set into its own,
+ and broadcasts only the union. This is a cooperative operation — all
+ ranks must call it simultaneously.
"""
new_loras = [(lora.path, lora.strength, lora.sd_ops.name) for lora in loras]
@@ -138,13 +174,19 @@ class TransformerWeightTracker:
if self.stored_sd is None or self.broadcast_sd is None:
raise RuntimeError("ShardedSDs must be initialised before apply_loras_ (call build first)")
- source = self.stored_sd.local_shard
- target = {k: v for k, v in target_sd.items() if k in source}
- lora_keys = self._local_lora_keys()
- _apply_loras_inplace(source, target, loras, builder, lora_keys)
-
- self.broadcast_sd.broadcast_shards_into(target_sd, self.staging)
+ footprint = lora_footprint(self.stored_sd.keys, loras)
+ touched = footprint | self.dirty
+ source = {k: v for k, v in self.stored_sd.local_shard.items() if k in touched}
+ target = {k: target_sd[k] for k in source}
+ written = _apply_loras_inplace(source, target, loras, builder, self._local_lora_keys()) if target else []
+ stray = [key for key in written if key not in footprint]
+ if stray:
+ # Other ranks would never receive these: refuse rather than leave the ranks' weights apart.
+ raise RuntimeError(f"LoRA fusion wrote keys outside the LoRA set's footprint: {stray[:5]}")
+
+ self.broadcast_sd.subset(touched).broadcast_shards_into(target_sd, self.staging)
self.loras = new_loras
+ self.dirty = footprint
def build(
self,
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/retake.py b/packages/ltx-pipelines/src/ltx_pipelines/retake.py
index cdf2ad4..6f9d3eb 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/retake.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/retake.py
@@ -19,7 +19,10 @@ from ltx_core.types import (
SpatioTemporalScaleFactors,
VideoPixelShape,
)
-from ltx_pipelines.utils.args import video_editing_arg_parser
+from ltx_pipelines.utils.args import (
+ markers_from_args,
+ video_editing_arg_parser,
+)
from ltx_pipelines.utils.blocks import (
AudioConditioner,
AudioDecoder,
@@ -340,6 +343,8 @@ def main() -> None:
parser = video_editing_arg_parser(distilled=True)
parser.description = "Retake: regenerate a time region of a video with LTX-2."
args = parser.parse_args()
+ hdr = resolve_hdr_color_space(video_paths=[args.video_path], hdr=args.hdr)
+ video_marker, audio_marker = markers_from_args(args, device=get_device(), color_space=hdr)
if args.start_time >= args.end_time:
raise ValueError("start_time must be less than end_time")
@@ -367,7 +372,6 @@ def main() -> None:
diffvae_optimization=args.diffvae_optimization,
)
params = detect_params(args.model_paths.transformer())
- hdr = resolve_hdr_color_space(video_paths=[args.video_path], hdr=args.hdr)
vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)
result = pipeline(
video_path=args.video_path,
@@ -393,6 +397,8 @@ def main() -> None:
output_path=args.output_path,
video_chunks_number=get_video_chunks_number(result.num_frames, result.tiling_config),
color_space=hdr,
+ video_marker=video_marker,
+ audio_marker=audio_marker,
)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/t2a_one_stage.py b/packages/ltx-pipelines/src/ltx_pipelines/t2a_one_stage.py
index 4c1848b..fafdb70 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/t2a_one_stage.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/t2a_one_stage.py
@@ -19,6 +19,7 @@ from ltx_core.types import Audio, AudioLatentShape
from ltx_pipelines.utils import get_device
from ltx_pipelines.utils.args import (
default_1_stage_t2a_arg_parser,
+ markers_from_args,
resolve_cli_params,
)
from ltx_pipelines.utils.blocks import (
@@ -179,6 +180,7 @@ def main() -> None:
params = resolve_cli_params()
parser = default_1_stage_t2a_arg_parser(params=params)
args = parser.parse_args()
+ _, audio_marker = markers_from_args(args, device=get_device())
pipeline = T2AOneStagePipeline(
model_paths=args.model_paths,
loras=tuple(args.lora) if args.lora else (),
@@ -209,7 +211,7 @@ def main() -> None:
max_batch_size=args.max_batch_size,
)
- encode_audio(audio=audio, output_path=args.output_path)
+ encode_audio(audio=audio, output_path=args.output_path, audio_marker=audio_marker)
if __name__ == "__main__":
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_one_stage.py b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_one_stage.py
index a9dd0cc..56cb271 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_one_stage.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_one_stage.py
@@ -22,6 +22,7 @@ from ltx_pipelines.utils.args import (
add_generated_keyframes_arg,
add_keyframe_decode_arg,
default_1_stage_arg_parser,
+ markers_from_args,
resolve_cli_params,
)
from ltx_pipelines.utils.blocks import (
@@ -38,12 +39,12 @@ from ltx_pipelines.utils.denoisers import FactoryGuidedDenoiser
from ltx_pipelines.utils.helpers import (
assert_generated_keyframes_request,
assert_resolution,
- combined_image_conditionings,
create_initial_av_latents,
decode_keyframes_from_slots,
ensure_tiling_config,
generated_keyframe_conditionings,
get_device,
+ image_conditionings,
resolve_generated_keyframes,
tiling_scale_factors_for_vae,
)
@@ -164,7 +165,8 @@ class TI2VidOneStagePipeline:
decode_with_keyframes: bool = False,
) -> PipelineOutput:
require_num_frames_source(num_frames, self.duration_predictor)
- images = self.image_conditioner.resolve_crf(images)
+ enhance_prompt_image = images[0].path if images else None
+ images = self.image_conditioner.load_conditionings(images, color_space)
assert_resolution(height=height, width=width, is_two_stage=False)
assert_generated_keyframes_request(decode_with_keyframes, generated_keyframes, self.stage)
@@ -178,7 +180,7 @@ class TI2VidOneStagePipeline:
[prompt, negative_prompt],
enhance_first_prompt=enhance_prompt,
enhance_static_cache=enhance_static_cache,
- enhance_prompt_image=images[0][0] if len(images) > 0 else None,
+ enhance_prompt_image=enhance_prompt_image,
)
v_context_p, a_context_p = ctx_p.video_encoding, ctx_p.audio_encoding
v_context_n, a_context_n = ctx_n.video_encoding, ctx_n.audio_encoding
@@ -203,14 +205,13 @@ class TI2VidOneStagePipeline:
)
stage_1_conditionings = self.image_conditioner(
- lambda enc: combined_image_conditionings(
+ lambda enc: image_conditionings(
images=images,
height=height,
width=width,
video_encoder=enc,
dtype=dtype,
device=self.device,
- color_space=color_space,
)
)
stage_1_conditionings.extend(generated_keyframe_conditionings(generated_keyframes, num_frames))
@@ -292,6 +293,8 @@ def main() -> None:
add_generated_keyframes_arg(default_1_stage_arg_parser(params=params, supports_auto_duration=True))
)
args = parser.parse_args()
+ hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)
+ video_marker, audio_marker = markers_from_args(args, device=get_device(), color_space=hdr)
pipeline = TI2VidOneStagePipeline(
model_paths=args.model_paths,
loras=tuple(args.lora) if args.lora else (),
@@ -301,7 +304,6 @@ def main() -> None:
prompt_enhancer_gemma_root=args.prompt_enhancer_gemma_root,
diffvae_optimization=args.diffvae_optimization,
)
- hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)
vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)
result = pipeline(
prompt=args.prompt,
@@ -346,6 +348,8 @@ def main() -> None:
output_path=args.output_path,
video_chunks_number=get_video_chunks_number(result.num_frames, result.tiling_config),
color_space=hdr,
+ video_marker=video_marker,
+ audio_marker=audio_marker,
)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py
index 8238658..611e847 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py
@@ -43,6 +43,7 @@ from ltx_pipelines.utils.args import (
add_keyframe_decode_arg,
chunk_config_from_args,
default_2_stage_arg_parser,
+ markers_from_args,
resolve_cli_params,
)
from ltx_pipelines.utils.blocks import (
@@ -74,6 +75,7 @@ from ltx_pipelines.utils.types import (
DEFAULT_AUTO_DURATION,
AutoDuration,
ImageConditioningInput,
+ LoadedImageConditioning,
OffloadMode,
PipelineOutput,
VideoAudio,
@@ -90,8 +92,8 @@ class _TI2VidRunContext:
noiser: GaussianNoiser
dtype: torch.dtype
vae_dtype: torch.dtype
- #: ``images`` with every unset CRF filled in, so callers do not re-resolve it.
- images: list[ImageConditioningInput]
+ #: ``images`` loaded once for the run (CRF resolved, stills decoded), so callers do not reload them.
+ images: list[LoadedImageConditioning]
v_context_p: torch.Tensor
a_context_p: torch.Tensor
v_context_n: torch.Tensor
@@ -211,6 +213,7 @@ class TI2VidTwoStagesPipeline:
width: int,
frame_rate: float,
images: list[ImageConditioningInput],
+ color_space: EXRColorSpace | None,
num_frames: int | AutoDuration,
vae_dtype: torch.dtype | None,
tiling_config: TilingConfig | AutoTiling | None,
@@ -222,7 +225,8 @@ class TI2VidTwoStagesPipeline:
) -> _TI2VidRunContext:
"""Validate inputs, encode prompts, and resolve frame count / tiling before chunk planning."""
require_num_frames_source(num_frames, self.duration_predictor)
- images = self.image_conditioner.resolve_crf(images)
+ enhance_prompt_image = images[0].path if images else None
+ images = self.image_conditioner.load_conditionings(images, color_space)
assert_resolution(height=height, width=width, is_two_stage=True)
assert_generated_keyframes_request(decode_with_keyframes, generated_keyframes, self.stage_1)
@@ -235,7 +239,7 @@ class TI2VidTwoStagesPipeline:
[prompt, negative_prompt],
enhance_first_prompt=enhance_prompt,
enhance_static_cache=enhance_static_cache,
- enhance_prompt_image=images[0][0] if len(images) > 0 else None,
+ enhance_prompt_image=enhance_prompt_image,
)
v_context_p, a_context_p = ctx_p.video_encoding, ctx_p.audio_encoding
v_context_n, a_context_n = ctx_n.video_encoding, ctx_n.audio_encoding
@@ -289,9 +293,8 @@ class TI2VidTwoStagesPipeline:
def _video_conditionings(
self,
- images: list[ImageConditioningInput],
+ images: list[LoadedImageConditioning],
num_frames: int,
- color_space: EXRColorSpace | None,
) -> MakeVideoConditionings:
assert_image_frames_in_clip(images, num_frames)
@@ -303,7 +306,6 @@ class TI2VidTwoStagesPipeline:
chunk,
images=images,
video_encoder=enc,
- color_space=color_space,
)
)
@@ -333,9 +335,11 @@ class TI2VidTwoStagesPipeline:
generated_keyframes: int | Sequence[int] = 0,
decode_with_keyframes: bool = False,
chunk_config: ChunkConfig | None = None,
+ refine_audio: bool = False,
) -> PipelineOutput:
"""Generate a video.
``chunk_config=None`` plans a single chunk covering the snapped clip.
+ Set ``refine_audio=True`` to denoise audio in stage 2; by default stage-1 audio is preserved.
"""
chunks, num_frames, tiling_config = self.stream_chunks(
prompt=prompt,
@@ -360,6 +364,7 @@ class TI2VidTwoStagesPipeline:
generated_keyframes=generated_keyframes,
decode_with_keyframes=decode_with_keyframes,
chunk_config=chunk_config,
+ refine_audio=refine_audio,
)
return pipeline_output_from_chunks(chunks, num_frames=num_frames, tiling_config=tiling_config)
@@ -387,9 +392,11 @@ class TI2VidTwoStagesPipeline:
generated_keyframes: int | Sequence[int] = 0,
decode_with_keyframes: bool = False,
chunk_config: ChunkConfig | None = None,
+ refine_audio: bool = False,
) -> tuple[Iterator[DecodedChunk], int, TilingConfig]:
"""Generate a video as a stream of decoded chunks.
``chunk_config=None`` plans a single chunk covering the snapped clip.
+ Set ``refine_audio=True`` to denoise audio in stage 2; by default stage-1 audio is preserved.
"""
ctx = self._prepare_run(
prompt=prompt,
@@ -399,6 +406,7 @@ class TI2VidTwoStagesPipeline:
width=width,
frame_rate=frame_rate,
images=images,
+ color_space=color_space,
num_frames=num_frames,
vae_dtype=vae_dtype,
tiling_config=tiling_config,
@@ -440,7 +448,7 @@ class TI2VidTwoStagesPipeline:
dtype=ctx.dtype,
config=ctx.chunk_config,
video_scale_factors=self.stage_1.video_scale_factors,
- make_video_conditionings=self._video_conditionings(ctx.images, ctx.num_frames, color_space),
+ make_video_conditionings=self._video_conditionings(ctx.images, ctx.num_frames),
generated_keyframes=generated_keyframes,
)
base = denoise_chunks(
@@ -459,7 +467,7 @@ class TI2VidTwoStagesPipeline:
sigmas=stage_2_sigmas,
noiser=ctx.noiser,
fps=frame_rate,
- freeze_audio=True,
+ freeze_audio=not refine_audio,
)
return (
decode_chunks(
@@ -486,7 +494,14 @@ def main() -> None:
add_generated_keyframes_arg(default_2_stage_arg_parser(params=params, supports_auto_duration=True))
)
)
+ parser.add_argument(
+ "--refine-audio",
+ action="store_true",
+ help="Denoise audio in stage 2 instead of preserving the stage-1 audio.",
+ )
args = parser.parse_args()
+ hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)
+ video_marker, audio_marker = markers_from_args(args, device=get_device(), color_space=hdr)
pipeline = TI2VidTwoStagesPipeline(
model_paths=args.model_paths,
distilled_lora=args.distilled_lora,
@@ -498,7 +513,6 @@ def main() -> None:
prompt_enhancer_gemma_root=args.prompt_enhancer_gemma_root,
diffvae_optimization=args.diffvae_optimization,
)
- hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)
vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)
chunk_config = chunk_config_from_args(args)
video_guider_params = MultiModalGuiderParams(
@@ -538,6 +552,7 @@ def main() -> None:
generated_keyframes=args.num_generated_keyframes,
decode_with_keyframes=args.decode_with_keyframes,
chunk_config=chunk_config,
+ refine_audio=args.refine_audio,
)
if chunk_config is not None:
video, audio, audio_sampling_rate = split_decoded_chunks(chunks)
@@ -558,6 +573,8 @@ def main() -> None:
video_chunks_number=video_chunks_number,
color_space=hdr,
audio_sampling_rate=audio_sampling_rate,
+ video_marker=video_marker,
+ audio_marker=audio_marker,
)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq.py b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq.py
index 854ea94..d0a890b 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq.py
@@ -19,6 +19,7 @@ from ltx_pipelines.utils.args import (
add_generated_keyframes_arg,
add_keyframe_decode_arg,
hq_2_stage_arg_parser,
+ markers_from_args,
)
from ltx_pipelines.utils.blocks import (
AudioDecoder,
@@ -39,12 +40,12 @@ from ltx_pipelines.utils.denoisers import GuidedDenoiser, SimpleDenoiser
from ltx_pipelines.utils.helpers import (
assert_generated_keyframes_request,
assert_resolution,
- combined_image_conditionings,
create_initial_av_latents,
decode_keyframes_from_slots,
ensure_tiling_config,
generated_keyframe_conditionings,
get_device,
+ image_conditionings,
resolve_generated_keyframes,
tiling_scale_factors_for_vae,
)
@@ -207,7 +208,8 @@ class TI2VidTwoStagesHQPipeline:
decode_with_keyframes: bool = False,
) -> PipelineOutput:
require_num_frames_source(num_frames, self.duration_predictor)
- images = self.image_conditioner.resolve_crf(images)
+ enhance_prompt_image = images[0].path if images else None
+ images = self.image_conditioner.load_conditionings(images, color_space)
assert_resolution(height=height, width=width, is_two_stage=True)
assert_generated_keyframes_request(decode_with_keyframes, generated_keyframes, self.stage_1)
@@ -221,7 +223,7 @@ class TI2VidTwoStagesHQPipeline:
[prompt, negative_prompt],
enhance_first_prompt=enhance_prompt,
enhance_static_cache=enhance_static_cache,
- enhance_prompt_image=images[0][0] if len(images) > 0 else None,
+ enhance_prompt_image=enhance_prompt_image,
)
v_context_p, a_context_p = ctx_p.video_encoding, ctx_p.audio_encoding
v_context_n, a_context_n = ctx_n.video_encoding, ctx_n.audio_encoding
@@ -256,14 +258,13 @@ class TI2VidTwoStagesHQPipeline:
fps=frame_rate,
)
stage_1_conditionings = self.image_conditioner(
- lambda enc: combined_image_conditionings(
+ lambda enc: image_conditionings(
images=images,
height=stage_1_output_shape.height,
width=stage_1_output_shape.width,
video_encoder=enc,
dtype=dtype,
device=self.device,
- color_space=color_space,
)
)
stage_1_conditionings.extend(generated_keyframe_conditionings(generated_keyframes, num_frames))
@@ -325,14 +326,13 @@ class TI2VidTwoStagesHQPipeline:
stage_2_sigmas = stage_2_sigmas.to(dtype=torch.float32, device=self.device)
stage_2_output_shape = VideoPixelShape(batch=1, frames=num_frames, width=width, height=height, fps=frame_rate)
stage_2_conditionings = self.image_conditioner(
- lambda enc: combined_image_conditionings(
+ lambda enc: image_conditionings(
images=images,
height=stage_2_output_shape.height,
width=stage_2_output_shape.width,
video_encoder=enc,
dtype=dtype,
device=self.device,
- color_space=color_space,
)
)
if decode_with_keyframes:
@@ -397,6 +397,8 @@ def main() -> None:
add_generated_keyframes_arg(hq_2_stage_arg_parser(params=LTX_2_3_HQ_PARAMS, supports_auto_duration=True))
)
args = parser.parse_args()
+ hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)
+ video_marker, audio_marker = markers_from_args(args, device=get_device(), color_space=hdr)
pipeline = TI2VidTwoStagesHQPipeline(
model_paths=args.model_paths,
distilled_lora=args.distilled_lora,
@@ -410,7 +412,6 @@ def main() -> None:
prompt_enhancer_gemma_root=args.prompt_enhancer_gemma_root,
diffvae_optimization=args.diffvae_optimization,
)
- hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)
vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)
result = pipeline(
prompt=args.prompt,
@@ -455,6 +456,8 @@ def main() -> None:
output_path=args.output_path,
video_chunks_number=get_video_chunks_number(result.num_frames, result.tiling_config),
color_space=hdr,
+ video_marker=video_marker,
+ audio_marker=audio_marker,
)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq_mgpu.py b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq_mgpu.py
index 273fb35..96e742d 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq_mgpu.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq_mgpu.py
@@ -15,6 +15,7 @@ from __future__ import annotations
import logging
from collections.abc import Iterator, Sequence
from multiprocessing import SimpleQueue
+from types import SimpleNamespace
import torch
import torch.distributed as dist
@@ -37,7 +38,9 @@ from ltx_pipelines.multigpu.tdp_builder import TiledDataParallelBuilder
from ltx_pipelines.multigpu.vae_builders import DistributedDecoderBuilder
from ltx_pipelines.multigpu.weight_tracker import TransformerWeightTracker
from ltx_pipelines.ti2vid_two_stages_hq import TI2VidTwoStagesHQPipeline
+from ltx_pipelines.utils.args import markers_from_args
from ltx_pipelines.utils.constants import TDP_DISTILLED_SIGMAS
+from ltx_pipelines.utils.helpers import get_device
from ltx_pipelines.utils.media_io import EXRColorSpace, encode_video, resolve_hdr_color_space, vae_dtype_for_hdr
from ltx_pipelines.utils.model_paths import ModelPaths
from ltx_pipelines.utils.types import DEFAULT_AUTO_DURATION, AutoDuration
@@ -181,10 +184,16 @@ class TI2VidTwoStagesHQRunner(MGPURunner):
hdr: EXRColorSpace | None = None,
generated_keyframes: int | Sequence[int] = 0,
decode_with_keyframes: bool = False,
+ marker_root_path: str | None = None,
) -> Iterator[str | None]:
# The pipeline raises ValueError on invalid input (symmetric across ranks); the controller
# catches that and turns it into a recoverable RunnerError. Anything else is fatal.
hdr = resolve_hdr_color_space(images=images or [], hdr=hdr)
+ video_marker, audio_marker = markers_from_args(
+ SimpleNamespace(marker_root_path=marker_root_path),
+ device=get_device(),
+ color_space=hdr,
+ )
vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)
result = self._pipeline(
prompt=prompt,
@@ -217,12 +226,18 @@ class TI2VidTwoStagesHQRunner(MGPURunner):
output_path=output_path,
video_chunks_number=get_video_chunks_number(result.num_frames, result.tiling_config),
color_space=hdr,
+ video_marker=video_marker,
+ audio_marker=audio_marker,
)
yield output_path
if __name__ == "__main__":
- from ltx_pipelines.utils.args import add_generated_keyframes_arg, add_keyframe_decode_arg, hq_2_stage_arg_parser
+ from ltx_pipelines.utils.args import (
+ add_generated_keyframes_arg,
+ add_keyframe_decode_arg,
+ hq_2_stage_arg_parser,
+ )
from ltx_pipelines.utils.constants import LTX_2_3_HQ_PARAMS
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
@@ -277,6 +292,7 @@ if __name__ == "__main__":
hdr=args.hdr,
generated_keyframes=args.num_generated_keyframes,
decode_with_keyframes=args.decode_with_keyframes,
+ marker_root_path=args.marker_root_path,
):
pass # drive the job to completion; the runner writes the file as a side effect
finally:
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_mgpu.py b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_mgpu.py
index d1f8196..344b777 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_mgpu.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_mgpu.py
@@ -12,6 +12,7 @@ from __future__ import annotations
import logging
from collections.abc import Callable, Iterator, Sequence
from multiprocessing import SimpleQueue
+from types import SimpleNamespace
import torch
import torch.distributed as dist
@@ -35,7 +36,9 @@ from ltx_pipelines.multigpu.tdp_builder import TiledDataParallelBuilder
from ltx_pipelines.multigpu.vae_builders import DistributedDecoderBuilder
from ltx_pipelines.multigpu.weight_tracker import TransformerWeightTracker
from ltx_pipelines.ti2vid_two_stages import TI2VidTwoStagesPipeline
+from ltx_pipelines.utils.args import markers_from_args
from ltx_pipelines.utils.constants import TDP_DISTILLED_SIGMAS
+from ltx_pipelines.utils.helpers import get_device
from ltx_pipelines.utils.media_io import EXRColorSpace, encode_video, resolve_hdr_color_space, vae_dtype_for_hdr
from ltx_pipelines.utils.model_paths import ModelPaths
from ltx_pipelines.utils.types import DEFAULT_AUTO_DURATION, AutoDuration
@@ -181,10 +184,17 @@ class TI2VidTwoStagesRunner(MGPURunner):
generated_keyframes: int | Sequence[int] = 0,
decode_with_keyframes: bool = False,
chunk_config: ChunkConfig | None = None,
+ refine_audio: bool = False,
+ marker_root_path: str | None = None,
) -> Iterator[str | None]:
# The pipeline raises ValueError on invalid input (symmetric across ranks); the controller
# catches that and turns it into a recoverable RunnerError. Anything else is fatal.
hdr = resolve_hdr_color_space(images=images or [], hdr=hdr)
+ video_marker, audio_marker = markers_from_args(
+ SimpleNamespace(marker_root_path=marker_root_path),
+ device=get_device(),
+ color_space=hdr,
+ )
vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)
chunks, _, _ = self._pipeline.stream_chunks(
prompt=prompt,
@@ -207,6 +217,7 @@ class TI2VidTwoStagesRunner(MGPURunner):
generated_keyframes=generated_keyframes,
decode_with_keyframes=decode_with_keyframes,
chunk_config=chunk_config,
+ refine_audio=refine_audio,
)
if dist.get_rank() != _DRIVER_RANK:
for _ in chunks:
@@ -222,6 +233,8 @@ class TI2VidTwoStagesRunner(MGPURunner):
video_chunks_number=1,
color_space=hdr,
audio_sampling_rate=audio_sampling_rate,
+ video_marker=video_marker,
+ audio_marker=audio_marker,
)
yield output_path
@@ -243,6 +256,11 @@ if __name__ == "__main__":
add_generated_keyframes_arg(default_2_stage_arg_parser(params=params, supports_auto_duration=True))
)
)
+ parser.add_argument(
+ "--refine-audio",
+ action="store_true",
+ help="Denoise audio in stage 2 instead of preserving the stage-1 audio.",
+ )
args = parser.parse_args()
vae_queue = torch.multiprocessing.get_context("spawn").SimpleQueue()
@@ -290,6 +308,8 @@ if __name__ == "__main__":
generated_keyframes=args.num_generated_keyframes,
decode_with_keyframes=args.decode_with_keyframes,
chunk_config=chunk_config_from_args(args),
+ refine_audio=args.refine_audio,
+ marker_root_path=args.marker_root_path,
):
pass # drive the job to completion; the runner writes the file as a side effect
finally:
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/__init__.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/__init__.py
index 6d0cc5e..bead6ed 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/__init__.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/__init__.py
@@ -13,12 +13,11 @@ from ltx_pipelines.utils.helpers import (
assert_resolution,
assert_stage_supports_generated_keyframes,
cleanup_memory,
- combined_image_conditionings,
decode_keyframes_from_slots,
evenly_spaced_keyframe_positions,
generated_keyframe_conditionings,
get_device,
- image_conditionings_by_adding_guiding_latent,
+ image_conditionings,
resolve_generated_keyframes,
)
from ltx_pipelines.utils.samplers import (
@@ -58,7 +57,6 @@ __all__ = [
"assert_resolution",
"assert_stage_supports_generated_keyframes",
"cleanup_memory",
- "combined_image_conditionings",
"decode_keyframes_from_slots",
"euler_ancestral_denoising_loop",
"euler_cfg_pp_denoising_loop",
@@ -67,7 +65,7 @@ __all__ = [
"generated_keyframe_conditionings",
"get_device",
"gradient_estimating_euler_denoising_loop",
- "image_conditionings_by_adding_guiding_latent",
+ "image_conditionings",
"res2s_audio_video_denoising_loop",
"resolve_generated_keyframes",
]
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/args.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/args.py
index e369035..3ca30bf 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/args.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/args.py
@@ -7,17 +7,21 @@ from collections.abc import Sequence
from pathlib import Path
from typing import Any
+import torch
+
from ltx_core.loader import LTXV_LORA_COMFY_RENAMING_MAP, LoraPathStrengthAndSDOps
from ltx_core.model.transformer.compiling import CompilationConfig
from ltx_core.model.video_vae.transformer import DiffVAEMode
from ltx_core.quantization import QuantizationPolicy
from ltx_pipelines.chunks import ChunkConfig
+from ltx_pipelines.marking import AudioMarker, VideoMarker, create_markers
from ltx_pipelines.utils.constants import (
DEFAULT_LORA_STRENGTH,
DEFAULT_NEGATIVE_PROMPT,
LTX_2_3_HQ_PARAMS,
LTX_2_3_PARAMS,
LTX_2_4_PARAMS,
+ VIDEO_SCALE_FACTORS,
PipelineParams,
detect_params,
)
@@ -78,6 +82,44 @@ def add_hdr_args(parser: argparse.ArgumentParser) -> argparse.ArgumentParser:
return parser
+def add_marker_root_args(parser: argparse.ArgumentParser) -> argparse.ArgumentParser:
+ """Add ``--marker-root-path``. Requires the ``videoseal`` and ``audioseal`` extras."""
+ parser.add_argument(
+ "--marker-root-path",
+ dest="marker_root_path",
+ type=resolve_existing_dir,
+ default=None,
+ help=(
+ "Path to a local path/to/marking pack directory (VideoSeal checkpoint/card/"
+ "attenuation and AudioSeal generator/detector). Stamps SDR pixels and stereo PCM. "
+ "HDR encode skips video and still stamps audio. "
+ "Requires the videoseal and audioseal extras. "
+ "Default: no marking."
+ ),
+ )
+ return parser
+
+
+def markers_from_args(
+ args: object,
+ device: torch.device,
+ color_space: EXRColorSpace | None = None,
+) -> tuple[VideoMarker | None, AudioMarker | None]:
+ """Load resident markers for this CLI, or ``(None, None)`` when the flag is omitted.
+ When ``--marker-root-path`` is set, both extras and pack files are checked
+ immediately. HDR (``color_space`` set) drops the video marker, stamps audio,
+ and logs that video marking is unavailable with HDR.
+ """
+ path = getattr(args, "marker_root_path", None)
+ if path is None:
+ return None, None
+ video_marker, audio_marker = create_markers(path, device)
+ if color_space is not None:
+ logger.warning("video marking is unavailable with HDR")
+ return None, audio_marker
+ return video_marker, audio_marker
+
+
class VideoConditioningAction(argparse.Action):
def __call__(
self,
@@ -135,10 +177,21 @@ class AutoDurationAction(argparse.Action):
setattr(namespace, self.dest, AutoDuration(min_seconds=min_seconds, max_seconds=max_seconds))
+def _append_image_conditioning(namespace: argparse.Namespace, dest: str, item: ImageConditioningInput) -> None:
+ current = getattr(namespace, dest) or []
+ current.append(item)
+ setattr(namespace, dest, current)
+
+
+_IMAGE_BINDING_KEYWORDS = frozenset({"auto", "replace", "guide"})
+
+
class ImageAction(argparse.Action):
- """Parse ``--image PATH FRAME_IDX STRENGTH [CRF]``.
- An omitted CRF stays ``None``, which the pipeline resolves from the checkpoint it runs
- (see ``ImageConditioner.resolve_crf``) -- the CLI does not need to know the value.
+ """Parse ``--image PATH FRAME_IDX STRENGTH [BINDING] [CRF]``.
+ ``BINDING`` is ``auto`` (default when omitted), ``replace``, or ``guide``. With ``auto``, frame 0
+ replaces the latent and later frames guide as keyframes. When only one trailing token is given,
+ a keyword selects ``BINDING`` and an integer selects ``CRF``. An omitted CRF stays ``None`` for
+ the pipeline to resolve from its checkpoint.
"""
def __call__(
@@ -148,19 +201,43 @@ class ImageAction(argparse.Action):
values: list[str],
option_string: str | None = None,
) -> None:
- if len(values) not in (3, 4):
- msg = f"{option_string} requires 3 or 4 arguments (PATH FRAME_IDX STRENGTH [CRF]), got {len(values)}"
+ if len(values) not in (3, 4, 5):
+ msg = (
+ f"{option_string} requires 3 to 5 arguments "
+ f"(PATH FRAME_IDX STRENGTH [BINDING] [CRF]), got {len(values)}"
+ )
raise argparse.ArgumentError(self, msg)
- conditioning = ImageConditioningInput(
- path=resolve_existing_path(values[0]),
- frame_idx=int(values[1]),
- strength=float(values[2]),
- crf=int(values[3]) if len(values) > 3 else None,
- )
- current = getattr(namespace, self.dest) or []
- current.append(conditioning)
- setattr(namespace, self.dest, current)
+ path = resolve_existing_path(values[0])
+ frame_idx = int(values[1])
+ strength = float(values[2])
+ binding_mode = "auto"
+ crf: int | None = None
+
+ if len(values) == 4:
+ if values[3] in _IMAGE_BINDING_KEYWORDS:
+ binding_mode = values[3]
+ else:
+ crf = int(values[3])
+ elif len(values) == 5:
+ if values[3] not in _IMAGE_BINDING_KEYWORDS:
+ msg = f"{option_string} BINDING must be auto, replace, or guide, got {values[3]!r}"
+ raise argparse.ArgumentError(self, msg)
+ binding_mode = values[3]
+ crf = int(values[4])
+
+ match binding_mode:
+ case "auto":
+ image = ImageConditioningInput.at(path, frame_idx, strength, crf)
+ case "replace":
+ image = ImageConditioningInput.replace(path, frame_idx, strength, crf)
+ case "guide":
+ image = ImageConditioningInput.guide(path, frame_idx, strength, crf)
+ case _:
+ msg = f"{option_string} BINDING must be auto, replace, or guide, got {binding_mode!r}"
+ raise argparse.ArgumentError(self, msg)
+
+ _append_image_conditioning(namespace, self.dest, image)
class LoraAction(argparse.Action):
@@ -319,6 +396,14 @@ def resolve_existing_path(path: str) -> str:
return resolved
+def resolve_existing_dir(path: str) -> str:
+ """Resolve *path* and verify it is an existing directory."""
+ resolved = resolve_existing_path(path)
+ if not Path(resolved).is_dir():
+ raise argparse.ArgumentError(None, f"Not a directory: {resolved}")
+ return resolved
+
+
QUANTIZATION_POLICIES = tuple(k.value for k in QuantizationKind)
@@ -922,17 +1007,21 @@ def new_video_gen_arg_parser(
metavar="ARG",
default=[],
help=(
- "Image conditioning input: PATH FRAME_IDX STRENGTH [CRF]. "
+ "Image conditioning: PATH FRAME_IDX STRENGTH [BINDING] [CRF]. "
"PATH is a still file — PNG/JPEG (SDR) or scene-linear .exr (HDR). "
- "FRAME_IDX is the target frame index, STRENGTH is the conditioning strength "
- "(all three required). "
- "CRF is the optional H.264 compression quality for SDR stills (0=lossless); when "
- "omitted, the value matching the checkpoint's model version is used. "
- "Can be specified multiple times. Example: --image path/to/image1.jpg 0 0.8 "
- "--image path/to/plate.exr 0 1.0 --hdr SRGB_LINEAR"
+ "BINDING is auto (default when omitted), replace, or guide. With auto, FRAME_IDX 0 "
+ "replaces the latent and later indices guide as keyframes. replace pins a still on a "
+ f"causal latent start (0 or 1 + {VIDEO_SCALE_FACTORS.time} * k pixel frames). guide adds "
+ "a keyframe at any pixel frame. When only one trailing token is given, a binding keyword "
+ "selects BINDING and an integer selects CRF. STRENGTH is the conditioning strength. CRF "
+ "optionally sets H.264 compression quality for SDR stills (0=lossless); when omitted, "
+ "the checkpoint's default is used. Examples: --image path/to/start.jpg 0 1.0 or "
+ "--image path/to/end.jpg 160 1.0 guide or --image path/to/middle.jpg 65 1.0 replace or "
+ "--image path/to/start.jpg 0 1.0 7 or --image path/to/plate.exr 0 1.0 --hdr SRGB_LINEAR."
),
)
add_hdr_args(parser)
+ add_marker_root_args(parser)
return parser
@@ -1069,6 +1158,7 @@ def video_editing_arg_parser(
parser.add_argument("--start-time", type=float, required=True, help="Start time of the region to regenerate (s).")
parser.add_argument("--end-time", type=float, required=True, help="End time of the region to regenerate (s).")
add_hdr_args(parser)
+ add_marker_root_args(parser)
return parser
@@ -1183,6 +1273,7 @@ def hdr_ic_lora_arg_parser(params: PipelineParams = LTX_2_4_PARAMS) -> argparse.
"ignored with --no-keyframes). Default: 0.95."
),
)
+ add_marker_root_args(parser)
return parser
@@ -1233,6 +1324,7 @@ def dubit_arg_parser(
default=1.0,
help="Strength for IC-LoRA video reference conditioning (default: 1.0).",
)
+ add_marker_root_args(parser)
return parser
@@ -1426,6 +1518,7 @@ def default_1_stage_t2a_arg_parser(params: PipelineParams = LTX_2_3_PARAMS) -> a
default=audio_guider.skip_step,
help=f"Audio skip step (default: {audio_guider.skip_step}).",
)
+ add_marker_root_args(parser)
return parser
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks.py
deleted file mode 100644
index e279c41..0000000
--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks.py
+++ /dev/null
@@ -1,1300 +0,0 @@
-"""Pipeline blocks — each block owns its model lifecycle.
-Blocks build a model on each ``__call__``, use it, then free GPU memory.
-This eliminates manual ``del model; cleanup_memory()`` in pipelines: each
-block is self-contained, so no central model-coordinator object is needed.
-"""
-
-from __future__ import annotations
-
-import copy
-import logging
-from collections.abc import Iterator, Sequence
-from contextlib import AbstractContextManager, contextmanager
-from dataclasses import replace
-from typing import Callable, TypeVar
-
-import torch
-
-from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy
-from ltx_core.batch_split import BatchSplitAdapter
-from ltx_core.block_streaming import DISK_CPU_SLOTS, StreamingModelBuilder
-from ltx_core.components.noisers import Noiser
-from ltx_core.components.patchifiers import AudioPatchifier, VideoLatentPatchifier
-from ltx_core.conditioning import ConditioningItem, VideoGeneratedKeyframeSlots
-from ltx_core.duration_head import (
- DURATION_HEAD_KEY_OPS,
- DurationHead,
- DurationHeadConfigurator,
-)
-from ltx_core.loader import SDOps
-from ltx_core.loader.attention_ops import set_attention_module_op
-from ltx_core.loader.fuse_loras import bf16_fuse_rule
-from ltx_core.loader.module_ops import ModuleOps
-from ltx_core.loader.primitives import BuilderProtocol, LoraPathStrengthAndSDOps, ModelBuilderProtocol
-from ltx_core.loader.registry import ModelRegistry, Registry
-from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder
-from ltx_core.model.audio_vae import (
- AUDIO_VAE_DECODER_COMFY_KEYS_FILTER,
- AUDIO_VAE_ENCODER_COMFY_KEYS_FILTER,
- VOCODER_COMFY_KEYS_FILTER,
- AudioDecoderConfigurator,
- AudioEncoderConfigurator,
- VocoderConfigurator,
-)
-from ltx_core.model.audio_vae import (
- decode_audio as vae_decode_audio,
-)
-from ltx_core.model.model_protocol import LTXModelProtocol, ModelConfigurator
-from ltx_core.model.transformer import (
- LTXV_MODEL_COMFY_RENAMING_MAP,
- LTXModelConfigurator,
- X0Model,
-)
-from ltx_core.model.transformer.attention import (
- AttentionCallable,
- AttentionFunction,
-)
-from ltx_core.model.transformer.compiling import (
- CompilationConfig,
- build_compile_transformer_op,
- modify_sd_ops_for_compilation,
-)
-from ltx_core.model.upsampler import LatentUpsamplerConfigurator, upsample_video
-from ltx_core.model.video_vae import (
- CHANNELS_LAST_3D_WEIGHTS,
- MEMORY_EFFICIENT_DECODE,
- VAE_DECODER_COMFY_KEYS_FILTER,
- VAE_ENCODER_COMFY_KEYS_FILTER,
- TilingConfig,
- VideoDecoderConfigurator,
- VideoEncoder,
- VideoEncoderConfigurator,
- is_diffusion_video_vae,
- video_decoder_sd_ops_for_checkpoint,
-)
-from ltx_core.model.video_vae.keyframes import DecodeKeyframes
-from ltx_core.model.video_vae.transformer import (
- DiffVAEMode,
- NAttentionKind,
- build_diffvae_mode_op,
- na_dsl_available,
-)
-from ltx_core.quantization import QuantizationPolicy, fp8_cast_fuse_rule
-from ltx_core.text_encoders.gemma import (
- EMBEDDINGS_PROCESSOR_KEY_OPS,
- EmbeddingsProcessorConfigurator,
- GemmaTextEncoderConfigurator,
- gemma_model_type,
- get_gemma_ops,
- resolve_gemma_weight_paths,
-)
-from ltx_core.text_encoders.gemma.embeddings_processor import EmbeddingsProcessor, EmbeddingsProcessorOutput
-from ltx_core.tools import AudioLatentTools, LatentTools, VideoLatentTools
-from ltx_core.types import (
- VIDEO_SCALE_FACTORS,
- Audio,
- AudioLatentShape,
- LatentState,
- SpatioTemporalScaleFactors,
- VideoLatentShape,
-)
-from ltx_pipelines.utils.constants import detect_params
-from ltx_pipelines.utils.gpu_model import gpu_model
-from ltx_pipelines.utils.helpers import (
- assert_stage_supports_generated_keyframes,
- cleanup_memory,
- create_noised_state,
- generate_enhanced_prompt,
- seconds_to_clamped_num_frames,
-)
-from ltx_pipelines.utils.model_paths import ModelPaths
-from ltx_pipelines.utils.samplers import euler_denoising_loop
-from ltx_pipelines.utils.types import (
- AutoDuration,
- Denoiser,
- ImageConditioningInput,
- ModalitySpec,
- OffloadMode,
- VideoAudio,
-)
-
-_ENCODE_MODEL_TYPES = frozenset({"gemma3", "gemma4", "gemma4_unified"})
-
-logger = logging.getLogger(__name__)
-
-ModelWrapper = Callable[[torch.nn.Module, "LatentTools | None"], torch.nn.Module]
-"""Wraps a built transformer for one :meth:`DiffusionStage.__call__` (see
-:meth:`DiffusionStage.with_model_wrapper`). Must keep the
-``(video, audio, perturbations) -> (video, audio)`` forward signature."""
-
-T = TypeVar("T")
-_M = TypeVar("_M", bound=torch.nn.Module)
-
-DenoisingLoop = Callable[..., VideoAudio[LatentState]]
-
-
-# ---------------------------------------------------------------------------
-# Internal helpers
-# ---------------------------------------------------------------------------
-
-
-def _chain_quantization(
- sd_ops: SDOps,
- module_ops: tuple[ModuleOps, ...],
- quantization: QuantizationPolicy,
-) -> tuple[SDOps, tuple[ModuleOps, ...]]:
- chained_sd_ops = sd_ops
- if quantization.sd_ops is not None:
- chained_sd_ops = SDOps(
- name=f"sd_ops_chain_{sd_ops.name}+{quantization.sd_ops.name}",
- mapping=(*sd_ops.mapping, *quantization.sd_ops.mapping),
- )
- return chained_sd_ops, (*module_ops, *quantization.module_ops)
-
-
-def _apply_compile_ops(
- sd_ops: SDOps,
- module_ops: tuple[ModuleOps, ...],
- loras: tuple[LoraPathStrengthAndSDOps, ...],
- number_of_layers: int,
- compilation_config: CompilationConfig,
-) -> tuple[SDOps, tuple[ModuleOps, ...], tuple[LoraPathStrengthAndSDOps, ...]]:
- """Rewrite sd_ops/module_ops/LoRAs for compiled blocks (params land under ``_orig_mod``)."""
- sd_ops = modify_sd_ops_for_compilation(sd_ops, number_of_layers)
- compile_op = build_compile_transformer_op(compilation_config)
- module_ops = (*module_ops, compile_op)
- loras = tuple(
- LoraPathStrengthAndSDOps(
- lora.path,
- lora.strength,
- modify_sd_ops_for_compilation(lora.sd_ops, number_of_layers),
- )
- for lora in loras
- )
- return sd_ops, module_ops, loras
-
-
-def _ensure_cudagraph_compatible_builder(
- builder: ModelBuilderProtocol[LTXModelProtocol], compilation_config: CompilationConfig | None
-) -> None:
- """CUDA-graph compile paths need GPU-resident weights across dispose/rebuild.
- Covers ``capture=True`` and inductor cudagraph modes (``reduce-overhead`` /
- ``max-autotune``). Checked lazily at prepare/build time so MGPU runners can
- wrap the stage's builder with SP/TDP (+ weight tracker) after construction.
- """
- if compilation_config is None:
- return
- cudagraph_modes = frozenset({"reduce-overhead", "max-autotune"})
- needs_resident = compilation_config.capture or compilation_config.mode in cudagraph_modes
- if not needs_resident or builder.keeps_gpu_resident_weights:
- return
- raise ValueError(
- "CompilationConfig capture / mode=reduce-overhead|max-autotune requires a builder "
- "with keeps_gpu_resident_weights=True (SequenceParallelBuilder / "
- "TiledDataParallelBuilder with a weight tracker, or StreamingModelBuilder). "
- "Plain SingleGPUModelBuilder reloads weights onto fresh GPU storages each stage, "
- "which invalidates CUDA graphs (or repeatedly recaptures until Dynamo gives up)."
- )
-
-
-@contextmanager
-def _streaming_model(
- builder: StreamingModelBuilder,
- target_device: torch.device,
- dtype: torch.dtype,
- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
-) -> Iterator:
- """Build a streaming wrapper, yield it, then tear down and free memory.
- The builder's own ``cpu_slots_count`` selects RAM vs disk streaming.
- ``teardown()`` always runs -- it releases non-memory resources (forward
- hooks, the disk I/O worker thread, open file handles) that GC would not
- reclaim promptly. ``dispose`` always metas parameter storage; ``TRIM``
- also runs ``cleanup_memory()``, while ``DEFER`` leaves the CUDA cache warm.
- """
- wrapped = builder.build(device=target_device, dtype=dtype)
- try:
- yield wrapped
- finally:
- wrapped.teardown()
- wrapped.dispose()
- if alloc_trim_strategy == AllocatorTrimStrategy.TRIM:
- cleanup_memory()
-
-
-def _build_state(
- spec: ModalitySpec,
- tools: LatentTools,
- noiser: Noiser,
- dtype: torch.dtype,
- device: torch.device,
-) -> LatentState:
- """Create a noised latent state from a modality spec and tools."""
- state = create_noised_state(
- tools=tools,
- conditionings=spec.conditionings,
- noiser=noiser,
- dtype=dtype,
- device=device,
- noise_scale=spec.noise_scale,
- initial_latent=spec.latent,
- )
- if spec.frozen:
- state = replace(
- state,
- denoise_mask=torch.zeros_like(state.denoise_mask),
- frozen=True,
- )
- return state
-
-
-def _cleanup_iter(
- it: Iterator[torch.Tensor],
- model: torch.nn.Module,
- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
-) -> Iterator[torch.Tensor]:
- """Wrap an iterator to release *model* memory (per ``alloc_trim_strategy``) once exhausted or abandoned."""
- with gpu_model(model, alloc_trim_strategy=alloc_trim_strategy):
- yield from it
-
-
-# ---------------------------------------------------------------------------
-# DiffusionStage
-# ---------------------------------------------------------------------------
-
-
-class DiffusionStage:
- """Owns transformer lifecycle. Builds on each call, frees on exit.
- Replaces the manual build-transformer / ``del transformer`` pattern that
- every pipeline previously repeated.
- """
-
- def __init__(
- self,
- transformer_builder: ModelBuilderProtocol[LTXModelProtocol],
- dtype: torch.dtype,
- device: torch.device,
- *,
- quantization: QuantizationPolicy | None = None,
- compilation_config: CompilationConfig | None = None,
- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
- scale_factors: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,
- ) -> None:
- """Construct a stage from a single pre-built transformer ``builder``.
- Holds only that builder plus build-time configuration (dtype, device,
- quantization, compilation). Turning a checkpoint path + LoRA set into a
- builder -- and choosing a :class:`StreamingModelBuilder` when offloading --
- lives in :meth:`from_checkpoint`, which is how pipelines normally create a
- stage. A :class:`StreamingModelBuilder` selects the block-streaming build
- path; any other builder uses the standard (all-on-GPU) path.
- ``quantization`` and ``compilation_config`` are applied lazily by
- :meth:`_prepared_builder` on both the standard and streaming paths.
- ``scale_factors`` are the video VAE's spatiotemporal downscaling factors,
- defaulting to the 32x32x8 layout.
- """
- self._transformer_builder = transformer_builder
- self._dtype = dtype
- self._device = device
- self._quantization = quantization
- self._compilation_config = compilation_config
- self._alloc_trim_strategy = alloc_trim_strategy
- self.video_scale_factors = scale_factors
- self._model_wrapper: ModelWrapper | None = None
-
- @classmethod
- def from_checkpoint( # noqa: PLR0913
- cls,
- checkpoint_path: str,
- dtype: torch.dtype,
- device: torch.device,
- loras: tuple[LoraPathStrengthAndSDOps, ...] = (),
- quantization: QuantizationPolicy | None = None,
- registry: Registry | None = None,
- compilation_config: CompilationConfig | None = None,
- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
- offload_mode: OffloadMode = OffloadMode.NONE,
- model_configurator: type[ModelConfigurator] = LTXModelConfigurator,
- model_sd_ops: SDOps = LTXV_MODEL_COMFY_RENAMING_MAP,
- scale_factors: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,
- ) -> "DiffusionStage":
- """Build a stage from a checkpoint path and LoRA set.
- Constructs a single transformer builder from ``checkpoint_path`` +
- ``loras`` and delegates to ``__init__``. When ``offload_mode != OffloadMode.NONE``
- that builder is a :class:`StreamingModelBuilder` (fuse rule +
- ``cpu_slots_count`` for the mode); otherwise it is the standard single-GPU
- builder. Quantization and compilation stay on the stage and are applied
- lazily at build time. This is the high-level entry point used by pipelines;
- ``__init__`` itself takes an already-built builder.
- ``model_configurator`` / ``model_sd_ops`` let callers (e.g. the audio-only
- T2A pipeline) override the model class configurator and the state-dict key
- mapping.
- """
- transformer_builder: ModelBuilderProtocol[LTXModelProtocol]
- if offload_mode == OffloadMode.NONE:
- transformer_builder = Builder(
- model_path=checkpoint_path,
- model_class_configurator=model_configurator,
- model_sd_ops=model_sd_ops,
- loras=tuple(loras),
- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
- )
- else:
- transformer_builder = cls._build_streaming_builder(
- checkpoint_path=checkpoint_path,
- configurator=model_configurator,
- model_sd_ops=model_sd_ops,
- loras=tuple(loras),
- quantization=quantization,
- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
- offload_mode=offload_mode,
- )
-
- return cls(
- transformer_builder,
- dtype,
- device,
- quantization=quantization,
- compilation_config=compilation_config,
- alloc_trim_strategy=alloc_trim_strategy,
- scale_factors=scale_factors,
- )
-
- @staticmethod
- def _build_streaming_builder(
- *,
- checkpoint_path: str,
- configurator: type[ModelConfigurator],
- model_sd_ops: SDOps,
- loras: tuple[LoraPathStrengthAndSDOps, ...],
- quantization: QuantizationPolicy | None,
- registry: Registry,
- offload_mode: OffloadMode,
- ) -> StreamingModelBuilder:
- """Construct the streaming transformer builder for an offloading stage.
- Returns a :class:`StreamingModelBuilder` with the checkpoint path, LoRAs,
- fuse rule (bf16 or fp8_cast), and ``cpu_slots_count`` for ``offload_mode``.
- Compilation / quantization sd_ops and module_ops are not applied here --
- the stage applies them lazily via :meth:`_prepared_builder`.
- """
- # WeightsProvider currently only supports plain bf16 + fp8_cast LoRA fusion
- # (no companion-key emission). Quantization policies that emit
- # companion keys (e.g. ``.weight_scale``) cannot be streamed yet.
- if quantization is not None and quantization.fuse_rule is not fp8_cast_fuse_rule:
- raise ValueError(
- "Block streaming is not supported with this quantization policy "
- "(only bf16 and fp8_cast are currently supported)."
- )
- return StreamingModelBuilder(
- model_class_configurator=configurator,
- model_path=checkpoint_path,
- model_sd_ops=model_sd_ops,
- loras=loras,
- registry=registry,
- fuse_rule=quantization.fuse_rule if quantization is not None else bf16_fuse_rule,
- blocks_attr="transformer_blocks",
- blocks_prefix="transformer_blocks",
- cpu_slots_count=DISK_CPU_SLOTS if offload_mode == OffloadMode.DISK else None,
- )
-
- @property
- def supports_generated_keyframes(self) -> bool:
- """Whether this stage's checkpoint was trained with the keyframe absolute-position embedding.
- Read from the checkpoint's transformer config, so it is answered before any weights are
- built. Generated keyframe slots on a checkpoint without it would be denoised as unmarked
- tokens while still costing a full latent frame of tokens each, so pipelines refuse rather
- than silently degrade.
- """
- transformer_config = self._transformer_builder.model_config().get("transformer", {})
- return bool(transformer_config.get("use_keyframes_abs_pos_embedding", False))
-
- def _assert_supports_conditionings(self, conditionings: Sequence[ConditioningItem] | None) -> None:
- if conditionings is None or self.supports_generated_keyframes:
- return
- if any(isinstance(item, VideoGeneratedKeyframeSlots) for item in conditionings):
- assert_stage_supports_generated_keyframes(self)
-
- @property
- def dtype(self) -> torch.dtype:
- """Dtype this stage builds its transformer and sizes its latents in."""
- return self._dtype
-
- @property
- def device(self) -> torch.device:
- """Device this stage builds its transformer and sizes its latents on."""
- return self._device
-
- def with_attention(self, attention: AttentionFunction | AttentionCallable | None) -> "DiffusionStage":
- """Return a new ``DiffusionStage`` that pins the transformer build to ``attention``.
- Functional: never mutates ``self``. The returned stage shares all other
- configuration with the original; only the underlying builders' ``module_ops``
- gain a ``set_attention_module_op(attention)`` entry so subsequent transformer
- builds use that kernel. ``attention=None`` is a no-op (returns ``self``).
- """
- if attention is None:
- return self
- op = set_attention_module_op(attention)
- new = copy.copy(self)
- new._transformer_builder = self._transformer_builder.with_module_ops(
- (*self._transformer_builder.module_ops, op),
- )
- return new
-
- def with_builder(self, builder: ModelBuilderProtocol[LTXModelProtocol]) -> "DiffusionStage":
- """Return a new ``DiffusionStage`` that builds its transformer from ``builder``.
- Functional: never mutates ``self``; shares all other configuration (dtype, device,
- quantization, compilation). Affects the standard (non-offload) build path.
- """
- new = copy.copy(self)
- new._transformer_builder = builder
- return new
-
- def with_model_wrapper(self, wrapper: "ModelWrapper | None") -> "DiffusionStage":
- """Return a new stage that passes each built transformer through ``wrapper``.
- Functional: never mutates ``self``. The wrapper is applied inside the transformer
- context -- after the model is built, before the denoising loop -- on both the standard
- and block-streaming paths, and it receives the call's video tools. Wrapping *here*
- rather than around the builder is what lets a wrapper compose with offloading: a
- wrapping builder would hide a :class:`StreamingModelBuilder` from :attr:`_is_streaming`,
- so the stage would take the plain ``gpu_model`` path and never tear the stream down.
- ``wrapper=None`` clears any previously set wrapper.
- """
- new = copy.copy(self)
- new._model_wrapper = wrapper
- return new
-
- def with_loras(self, loras: tuple[LoraPathStrengthAndSDOps, ...]) -> "DiffusionStage":
- """Return a new ``DiffusionStage`` built with exactly ``loras`` (replacing the current set)."""
- return self.with_builder(self._transformer_builder.with_loras(loras))
-
- def _prepared_builder(self) -> ModelBuilderProtocol[LTXModelProtocol]:
- """Return the configured builder with the stage's build-time ops applied.
- Compilation and quantization live on the stage (not on the builder) and are
- applied here, lazily, for both the standard and streaming paths. This keeps
- the builder holding only raw sd_ops/module_ops/LoRAs, so ``with_loras`` /
- ``with_builder`` swap them consistently regardless of the build path. The
- returned copy preserves the builder's concrete type (e.g. a
- ``StreamingModelBuilder`` stays one).
- """
- builder = self._transformer_builder
- _ensure_cudagraph_compatible_builder(builder, self._compilation_config)
- sd_ops = builder.model_sd_ops
- module_ops = builder.module_ops
- loras = builder.loras
- if self._quantization is not None:
- sd_ops, module_ops = _chain_quantization(sd_ops, module_ops, self._quantization)
- builder = builder.with_fuse_rule(self._quantization.fuse_rule)
- if self._compilation_config is not None:
- number_of_layers = builder.model_config()["transformer"]["num_layers"]
- sd_ops, module_ops, loras = _apply_compile_ops(
- sd_ops, module_ops, loras, number_of_layers, self._compilation_config
- )
- return builder.with_module_ops(module_ops).with_sd_ops(sd_ops).with_loras(loras)
-
- def _build_transformer(self, *, device: torch.device | None = None, **kwargs: object) -> X0Model:
- target = device or self._device
- return X0Model(self._prepared_builder().build(device=target, **kwargs)).to(target).eval()
-
- @property
- def _is_streaming(self) -> bool:
- """Whether the configured builder uses the block-streaming build path."""
- return isinstance(self._transformer_builder, StreamingModelBuilder)
-
- @contextmanager
- def _streaming_transformer_ctx(self) -> Iterator[X0Model]:
- builder = self._prepared_builder()
- assert isinstance(builder, StreamingModelBuilder)
- with _streaming_model(builder, self._device, self._dtype, self._alloc_trim_strategy) as streaming_wrapper:
- yield X0Model(streaming_wrapper).eval()
-
- def _transformer_ctx(self, **kwargs: object) -> AbstractContextManager:
- if self._is_streaming:
- return self._streaming_transformer_ctx()
- return gpu_model(self._build_transformer(**kwargs), alloc_trim_strategy=self._alloc_trim_strategy)
-
- def __call__(
- self,
- denoiser: Denoiser,
- sigmas: torch.Tensor,
- noiser: Noiser,
- modalities: VideoAudio[ModalitySpec],
- loop: DenoisingLoop | None = None,
- max_batch_size: int = 1,
- ) -> VideoAudio[LatentState]:
- """Build transformer -> run denoising loop -> free transformer.
- Returns a :class:`VideoAudio` of unpatchified :class:`LatentState` with cleared
- conditionings for present modalities.
- """
- video, audio = modalities
- if video is None and audio is None:
- raise ValueError("At least one of `modalities.video` or `modalities.audio` must be provided")
- if video is not None:
- self._assert_supports_conditionings(video.conditionings)
-
- if loop is None:
- loop = euler_denoising_loop
-
- video_latent = video.latent if video is not None else None
- audio_latent = audio.latent if audio is not None else None
- conditioning_fps = video.conditioning_fps if video is not None else None
-
- # Build video_tools up front so it can be forwarded to the transformer
- # context (required by TiledDataParallelBuilder in multi-GPU mode).
- video_tools = (
- VideoLatentTools(
- VideoLatentPatchifier(patch_size=1),
- VideoLatentShape.from_torch_shape(video_latent.shape),
- conditioning_fps,
- scale_factors=self.video_scale_factors,
- )
- if video_latent is not None
- else None
- )
-
- mode = "streaming" if self._is_streaming else "standard"
- logger.info("Building transformer (%s) from %s", mode, self._transformer_builder.checkpoint)
- with self._transformer_ctx(video_tools=video_tools) as built:
- transformer = built if self._model_wrapper is None else self._model_wrapper(built, video_tools)
- logger.info(
- "Running denoising loop (%d steps, video latent %s, audio latent %s @ %.1f fps)",
- len(sigmas) - 1,
- tuple(video_latent.shape) if video_latent is not None else None,
- tuple(audio_latent.shape) if audio_latent is not None else None,
- conditioning_fps,
- )
- video_state: LatentState | None = None
- if video is not None and video_tools is not None:
- video_state = _build_state(video, video_tools, noiser, self._dtype, self._device)
-
- audio_tools = (
- AudioLatentTools(
- AudioPatchifier(patch_size=1),
- AudioLatentShape.from_torch_shape(audio_latent.shape),
- )
- if audio_latent is not None
- else None
- )
- audio_state: LatentState | None = None
- if audio is not None and audio_tools is not None:
- audio_state = _build_state(audio, audio_tools, noiser, self._dtype, self._device)
-
- wrapped = BatchSplitAdapter(transformer, max_batch_size=max_batch_size) # type: ignore[arg-type]
- video_state, audio_state = loop(
- sigmas=sigmas,
- video_state=video_state,
- audio_state=audio_state,
- transformer=wrapped,
- denoiser=denoiser,
- )
-
- if video_state is not None and video_tools is not None:
- video_state = video_tools.clear_conditioning(video_state)
- video_state = video_tools.unpatchify(video_state)
- if audio_state is not None and audio_tools is not None:
- audio_state = audio_tools.clear_conditioning(audio_state)
- audio_state = audio_tools.unpatchify(audio_state)
-
- return VideoAudio(video_state, audio_state)
-
-
-class RecordingDiffusionStage:
- """Wraps a :class:`DiffusionStage` and records the states each call produced.
- Diagnostics wrapper for a stage's intermediate ``LatentState``s. Use this to inspect
- per-stage states (including half-res stage-1 slots) without a lifecycle callback. Pipelines
- hold their stage in a plain attribute, so a caller substitutes this in::
- pipeline.stage = RecordingDiffusionStage(pipeline.stage)
- result = pipeline(...)
- keyframes = pipeline.stage.generated_keyframes
- Delegates every other attribute to the wrapped stage, so a substituted stage stays usable
- wherever the original was. Recording holds references to the returned states, which keeps
- their latents alive -- fine for diagnostics on one run, not something to leave enabled in a
- long-lived serving process.
- """
-
- def __init__(self, stage: DiffusionStage) -> None:
- self._stage = stage
- self.video_states: list[LatentState] = []
- self.audio_states: list[LatentState | None] = []
-
- def __call__(self, *args: object, **kwargs: object) -> VideoAudio[LatentState]:
- video_state, audio_state = self._stage(*args, **kwargs) # type: ignore[arg-type]
- if video_state is not None:
- self.video_states.append(video_state)
- self.audio_states.append(audio_state)
- return VideoAudio(video_state, audio_state)
-
- @property
- def generated_keyframes(self) -> torch.Tensor | None:
- """Generated keyframe latents from the first recorded call, if it produced any."""
- for state in self.video_states:
- if state.generated_keyframes is not None:
- return state.generated_keyframes
- return None
-
- def __getattr__(self, name: str) -> object:
- return getattr(self._stage, name)
-
-
-# ---------------------------------------------------------------------------
-# PromptEncoder
-# ---------------------------------------------------------------------------
-
-
-class PromptEncoder:
- """Owns text encoder + embeddings processor lifecycle.
- Loads Gemma, optionally enhances then encodes prompts, frees Gemma, then loads the
- embeddings processor. ``_enhancer_text_encoder_builder`` is always set: when enhance
- and encode share a checkpoint it aliases ``_text_encoder_builder`` (same object /
- residency); a distinct builder means sequential enhance-then-encode contexts.
- """
-
- def __init__(
- self,
- model_paths: ModelPaths,
- dtype: torch.dtype,
- device: torch.device,
- registry: Registry | None = None,
- offload_mode: OffloadMode = OffloadMode.NONE,
- text_encoder_builder: BuilderProtocol | None = None,
- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
- prompt_enhancer_gemma_root: str | None = None,
- enhancer_text_encoder_builder: BuilderProtocol | None = None,
- ) -> None:
- text_encoder_path = model_paths.text_encoder()
- embeddings_paths = model_paths.embeddings_weight_paths
- if not embeddings_paths:
- raise ValueError(
- "PromptEncoder requires ModelPaths.embeddings_weight_paths "
- "(transformer checkpoint and, in split mode, the text-encoder file)."
- )
- self._text_encoder_path = text_encoder_path
- self._embeddings_paths = embeddings_paths
- self._dtype = dtype
- self._device = device
- self._offload_mode = offload_mode
- self._alloc_trim_strategy = alloc_trim_strategy
- self._prompt_enhancer_gemma_root = prompt_enhancer_gemma_root
-
- registry = registry or ModelRegistry(cache_models=True, cache_weights=False)
- self._registry = registry
-
- if text_encoder_builder is not None:
- encode_type = text_encoder_builder.model_config().get("model_type") or gemma_model_type(text_encoder_path)
- else:
- encode_type = gemma_model_type(text_encoder_path)
- if encode_type not in _ENCODE_MODEL_TYPES:
- raise ValueError(
- f"Encode root model_type={encode_type!r} is not supported for encoding; "
- f"expected one of {sorted(_ENCODE_MODEL_TYPES)}."
- )
- self._encode_model_type = encode_type
-
- if text_encoder_builder is not None:
- if offload_mode != OffloadMode.NONE:
- raise ValueError(
- "text_encoder_builder cannot be used with offload_mode != OffloadMode.NONE "
- "because no streaming text encoder builder is available."
- )
- self._text_encoder_builder = text_encoder_builder
- self._streaming_text_encoder_builder = None
- else:
- gemma_sd_ops, gemma_module_ops = get_gemma_ops(text_encoder_path)
- weight_paths = resolve_gemma_weight_paths(text_encoder_path)
- self._text_encoder_builder = Builder(
- model_path=weight_paths,
- model_class_configurator=GemmaTextEncoderConfigurator.with_gemma_model_path(text_encoder_path),
- model_sd_ops=gemma_sd_ops,
- module_ops=gemma_module_ops,
- registry=registry,
- )
- self._streaming_text_encoder_builder = StreamingModelBuilder(
- model_path=weight_paths,
- model_class_configurator=GemmaTextEncoderConfigurator.with_gemma_model_path(text_encoder_path),
- model_sd_ops=gemma_sd_ops,
- module_ops=gemma_module_ops,
- registry=registry,
- blocks_attr="model.model.language_model.layers",
- blocks_prefix="model.model.language_model.layers",
- cpu_slots_count=DISK_CPU_SLOTS if offload_mode == OffloadMode.DISK else None,
- )
-
- # Always set: shared enhance aliases the encode builder (same object / residency).
- if enhancer_text_encoder_builder is not None:
- self._enhancer_text_encoder_builder = enhancer_text_encoder_builder
- elif prompt_enhancer_gemma_root is not None and prompt_enhancer_gemma_root != text_encoder_path:
- enhancer_sd_ops, enhancer_module_ops = get_gemma_ops(prompt_enhancer_gemma_root)
- enhancer_paths = resolve_gemma_weight_paths(prompt_enhancer_gemma_root)
- self._enhancer_text_encoder_builder = Builder(
- model_path=enhancer_paths,
- model_class_configurator=GemmaTextEncoderConfigurator.with_gemma_model_path(prompt_enhancer_gemma_root),
- model_sd_ops=enhancer_sd_ops,
- module_ops=enhancer_module_ops,
- registry=registry,
- )
- else:
- self._enhancer_text_encoder_builder = self._text_encoder_builder
-
- self._embeddings_processor_builder = Builder(
- model_path=embeddings_paths,
- model_class_configurator=EmbeddingsProcessorConfigurator.with_gemma_model_path(text_encoder_path),
- model_sd_ops=EMBEDDINGS_PROCESSOR_KEY_OPS,
- registry=registry,
- )
-
- def _build_text_encoder(self) -> torch.nn.Module:
- """Build the Gemma text encoder (non-streaming path)."""
- return self._text_encoder_builder.build(device=self._device, dtype=self._dtype).eval()
-
- def _build_enhancer_text_encoder(self) -> torch.nn.Module:
- return self._enhancer_text_encoder_builder.build(device=self._device, dtype=self._dtype).eval()
-
- def _build_embeddings_processor(self) -> EmbeddingsProcessor:
- """Build the embeddings processor on the target device."""
- return self._embeddings_processor_builder.build(device=self._device, dtype=self._dtype).eval()
-
- def _text_encoder_ctx(self) -> AbstractContextManager:
- if self._offload_mode != OffloadMode.NONE:
- return _streaming_model(
- self._streaming_text_encoder_builder, self._device, self._dtype, self._alloc_trim_strategy
- )
- return gpu_model(self._build_text_encoder(), alloc_trim_strategy=self._alloc_trim_strategy)
-
- def __call__(
- self,
- prompts: list[str],
- *,
- enhance_first_prompt: bool = False,
- enhance_prompt_image: str | None = None,
- enhance_prompt_seed: int = 42,
- enhance_static_cache: bool = False,
- ) -> list[EmbeddingsProcessorOutput]:
- """Encode *prompts* through Gemma -> embeddings processor, freeing each model after use."""
- prompts = list(prompts)
- enhance_kwargs = {
- "image_path": enhance_prompt_image,
- "seed": enhance_prompt_seed,
- "static_cache": enhance_static_cache,
- }
- separate_enhancer = self._enhancer_text_encoder_builder is not self._text_encoder_builder
-
- if enhance_first_prompt and separate_enhancer:
- logger.info(
- "Enhancing with separate Gemma%s",
- f" from {self._prompt_enhancer_gemma_root}" if self._prompt_enhancer_gemma_root else "",
- )
- with gpu_model(
- self._build_enhancer_text_encoder(),
- alloc_trim_strategy=self._alloc_trim_strategy,
- ) as enhancer:
- prompts[0] = generate_enhanced_prompt(enhancer, prompts[0], **enhance_kwargs)
- elif enhance_first_prompt and self._encode_model_type != "gemma3":
- raise ValueError(
- f"Prompt enhancement with encode root model_type={self._encode_model_type!r} "
- "requires --prompt-enhancer-gemma-root pointing at a generative instruct checkpoint "
- "(e.g. gemma3 or gemma4 E2B-it)."
- )
-
- logger.info("Building text encoder from %s", self._text_encoder_path)
- with self._text_encoder_ctx() as text_encoder:
- if enhance_first_prompt and not separate_enhancer:
- prompts[0] = generate_enhanced_prompt(text_encoder, prompts[0], **enhance_kwargs)
- raw_outputs = text_encoder.encode(prompts)
- logger.info("Text encoder done, building embeddings processor from %s", self._embeddings_paths)
-
- with gpu_model(
- self._build_embeddings_processor(), alloc_trim_strategy=self._alloc_trim_strategy
- ) as embeddings_processor:
- result = [embeddings_processor.process_hidden_states(hs, mask) for hs, mask in raw_outputs]
- logger.info("Prompt encoding complete")
- return result
-
-
-# ---------------------------------------------------------------------------
-# DurationPredictor
-# ---------------------------------------------------------------------------
-
-
-class DurationPredictor:
- """Predicts shot duration (in frames) from ``PromptEncoder`` output.
- Unlike most blocks, the model is held directly rather than rebuilt on every call:
- DurationHead is a few MB, so there's no memory pressure motivating the
- build-on-call / free-on-exit pattern used for the large transformer/VAE blocks.
- Construct via :meth:`from_checkpoint`.
- """
-
- def __init__(self, head: DurationHead) -> None:
- """Construct from an already-built, already-loaded head."""
- self._head = head
-
- @classmethod
- def from_checkpoint(
- cls,
- checkpoint_path: str | None,
- dtype: torch.dtype,
- device: torch.device,
- model_configurator: type[ModelConfigurator[DurationHead]] = DurationHeadConfigurator,
- model_sd_ops: SDOps = DURATION_HEAD_KEY_OPS,
- ) -> "DurationPredictor | None":
- """Build a predictor from a checkpoint path, or ``None`` if unavailable.
- Returns ``None`` when ``checkpoint_path`` is ``None`` (split pack without a
- duration-head file), or when the file has no DurationHead weights (pre-2.5 /
- gemma3 monoliths). The underlying loader uses ``strict=False``, so a missing
- head does not raise -- it silently leaves parameters on the meta device;
- checked here so callers get ``None`` instead of a predictor that would crash
- later.
- No registry: the head is a few MB, so there's no benefit to caching its weights or
- shell across pipeline instances the way the large transformer/VAE builders do.
- """
- if checkpoint_path is None:
- return None
- builder = Builder(
- model_path=checkpoint_path,
- model_class_configurator=model_configurator,
- model_sd_ops=model_sd_ops,
- )
- head = builder.build(device=device, dtype=dtype).eval()
- if any(param.is_meta for param in head.parameters()):
- logger.info(
- "No DurationHead weights found in %s; auto-duration prediction unavailable.",
- checkpoint_path,
- )
- return None
- return cls(head)
-
- def __call__(
- self,
- video_encoding: torch.Tensor | None,
- audio_encoding: torch.Tensor | None,
- *,
- frame_rate: float,
- min_seconds: float = 1.0,
- max_seconds: float = 20.0,
- ) -> int:
- """Predict a frame count from caption connector tokens, snapped to the VAE's grid.
- ``min_seconds``/``max_seconds`` clamp the prediction so a misbehaving prediction can't
- request a degenerate or OOM-sized generation; the defaults are 1s and 20s. The result is
- a frame count snapped to the VAE's ``8k + 1`` causal temporal grid. Callers doing offline
- analysis of raw predictions (rather than feeding a real generation) can pass a much wider
- range to see the unclamped duration.
- """
- if video_encoding is None and audio_encoding is None:
- raise ValueError("DurationPredictor requires at least one of video_encoding / audio_encoding")
- seconds_pred = self._head(video_encoding, audio_encoding)
- if seconds_pred.shape != (1,):
- raise ValueError(
- f"DurationPredictor only supports a single-item batch, got prediction shape {tuple(seconds_pred.shape)}"
- )
- seconds = seconds_pred.item()
- min_frames = round(min_seconds * frame_rate)
- max_frames = round(max_seconds * frame_rate)
- num_frames = seconds_to_clamped_num_frames(
- seconds, frame_rate=frame_rate, min_frames=min_frames, max_frames=max_frames
- )
- if seconds > max_seconds or seconds < min_seconds:
- logger.warning(
- "DurationHead prediction clamped: raw %.2fs outside [%.2fs, %.2fs], using %.2fs (%d frames) @ %.2f fps",
- seconds,
- min_seconds,
- max_seconds,
- num_frames / frame_rate,
- num_frames,
- frame_rate,
- )
- else:
- logger.info("DurationHead predicted %.2fs (%d frames @ %.2f fps)", seconds, num_frames, frame_rate)
- return num_frames
-
-
-def require_num_frames_source(num_frames: int | AutoDuration, duration_predictor: DurationPredictor | None) -> None:
- """Guard against an unsatisfiable auto-duration request.
- Call at the very top of a pipeline's ``__call__`` -- before prompt encoding or any other
- work -- so a checkpoint without DurationHead weights (anything predating 2.5) fails fast
- with a clear message instead of after paying for work whose result would be discarded.
- """
- if isinstance(num_frames, AutoDuration) and duration_predictor is None:
- raise ValueError(
- "num_frames was AutoDuration but this checkpoint has no DurationHead weights to "
- "auto-predict duration from (DurationHead ships from LTX 2.5 / gemma4 onward). "
- "Pass num_frames explicitly."
- )
-
-
-def resolve_num_frames(
- num_frames: int | AutoDuration,
- duration_predictor: DurationPredictor | None,
- *,
- video_encoding: torch.Tensor | None,
- audio_encoding: torch.Tensor | None,
- frame_rate: float,
-) -> int:
- """Resolve ``num_frames`` to a concrete frame count, predicting it if ``AutoDuration``.
- Call after prompt encoding (once ``video_encoding``/``audio_encoding`` exist) and after
- ``require_num_frames_source`` has already validated a predictor is available when needed.
- """
- if not isinstance(num_frames, AutoDuration):
- return num_frames
- return duration_predictor(
- video_encoding,
- audio_encoding,
- frame_rate=frame_rate,
- min_seconds=num_frames.min_seconds,
- max_seconds=num_frames.max_seconds,
- )
-
-
-# ---------------------------------------------------------------------------
-# ImageConditioner
-# ---------------------------------------------------------------------------
-
-
-class ImageConditioner:
- """Owns video encoder lifecycle, and the CRF image conditionings are re-compressed at.
- Builds the encoder, passes it to the user-supplied callable, then frees it.
- Also the single place that knows what H.264 CRF an image conditioning should be
- re-compressed at: that value is a property of the model generation, and this block
- already holds the checkpoint it belongs to. Pipelines call ``resolve_crf`` so callers
- may omit the CRF and still condition the way the model was trained.
- """
-
- def __init__(
- self,
- checkpoint_path: str,
- dtype: torch.dtype,
- device: torch.device,
- registry: Registry | None = None,
- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
- ) -> None:
- self._checkpoint_path = checkpoint_path
- self._default_image_crf: int | None = None
- self._dtype = dtype
- self._device = device
- self._encoder_builder = Builder(
- model_path=checkpoint_path,
- model_class_configurator=VideoEncoderConfigurator,
- model_sd_ops=VAE_ENCODER_COMFY_KEYS_FILTER,
- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
- )
- self._alloc_trim_strategy = alloc_trim_strategy
-
- @property
- def default_image_crf(self) -> int:
- """The H.264 CRF image conditionings should be re-compressed at for this checkpoint.
- Read from the checkpoint's ``model_version`` metadata (see ``detect_params``) rather
- than hardcoded, because the value the model was trained with changed between
- generations. Resolved on first use and cached, so a run without image conditionings
- never reads the header.
- """
- if self._default_image_crf is None:
- self._default_image_crf = detect_params(self._checkpoint_path).default_image_crf
- return self._default_image_crf
-
- def resolve_crf(self, images: Sequence[ImageConditioningInput]) -> list[ImageConditioningInput]:
- """Fill in ``default_image_crf`` for every conditioning that left its CRF unset (``crf=None``).
- Call near the top of a pipeline's ``__call__``: omitting the CRF then means "use what
- matches this model", while an explicitly passed CRF (including ``0`` for no
- re-compression) is always honoured.
- """
- return [image if image.crf is not None else image._replace(crf=self.default_image_crf) for image in images]
-
- def _build_encoder(self) -> VideoEncoder:
- return self._encoder_builder.build(device=self._device, dtype=self._dtype).eval()
-
- def __call__(self, fn: Callable[[VideoEncoder], T]) -> T:
- """Build video encoder → call *fn(encoder)* → free encoder."""
- with gpu_model(self._build_encoder(), alloc_trim_strategy=self._alloc_trim_strategy) as encoder:
- return fn(encoder)
-
-
-# ---------------------------------------------------------------------------
-# VideoUpsampler
-# ---------------------------------------------------------------------------
-
-
-class VideoUpsampler:
- """Owns video encoder + spatial upsampler lifecycle."""
-
- def __init__(
- self,
- checkpoint_path: str,
- upsampler_path: str,
- dtype: torch.dtype,
- device: torch.device,
- registry: Registry | None = None,
- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
- ) -> None:
- self._upsampler_path = upsampler_path
- self._dtype = dtype
- self._device = device
- self._encoder_builder = Builder(
- model_path=checkpoint_path,
- model_class_configurator=VideoEncoderConfigurator,
- model_sd_ops=VAE_ENCODER_COMFY_KEYS_FILTER,
- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
- )
- self._upsampler_builder = Builder(
- model_path=upsampler_path,
- model_class_configurator=LatentUpsamplerConfigurator,
- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
- )
- self._alloc_trim_strategy = alloc_trim_strategy
-
- def __call__(self, latent: torch.Tensor) -> torch.Tensor:
- """Upsample *latent* using video encoder + spatial upsampler, then free both."""
- logger.info("Building video encoder + spatial upsampler from %s", self._upsampler_path)
- with (
- gpu_model(
- self._encoder_builder.build(device=self._device, dtype=self._dtype).eval(),
- alloc_trim_strategy=self._alloc_trim_strategy,
- ) as encoder,
- gpu_model(
- self._upsampler_builder.build(device=self._device, dtype=self._dtype).eval(),
- alloc_trim_strategy=self._alloc_trim_strategy,
- ) as upsampler,
- ):
- return upsample_video(latent=latent, video_encoder=encoder, upsampler=upsampler)
-
-
-# ---------------------------------------------------------------------------
-# VideoDecoder
-# ---------------------------------------------------------------------------
-
-
-class VideoDecoder:
- """Owns video decoder lifecycle.
- Returns an iterator that cleans up the decoder after all chunks are consumed.
- """
-
- def __init__(
- self,
- checkpoint_path: str,
- dtype: torch.dtype,
- device: torch.device,
- registry: Registry | None = None,
- memory_efficient: bool = True,
- decoder_builder: BuilderProtocol | None = None,
- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
- diffvae_optimization: DiffVAEMode = DiffVAEMode.CHUNKED_EAGER,
- ) -> None:
- self._checkpoint_path = checkpoint_path
- self._dtype = dtype
- self._device = device
- self._diffvae_optimization = diffvae_optimization
- diffusion_vae = is_diffusion_video_vae(self._checkpoint_path)
- self._diffusion_vae = diffusion_vae
- cfg = diffvae_optimization.resolve() if diffusion_vae else None
- use_dsl = cfg is not None and cfg.attention is NAttentionKind.BLACKWELL_DSL
- if decoder_builder is not None:
- self._decoder_builder = decoder_builder
- else:
- if diffusion_vae and use_dsl and not na_dsl_available():
- raise RuntimeError(
- "diffvae_optimization=blackwell_dsl needs a datacenter Blackwell GPU and "
- "nvidia-cutlass-dsl (uv sync --group kernels)"
- )
- sd_ops = (
- video_decoder_sd_ops_for_checkpoint(
- self._checkpoint_path,
- diffusion_vae=True,
- na_dsl_kernel=use_dsl,
- )
- if diffusion_vae
- else VAE_DECODER_COMFY_KEYS_FILTER
- )
- module_ops: tuple[ModuleOps, ...] = ()
- # channels-last + memory-efficient decode apply to ConvVideoDecoder only
- if memory_efficient and not diffusion_vae:
- sd_ops = SDOps(
- name=f"sd_ops_chain_{sd_ops.name}+{CHANNELS_LAST_3D_WEIGHTS.name}",
- mapping=(*sd_ops.mapping, *CHANNELS_LAST_3D_WEIGHTS.mapping),
- )
- module_ops = (MEMORY_EFFICIENT_DECODE,)
- self._decoder_builder = Builder(
- model_path=self._checkpoint_path,
- model_class_configurator=VideoDecoderConfigurator,
- model_sd_ops=sd_ops,
- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
- module_ops=module_ops,
- )
- self._alloc_trim_strategy = alloc_trim_strategy
-
- @property
- def checkpoint_path(self) -> str:
- return self._checkpoint_path
-
- @property
- def diffvae_optimization(self) -> DiffVAEMode:
- return self._diffvae_optimization
-
- def with_builder(self, builder: BuilderProtocol) -> "VideoDecoder":
- """Return a new ``VideoDecoder`` that builds from ``builder``.
- Functional: never mutates ``self``; shares checkpoint, dtype, device, and
- trim strategy. Mirrors :meth:`DiffusionStage.with_builder`.
- """
- new = copy.copy(self)
- new._decoder_builder = builder
- return new
-
- def _prepared_builder(self) -> BuilderProtocol:
- """Return the builder with DiffVAE mode ops applied at build time.
- DiffVAE ``module_ops`` live on the decoder (not on the caller-held builder),
- matching :meth:`DiffusionStage._prepared_builder`. That keeps
- ``with_builder`` / Dist ``with_tiling`` swaps on the raw builder, while
- every build still installs the configured DiffVAE policy.
- """
- builder = self._decoder_builder
- if not self._diffusion_vae:
- return builder
- op = build_diffvae_mode_op(
- self._diffvae_optimization,
- CompilationConfig() if self._diffvae_optimization.resolve().compile_blocks else None,
- )
- return builder.with_module_ops((*builder.module_ops, op))
-
- def __call__(
- self,
- latent: torch.Tensor,
- tiling_config: TilingConfig | None = None,
- generator: torch.Generator | None = None,
- *,
- dtype: torch.dtype | None = None,
- keyframes: DecodeKeyframes | None = None,
- device_fn: Callable[[int], str | torch.device] | None = None,
- ) -> Iterator[torch.Tensor]:
- """Decode *latent* to pixel-space video chunks. Decoder freed after exhaustion.
- ``dtype`` overrides the constructor dtype for this call only (e.g. float32 for
- HDR raw-in/raw-out). Omitted → same dtype as on construction (SDR default).
- ``keyframes`` anchors the decode on already-encoded planes. Every decoder in this repo
- takes it, so this never branches on which VAE was built: a diffusion VAE uses the planes,
- a conv VAE logs that it is dropping them, and a distributed decoder splits them across
- its ranks. The keyword is only passed when there is something to pass, so a decoder
- written against the older signature still serves plain decodes.
- """
- build_dtype = self._dtype if dtype is None else dtype
- latent = latent.to(dtype=build_dtype)
- logger.info("Building video decoder from %s", self._checkpoint_path)
- decoder = self._prepared_builder().build(device=self._device, dtype=build_dtype).eval()
- extra = {} if keyframes is None else {"keyframes": keyframes}
- if device_fn is not None:
- extra["device_fn"] = device_fn
- chunks = decoder.decode_video(latent, tiling_config, generator, **extra)
- return _cleanup_iter(
- chunks,
- decoder,
- alloc_trim_strategy=self._alloc_trim_strategy,
- )
-
- def decode_single_frames(
- self,
- latents: Sequence[torch.Tensor],
- generator: torch.Generator | Sequence[torch.Generator | None] | None = None,
- *,
- dtype: torch.dtype | None = None,
- ) -> Iterator[torch.Tensor]:
- """Decode each latent as its own one-frame clip on one VAE build, then free it.
- A causal VAE cannot decode stacked independent planes without bleeding neighbours.
- A Dist build splits the list across its ranks and gathers the pixels back, so every
- rank returns the whole list; it does not go through the volume-splitting
- ``decode_video`` path, whose workers yield nothing.
- """
- if not latents:
- return iter(())
- build_dtype = self._dtype if dtype is None else dtype
- moved = [latent.to(dtype=build_dtype) for latent in latents]
- logger.info("Building video decoder from %s", self._checkpoint_path)
- decoder = self._prepared_builder().build(device=self._device, dtype=build_dtype).eval()
- chunks = decoder.decode_single_frames(moved, generator)
- return _cleanup_iter(
- chunks,
- decoder,
- alloc_trim_strategy=self._alloc_trim_strategy,
- )
-
-
-# ---------------------------------------------------------------------------
-# AudioDecoder
-# ---------------------------------------------------------------------------
-
-
-class AudioDecoder:
- """Owns audio decoder + vocoder lifecycle."""
-
- def __init__(
- self,
- checkpoint_path: str,
- dtype: torch.dtype,
- device: torch.device,
- registry: Registry | None = None,
- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
- ) -> None:
- self._checkpoint_path = checkpoint_path
- self._dtype = dtype
- self._device = device
- self._decoder_builder = Builder(
- model_path=checkpoint_path,
- model_class_configurator=AudioDecoderConfigurator,
- model_sd_ops=AUDIO_VAE_DECODER_COMFY_KEYS_FILTER,
- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
- )
- self._vocoder_builder = Builder(
- model_path=checkpoint_path,
- model_class_configurator=VocoderConfigurator,
- model_sd_ops=VOCODER_COMFY_KEYS_FILTER,
- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
- )
- self._alloc_trim_strategy = alloc_trim_strategy
-
- def __call__(self, latent: torch.Tensor) -> Audio:
- """Decode audio *latent* through VAE decoder + vocoder, then free both."""
- logger.info("Building audio decoder + vocoder from %s", self._checkpoint_path)
- # The vocoder always runs in fp32 (bf16 accumulation degrades spectral
- # metrics). On CUDA/CPU it is stored in bf16 and autocast upcasts per-op to
- # save memory; MPS has no fp32 autocast, so store it in fp32 directly and
- # avoid the per-call cast. Negligible footprint for this small model.
- vocoder_dtype = torch.float32 if self._device.type == "mps" else self._dtype
- with (
- gpu_model(
- self._decoder_builder.build(device=self._device, dtype=self._dtype).eval(),
- alloc_trim_strategy=self._alloc_trim_strategy,
- ) as decoder,
- gpu_model(
- self._vocoder_builder.build(device=self._device, dtype=vocoder_dtype).eval(),
- alloc_trim_strategy=self._alloc_trim_strategy,
- ) as vocoder,
- ):
- return vae_decode_audio(latent, decoder, vocoder)
-
-
-# ---------------------------------------------------------------------------
-# AudioEncoder
-# ---------------------------------------------------------------------------
-
-
-class AudioConditioner:
- """Owns audio encoder lifecycle.
- Builds the encoder, passes it to the user-supplied callable, then frees it.
- Mirrors :class:`ImageConditioner` for the audio modality.
- """
-
- def __init__(
- self,
- checkpoint_path: str,
- dtype: torch.dtype,
- device: torch.device,
- registry: Registry | None = None,
- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
- ) -> None:
- self._dtype = dtype
- self._device = device
- self._alloc_trim_strategy = alloc_trim_strategy
- self._encoder_builder = Builder(
- model_path=checkpoint_path,
- model_class_configurator=AudioEncoderConfigurator,
- model_sd_ops=AUDIO_VAE_ENCODER_COMFY_KEYS_FILTER,
- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
- )
-
- def __call__(self, fn: Callable[[torch.nn.Module], T]) -> T:
- """Build audio encoder → call *fn(encoder)* → free encoder."""
- with gpu_model(
- self._encoder_builder.build(device=self._device, dtype=self._dtype).eval(),
- alloc_trim_strategy=self._alloc_trim_strategy,
- ) as encoder:
- return fn(encoder)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/__init__.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/__init__.py
new file mode 100644
index 0000000..f79afe5
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/__init__.py
@@ -0,0 +1,47 @@
+"""Pipeline blocks — each block owns its model lifecycle.
+Blocks build a model on each ``__call__``, use it, then free GPU memory.
+This eliminates manual ``del model; cleanup_memory()`` in pipelines: each
+block is self-contained, so no central model-coordinator object is needed.
+"""
+
+from ltx_core.components.noisers import Noiser
+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder
+from ltx_pipelines.utils.blocks.audio_conditioner import AudioConditioner
+from ltx_pipelines.utils.blocks.audio_decoder import AudioDecoder
+from ltx_pipelines.utils.blocks.diffusion_stage import (
+ DenoisingLoop,
+ DiffusionStage,
+ ModelWrapper,
+ RecordingDiffusionStage,
+)
+from ltx_pipelines.utils.blocks.duration_predictor import (
+ DurationPredictor,
+ require_num_frames_source,
+ resolve_num_frames,
+)
+from ltx_pipelines.utils.blocks.image_conditioner import ImageConditioner
+from ltx_pipelines.utils.blocks.prompt_encoder import PromptEncoder
+from ltx_pipelines.utils.blocks.video_decoder import VideoDecoder
+from ltx_pipelines.utils.blocks.video_upsampler import VideoUpsampler
+from ltx_pipelines.utils.constants import detect_params
+from ltx_pipelines.utils.helpers import cleanup_memory
+
+__all__ = [
+ "AudioConditioner",
+ "AudioDecoder",
+ "Builder",
+ "DenoisingLoop",
+ "DiffusionStage",
+ "DurationPredictor",
+ "ImageConditioner",
+ "ModelWrapper",
+ "Noiser",
+ "PromptEncoder",
+ "RecordingDiffusionStage",
+ "VideoDecoder",
+ "VideoUpsampler",
+ "cleanup_memory",
+ "detect_params",
+ "require_num_frames_source",
+ "resolve_num_frames",
+]
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/audio_conditioner.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/audio_conditioner.py
new file mode 100644
index 0000000..bd4775d
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/audio_conditioner.py
@@ -0,0 +1,49 @@
+from __future__ import annotations
+
+from collections.abc import Callable
+from typing import TypeVar
+
+import torch
+
+from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy
+from ltx_core.loader.registry import ModelRegistry, Registry
+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder
+from ltx_core.model.audio_vae import AUDIO_VAE_ENCODER_COMFY_KEYS_FILTER, AudioEncoderConfigurator
+from ltx_pipelines.utils.gpu_model import gpu_model
+
+T = TypeVar("T")
+
+__all__ = ["AudioConditioner"]
+
+
+class AudioConditioner:
+ """Owns audio encoder lifecycle.
+ Builds the encoder, passes it to the user-supplied callable, then frees it.
+ Mirrors :class:`ImageConditioner` for the audio modality.
+ """
+
+ def __init__(
+ self,
+ checkpoint_path: str,
+ dtype: torch.dtype,
+ device: torch.device,
+ registry: Registry | None = None,
+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
+ ) -> None:
+ self._dtype = dtype
+ self._device = device
+ self._alloc_trim_strategy = alloc_trim_strategy
+ self._encoder_builder = Builder(
+ model_path=checkpoint_path,
+ model_class_configurator=AudioEncoderConfigurator,
+ model_sd_ops=AUDIO_VAE_ENCODER_COMFY_KEYS_FILTER,
+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
+ )
+
+ def __call__(self, fn: Callable[[torch.nn.Module], T]) -> T:
+ """Build audio encoder → call *fn(encoder)* → free encoder."""
+ with gpu_model(
+ self._encoder_builder.build(device=self._device, dtype=self._dtype).eval(),
+ alloc_trim_strategy=self._alloc_trim_strategy,
+ ) as encoder:
+ return fn(encoder)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/audio_decoder.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/audio_decoder.py
new file mode 100644
index 0000000..96a2624
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/audio_decoder.py
@@ -0,0 +1,71 @@
+from __future__ import annotations
+
+import logging
+
+import torch
+
+from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy
+from ltx_core.loader.registry import ModelRegistry, Registry
+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder
+from ltx_core.model.audio_vae import (
+ AUDIO_VAE_DECODER_COMFY_KEYS_FILTER,
+ VOCODER_COMFY_KEYS_FILTER,
+ AudioDecoderConfigurator,
+ VocoderConfigurator,
+)
+from ltx_core.model.audio_vae import decode_audio as vae_decode_audio
+from ltx_core.types import Audio
+from ltx_pipelines.utils.gpu_model import gpu_model
+
+logger = logging.getLogger(__name__)
+
+__all__ = ["AudioDecoder"]
+
+
+class AudioDecoder:
+ """Owns audio decoder + vocoder lifecycle."""
+
+ def __init__(
+ self,
+ checkpoint_path: str,
+ dtype: torch.dtype,
+ device: torch.device,
+ registry: Registry | None = None,
+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
+ ) -> None:
+ self._checkpoint_path = checkpoint_path
+ self._dtype = dtype
+ self._device = device
+ self._decoder_builder = Builder(
+ model_path=checkpoint_path,
+ model_class_configurator=AudioDecoderConfigurator,
+ model_sd_ops=AUDIO_VAE_DECODER_COMFY_KEYS_FILTER,
+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
+ )
+ self._vocoder_builder = Builder(
+ model_path=checkpoint_path,
+ model_class_configurator=VocoderConfigurator,
+ model_sd_ops=VOCODER_COMFY_KEYS_FILTER,
+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
+ )
+ self._alloc_trim_strategy = alloc_trim_strategy
+
+ def __call__(self, latent: torch.Tensor) -> Audio:
+ """Decode audio *latent* through VAE decoder + vocoder, then free both."""
+ logger.info("Building audio decoder + vocoder from %s", self._checkpoint_path)
+ # The vocoder always runs in fp32 (bf16 accumulation degrades spectral
+ # metrics). On CUDA/CPU it is stored in bf16 and autocast upcasts per-op to
+ # save memory; MPS has no fp32 autocast, so store it in fp32 directly and
+ # avoid the per-call cast. Negligible footprint for this small model.
+ vocoder_dtype = torch.float32 if self._device.type == "mps" else self._dtype
+ with (
+ gpu_model(
+ self._decoder_builder.build(device=self._device, dtype=self._dtype).eval(),
+ alloc_trim_strategy=self._alloc_trim_strategy,
+ ) as decoder,
+ gpu_model(
+ self._vocoder_builder.build(device=self._device, dtype=vocoder_dtype).eval(),
+ alloc_trim_strategy=self._alloc_trim_strategy,
+ ) as vocoder,
+ ):
+ return vae_decode_audio(latent, decoder, vocoder)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/diffusion_stage.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/diffusion_stage.py
new file mode 100644
index 0000000..1d51688
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/diffusion_stage.py
@@ -0,0 +1,588 @@
+from __future__ import annotations
+
+import copy
+import logging
+from collections.abc import Callable, Iterator, Sequence
+from contextlib import AbstractContextManager, ExitStack, contextmanager, nullcontext
+from dataclasses import replace
+
+import torch
+
+from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy
+from ltx_core.batch_split import BatchSplitAdapter
+from ltx_core.block_streaming import DISK_CPU_SLOTS, StreamingModelBuilder
+from ltx_core.components.noisers import Noiser
+from ltx_core.components.patchifiers import AudioPatchifier, VideoLatentPatchifier
+from ltx_core.conditioning import ConditioningItem, VideoGeneratedKeyframeSlots
+from ltx_core.loader import SDOps
+from ltx_core.loader.attention_ops import set_attention_module_op
+from ltx_core.loader.fuse_loras import bf16_fuse_rule
+from ltx_core.loader.helpers import to_device
+from ltx_core.loader.module_ops import ModuleOps
+from ltx_core.loader.primitives import LoraPathStrengthAndSDOps, ModelBuilderProtocol
+from ltx_core.loader.registry import ModelRegistry, Registry
+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder
+from ltx_core.model.model_protocol import LTXModelProtocol, ModelConfigurator
+from ltx_core.model.transformer import (
+ LTXV_MODEL_COMFY_RENAMING_MAP,
+ LTXModelConfigurator,
+ X0Model,
+)
+from ltx_core.model.transformer.attention import AttentionCallable, AttentionFunction
+from ltx_core.model.transformer.compiling import (
+ CompilationConfig,
+ build_compile_transformer_op,
+ modify_sd_ops_for_compilation,
+)
+from ltx_core.quantization import QuantizationPolicy, fp8_cast_fuse_rule
+from ltx_core.tools import AudioLatentTools, LatentTools, VideoLatentTools
+from ltx_core.types import (
+ VIDEO_SCALE_FACTORS,
+ AudioLatentShape,
+ DenoiseMask,
+ LatentState,
+ SpatioTemporalScaleFactors,
+ VideoLatentShape,
+)
+from ltx_pipelines.utils.gpu_model import gpu_model
+from ltx_pipelines.utils.helpers import assert_stage_supports_generated_keyframes, create_noised_state
+from ltx_pipelines.utils.samplers import euler_denoising_loop
+from ltx_pipelines.utils.streaming_model import streaming_model
+from ltx_pipelines.utils.types import Denoiser, ModalitySpec, OffloadMode, VideoAudio
+
+logger = logging.getLogger(__name__)
+
+ModelWrapper = Callable[[torch.nn.Module, LatentTools | None], torch.nn.Module]
+"""Wraps a built transformer for one :meth:`DiffusionStage.__call__` (see
+:meth:`DiffusionStage.with_model_wrapper`). Must keep the
+``(video, audio, perturbations) -> (video, audio)`` forward signature."""
+
+DenoisingLoop = Callable[..., VideoAudio[LatentState]]
+
+__all__ = ["DenoisingLoop", "DiffusionStage", "ModelWrapper", "RecordingDiffusionStage"]
+
+
+class _ResidentTransformer:
+ """The transformer a :meth:`DiffusionStage.resident` block holds: built by its first call, released at its end."""
+
+ def __init__(self) -> None:
+ self.model: X0Model | None = None
+ self.active = True
+ self.lifetime = ExitStack()
+
+ def release(self) -> None:
+ self.active = False
+ self.model = None
+ self.lifetime.close()
+
+
+class DiffusionStage:
+ """Owns transformer lifecycle. Builds on each call, frees on exit.
+ Under ``AllocatorTrimStrategy.DEFER`` a builder whose weights the registry keeps GPU-resident
+ (``keeps_gpu_resident_weights``) leaves its build assigned instead: every build reassigns those
+ weights, so a dispose would release no memory.
+ Replaces the manual build-transformer / ``del transformer`` pattern that
+ every pipeline previously repeated.
+ """
+
+ def __init__(
+ self,
+ transformer_builder: ModelBuilderProtocol[LTXModelProtocol],
+ dtype: torch.dtype,
+ device: torch.device,
+ *,
+ quantization: QuantizationPolicy | None = None,
+ compilation_config: CompilationConfig | None = None,
+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
+ scale_factors: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,
+ ) -> None:
+ """Construct a stage from a single pre-built transformer ``builder``.
+ Holds only that builder plus build-time configuration (dtype, device,
+ quantization, compilation). Turning a checkpoint path + LoRA set into a
+ builder -- and choosing a :class:`StreamingModelBuilder` when offloading --
+ lives in :meth:`from_checkpoint`, which is how pipelines normally create a
+ stage. A :class:`StreamingModelBuilder` selects the block-streaming build
+ path; any other builder uses the standard (all-on-GPU) path.
+ ``quantization`` and ``compilation_config`` are applied lazily by
+ :meth:`_prepared_builder` on both the standard and streaming paths.
+ ``scale_factors`` are the video VAE's spatiotemporal downscaling factors,
+ defaulting to the 32x32x8 layout.
+ """
+ self._transformer_builder = transformer_builder
+ self._dtype = dtype
+ self._device = device
+ self._quantization = quantization
+ self._compilation_config = compilation_config
+ self._alloc_trim_strategy = alloc_trim_strategy
+ self.video_scale_factors = scale_factors
+ self._model_wrapper: ModelWrapper | None = None
+ self._resident: _ResidentTransformer | None = None
+
+ @classmethod
+ def from_checkpoint( # noqa: PLR0913
+ cls,
+ checkpoint_path: str,
+ dtype: torch.dtype,
+ device: torch.device,
+ loras: tuple[LoraPathStrengthAndSDOps, ...] = (),
+ quantization: QuantizationPolicy | None = None,
+ registry: Registry | None = None,
+ compilation_config: CompilationConfig | None = None,
+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
+ offload_mode: OffloadMode = OffloadMode.NONE,
+ model_configurator: type[ModelConfigurator] = LTXModelConfigurator,
+ model_sd_ops: SDOps = LTXV_MODEL_COMFY_RENAMING_MAP,
+ scale_factors: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,
+ ) -> "DiffusionStage":
+ """Build a stage from a checkpoint path and LoRA set.
+ Constructs a single transformer builder from ``checkpoint_path`` +
+ ``loras`` and delegates to ``__init__``. When ``offload_mode != OffloadMode.NONE``
+ that builder is a :class:`StreamingModelBuilder` (fuse rule +
+ ``cpu_slots_count`` for the mode); otherwise it is the standard single-GPU
+ builder. Quantization and compilation stay on the stage and are applied
+ lazily at build time. This is the high-level entry point used by pipelines;
+ ``__init__`` itself takes an already-built builder.
+ ``model_configurator`` / ``model_sd_ops`` let callers (e.g. the audio-only
+ T2A pipeline) override the model class configurator and the state-dict key
+ mapping.
+ """
+ transformer_builder: ModelBuilderProtocol[LTXModelProtocol]
+ if offload_mode == OffloadMode.NONE:
+ transformer_builder = Builder(
+ model_path=checkpoint_path,
+ model_class_configurator=model_configurator,
+ model_sd_ops=model_sd_ops,
+ loras=tuple(loras),
+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
+ )
+ else:
+ transformer_builder = cls._build_streaming_builder(
+ checkpoint_path=checkpoint_path,
+ configurator=model_configurator,
+ model_sd_ops=model_sd_ops,
+ loras=tuple(loras),
+ quantization=quantization,
+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
+ offload_mode=offload_mode,
+ )
+
+ return cls(
+ transformer_builder,
+ dtype,
+ device,
+ quantization=quantization,
+ compilation_config=compilation_config,
+ alloc_trim_strategy=alloc_trim_strategy,
+ scale_factors=scale_factors,
+ )
+
+ @staticmethod
+ def _build_streaming_builder(
+ *,
+ checkpoint_path: str,
+ configurator: type[ModelConfigurator],
+ model_sd_ops: SDOps,
+ loras: tuple[LoraPathStrengthAndSDOps, ...],
+ quantization: QuantizationPolicy | None,
+ registry: Registry,
+ offload_mode: OffloadMode,
+ ) -> StreamingModelBuilder:
+ """Construct the streaming transformer builder for an offloading stage.
+ Returns a :class:`StreamingModelBuilder` with the checkpoint path, LoRAs,
+ fuse rule (bf16 or fp8_cast), and ``cpu_slots_count`` for ``offload_mode``.
+ Compilation / quantization sd_ops and module_ops are not applied here --
+ the stage applies them lazily via :meth:`_prepared_builder`.
+ """
+ # WeightsProvider currently only supports plain bf16 + fp8_cast LoRA fusion
+ # (no companion-key emission). Quantization policies that emit
+ # companion keys (e.g. ``.weight_scale``) cannot be streamed yet.
+ if quantization is not None and quantization.fuse_rule is not fp8_cast_fuse_rule:
+ raise ValueError(
+ "Block streaming is not supported with this quantization policy "
+ "(only bf16 and fp8_cast are currently supported)."
+ )
+ return StreamingModelBuilder(
+ model_class_configurator=configurator,
+ model_path=checkpoint_path,
+ model_sd_ops=model_sd_ops,
+ loras=loras,
+ registry=registry,
+ fuse_rule=quantization.fuse_rule if quantization is not None else bf16_fuse_rule,
+ blocks_attr="transformer_blocks",
+ blocks_prefix="transformer_blocks",
+ cpu_slots_count=DISK_CPU_SLOTS if offload_mode == OffloadMode.DISK else None,
+ )
+
+ @property
+ def supports_generated_keyframes(self) -> bool:
+ """Whether this stage's checkpoint was trained with the keyframe absolute-position embedding.
+ Read from the checkpoint's transformer config, so it is answered before any weights are
+ built. Generated keyframe slots on a checkpoint without it would be denoised as unmarked
+ tokens while still costing a full latent frame of tokens each, so pipelines refuse rather
+ than silently degrade.
+ """
+ transformer_config = self._transformer_builder.model_config().get("transformer", {})
+ return bool(transformer_config.get("use_keyframes_abs_pos_embedding", False))
+
+ def _assert_supports_conditionings(self, conditionings: Sequence[ConditioningItem] | None) -> None:
+ if conditionings is None or self.supports_generated_keyframes:
+ return
+ if any(isinstance(item, VideoGeneratedKeyframeSlots) for item in conditionings):
+ assert_stage_supports_generated_keyframes(self)
+
+ @property
+ def dtype(self) -> torch.dtype:
+ """Dtype this stage builds its transformer and sizes its latents in."""
+ return self._dtype
+
+ @property
+ def device(self) -> torch.device:
+ """Device this stage builds its transformer and sizes its latents on."""
+ return self._device
+
+ def with_attention(self, attention: AttentionFunction | AttentionCallable | None) -> "DiffusionStage":
+ """Return a new ``DiffusionStage`` that pins the transformer build to ``attention``.
+ Functional: never mutates ``self``. The returned stage shares all other
+ configuration with the original; only the underlying builders' ``module_ops``
+ gain a ``set_attention_module_op(attention)`` entry so subsequent transformer
+ builds use that kernel. ``attention=None`` is a no-op (returns ``self``).
+ """
+ if attention is None:
+ return self
+ op = set_attention_module_op(attention)
+ new = copy.copy(self)
+ new._transformer_builder = self._transformer_builder.with_module_ops(
+ (*self._transformer_builder.module_ops, op),
+ )
+ new._resident = None
+ return new
+
+ def with_builder(self, builder: ModelBuilderProtocol[LTXModelProtocol]) -> "DiffusionStage":
+ """Return a new ``DiffusionStage`` that builds its transformer from ``builder``.
+ Functional: never mutates ``self``; shares all other configuration (dtype, device,
+ quantization, compilation). Affects the standard (non-offload) build path.
+ """
+ new = copy.copy(self)
+ new._transformer_builder = builder
+ new._resident = None
+ return new
+
+ def with_model_wrapper(self, wrapper: "ModelWrapper | None") -> "DiffusionStage":
+ """Return a new stage that passes each built transformer through ``wrapper``.
+ Functional: never mutates ``self``. The wrapper is applied inside the transformer
+ context -- after the model is built, before the denoising loop -- on both the standard
+ and block-streaming paths, and it receives the call's video tools. Wrapping *here*
+ rather than around the builder is what lets a wrapper compose with offloading: a
+ wrapping builder would hide a :class:`StreamingModelBuilder` from :attr:`_is_streaming`,
+ so the stage would take the plain ``gpu_model`` path and never tear the stream down.
+ ``wrapper=None`` clears any previously set wrapper.
+ """
+ new = copy.copy(self)
+ new._model_wrapper = wrapper
+ return new
+
+ @property
+ def quantization(self) -> QuantizationPolicy | None:
+ """The quantization policy applied to every transformer build."""
+ return self._quantization
+
+ def with_quantization(self, quantization: QuantizationPolicy | None) -> "DiffusionStage":
+ """Return a new ``DiffusionStage`` whose builds apply ``quantization`` instead.
+ Functional: never mutates ``self``. A policy with the same ``sd_ops`` loads the same
+ weights, so the two stages share the registry's tensors. Like a copy with another builder,
+ the copy is outside any :meth:`resident` block of ``self``: its builds apply its own policy.
+ """
+ new = copy.copy(self)
+ new._quantization = quantization
+ new._resident = None
+ return new
+
+ def with_loras(self, loras: tuple[LoraPathStrengthAndSDOps, ...]) -> "DiffusionStage":
+ """Return a new ``DiffusionStage`` built with exactly ``loras`` (replacing the current set)."""
+ return self.with_builder(self._transformer_builder.with_loras(loras))
+
+ def _prepared_builder(self) -> ModelBuilderProtocol[LTXModelProtocol]:
+ """Return the configured builder with the stage's build-time ops applied.
+ Compilation and quantization live on the stage (not on the builder) and are
+ applied here, lazily, for both the standard and streaming paths. This keeps
+ the builder holding only raw sd_ops/module_ops/LoRAs, so ``with_loras`` /
+ ``with_builder`` swap them consistently regardless of the build path. The
+ returned copy preserves the builder's concrete type (e.g. a
+ ``StreamingModelBuilder`` stays one).
+ """
+ builder = self._transformer_builder
+ _ensure_cudagraph_compatible_builder(builder, self._compilation_config)
+ sd_ops = builder.model_sd_ops
+ module_ops = builder.module_ops
+ loras = builder.loras
+ if self._quantization is not None:
+ sd_ops, module_ops = _chain_quantization(sd_ops, module_ops, self._quantization)
+ builder = builder.with_fuse_rule(self._quantization.fuse_rule)
+ if self._compilation_config is not None:
+ number_of_layers = builder.model_config()["transformer"]["num_layers"]
+ sd_ops, module_ops, loras = _apply_compile_ops(
+ sd_ops, module_ops, loras, number_of_layers, self._compilation_config
+ )
+ return builder.with_module_ops(module_ops).with_sd_ops(sd_ops).with_loras(loras)
+
+ def _build_transformer(self, *, device: torch.device | None = None, **kwargs: object) -> X0Model:
+ target = device or self._device
+ return to_device(X0Model(self._prepared_builder().build(device=target, **kwargs)), target).eval()
+
+ @property
+ def _is_streaming(self) -> bool:
+ """Whether the configured builder uses the block-streaming build path."""
+ return isinstance(self._transformer_builder, StreamingModelBuilder)
+
+ @contextmanager
+ def _streaming_transformer_ctx(self) -> Iterator[X0Model]:
+ builder = self._prepared_builder()
+ assert isinstance(builder, StreamingModelBuilder)
+ with streaming_model(builder, self._device, self._dtype, self._alloc_trim_strategy) as streaming_wrapper:
+ yield X0Model(streaming_wrapper).eval()
+
+ def _transformer_ctx(self, **kwargs: object) -> AbstractContextManager:
+ if self._is_streaming:
+ return self._streaming_transformer_ctx()
+ if self._resident is None or not self._resident.active or self._transformer_builder.builds_per_shape:
+ return self._transformer_lifetime(self._build_transformer(**kwargs))
+ if self._resident.model is None:
+ self._resident.model = self._resident.lifetime.enter_context(
+ self._transformer_lifetime(self._build_transformer(**kwargs))
+ )
+ return nullcontext(self._resident.model)
+
+ def _transformer_lifetime(self, transformer: X0Model) -> AbstractContextManager:
+ if self._transformer_builder.keeps_gpu_resident_weights and (
+ self._alloc_trim_strategy == AllocatorTrimStrategy.DEFER
+ ):
+ # The registry keeps these weights resident and every build reassigns them, so a
+ # dispose would release no memory.
+ return nullcontext(transformer)
+ return gpu_model(transformer, alloc_trim_strategy=self._alloc_trim_strategy)
+
+ @contextmanager
+ def resident(self) -> Iterator[None]:
+ """Hold one built transformer for every call on this stage inside the block.
+ The first call builds; later calls -- on this stage or on the ``with_model_wrapper`` copies
+ made of it inside the block, e.g. a tiling wrapper per window -- reuse that model, and the
+ block's end releases it. A copy with another builder or policy (``with_builder``,
+ ``with_loras``, ``with_attention``, ``with_quantization``) is outside the block and builds
+ per call. Every call inside must want the stage's weights: the block does not track LoRA
+ sets, so a call on a stage with other LoRAs must not happen inside it. A streaming stage, a
+ builder that builds per latent shape (tiled data parallel), or a nested block just runs the
+ body.
+ """
+ if self._is_streaming or self._resident is not None:
+ yield
+ return
+ holder = _ResidentTransformer()
+ self._resident = holder
+ try:
+ yield
+ finally:
+ self._resident = None
+ holder.release()
+
+ def __call__(
+ self,
+ denoiser: Denoiser,
+ sigmas: torch.Tensor,
+ noiser: Noiser,
+ modalities: VideoAudio[ModalitySpec],
+ loop: DenoisingLoop | None = None,
+ max_batch_size: int = 1,
+ ) -> VideoAudio[LatentState]:
+ """Build transformer -> run denoising loop -> free transformer (see the class docstring for DEFER).
+ Returns a :class:`VideoAudio` of unpatchified :class:`LatentState` with cleared
+ conditionings for present modalities.
+ """
+ video, audio = modalities
+ if video is None and audio is None:
+ raise ValueError("At least one of `modalities.video` or `modalities.audio` must be provided")
+ if video is not None:
+ self._assert_supports_conditionings(video.conditionings)
+
+ if loop is None:
+ loop = euler_denoising_loop
+
+ video_latent = video.latent if video is not None else None
+ audio_latent = audio.latent if audio is not None else None
+ conditioning_fps = video.conditioning_fps if video is not None else None
+
+ # Build video_tools up front so it can be forwarded to the transformer
+ # context (required by TiledDataParallelBuilder in multi-GPU mode).
+ video_tools = (
+ VideoLatentTools(
+ VideoLatentPatchifier(patch_size=1),
+ VideoLatentShape.from_torch_shape(video_latent.shape),
+ conditioning_fps,
+ scale_factors=self.video_scale_factors,
+ )
+ if video_latent is not None
+ else None
+ )
+
+ mode = "streaming" if self._is_streaming else "standard"
+ logger.info("Building transformer (%s) from %s", mode, self._transformer_builder.checkpoint)
+ with self._transformer_ctx(video_tools=video_tools) as built:
+ transformer = built if self._model_wrapper is None else self._model_wrapper(built, video_tools)
+ logger.info(
+ "Running denoising loop (%d steps, video latent %s, audio latent %s @ %.1f fps)",
+ len(sigmas) - 1,
+ tuple(video_latent.shape) if video_latent is not None else None,
+ tuple(audio_latent.shape) if audio_latent is not None else None,
+ conditioning_fps,
+ )
+ video_state: LatentState | None = None
+ if video is not None and video_tools is not None:
+ video_state = _build_state(video, video_tools, noiser, self._dtype, self._device)
+
+ audio_tools = (
+ AudioLatentTools(
+ AudioPatchifier(patch_size=1),
+ AudioLatentShape.from_torch_shape(audio_latent.shape),
+ )
+ if audio_latent is not None
+ else None
+ )
+ audio_state: LatentState | None = None
+ if audio is not None and audio_tools is not None:
+ audio_state = _build_state(audio, audio_tools, noiser, self._dtype, self._device)
+
+ wrapped = BatchSplitAdapter(transformer, max_batch_size=max_batch_size) # type: ignore[arg-type]
+ video_state, audio_state = loop(
+ sigmas=sigmas,
+ video_state=video_state,
+ audio_state=audio_state,
+ transformer=wrapped,
+ denoiser=denoiser,
+ )
+
+ if video_state is not None and video_tools is not None:
+ video_state = video_tools.clear_conditioning(video_state)
+ video_state = video_tools.unpatchify(video_state)
+ if audio_state is not None and audio_tools is not None:
+ audio_state = audio_tools.clear_conditioning(audio_state)
+ audio_state = audio_tools.unpatchify(audio_state)
+
+ return VideoAudio(video_state, audio_state)
+
+
+class RecordingDiffusionStage:
+ """Wraps a :class:`DiffusionStage` and records the states each call produced.
+ Diagnostics wrapper for a stage's intermediate ``LatentState``s. Use this to inspect
+ per-stage states (including half-res stage-1 slots) without a lifecycle callback. Pipelines
+ hold their stage in a plain attribute, so a caller substitutes this in::
+ pipeline.stage = RecordingDiffusionStage(pipeline.stage)
+ result = pipeline(...)
+ keyframes = pipeline.stage.generated_keyframes
+ Delegates every other attribute to the wrapped stage, so a substituted stage stays usable
+ wherever the original was. Recording holds references to the returned states, which keeps
+ their latents alive -- fine for diagnostics on one run, not something to leave enabled in a
+ long-lived serving process.
+ """
+
+ def __init__(self, stage: DiffusionStage) -> None:
+ self._stage = stage
+ self.video_states: list[LatentState] = []
+ self.audio_states: list[LatentState | None] = []
+
+ def __call__(self, *args: object, **kwargs: object) -> VideoAudio[LatentState]:
+ video_state, audio_state = self._stage(*args, **kwargs) # type: ignore[arg-type]
+ if video_state is not None:
+ self.video_states.append(video_state)
+ self.audio_states.append(audio_state)
+ return VideoAudio(video_state, audio_state)
+
+ @property
+ def generated_keyframes(self) -> torch.Tensor | None:
+ """Generated keyframe latents from the first recorded call, if it produced any."""
+ for state in self.video_states:
+ if state.generated_keyframes is not None:
+ return state.generated_keyframes
+ return None
+
+ def __getattr__(self, name: str) -> object:
+ return getattr(self._stage, name)
+
+
+def _chain_quantization(
+ sd_ops: SDOps,
+ module_ops: tuple[ModuleOps, ...],
+ quantization: QuantizationPolicy,
+) -> tuple[SDOps, tuple[ModuleOps, ...]]:
+ chained_sd_ops = sd_ops
+ if quantization.sd_ops is not None:
+ chained_sd_ops = SDOps(
+ name=f"sd_ops_chain_{sd_ops.name}+{quantization.sd_ops.name}",
+ mapping=(*sd_ops.mapping, *quantization.sd_ops.mapping),
+ )
+ return chained_sd_ops, (*module_ops, *quantization.module_ops)
+
+
+def _apply_compile_ops(
+ sd_ops: SDOps,
+ module_ops: tuple[ModuleOps, ...],
+ loras: tuple[LoraPathStrengthAndSDOps, ...],
+ number_of_layers: int,
+ compilation_config: CompilationConfig,
+) -> tuple[SDOps, tuple[ModuleOps, ...], tuple[LoraPathStrengthAndSDOps, ...]]:
+ """Rewrite sd_ops/module_ops/LoRAs for compiled blocks (params land under ``_orig_mod``)."""
+ sd_ops = modify_sd_ops_for_compilation(sd_ops, number_of_layers)
+ compile_op = build_compile_transformer_op(compilation_config)
+ module_ops = (*module_ops, compile_op)
+ loras = tuple(
+ LoraPathStrengthAndSDOps(
+ lora.path,
+ lora.strength,
+ modify_sd_ops_for_compilation(lora.sd_ops, number_of_layers),
+ )
+ for lora in loras
+ )
+ return sd_ops, module_ops, loras
+
+
+def _ensure_cudagraph_compatible_builder(
+ builder: ModelBuilderProtocol[LTXModelProtocol], compilation_config: CompilationConfig | None
+) -> None:
+ """CUDA-graph compile paths need GPU-resident weights across dispose/rebuild."""
+ if compilation_config is None:
+ return
+ cudagraph_modes = frozenset({"reduce-overhead", "max-autotune"})
+ needs_resident = compilation_config.capture or compilation_config.mode in cudagraph_modes
+ if not needs_resident or builder.keeps_gpu_resident_weights:
+ return
+ raise ValueError(
+ "CompilationConfig capture / mode=reduce-overhead|max-autotune requires a builder "
+ "with keeps_gpu_resident_weights=True (SequenceParallelBuilder / "
+ "TiledDataParallelBuilder with a weight tracker, or StreamingModelBuilder). "
+ "Plain SingleGPUModelBuilder reloads weights onto fresh GPU storages each stage, "
+ "which invalidates CUDA graphs (or repeatedly recaptures until Dynamo gives up)."
+ )
+
+
+def _build_state(
+ spec: ModalitySpec,
+ tools: LatentTools,
+ noiser: Noiser,
+ dtype: torch.dtype,
+ device: torch.device,
+) -> LatentState:
+ """Create a noised latent state from a modality spec and tools."""
+ state = create_noised_state(
+ tools=tools,
+ conditionings=spec.conditionings,
+ noiser=noiser,
+ dtype=dtype,
+ device=device,
+ noise_scale=spec.noise_scale,
+ initial_latent=spec.latent,
+ )
+ if spec.frozen:
+ state = replace(
+ state,
+ denoise_mask=DenoiseMask(torch.zeros(state.denoise_mask.tensor.shape)).compact(device),
+ frozen=True,
+ )
+ return state
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/duration_predictor.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/duration_predictor.py
new file mode 100644
index 0000000..75c75c1
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/duration_predictor.py
@@ -0,0 +1,148 @@
+from __future__ import annotations
+
+import logging
+
+import torch
+
+from ltx_core.duration_head import (
+ DURATION_HEAD_KEY_OPS,
+ DurationHead,
+ DurationHeadConfigurator,
+)
+from ltx_core.loader import SDOps
+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder
+from ltx_core.model.model_protocol import ModelConfigurator
+from ltx_pipelines.utils.helpers import seconds_to_clamped_num_frames
+from ltx_pipelines.utils.types import AutoDuration
+
+logger = logging.getLogger(__name__)
+
+__all__ = ["DurationPredictor", "require_num_frames_source", "resolve_num_frames"]
+
+
+class DurationPredictor:
+ """Predicts shot duration (in frames) from ``PromptEncoder`` output.
+ Unlike most blocks, the model is held directly rather than rebuilt on every call:
+ DurationHead is a few MB, so there's no memory pressure motivating the
+ build-on-call / free-on-exit pattern used for the large transformer/VAE blocks.
+ Construct via :meth:`from_checkpoint`.
+ """
+
+ def __init__(self, head: DurationHead) -> None:
+ """Construct from an already-built, already-loaded head."""
+ self._head = head
+
+ @classmethod
+ def from_checkpoint(
+ cls,
+ checkpoint_path: str | None,
+ dtype: torch.dtype,
+ device: torch.device,
+ model_configurator: type[ModelConfigurator[DurationHead]] = DurationHeadConfigurator,
+ model_sd_ops: SDOps = DURATION_HEAD_KEY_OPS,
+ ) -> "DurationPredictor | None":
+ """Build a predictor from a checkpoint path, or ``None`` if unavailable.
+ Returns ``None`` when ``checkpoint_path`` is ``None`` (split pack without a
+ duration-head file), or when the file has no DurationHead weights (pre-2.5 /
+ gemma3 monoliths). The underlying loader uses ``strict=False``, so a missing
+ head does not raise -- it silently leaves parameters on the meta device;
+ checked here so callers get ``None`` instead of a predictor that would crash
+ later.
+ No registry: the head is a few MB, so there's no benefit to caching its weights or
+ shell across pipeline instances the way the large transformer/VAE builders do.
+ """
+ if checkpoint_path is None:
+ return None
+ builder = Builder(
+ model_path=checkpoint_path,
+ model_class_configurator=model_configurator,
+ model_sd_ops=model_sd_ops,
+ )
+ head = builder.build(device=device, dtype=dtype).eval()
+ if any(param.is_meta for param in head.parameters()):
+ logger.info(
+ "No DurationHead weights found in %s; auto-duration prediction unavailable.",
+ checkpoint_path,
+ )
+ return None
+ return cls(head)
+
+ def __call__(
+ self,
+ video_encoding: torch.Tensor | None,
+ audio_encoding: torch.Tensor | None,
+ *,
+ frame_rate: float,
+ min_seconds: float = 1.0,
+ max_seconds: float = 20.0,
+ ) -> int:
+ """Predict a frame count from caption connector tokens, snapped to the VAE's grid.
+ ``min_seconds``/``max_seconds`` clamp the prediction so a misbehaving prediction can't
+ request a degenerate or OOM-sized generation; the defaults are 1s and 20s. The result is
+ a frame count snapped to the VAE's ``8k + 1`` causal temporal grid. Callers doing offline
+ analysis of raw predictions (rather than feeding a real generation) can pass a much wider
+ range to see the unclamped duration.
+ """
+ if video_encoding is None and audio_encoding is None:
+ raise ValueError("DurationPredictor requires at least one of video_encoding / audio_encoding")
+ seconds_pred = self._head(video_encoding, audio_encoding)
+ if seconds_pred.shape != (1,):
+ raise ValueError(
+ f"DurationPredictor only supports a single-item batch, got prediction shape {tuple(seconds_pred.shape)}"
+ )
+ seconds = seconds_pred.item()
+ min_frames = round(min_seconds * frame_rate)
+ max_frames = round(max_seconds * frame_rate)
+ num_frames = seconds_to_clamped_num_frames(
+ seconds, frame_rate=frame_rate, min_frames=min_frames, max_frames=max_frames
+ )
+ if seconds > max_seconds or seconds < min_seconds:
+ logger.warning(
+ "DurationHead prediction clamped: raw %.2fs outside [%.2fs, %.2fs], using %.2fs (%d frames) @ %.2f fps",
+ seconds,
+ min_seconds,
+ max_seconds,
+ num_frames / frame_rate,
+ num_frames,
+ frame_rate,
+ )
+ else:
+ logger.info("DurationHead predicted %.2fs (%d frames @ %.2f fps)", seconds, num_frames, frame_rate)
+ return num_frames
+
+
+def require_num_frames_source(num_frames: int | AutoDuration, duration_predictor: DurationPredictor | None) -> None:
+ """Guard against an unsatisfiable auto-duration request.
+ Call at the very top of a pipeline's ``__call__`` -- before prompt encoding or any other
+ work -- so a checkpoint without DurationHead weights (anything predating 2.5) fails fast
+ with a clear message instead of after paying for work whose result would be discarded.
+ """
+ if isinstance(num_frames, AutoDuration) and duration_predictor is None:
+ raise ValueError(
+ "num_frames was AutoDuration but this checkpoint has no DurationHead weights to "
+ "auto-predict duration from (DurationHead ships from LTX 2.5 / gemma4 onward). "
+ "Pass num_frames explicitly."
+ )
+
+
+def resolve_num_frames(
+ num_frames: int | AutoDuration,
+ duration_predictor: DurationPredictor | None,
+ *,
+ video_encoding: torch.Tensor | None,
+ audio_encoding: torch.Tensor | None,
+ frame_rate: float,
+) -> int:
+ """Resolve ``num_frames`` to a concrete frame count, predicting it if ``AutoDuration``.
+ Call after prompt encoding (once ``video_encoding``/``audio_encoding`` exist) and after
+ ``require_num_frames_source`` has already validated a predictor is available when needed.
+ """
+ if not isinstance(num_frames, AutoDuration):
+ return num_frames
+ return duration_predictor(
+ video_encoding,
+ audio_encoding,
+ frame_rate=frame_rate,
+ min_seconds=num_frames.min_seconds,
+ max_seconds=num_frames.max_seconds,
+ )
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/image_conditioner.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/image_conditioner.py
new file mode 100644
index 0000000..cb7513c
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/image_conditioner.py
@@ -0,0 +1,119 @@
+from __future__ import annotations
+
+from collections.abc import Callable, Sequence
+from concurrent.futures import ThreadPoolExecutor
+from dataclasses import replace
+from typing import TypeVar
+
+import torch
+
+from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy
+from ltx_core.loader import SDOps
+from ltx_core.loader.registry import ModelRegistry, Registry
+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder
+from ltx_core.model.video_vae import (
+ CHANNELS_LAST_3D_WEIGHTS,
+ VAE_ENCODER_COMFY_KEYS_FILTER,
+ VideoEncoder,
+ VideoEncoderConfigurator,
+)
+from ltx_pipelines.utils.constants import detect_params
+from ltx_pipelines.utils.gpu_model import gpu_model
+from ltx_pipelines.utils.media_io.color_config import EXRColorSpace
+from ltx_pipelines.utils.media_io.decode import BackgroundStill, check_conditioning_still, load_conditioning_still
+from ltx_pipelines.utils.types import DecodedStill, ImageConditioningInput, LoadedImageConditioning
+
+# Shared channels-last encoder sd_ops so the registry caches one copy across encoder builds.
+VAE_ENCODER_SD_OPS = SDOps(
+ name=f"sd_ops_chain_{VAE_ENCODER_COMFY_KEYS_FILTER.name}+{CHANNELS_LAST_3D_WEIGHTS.name}",
+ mapping=(*VAE_ENCODER_COMFY_KEYS_FILTER.mapping, *CHANNELS_LAST_3D_WEIGHTS.mapping),
+)
+
+T = TypeVar("T")
+
+__all__ = ["ImageConditioner"]
+
+
+class ImageConditioner:
+ """Owns video encoder lifecycle, the CRF image conditionings are re-compressed at, and their stills.
+ Builds the encoder, passes it to the user-supplied callable, then frees it.
+ Also the single place that knows what H.264 CRF an image conditioning should be
+ re-compressed at: that value is a property of the model generation, and this block
+ already holds the checkpoint it belongs to. Pipelines call ``load_conditionings`` so callers may
+ omit the CRF and still condition the way the model was trained, and so each still is
+ decoded once per run however many stages, tiles or windows encode it.
+ """
+
+ def __init__(
+ self,
+ checkpoint_path: str,
+ dtype: torch.dtype,
+ device: torch.device,
+ registry: Registry | None = None,
+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
+ ) -> None:
+ self._checkpoint_path = checkpoint_path
+ self._default_image_crf: int | None = None
+ self._dtype = dtype
+ self._device = device
+ self._encoder_builder = Builder(
+ model_path=checkpoint_path,
+ model_class_configurator=VideoEncoderConfigurator,
+ model_sd_ops=VAE_ENCODER_SD_OPS,
+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
+ )
+ self._alloc_trim_strategy = alloc_trim_strategy
+ self._decode_pool = ThreadPoolExecutor(max_workers=2, thread_name_prefix="image-decode")
+
+ @property
+ def default_image_crf(self) -> int:
+ """The H.264 CRF image conditionings should be re-compressed at for this checkpoint.
+ Read from the checkpoint's ``model_version`` metadata (see ``detect_params``) rather
+ than hardcoded, because the value the model was trained with changed between
+ generations. Resolved on first use and cached, so a run without image conditionings
+ never reads the header.
+ """
+ if self._default_image_crf is None:
+ self._default_image_crf = detect_params(self._checkpoint_path).default_image_crf
+ return self._default_image_crf
+
+ def resolve_crf(self, images: Sequence[ImageConditioningInput]) -> list[ImageConditioningInput]:
+ """Fill in ``default_image_crf`` for every conditioning that left its CRF unset (``crf=None``).
+ Call near the top of a pipeline's ``__call__``: omitting the CRF then means "use what
+ matches this model", while an explicitly passed CRF (including ``0`` for no
+ re-compression) is always honoured.
+ """
+ return [image if image.crf is not None else replace(image, crf=self.default_image_crf) for image in images]
+
+ def load_conditionings(
+ self, images: Sequence[ImageConditioningInput], color_space: EXRColorSpace | None = None
+ ) -> list[LoadedImageConditioning]:
+ """Resolve every CRF (see ``resolve_crf``) and start decoding each distinct still once for the run.
+ Call near the top of a pipeline's ``__call__``. Every input comes back as a
+ :class:`LoadedImageConditioning` whose still is shared by every input with the same path and
+ CRF and decodes on a worker thread while the run goes on (prompt encoding, model builds);
+ every stage, tile, window and chunk that encodes it (or a rebased copy) waits for that still
+ and cuts its resolution from it instead of decoding the file again. ``.exr`` stills need
+ ``color_space`` (from ``--hdr``). A missing file, or an EXR without ``color_space``, raises
+ ``ValueError`` here, before any model is built; an undecodable one raises the same
+ ``ValueError`` at its first encode.
+ """
+ stills: dict[tuple[str, int], DecodedStill] = {}
+ loaded: list[LoadedImageConditioning] = []
+ for image in self.resolve_crf(images):
+ key = (image.path, image.crf)
+ if key not in stills:
+ check_conditioning_still(image.path, color_space)
+ stills[key] = BackgroundStill(
+ self._decode_pool.submit(load_conditioning_still, image.path, image.crf, color_space)
+ )
+ loaded.append(LoadedImageConditioning(still=stills[key], strength=image.strength, binding=image.binding))
+ return loaded
+
+ def _build_encoder(self) -> VideoEncoder:
+ return self._encoder_builder.build(device=self._device, dtype=self._dtype).eval()
+
+ def __call__(self, fn: Callable[[VideoEncoder], T]) -> T:
+ """Build video encoder → call *fn(encoder)* → free encoder."""
+ with gpu_model(self._build_encoder(), alloc_trim_strategy=self._alloc_trim_strategy) as encoder:
+ return fn(encoder)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/prompt_encoder.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/prompt_encoder.py
new file mode 100644
index 0000000..ec574a1
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/prompt_encoder.py
@@ -0,0 +1,202 @@
+from __future__ import annotations
+
+import logging
+from contextlib import AbstractContextManager
+
+import torch
+
+from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy
+from ltx_core.block_streaming import DISK_CPU_SLOTS, StreamingModelBuilder
+from ltx_core.loader.primitives import BuilderProtocol
+from ltx_core.loader.registry import ModelRegistry, Registry
+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder
+from ltx_core.text_encoders.gemma import (
+ EMBEDDINGS_PROCESSOR_KEY_OPS,
+ EmbeddingsProcessorConfigurator,
+ GemmaTextEncoderConfigurator,
+ gemma_model_type,
+ get_gemma_ops,
+ resolve_gemma_weight_paths,
+)
+from ltx_core.text_encoders.gemma.embeddings_processor import EmbeddingsProcessor, EmbeddingsProcessorOutput
+from ltx_pipelines.utils.gpu_model import gpu_model
+from ltx_pipelines.utils.helpers import generate_enhanced_prompt
+from ltx_pipelines.utils.model_paths import ModelPaths
+from ltx_pipelines.utils.streaming_model import streaming_model
+from ltx_pipelines.utils.types import OffloadMode
+
+logger = logging.getLogger(__name__)
+
+__all__ = ["PromptEncoder"]
+
+
+class PromptEncoder:
+ """Owns text encoder + embeddings processor lifecycle.
+ Loads Gemma, optionally enhances then encodes prompts, frees Gemma, then loads the
+ embeddings processor. ``_enhancer_text_encoder_builder`` is always set: when enhance
+ and encode share a checkpoint it aliases ``_text_encoder_builder`` (same object /
+ residency); a distinct builder means sequential enhance-then-encode contexts.
+ """
+
+ def __init__(
+ self,
+ model_paths: ModelPaths,
+ dtype: torch.dtype,
+ device: torch.device,
+ registry: Registry | None = None,
+ offload_mode: OffloadMode = OffloadMode.NONE,
+ text_encoder_builder: BuilderProtocol | None = None,
+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
+ prompt_enhancer_gemma_root: str | None = None,
+ enhancer_text_encoder_builder: BuilderProtocol | None = None,
+ ) -> None:
+ text_encoder_path = model_paths.text_encoder()
+ embeddings_paths = model_paths.embeddings_weight_paths
+ if not embeddings_paths:
+ raise ValueError(
+ "PromptEncoder requires ModelPaths.embeddings_weight_paths "
+ "(transformer checkpoint and, in split mode, the text-encoder file)."
+ )
+ self._text_encoder_path = text_encoder_path
+ self._embeddings_paths = embeddings_paths
+ self._dtype = dtype
+ self._device = device
+ self._offload_mode = offload_mode
+ self._alloc_trim_strategy = alloc_trim_strategy
+ self._prompt_enhancer_gemma_root = prompt_enhancer_gemma_root
+
+ registry = registry or ModelRegistry(cache_models=True, cache_weights=False)
+ self._registry = registry
+
+ if text_encoder_builder is not None:
+ encode_type = text_encoder_builder.model_config().get("model_type") or gemma_model_type(text_encoder_path)
+ else:
+ encode_type = gemma_model_type(text_encoder_path)
+ if encode_type not in _ENCODE_MODEL_TYPES:
+ raise ValueError(
+ f"Encode root model_type={encode_type!r} is not supported for encoding; "
+ f"expected one of {sorted(_ENCODE_MODEL_TYPES)}."
+ )
+ self._encode_model_type = encode_type
+
+ if text_encoder_builder is not None:
+ if offload_mode != OffloadMode.NONE:
+ raise ValueError(
+ "text_encoder_builder cannot be used with offload_mode != OffloadMode.NONE "
+ "because no streaming text encoder builder is available."
+ )
+ self._text_encoder_builder = text_encoder_builder
+ self._streaming_text_encoder_builder = None
+ else:
+ gemma_sd_ops, gemma_module_ops = get_gemma_ops(text_encoder_path)
+ weight_paths = resolve_gemma_weight_paths(text_encoder_path)
+ self._text_encoder_builder = Builder(
+ model_path=weight_paths,
+ model_class_configurator=GemmaTextEncoderConfigurator.with_gemma_model_path(text_encoder_path),
+ model_sd_ops=gemma_sd_ops,
+ module_ops=gemma_module_ops,
+ registry=registry,
+ )
+ self._streaming_text_encoder_builder = StreamingModelBuilder(
+ model_path=weight_paths,
+ model_class_configurator=GemmaTextEncoderConfigurator.with_gemma_model_path(text_encoder_path),
+ model_sd_ops=gemma_sd_ops,
+ module_ops=gemma_module_ops,
+ registry=registry,
+ blocks_attr="model.model.language_model.layers",
+ blocks_prefix="model.model.language_model.layers",
+ cpu_slots_count=DISK_CPU_SLOTS if offload_mode == OffloadMode.DISK else None,
+ )
+
+ # Always set: shared enhance aliases the encode builder (same object / residency).
+ if enhancer_text_encoder_builder is not None:
+ self._enhancer_text_encoder_builder = enhancer_text_encoder_builder
+ elif prompt_enhancer_gemma_root is not None and prompt_enhancer_gemma_root != text_encoder_path:
+ enhancer_sd_ops, enhancer_module_ops = get_gemma_ops(prompt_enhancer_gemma_root)
+ enhancer_paths = resolve_gemma_weight_paths(prompt_enhancer_gemma_root)
+ self._enhancer_text_encoder_builder = Builder(
+ model_path=enhancer_paths,
+ model_class_configurator=GemmaTextEncoderConfigurator.with_gemma_model_path(prompt_enhancer_gemma_root),
+ model_sd_ops=enhancer_sd_ops,
+ module_ops=enhancer_module_ops,
+ registry=registry,
+ )
+ else:
+ self._enhancer_text_encoder_builder = self._text_encoder_builder
+
+ self._embeddings_processor_builder = Builder(
+ model_path=embeddings_paths,
+ model_class_configurator=EmbeddingsProcessorConfigurator.with_gemma_model_path(text_encoder_path),
+ model_sd_ops=EMBEDDINGS_PROCESSOR_KEY_OPS,
+ registry=registry,
+ )
+
+ def _build_text_encoder(self) -> torch.nn.Module:
+ """Build the Gemma text encoder (non-streaming path)."""
+ return self._text_encoder_builder.build(device=self._device, dtype=self._dtype).eval()
+
+ def _build_enhancer_text_encoder(self) -> torch.nn.Module:
+ return self._enhancer_text_encoder_builder.build(device=self._device, dtype=self._dtype).eval()
+
+ def _build_embeddings_processor(self) -> EmbeddingsProcessor:
+ """Build the embeddings processor on the target device."""
+ return self._embeddings_processor_builder.build(device=self._device, dtype=self._dtype).eval()
+
+ def _text_encoder_ctx(self) -> AbstractContextManager:
+ if self._offload_mode != OffloadMode.NONE:
+ return streaming_model(
+ self._streaming_text_encoder_builder, self._device, self._dtype, self._alloc_trim_strategy
+ )
+ return gpu_model(self._build_text_encoder(), alloc_trim_strategy=self._alloc_trim_strategy)
+
+ def __call__(
+ self,
+ prompts: list[str],
+ *,
+ enhance_first_prompt: bool = False,
+ enhance_prompt_image: str | None = None,
+ enhance_prompt_seed: int = 42,
+ enhance_static_cache: bool = False,
+ ) -> list[EmbeddingsProcessorOutput]:
+ """Encode *prompts* through Gemma -> embeddings processor, freeing each model after use."""
+ prompts = list(prompts)
+ enhance_kwargs = {
+ "image_path": enhance_prompt_image,
+ "seed": enhance_prompt_seed,
+ "static_cache": enhance_static_cache,
+ }
+ separate_enhancer = self._enhancer_text_encoder_builder is not self._text_encoder_builder
+
+ if enhance_first_prompt and separate_enhancer:
+ logger.info(
+ "Enhancing with separate Gemma%s",
+ f" from {self._prompt_enhancer_gemma_root}" if self._prompt_enhancer_gemma_root else "",
+ )
+ with gpu_model(
+ self._build_enhancer_text_encoder(),
+ alloc_trim_strategy=self._alloc_trim_strategy,
+ ) as enhancer:
+ prompts[0] = generate_enhanced_prompt(enhancer, prompts[0], **enhance_kwargs)
+ elif enhance_first_prompt and self._encode_model_type != "gemma3":
+ raise ValueError(
+ f"Prompt enhancement with encode root model_type={self._encode_model_type!r} "
+ "requires --prompt-enhancer-gemma-root pointing at a generative instruct checkpoint "
+ "(e.g. gemma3 or gemma4 E2B-it)."
+ )
+
+ logger.info("Building text encoder from %s", self._text_encoder_path)
+ with self._text_encoder_ctx() as text_encoder:
+ if enhance_first_prompt and not separate_enhancer:
+ prompts[0] = generate_enhanced_prompt(text_encoder, prompts[0], **enhance_kwargs)
+ raw_outputs = text_encoder.encode(prompts)
+ logger.info("Text encoder done, building embeddings processor from %s", self._embeddings_paths)
+
+ with gpu_model(
+ self._build_embeddings_processor(), alloc_trim_strategy=self._alloc_trim_strategy
+ ) as embeddings_processor:
+ result = [embeddings_processor.process_hidden_states(hs, mask) for hs, mask in raw_outputs]
+ logger.info("Prompt encoding complete")
+ return result
+
+
+_ENCODE_MODEL_TYPES = frozenset({"gemma3", "gemma4", "gemma4_unified"})
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/video_decoder.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/video_decoder.py
new file mode 100644
index 0000000..9127487
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/video_decoder.py
@@ -0,0 +1,224 @@
+from __future__ import annotations
+
+import copy
+import logging
+from collections.abc import Callable, Iterator, Sequence
+
+import torch
+
+from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy
+from ltx_core.loader import SDOps
+from ltx_core.loader.module_ops import ModuleOps
+from ltx_core.loader.primitives import BuilderProtocol
+from ltx_core.loader.registry import ModelRegistry, Registry
+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder
+from ltx_core.model.transformer.compiling import CompilationConfig
+from ltx_core.model.video_vae import (
+ CHANNELS_LAST_3D_WEIGHTS,
+ MEMORY_EFFICIENT_DECODE,
+ VAE_DECODER_COMFY_KEYS_FILTER,
+ TilingConfig,
+ VideoDecoderConfigurator,
+ is_diffusion_video_vae,
+ video_decoder_sd_ops_for_checkpoint,
+)
+from ltx_core.model.video_vae.keyframes import DecodeKeyframes
+from ltx_core.model.video_vae.transformer import (
+ DiffVAEMode,
+ NAttentionKind,
+ build_diffvae_mode_op,
+ na_dsl_available,
+)
+from ltx_core.multigpu.vae.distributed_decoder import DistributedVideoDecoder, OwnedSingleFrames
+from ltx_pipelines.utils.gpu_model import gpu_model
+
+logger = logging.getLogger(__name__)
+
+__all__ = ["VideoDecoder"]
+
+
+class VideoDecoder:
+ """Owns video decoder lifecycle.
+ Returns an iterator that cleans up the decoder after all chunks are consumed.
+ """
+
+ def __init__(
+ self,
+ checkpoint_path: str,
+ dtype: torch.dtype,
+ device: torch.device,
+ registry: Registry | None = None,
+ memory_efficient: bool = True,
+ decoder_builder: BuilderProtocol | None = None,
+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
+ diffvae_optimization: DiffVAEMode = DiffVAEMode.CHUNKED_EAGER,
+ ) -> None:
+ self._checkpoint_path = checkpoint_path
+ self._dtype = dtype
+ self._device = device
+ self._diffvae_optimization = diffvae_optimization
+ diffusion_vae = is_diffusion_video_vae(self._checkpoint_path)
+ self._diffusion_vae = diffusion_vae
+ cfg = diffvae_optimization.resolve() if diffusion_vae else None
+ use_dsl = cfg is not None and cfg.attention is NAttentionKind.BLACKWELL_DSL
+ if decoder_builder is not None:
+ self._decoder_builder = decoder_builder
+ else:
+ if diffusion_vae and use_dsl and not na_dsl_available():
+ raise RuntimeError(
+ "diffvae_optimization=blackwell_dsl needs a datacenter Blackwell GPU and "
+ "nvidia-cutlass-dsl (uv sync --group kernels)"
+ )
+ sd_ops = (
+ video_decoder_sd_ops_for_checkpoint(
+ self._checkpoint_path,
+ diffusion_vae=True,
+ na_dsl_kernel=use_dsl,
+ )
+ if diffusion_vae
+ else VAE_DECODER_COMFY_KEYS_FILTER
+ )
+ module_ops: tuple[ModuleOps, ...] = ()
+ # channels-last + memory-efficient decode apply to ConvVideoDecoder only
+ if memory_efficient and not diffusion_vae:
+ sd_ops = SDOps(
+ name=f"sd_ops_chain_{sd_ops.name}+{CHANNELS_LAST_3D_WEIGHTS.name}",
+ mapping=(*sd_ops.mapping, *CHANNELS_LAST_3D_WEIGHTS.mapping),
+ )
+ module_ops = (MEMORY_EFFICIENT_DECODE,)
+ self._decoder_builder = Builder(
+ model_path=self._checkpoint_path,
+ model_class_configurator=VideoDecoderConfigurator,
+ model_sd_ops=sd_ops,
+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
+ module_ops=module_ops,
+ )
+ self._alloc_trim_strategy = alloc_trim_strategy
+
+ @property
+ def checkpoint_path(self) -> str:
+ return self._checkpoint_path
+
+ @property
+ def diffvae_optimization(self) -> DiffVAEMode:
+ return self._diffvae_optimization
+
+ def with_builder(self, builder: BuilderProtocol) -> "VideoDecoder":
+ """Return a new ``VideoDecoder`` that builds from ``builder``.
+ Functional: never mutates ``self``; shares checkpoint, dtype, device, and
+ trim strategy. Mirrors :meth:`DiffusionStage.with_builder`.
+ """
+ new = copy.copy(self)
+ new._decoder_builder = builder
+ return new
+
+ def _prepared_builder(self) -> BuilderProtocol:
+ """Return the builder with DiffVAE mode ops applied at build time.
+ DiffVAE ``module_ops`` live on the decoder (not on the caller-held builder),
+ matching :meth:`DiffusionStage._prepared_builder`. That keeps
+ ``with_builder`` / Dist ``with_tiling`` swaps on the raw builder, while
+ every build still installs the configured DiffVAE policy.
+ """
+ builder = self._decoder_builder
+ if not self._diffusion_vae:
+ return builder
+ op = build_diffvae_mode_op(
+ self._diffvae_optimization,
+ CompilationConfig() if self._diffvae_optimization.resolve().compile_blocks else None,
+ )
+ return builder.with_module_ops((*builder.module_ops, op))
+
+ def __call__(
+ self,
+ latent: torch.Tensor,
+ tiling_config: TilingConfig | None = None,
+ generator: torch.Generator | None = None,
+ *,
+ dtype: torch.dtype | None = None,
+ keyframes: DecodeKeyframes | None = None,
+ device_fn: Callable[[int], str | torch.device] | None = None,
+ ) -> Iterator[torch.Tensor]:
+ """Decode *latent* to pixel-space video chunks. Decoder freed after exhaustion.
+ ``dtype`` overrides the constructor dtype for this call only (e.g. float32 for
+ HDR raw-in/raw-out). Omitted → same dtype as on construction (SDR default).
+ ``keyframes`` anchors the decode on already-encoded planes. Every decoder in this repo
+ takes it, so this never branches on which VAE was built: a diffusion VAE uses the planes,
+ a conv VAE logs that it is dropping them, and a distributed decoder splits them across
+ its ranks. The keyword is only passed when there is something to pass, so a decoder
+ written against the older signature still serves plain decodes.
+ """
+ build_dtype = self._dtype if dtype is None else dtype
+ latent = latent.to(dtype=build_dtype)
+ logger.info("Building video decoder from %s", self._checkpoint_path)
+ decoder = self._prepared_builder().build(device=self._device, dtype=build_dtype).eval()
+ extra = {} if keyframes is None else {"keyframes": keyframes}
+ if device_fn is not None:
+ extra["device_fn"] = device_fn
+ chunks = decoder.decode_video(latent, tiling_config, generator, **extra)
+ return _cleanup_iter(
+ chunks,
+ decoder,
+ alloc_trim_strategy=self._alloc_trim_strategy,
+ )
+
+ def decode_single_frames(
+ self,
+ latents: Sequence[torch.Tensor],
+ generator: torch.Generator | Sequence[torch.Generator | None] | None = None,
+ *,
+ dtype: torch.dtype | None = None,
+ ) -> Iterator[torch.Tensor]:
+ """Decode each latent as its own one-frame clip on one VAE build, then free it.
+ A causal VAE cannot decode stacked independent planes without bleeding neighbours.
+ A Dist build splits the list across its ranks and gathers the pixels back, so every
+ rank returns the whole list; it does not go through the volume-splitting
+ ``decode_video`` path, whose workers yield nothing.
+ """
+ if not latents:
+ return iter(())
+ build_dtype = self._dtype if dtype is None else dtype
+ moved = [latent.to(dtype=build_dtype) for latent in latents]
+ logger.info("Building video decoder from %s", self._checkpoint_path)
+ decoder = self._prepared_builder().build(device=self._device, dtype=build_dtype).eval()
+ chunks = decoder.decode_single_frames(moved, generator)
+ return _cleanup_iter(
+ chunks,
+ decoder,
+ alloc_trim_strategy=self._alloc_trim_strategy,
+ )
+
+ def decode_owned_single_frames(
+ self,
+ latents: Sequence[torch.Tensor],
+ generator: torch.Generator | Sequence[torch.Generator | None] | None = None,
+ *,
+ dtype: torch.dtype | None = None,
+ ) -> OwnedSingleFrames:
+ """``decode_single_frames`` without the pixel gather: only the planes this build decodes.
+ A Dist build yields its round-robin share and a collective ``gather`` every rank calls
+ once with what it derived from its planes; an SGPU build owns every plane and its
+ ``gather`` returns its argument. The decoder is freed once the planes are exhausted.
+ """
+ if not latents:
+ return OwnedSingleFrames([], iter(()), dict)
+ build_dtype = self._dtype if dtype is None else dtype
+ moved = [latent.to(dtype=build_dtype) for latent in latents]
+ logger.info("Building video decoder from %s", self._checkpoint_path)
+ decoder = self._prepared_builder().build(device=self._device, dtype=build_dtype).eval()
+ if isinstance(decoder, DistributedVideoDecoder):
+ owned = decoder.owned_single_frames(moved, generator)
+ else:
+ owned = OwnedSingleFrames(list(range(len(moved))), decoder.decode_single_frames(moved, generator), dict)
+ return owned._replace(
+ planes=_cleanup_iter(owned.planes, decoder, alloc_trim_strategy=self._alloc_trim_strategy)
+ )
+
+
+def _cleanup_iter(
+ it: Iterator[torch.Tensor],
+ model: torch.nn.Module,
+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
+) -> Iterator[torch.Tensor]:
+ """Wrap an iterator to release *model* memory once exhausted or abandoned."""
+ with gpu_model(model, alloc_trim_strategy=alloc_trim_strategy):
+ yield from it
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/video_upsampler.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/video_upsampler.py
new file mode 100644
index 0000000..f434726
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/video_upsampler.py
@@ -0,0 +1,61 @@
+from __future__ import annotations
+
+import logging
+
+import torch
+
+from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy
+from ltx_core.loader.registry import ModelRegistry, Registry
+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder
+from ltx_core.model.upsampler import LatentUpsamplerConfigurator, upsample_video
+from ltx_core.model.video_vae import VideoEncoderConfigurator
+from ltx_pipelines.utils.blocks.image_conditioner import VAE_ENCODER_SD_OPS
+from ltx_pipelines.utils.gpu_model import gpu_model
+
+logger = logging.getLogger(__name__)
+
+__all__ = ["VideoUpsampler"]
+
+
+class VideoUpsampler:
+ """Owns video encoder + spatial upsampler lifecycle."""
+
+ def __init__(
+ self,
+ checkpoint_path: str,
+ upsampler_path: str,
+ dtype: torch.dtype,
+ device: torch.device,
+ registry: Registry | None = None,
+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
+ ) -> None:
+ self._upsampler_path = upsampler_path
+ self._dtype = dtype
+ self._device = device
+ self._encoder_builder = Builder(
+ model_path=checkpoint_path,
+ model_class_configurator=VideoEncoderConfigurator,
+ model_sd_ops=VAE_ENCODER_SD_OPS,
+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
+ )
+ self._upsampler_builder = Builder(
+ model_path=upsampler_path,
+ model_class_configurator=LatentUpsamplerConfigurator,
+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),
+ )
+ self._alloc_trim_strategy = alloc_trim_strategy
+
+ def __call__(self, latent: torch.Tensor) -> torch.Tensor:
+ """Upsample *latent* using video encoder + spatial upsampler, then free both."""
+ logger.info("Building video encoder + spatial upsampler from %s", self._upsampler_path)
+ with (
+ gpu_model(
+ self._encoder_builder.build(device=self._device, dtype=self._dtype).eval(),
+ alloc_trim_strategy=self._alloc_trim_strategy,
+ ) as encoder,
+ gpu_model(
+ self._upsampler_builder.build(device=self._device, dtype=self._dtype).eval(),
+ alloc_trim_strategy=self._alloc_trim_strategy,
+ ) as upsampler,
+ ):
+ return upsample_video(latent=latent, video_encoder=encoder, upsampler=upsampler)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/denoisers.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/denoisers.py
index bc41cfd..b734612 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/denoisers.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/denoisers.py
@@ -47,7 +47,7 @@ def _repeat_state(state: LatentState, n: int) -> LatentState:
return LatentState(
latent=_repeat(state.latent),
- denoise_mask=_repeat(state.denoise_mask),
+ denoise_mask=state.denoise_mask.with_same_values(_repeat(state.denoise_mask.tensor)),
positions=_repeat(state.positions),
clean_latent=_repeat(state.clean_latent),
attention_mask=_repeat(state.attention_mask) if state.attention_mask is not None else None,
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/helpers.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/helpers.py
index bc54583..22e1d00 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/helpers.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/helpers.py
@@ -2,6 +2,7 @@ from __future__ import annotations
import logging
from collections.abc import Sequence
+from dataclasses import replace
from typing import TYPE_CHECKING
import torch
@@ -9,14 +10,12 @@ import torch
from ltx_core.components.noisers import Noiser
from ltx_core.conditioning import (
ConditioningItem,
- VideoConditionByKeyframeIndex,
- VideoConditionByLatentIndex,
VideoGeneratedKeyframeSlots,
)
from ltx_core.devices import activation_budget_bytes, cleanup_accelerator_memory, get_preferred_device
from ltx_core.loader.sft_loader import SafetensorsModelStateDictLoader
from ltx_core.model.audio_vae import encode_audio
-from ltx_core.model.transformer import Modality
+from ltx_core.model.transformer import Modality, TokenTimesteps
from ltx_core.model.video_vae import AutoTiling, TileSizeConfig, TilingConfig, VideoEncoder
from ltx_core.model.video_vae.diffusion_tiling import recommended_decode_tiling_config
from ltx_core.model.video_vae.keyframes import DecodeKeyframes
@@ -47,12 +46,11 @@ from ltx_pipelines.utils.media_io import (
get_videostream_fps,
is_exr_dir,
load_exr_as_hdr_conditioning,
- load_image_and_preprocess,
resize_aspect_ratio_preserving,
video_preprocess,
)
from ltx_pipelines.utils.media_io.color_config import EXRColorSpace
-from ltx_pipelines.utils.types import ImageConditioningInput
+from ltx_pipelines.utils.types import LoadedImageConditioning
if TYPE_CHECKING:
from ltx_pipelines.utils.blocks import DiffusionStage
@@ -352,102 +350,16 @@ def audio_latent_from_file(
return _conform_latent_length(latents, required_latent_frames)
-def combined_image_conditionings(
- images: list[ImageConditioningInput],
+def image_conditionings(
+ images: Sequence[LoadedImageConditioning],
height: int,
width: int,
video_encoder: VideoEncoder,
dtype: torch.dtype,
device: torch.device,
- color_space: EXRColorSpace | None = None,
-) -> list[ConditioningItem]:
- """Create a list of conditionings by replacing the latent at the first frame with the encoded image if present
- and using other encoded images as the keyframe conditionings."""
- conditionings = []
- for img in images:
- image = load_image_and_preprocess(
- image_path=img.path,
- height=height,
- width=width,
- dtype=dtype,
- device=device,
- crf=img.crf,
- color_space=color_space,
- )
- encoded_image = video_encoder(image)
- if img.frame_idx == 0:
- conditioning = VideoConditionByLatentIndex(
- latent=encoded_image,
- strength=img.strength,
- latent_idx=0,
- )
- else:
- conditioning = VideoConditionByKeyframeIndex(
- keyframes=encoded_image,
- strength=img.strength,
- frame_idx=img.frame_idx,
- )
- conditionings.append(conditioning)
- return conditionings
-
-
-def image_conditionings_by_replacing_latent(
- images: list[ImageConditioningInput],
- height: int,
- width: int,
- video_encoder: VideoEncoder,
- dtype: torch.dtype,
- device: torch.device,
- color_space: EXRColorSpace | None = None,
-) -> list[ConditioningItem]:
- conditionings = []
- for img in images:
- image = load_image_and_preprocess(
- image_path=img.path,
- height=height,
- width=width,
- dtype=dtype,
- device=device,
- crf=img.crf,
- color_space=color_space,
- )
- encoded_image = video_encoder(image)
- conditionings.append(
- VideoConditionByLatentIndex(
- latent=encoded_image,
- strength=img.strength,
- latent_idx=img.frame_idx,
- )
- )
-
- return conditionings
-
-
-def image_conditionings_by_adding_guiding_latent(
- images: list[ImageConditioningInput],
- height: int,
- width: int,
- video_encoder: VideoEncoder,
- dtype: torch.dtype,
- device: torch.device,
- color_space: EXRColorSpace | None = None,
) -> list[ConditioningItem]:
- conditionings = []
- for img in images:
- image = load_image_and_preprocess(
- image_path=img.path,
- height=height,
- width=width,
- dtype=dtype,
- device=device,
- crf=img.crf,
- color_space=color_space,
- )
- encoded_image = video_encoder(image)
- conditionings.append(
- VideoConditionByKeyframeIndex(keyframes=encoded_image, frame_idx=img.frame_idx, strength=img.strength)
- )
- return conditionings
+ """Encode loaded stills at ``height`` x ``width`` into conditioning items using each ``binding``."""
+ return [img.to_item(video_encoder(img.still.tensor(height, width, dtype, device))) for img in images]
def evenly_spaced_keyframe_positions(num_keyframes: int, num_frames: int) -> list[int]:
@@ -574,9 +486,8 @@ def create_noised_state(
"""
state = tools.create_initial_state(device, dtype, initial_latent)
state = state_with_conditionings(state, conditionings, tools)
- state = noiser(state, noise_scale)
-
- return state
+ state = replace(state, denoise_mask=state.denoise_mask.compact(device))
+ return noiser(state, noise_scale)
def state_with_conditionings(
@@ -612,11 +523,18 @@ def modality_from_latent_state(
"""
if state.frozen:
sigma = torch.zeros_like(sigma)
+ values = state.denoise_mask.values
+ timesteps = timesteps_from_mask(values.reshape(1, -1), sigma)
+ # Clamped because a mask value above the table's last entry would otherwise index one
+ # past it, and that read happens inside a captured CUDA graph.
+ timestep_indices = torch.searchsorted(
+ values, state.denoise_mask.tensor.reshape(state.latent.shape[0], -1), out_int32=True
+ ).clamp_(max=values.numel() - 1)
return Modality(
enabled=enabled,
latent=state.latent,
sigma=sigma,
- timesteps=timesteps_from_mask(state.denoise_mask, sigma),
+ timesteps=TokenTimesteps(timesteps, timestep_indices, state.denoise_mask.zero_index),
positions=state.positions,
context=context,
context_mask=None,
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/decode.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/decode.py
index 010248a..74e32d1 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/decode.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/decode.py
@@ -4,8 +4,11 @@ from __future__ import annotations
import logging
from collections.abc import Generator, Iterator
+from concurrent.futures import Future
+from dataclasses import dataclass
from io import BytesIO
from pathlib import Path
+from typing import TYPE_CHECKING
import av
import numpy as np
@@ -15,10 +18,11 @@ from PIL import ExifTags, Image, ImageCms, UnidentifiedImageError
from torch._prims_common import DeviceLikeType
from ltx_core.types import Audio, VideoPixelShape
-from ltx_pipelines.utils.media_io.color_config import EXRColorSpace, srgb_to_acescct
+from ltx_pipelines.utils.media_io.color_config import EXRColorSpace, srgb_to_acescct, to_working_space
from ltx_pipelines.utils.media_io.exr import (
+ _exr_paths_for_conditioning,
is_exr_dir,
- load_exr_image_as_hdr_conditioning,
+ read_exr,
)
from ltx_pipelines.utils.media_io.range_map import normalize_images, to_vae_range
from ltx_pipelines.utils.media_io.resize import (
@@ -27,6 +31,9 @@ from ltx_pipelines.utils.media_io.resize import (
resize_and_reflect_pad,
)
+if TYPE_CHECKING:
+ from ltx_pipelines.utils.types import DecodedStill
+
logger = logging.getLogger(__name__)
_ORIENTATION_EXIF_KEY = next(key for key, value in ExifTags.TAGS.items() if value == "Orientation")
@@ -45,6 +52,9 @@ _INT_FORMAT_MAX: dict[str, float] = {
}
+_EXR_NEEDS_COLOR_SPACE = "EXR input requires --hdr {SRGB_LINEAR,ACESCG,ACESCCT} to declare the source colour space."
+
+
def load_image_and_preprocess(
image_path: str,
height: int,
@@ -60,25 +70,70 @@ def load_image_and_preprocess(
An ``.exr`` still requires ``color_space`` (from ``--hdr``) and is loaded via the
HDR conditioning path instead of the SDR decode/normalize path.
"""
+ return load_conditioning_still(image_path, crf, color_space).tensor(height, width, dtype, device)
+
+
+def load_conditioning_still(image_path: str, crf: int, color_space: EXRColorSpace | None = None) -> DecodedStill:
+ """Decode an image conditioning's still once, for every resolution a run encodes it at.
+ An SDR image is decoded and re-compressed at ``crf``; an ``.exr`` still is read as linear HDR and
+ needs ``color_space`` (from ``--hdr``). A missing or undecodable file of either kind raises
+ ``ValueError("Cannot decode image file '<path>'.")``.
+ """
if str(image_path).lower().endswith(".exr"):
if color_space is None:
- raise ValueError(
- "EXR input requires --hdr {SRGB_LINEAR,ACESCG,ACESCCT} to declare the source colour space."
- )
- return load_exr_image_as_hdr_conditioning(
- exr_path=image_path,
- height=height,
- width=width,
- dtype=dtype,
- device=device,
- color_space=color_space,
- )
- image = decode_image(image_path=image_path)
- image = preprocess(image=image, crf=crf)
- image = torch.tensor(image, dtype=torch.float32, device=device)
- image = resize_and_center_crop(image, height, width)
- image = normalize_images(image, device, dtype)
- return image
+ raise ValueError(_EXR_NEEDS_COLOR_SPACE)
+ try:
+ pixels = read_exr(_exr_paths_for_conditioning(image_path)[0])
+ except (OSError, RuntimeError, ValueError) as err:
+ raise ValueError(f"Cannot decode image file '{image_path}'.") from err
+ return ExrStill(pixels, color_space)
+ image = preprocess(image=decode_image(image_path=image_path), crf=crf)
+ image.setflags(write=False)
+ return SdrStill(image)
+
+
+def check_conditioning_still(image_path: str, color_space: EXRColorSpace | None = None) -> None:
+ """Raise what :func:`load_conditioning_still` would for a missing file or an EXR without ``color_space``.
+ Cheap enough for a pipeline's prologue when the decode itself runs elsewhere.
+ """
+ if str(image_path).lower().endswith(".exr") and color_space is None:
+ raise ValueError(_EXR_NEEDS_COLOR_SPACE)
+ if not Path(image_path).exists():
+ raise ValueError(f"Cannot decode image file '{image_path}'.")
+
+
+@dataclass(frozen=True, eq=False)
+class BackgroundStill:
+ """A still decoding on a worker thread; :meth:`tensor` waits for it and re-raises its decode error."""
+
+ future: Future[DecodedStill]
+
+ def tensor(self, height: int, width: int, dtype: torch.dtype, device: torch.device) -> torch.Tensor:
+ return self.future.result().tensor(height, width, dtype, device)
+
+
+@dataclass(frozen=True, eq=False)
+class SdrStill:
+ """A decoded SDR still, re-compressed at its conditioning CRF: read-only ``(H, W, 3)`` uint8."""
+
+ pixels: np.ndarray
+
+ def tensor(self, height: int, width: int, dtype: torch.dtype, device: torch.device) -> torch.Tensor:
+ image = torch.tensor(self.pixels, dtype=torch.float32, device=device)
+ image = resize_and_center_crop(image, height, width)
+ return normalize_images(image, device, dtype)
+
+
+@dataclass(frozen=True, eq=False)
+class ExrStill:
+ """A linear HDR still as read from EXR, ``(H, W, 3)`` float, mapped to the VAE working space per resolution."""
+
+ pixels: torch.Tensor
+ color_space: EXRColorSpace
+
+ def tensor(self, height: int, width: int, dtype: torch.dtype, device: torch.device) -> torch.Tensor:
+ frame = resize_and_center_crop(self.pixels.to(device=device, dtype=torch.float32), height, width)
+ return to_vae_range(to_working_space(frame, self.color_space)).to(device=device, dtype=dtype)
def video_preprocess(
@@ -416,12 +471,12 @@ def preprocess(image: np.array, crf: int) -> np.array:
``crf`` is required: the correct value is a property of the model generation, so a
code-level default here would silently condition on the wrong compression. Reaching this
with ``crf=None`` means a conditioning skipped resolution -- see
- ``ImageConditioner.resolve_crf``, which every pipeline runs over its image inputs.
+ ``ImageConditioner.load_conditionings``, which every pipeline runs over its image inputs.
"""
if crf is None:
raise ValueError(
"Image conditioning CRF is unresolved (crf=None). Resolve it against the checkpoint "
- "first -- ImageConditioner.resolve_crf(images), or detect_params(checkpoint_path).default_image_crf."
+ "first -- ImageConditioner.load_conditionings(images), or detect_params(checkpoint_path).default_image_crf."
)
if crf == 0:
return image
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/encode.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/encode.py
index bd70b14..5b25e66 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/encode.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/encode.py
@@ -23,6 +23,7 @@ from ltx_core.color.hlg import encode_linear_hdr_frames_to_hlg_mp4
from ltx_core.color.primaries import Primaries
from ltx_core.color.yuv import FrameConverter, PixelFormat, yuv420p_bt709_converter_
from ltx_core.types import Audio
+from ltx_pipelines.marking.types import AudioMarker, VideoMarker
from ltx_pipelines.utils.media_io.color_config import EXRColorSpace, decode_hdr_video
from ltx_pipelines.utils.media_io.exr import exr_colorspace_tag, exr_dir_label, save_exr_tensor
@@ -130,6 +131,35 @@ def _encode_hdr_video_outputs(
return exr_dir
+# Frames converted and handed to the encoder thread at a time: x264 starts on the first slice
+# while the rest are still being converted and copied off the GPU.
+_HOST_SLICE_FRAMES = 8
+
+# The encoder's conversion stream, one per device for the life of the process. cuBLAS keeps a workspace per
+# (handle, stream) until the process exits, so every new stream a GEMM runs on stays charged to the allocator.
+_CONVERT_STREAMS: dict[torch.device, torch.cuda.Stream] = {}
+
+
+def _convert_stream(device: torch.device) -> torch.cuda.Stream:
+ stream = _CONVERT_STREAMS.get(device)
+ if stream is None:
+ stream = _CONVERT_STREAMS[device] = torch.cuda.Stream(device=device)
+ return stream
+
+
+def _to_host(frames: torch.Tensor) -> np.ndarray:
+ """Copy converted frames to the host through a pinned buffer.
+ Pinned buffers come from torch's caching host allocator, so repeated encodes reuse them
+ instead of page-faulting a fresh pageable allocation per clip. The array keeps its buffer
+ alive until the encoder thread drops it.
+ """
+ if frames.device.type != "cuda":
+ return frames.to("cpu").numpy()
+ host = torch.empty(frames.shape, dtype=frames.dtype, pin_memory=True)
+ host.copy_(frames)
+ return host.numpy()
+
+
def _open_video_stream(
container: av.container.Container,
fps: int,
@@ -153,7 +183,7 @@ def _open_video_stream(
return stream
-def encode_sdr_h264(
+def encode_sdr_h264( # noqa: PLR0913
video: Iterator[torch.Tensor],
*,
fps: int,
@@ -165,26 +195,41 @@ def encode_sdr_h264(
crf: int = 19,
preset: str = "veryfast",
thread_count: int = 0,
+ chunks_complete: bool = False,
+ video_marker: VideoMarker | None = None,
) -> None:
"""Mux RGB frame chunks to SDR H.264 (optional AAC).
Pass :func:`~ltx_pipelines.chunks.deferred_stitch_audio` (and ``audio_sampling_rate``)
when audio is assembled while ``video`` is consumed. Pull the iterator only after
- video drain.
+ video drain. ``chunks_complete`` says each CUDA chunk is finished on every stream when
+ ``video`` yields it (see :func:`encode_video`). ``video_marker`` stamps each slice on the
+ stream that converts it.
"""
- def convert(chunk: torch.Tensor) -> torch.Tensor:
- return frame_converter(chunk.movedim(-1, -3))
+ def converted(piece: torch.Tensor) -> np.ndarray:
+ if video_marker is not None:
+ piece = video_marker.embed(piece)
+ return _to_host(frame_converter(piece.movedim(-1, -3)))
+
+ def host_frames(piece: torch.Tensor) -> np.ndarray:
+ """Stamp, convert and copy ``piece`` to the host; on the encoder's own stream for complete chunks."""
+ if not (chunks_complete and piece.is_cuda):
+ return converted(piece)
+ with torch.cuda.stream(_convert_stream(piece.device)):
+ return converted(piece)
first_raw_chunk = next(video, None)
if first_raw_chunk is None:
raise ValueError("video is empty; expected at least one frame chunk.")
- first_chunk = convert(first_raw_chunk)
+ # Converted a slice at a time, so the converter's float temporaries stay slice-sized.
+ first_slices = iter(first_raw_chunk.split(_HOST_SLICE_FRAMES))
+ first_slice = host_frames(next(first_slices))
if frame_converter.pixel_format == PixelFormat.RGB24:
- height, width = first_chunk.shape[-3], first_chunk.shape[-2]
+ height, width = first_slice.shape[-3], first_slice.shape[-2]
else:
- height = first_chunk.shape[-2] * 2 // 3
- width = first_chunk.shape[-1]
+ height = first_slice.shape[-2] * 2 // 3
+ width = first_slice.shape[-1]
Path(output_path).parent.mkdir(parents=True, exist_ok=True)
container = av.open(output_path, mode="w")
@@ -199,16 +244,20 @@ def encode_sdr_h264(
av_format = frame_converter.pixel_format.av_format
def cpu_chunks() -> Generator[np.ndarray, None, None]:
- yield first_chunk.to("cpu").numpy()
+ progress = tqdm(total=video_chunks_number)
+ yield first_slice
+ yield from (host_frames(piece) for piece in first_slices)
+ progress.update()
for chunk in video:
- yield convert(chunk).to("cpu").numpy()
+ yield from (host_frames(piece) for piece in chunk.split(_HOST_SLICE_FRAMES))
+ progress.update()
+ progress.close()
_encode_chunks_threaded(
container=container,
stream=stream,
av_format=av_format,
chunks=cpu_chunks(),
- progress_total=video_chunks_number,
)
mux_audio = next(audio, None) if audio is not None else None
@@ -237,6 +286,9 @@ def encode_video( # noqa: PLR0913
*,
color_space: EXRColorSpace | None = None,
audio_sampling_rate: int | None = None,
+ video_marker: VideoMarker | None = None,
+ audio_marker: AudioMarker | None = None,
+ chunks_complete: bool = False,
) -> Path | None:
"""Encode RGB frames to video; SDR H.264 or HDR EXR+HLG when ``color_space`` is set.
Args:
@@ -263,11 +315,21 @@ def encode_video( # noqa: PLR0913
When set, writes half EXR frames + HLG master.
audio_sampling_rate: AAC sample rate. Inferred when ``audio`` is an
:class:`~ltx_core.types.Audio`; required for an audio iterator.
+ video_marker: Optional pixel marker. ``None`` leaves pixels unchanged.
+ HDR encode must not pass one.
+ audio_marker: Optional audio marker. ``None`` leaves PCM unchanged.
+ Each yielded ``Audio`` is stamped before AAC mux (SDR and HDR).
+ chunks_complete: Each CUDA chunk ``video`` yields is already finished on every stream
+ (e.g. the distributed decoder's batches). SDR conversion and host copies then run
+ on a stream of their own, whose syncs do not wait for whatever else the device is
+ still running -- such as the rest of a streaming decode. Leave ``False`` when a
+ chunk may still be in flight on the current stream.
Returns:
EXR output directory when HDR, else ``None``.
Raises:
ValueError: On an empty ``video``, a non-stereo ``audio`` waveform, or conflicting
- explicit and embedded audio sampling rates.
+ explicit and embedded audio sampling rates. Also if ``video_marker`` is set
+ together with ``color_space``.
"""
if isinstance(audio, Audio):
validate_audio_waveform(audio)
@@ -281,38 +343,56 @@ def encode_video( # noqa: PLR0913
if isinstance(video, torch.Tensor):
video = iter([video])
+ if video_marker is not None and color_space is not None:
+ raise ValueError("video_marker is not supported on HDR encode")
+
+ marked_audio = _mark_audio(audio, audio_marker) if audio_marker is not None and audio is not None else audio
if color_space is not None:
return _encode_hdr_video_outputs(
video,
Path(output_path),
fps=float(fps),
color_space=color_space,
- audio=audio,
+ audio=marked_audio,
audio_sampling_rate=audio_sampling_rate,
)
-
encode_sdr_h264(
video,
fps=fps,
output_path=output_path,
video_chunks_number=video_chunks_number,
- audio=audio,
+ audio=marked_audio,
audio_sampling_rate=audio_sampling_rate,
frame_converter=frame_converter,
crf=crf,
preset=preset,
thread_count=thread_count,
+ chunks_complete=chunks_complete,
+ video_marker=video_marker,
)
return None
-def encode_audio(audio: Audio, output_path: str) -> None:
+def _mark_audio(audio: Iterator[Audio], marker: AudioMarker) -> Iterator[Audio]:
+ """Stamp each yielded ``Audio`` independently. Never stitches tracks."""
+ for track in audio:
+ yield marker.embed(track)
+
+
+def encode_audio(
+ audio: Audio,
+ output_path: str,
+ *,
+ audio_marker: AudioMarker | None = None,
+) -> None:
"""Save an audio waveform as a 16-bit PCM ``.wav`` file at the source sampling rate.
Reuses :func:`_write_audio` (the same muxing path used by :func:`encode_video`);
the only difference is a PCM (``pcm_s16le``) stream in a WAV container instead of
the AAC stream used for muxed video.
"""
validate_audio_waveform(audio)
+ if audio_marker is not None:
+ audio = audio_marker.embed(audio)
Path(output_path).parent.mkdir(parents=True, exist_ok=True)
container = av.open(output_path, mode="w")
audio_stream = container.add_stream("pcm_s16le", rate=audio.sampling_rate)
@@ -331,7 +411,6 @@ def _encode_chunks_threaded(
stream: av.video.stream.VideoStream,
av_format: str,
chunks: Iterator[np.ndarray],
- progress_total: int,
) -> None:
"""Run libx264 frame.encode + container.mux on a background thread while
the caller produces numpy chunks on the current thread. The 1-slot queue
@@ -369,7 +448,7 @@ def _encode_chunks_threaded(
encoder_thread = threading.Thread(target=encoder_worker, name="h264-encoder")
encoder_thread.start()
try:
- for arr in tqdm(chunks, total=progress_total):
+ for arr in chunks:
chunk_queue.put(arr)
finally:
chunk_queue.put(None)
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/range_map.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/range_map.py
index d6d1ded..31dc392 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/range_map.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/range_map.py
@@ -12,10 +12,14 @@ def normalize_images(images: torch.Tensor, device: torch.device, dtype: torch.dt
def to_vae_range(x: torch.Tensor) -> torch.Tensor:
"""Map [0, 1] working-space codes to [-1, 1] (VAE input convention).
Raises:
- ValueError: If ``x`` is outside ``[0, 1]`` (callers must compress or
- already hold valid working-space codes — this does not clamp).
+ ValueError: If a host ``x`` is outside ``[0, 1]`` (callers must compress or
+ already hold valid working-space codes — this does not clamp). A CUDA ``x`` is
+ checked asynchronously: a violation fails the stream with a device-side assert.
"""
- if bool((x < 0).any() or (x > 1).any()):
+ if x.is_cuda:
+ # Checked on the device: reading the result back would wait for every queued kernel.
+ torch._assert_async(((x >= 0) & (x <= 1)).all(), "to_vae_range expects values in [0, 1]")
+ elif bool((x < 0).any() or (x > 1).any()):
raise ValueError(f"to_vae_range expects values in [0, 1]; got min={float(x.min())}, max={float(x.max())}")
return x * 2.0 - 1.0
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/samplers.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/samplers.py
index 2130b55..b1e1ac7 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/samplers.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/samplers.py
@@ -15,6 +15,7 @@ from ltx_core.components.diffusion_steps import (
from ltx_core.components.protocols import DiffusionStepProtocol
from ltx_core.devices import highest_precision_float
from ltx_core.model.transformer import X0Model
+from ltx_core.types import HostDeviceTensorPair
from ltx_core.utils import to_denoised, to_velocity
from ltx_pipelines.utils.helpers import post_process_latent, timesteps_from_mask
from ltx_pipelines.utils.res2s import get_res2s_coefficients
@@ -23,6 +24,15 @@ from ltx_pipelines.utils.types import DenoisedLatentResult, Denoiser, LatentStat
logger = logging.getLogger(__name__)
+def _present_state(video_state: LatentState | None, audio_state: LatentState | None) -> LatentState:
+ """The video state, else the audio state; a loop needs at least one."""
+ if video_state is not None:
+ return video_state
+ if audio_state is not None:
+ return audio_state
+ raise ValueError("At least one of video_state or audio_state must be provided")
+
+
def _step_state(
state: LatentState | None,
denoised: torch.Tensor | None,
@@ -33,7 +43,7 @@ def _step_state(
"""Advance one diffusion step for a single modality, or return ``None`` if absent."""
if state is None or denoised is None:
return state
- denoised = post_process_latent(denoised, state.denoise_mask, state.clean_latent)
+ denoised = post_process_latent(denoised, state.denoise_mask.tensor, state.clean_latent)
return replace(state, latent=stepper.step(state.latent, denoised, sigmas, step_idx))
@@ -69,13 +79,14 @@ def euler_denoising_loop(
Final video and audio :class:`LatentState` after the denoising loop.
"""
stepper = EulerDiffusionStep()
- for step_idx, _ in enumerate(tqdm(sigmas[:-1])):
- video_result, audio_result = denoiser(transformer, video_state, audio_state, sigmas, step_idx)
+ schedule = HostDeviceTensorPair.on_device(sigmas, _present_state(video_state, audio_state).latent.device)
+ for step_idx in tqdm(range(len(schedule.device_tensor) - 1)):
+ video_result, audio_result = denoiser(transformer, video_state, audio_state, schedule.device_tensor, step_idx)
denoised_video = video_result.denoised if video_result is not None else None
denoised_audio = audio_result.denoised if audio_result is not None else None
- video_state = _step_state(video_state, denoised_video, stepper, sigmas, step_idx)
- audio_state = _step_state(audio_state, denoised_audio, stepper, sigmas, step_idx)
+ video_state = _step_state(video_state, denoised_video, stepper, schedule.host_tensor, step_idx)
+ audio_state = _step_state(audio_state, denoised_audio, stepper, schedule.host_tensor, step_idx)
return VideoAudio(video_state, audio_state)
@@ -104,6 +115,7 @@ def gradient_estimating_euler_denoising_loop(
stepper = EulerDiffusionStep()
previous_audio_velocity = None
previous_video_velocity = None
+ schedule = HostDeviceTensorPair.on_device(sigmas, _present_state(video_state, audio_state).latent.device)
def update_velocity_and_sample(
noisy_sample: torch.Tensor, denoised_sample: torch.Tensor, sigma: float, previous_velocity: torch.Tensor | None
@@ -115,17 +127,21 @@ def gradient_estimating_euler_denoising_loop(
denoised_sample = to_denoised(noisy_sample, total_velocity, sigma)
return current_velocity, denoised_sample
- for step_idx, _ in enumerate(tqdm(sigmas[:-1])):
- video_result, audio_result = denoiser(transformer, video_state, audio_state, sigmas, step_idx)
+ for step_idx in tqdm(range(len(schedule.device_tensor) - 1)):
+ video_result, audio_result = denoiser(transformer, video_state, audio_state, schedule.device_tensor, step_idx)
denoised_video = video_result.denoised if video_result is not None else None
denoised_audio = audio_result.denoised if audio_result is not None else None
if video_state is not None and denoised_video is not None:
- denoised_video = post_process_latent(denoised_video, video_state.denoise_mask, video_state.clean_latent)
+ denoised_video = post_process_latent(
+ denoised_video, video_state.denoise_mask.tensor, video_state.clean_latent
+ )
if audio_state is not None and denoised_audio is not None:
- denoised_audio = post_process_latent(denoised_audio, audio_state.denoise_mask, audio_state.clean_latent)
+ denoised_audio = post_process_latent(
+ denoised_audio, audio_state.denoise_mask.tensor, audio_state.clean_latent
+ )
- if sigmas[step_idx + 1] == 0:
+ if schedule.host_tensor[step_idx + 1] == 0:
if video_state is not None and denoised_video is not None:
video_state = replace(video_state, latent=denoised_video)
if audio_state is not None and denoised_audio is not None:
@@ -134,18 +150,20 @@ def gradient_estimating_euler_denoising_loop(
if video_state is not None and denoised_video is not None:
previous_video_velocity, denoised_video = update_velocity_and_sample(
- video_state.latent, denoised_video, sigmas[step_idx], previous_video_velocity
+ video_state.latent, denoised_video, schedule.host_tensor[step_idx], previous_video_velocity
)
video_state = replace(
- video_state, latent=stepper.step(video_state.latent, denoised_video, sigmas, step_idx)
+ video_state,
+ latent=stepper.step(video_state.latent, denoised_video, schedule.host_tensor, step_idx),
)
if audio_state is not None and denoised_audio is not None:
previous_audio_velocity, denoised_audio = update_velocity_and_sample(
- audio_state.latent, denoised_audio, sigmas[step_idx], previous_audio_velocity
+ audio_state.latent, denoised_audio, schedule.host_tensor[step_idx], previous_audio_velocity
)
audio_state = replace(
- audio_state, latent=stepper.step(audio_state.latent, denoised_audio, sigmas, step_idx)
+ audio_state,
+ latent=stepper.step(audio_state.latent, denoised_audio, schedule.host_tensor, step_idx),
)
return VideoAudio(video_state, audio_state)
@@ -182,11 +200,11 @@ def _inject_sde_noise(
eta: float = 0.5,
) -> torch.Tensor:
sigmas_copy = sigmas.clone()
- hp = highest_precision_float(state.denoise_mask.device)
+ hp = highest_precision_float(state.denoise_mask.tensor.device)
new_noise = new_noise_fn(state.latent, step_noise_generator)
if not legacy_mode:
- timesteps = timesteps_from_mask(state.denoise_mask.to(hp), sigmas_copy[step_idx].to(hp))
- next_timesteps = timesteps_from_mask(state.denoise_mask.to(hp), sigmas_copy[step_idx + 1].to(hp))
+ timesteps = timesteps_from_mask(state.denoise_mask.tensor.to(hp), sigmas_copy[step_idx].to(hp))
+ next_timesteps = timesteps_from_mask(state.denoise_mask.tensor.to(hp), sigmas_copy[step_idx + 1].to(hp))
sigmas = torch.stack([timesteps, next_timesteps])
step_idx = 0
x_next = stepper.step(
@@ -199,7 +217,7 @@ def _inject_sde_noise(
)
if legacy_mode:
- x_next = post_process_latent(x_next, state.denoise_mask, state.clean_latent)
+ x_next = post_process_latent(x_next, state.denoise_mask.tensor, state.clean_latent)
return x_next
@@ -273,19 +291,24 @@ def res2s_audio_video_denoising_loop( # noqa: PLR0913,PLR0915,PLR0912
substep_noise_injecting_fn = partial(sde_noise_injecting_fn, step_noise_generator=substep_noise_generator, eta=0.5)
n_full_steps = len(sigmas) - 1
+ host_sigmas = sigmas.cpu()
+ terminal_sigma = bool(host_sigmas[-1] == 0)
# inject minimal sigma value to avoid division by zero
- if sigmas[-1] == 0:
+ if terminal_sigma:
sigmas = torch.cat([sigmas[:-1], torch.tensor([0.0011, 0.0], device=sigmas.device)], dim=0)
+ host_sigmas = sigmas.cpu()
+ schedule = HostDeviceTensorPair(host_tensor=host_sigmas, device_tensor=sigmas)
# Compute step sizes in hyperbolic space
- hs = -torch.log(sigmas[1:].to(hp).cpu() / (sigmas[:-1].to(hp).cpu()))
+ host_sigmas_hp = schedule.host_tensor.to(hp)
+ hs = -torch.log(host_sigmas_hp[1:] / host_sigmas_hp[:-1])
# Initialize phi cache for reuse across loop iterations
phi_cache = {}
c2 = 0.5 # Midpoint for res_2s
for step_idx in tqdm(range(n_full_steps)):
- sigma = sigmas[step_idx].to(hp)
- sigma_next = sigmas[step_idx + 1].to(hp)
+ sigma = schedule.device_tensor[step_idx].to(hp)
+ sigma_next = schedule.device_tensor[step_idx + 1].to(hp)
# Initialize anchor point
x_anchor_video = video_state.latent.clone().to(hp) if video_state is not None else None
@@ -294,13 +317,17 @@ def res2s_audio_video_denoising_loop( # noqa: PLR0913,PLR0915,PLR0912
# ====================================================================
# STAGE 1: Evaluate at current point
# ====================================================================
- video_result, audio_result = denoiser(transformer, video_state, audio_state, sigmas, step_idx)
+ video_result, audio_result = denoiser(transformer, video_state, audio_state, schedule.device_tensor, step_idx)
denoised_video_1 = video_result.denoised if video_result is not None else None
denoised_audio_1 = audio_result.denoised if audio_result is not None else None
if video_state is not None and denoised_video_1 is not None:
- denoised_video_1 = post_process_latent(denoised_video_1, video_state.denoise_mask, video_state.clean_latent)
+ denoised_video_1 = post_process_latent(
+ denoised_video_1, video_state.denoise_mask.tensor, video_state.clean_latent
+ )
if audio_state is not None and denoised_audio_1 is not None:
- denoised_audio_1 = post_process_latent(denoised_audio_1, audio_state.denoise_mask, audio_state.clean_latent)
+ denoised_audio_1 = post_process_latent(
+ denoised_audio_1, audio_state.denoise_mask.tensor, audio_state.clean_latent
+ )
h = hs[step_idx].item()
@@ -350,7 +377,7 @@ def res2s_audio_video_denoising_loop( # noqa: PLR0913,PLR0915,PLR0912
# ====================================================================
# ITERATIVE REFINEMENT (Bong Iteration)
# ====================================================================
- if bongmath and h < 0.5 and sigma > 0.03:
+ if bongmath and h < 0.5 and schedule.host_tensor[step_idx] > 0.03:
for _ in range(bongmath_max_iter):
if x_mid_video is not None and eps_1_video is not None:
x_anchor_video = x_mid_video - h * a21 * eps_1_video
@@ -377,15 +404,19 @@ def res2s_audio_video_denoising_loop( # noqa: PLR0913,PLR0915,PLR0912
transformer,
video_state=mid_video_state,
audio_state=mid_audio_state,
- sigmas=torch.stack([sub_sigma]).to(sigmas.device),
+ sigmas=torch.stack([sub_sigma]).to(schedule.device_tensor.device),
step_index=0,
)
denoised_video_2 = video_result_2.denoised if video_result_2 is not None else None
denoised_audio_2 = audio_result_2.denoised if audio_result_2 is not None else None
if video_state is not None and denoised_video_2 is not None:
- denoised_video_2 = post_process_latent(denoised_video_2, video_state.denoise_mask, video_state.clean_latent)
+ denoised_video_2 = post_process_latent(
+ denoised_video_2, video_state.denoise_mask.tensor, video_state.clean_latent
+ )
if audio_state is not None and denoised_audio_2 is not None:
- denoised_audio_2 = post_process_latent(denoised_audio_2, audio_state.denoise_mask, audio_state.clean_latent)
+ denoised_audio_2 = post_process_latent(
+ denoised_audio_2, audio_state.denoise_mask.tensor, audio_state.clean_latent
+ )
# ====================================================================
# FINAL COMBINATION: Compute x_next using RK coefficients
@@ -410,7 +441,7 @@ def res2s_audio_video_denoising_loop( # noqa: PLR0913,PLR0915,PLR0912
state=video_state,
sample=x_anchor_video,
denoised_sample=x_next_video,
- sigmas=sigmas,
+ sigmas=schedule.device_tensor,
step_idx=step_idx,
)
if x_next_audio is not None and audio_state is not None:
@@ -418,7 +449,7 @@ def res2s_audio_video_denoising_loop( # noqa: PLR0913,PLR0915,PLR0912
state=audio_state,
sample=x_anchor_audio,
denoised_sample=x_next_audio,
- sigmas=sigmas,
+ sigmas=schedule.device_tensor,
step_idx=step_idx,
)
@@ -429,15 +460,21 @@ def res2s_audio_video_denoising_loop( # noqa: PLR0913,PLR0915,PLR0912
audio_state = replace(audio_state, latent=x_next_audio.to(model_dtype))
# Final step if we need to fully remove the noise
- if sigmas[-1] == 0:
- video_result_final, audio_result_final = denoiser(transformer, video_state, audio_state, sigmas, n_full_steps)
+ if terminal_sigma:
+ video_result_final, audio_result_final = denoiser(
+ transformer, video_state, audio_state, schedule.device_tensor, n_full_steps
+ )
denoised_video_1 = video_result_final.denoised if video_result_final is not None else None
denoised_audio_1 = audio_result_final.denoised if audio_result_final is not None else None
if video_state is not None and denoised_video_1 is not None:
- denoised_video_1 = post_process_latent(denoised_video_1, video_state.denoise_mask, video_state.clean_latent)
+ denoised_video_1 = post_process_latent(
+ denoised_video_1, video_state.denoise_mask.tensor, video_state.clean_latent
+ )
video_state = replace(video_state, latent=denoised_video_1.to(model_dtype))
if audio_state is not None and denoised_audio_1 is not None:
- denoised_audio_1 = post_process_latent(denoised_audio_1, audio_state.denoise_mask, audio_state.clean_latent)
+ denoised_audio_1 = post_process_latent(
+ denoised_audio_1, audio_state.denoise_mask.tensor, audio_state.clean_latent
+ )
audio_state = replace(audio_state, latent=denoised_audio_1.to(model_dtype))
return VideoAudio(video_state, audio_state)
@@ -467,17 +504,18 @@ class _ModalityStep:
modality: str,
state: LatentState,
result: DenoisedLatentResult | None,
- make_step_fn: Callable[[DenoisedLatentResult | None, str], Callable[..., torch.Tensor]],
+ schedule: HostDeviceTensorPair,
+ make_step_fn: Callable[[DenoisedLatentResult | None, str, HostDeviceTensorPair], Callable[..., torch.Tensor]],
) -> "_ModalityStep":
"""Resolve one modality's step inputs from a denoiser result.
``make_step_fn`` is invoked for every present modality, including one
whose result carries no denoised tensor, so that any validation it
performs fires in a stable order regardless of the denoiser's output.
"""
- step_fn = make_step_fn(result, modality)
+ step_fn = make_step_fn(result, modality, schedule)
denoised = None
if result is not None and result.denoised is not None:
- denoised = post_process_latent(result.denoised.float(), state.denoise_mask, state.clean_latent)
+ denoised = post_process_latent(result.denoised.float(), state.denoise_mask.tensor, state.clean_latent)
return cls(modality=modality, state=state, step_fn=step_fn, denoised=denoised)
@@ -491,7 +529,7 @@ def _ancestral_euler_denoising_loop(
new_noise_fn: Callable[[torch.Tensor, torch.Generator], torch.Tensor],
model_dtype: torch.dtype,
draw_noise: bool,
- make_step_fn: Callable[[DenoisedLatentResult | None, str], Callable[..., torch.Tensor]],
+ make_step_fn: Callable[[DenoisedLatentResult | None, str, HostDeviceTensorPair], Callable[..., torch.Tensor]],
) -> VideoAudio[LatentState]:
"""
Shared driver for the ancestral (noise-injecting) Euler denoising loops.
@@ -505,10 +543,10 @@ def _ancestral_euler_denoising_loop(
Whether to draw a noise tensor per modality per step. When ``False``,
``noise=None`` is passed to the stepper.
make_step_fn:
- ``(result, modality) -> step_fn`` factory returning the step callable
+ ``(result, modality, schedule) -> step_fn`` factory returning the step callable
for that modality, with any stepper-specific inputs already bound.
``step_fn`` is invoked as
- ``step_fn(sample=, denoised_sample=, sigmas=, step_index=, noise=)``.
+ ``step_fn(sample=, denoised_sample=, step_index=, noise=)``.
Called once per present modality per step right after the denoiser
returns (before the terminal-sigma check), so it can also validate the
denoiser output. ``modality`` is ``"video"`` or ``"audio"``.
@@ -516,17 +554,17 @@ def _ancestral_euler_denoising_loop(
VideoAudio[LatentState]
Final ``(video_state, audio_state)`` after the denoising loop.
"""
- if video_state is None and audio_state is None:
- raise ValueError("At least one of video_state or audio_state must be provided")
-
- present_state = video_state if video_state is not None else audio_state
+ present_state = _present_state(video_state, audio_state)
generator = torch.Generator(device=present_state.latent.device).manual_seed(noise_seed)
states: dict[str, LatentState | None] = {"video": video_state, "audio": audio_state}
+ schedule = HostDeviceTensorPair.on_device(sigmas, present_state.latent.device)
- for step_idx, _ in enumerate(tqdm(sigmas[:-1])):
- video_result, audio_result = denoiser(transformer, states["video"], states["audio"], sigmas, step_idx)
+ for step_idx in tqdm(range(len(schedule.device_tensor) - 1)):
+ video_result, audio_result = denoiser(
+ transformer, states["video"], states["audio"], schedule.device_tensor, step_idx
+ )
pending = [
- _ModalityStep.from_modality_result(modality, state, result, make_step_fn)
+ _ModalityStep.from_modality_result(modality, state, result, schedule, make_step_fn)
for modality, state, result in (
("video", states["video"], video_result),
("audio", states["audio"], audio_result),
@@ -534,7 +572,7 @@ def _ancestral_euler_denoising_loop(
if state is not None
]
- terminal_step = sigmas[step_idx + 1] == 0
+ terminal_step = bool(schedule.host_tensor[step_idx + 1] == 0)
for step in pending:
if step.denoised is None:
continue
@@ -545,12 +583,11 @@ def _ancestral_euler_denoising_loop(
x_next = step.step_fn(
sample=step.state.latent.float(),
denoised_sample=step.denoised,
- sigmas=sigmas,
step_index=step_idx,
noise=noise,
)
if draw_noise:
- x_next = post_process_latent(x_next, step.state.denoise_mask, step.state.clean_latent)
+ x_next = post_process_latent(x_next, step.state.denoise_mask.tensor, step.state.clean_latent)
states[step.modality] = replace(step.state, latent=x_next.to(model_dtype))
if terminal_step:
@@ -625,7 +662,9 @@ def euler_ancestral_denoising_loop(
# Gated on eta alone: at s_noise=0 the step still applies its variance-preserving
# rescale, so it must not fall back to the noise-free branch.
draw_noise=eta > 0,
- make_step_fn=lambda _result, _modality: partial(stepper.step, eta=eta, s_noise=s_noise),
+ make_step_fn=lambda _result, _modality, schedule: partial(
+ stepper.step, sigmas=schedule.host_tensor, eta=eta, s_noise=s_noise
+ ),
)
@@ -682,7 +721,9 @@ def euler_cfg_pp_denoising_loop(
"""
stepper = EulerCfgPpDiffusionStep()
- def make_step_fn(result: DenoisedLatentResult | None, modality: str) -> Callable[..., torch.Tensor]:
+ def make_step_fn(
+ result: DenoisedLatentResult | None, modality: str, schedule: HostDeviceTensorPair
+ ) -> Callable[..., torch.Tensor]:
"""Bind the modality's uncond prediction and the loop's noise scales into the CFG++ step."""
uncond = result.uncond if result is not None else None
if not isinstance(uncond, torch.Tensor):
@@ -691,7 +732,13 @@ def euler_cfg_pp_denoising_loop(
"Use GuidedDenoiser or FactoryGuidedDenoiser with cfg_scale != 1 "
"or force_uncond_pass=True and a negative_context."
)
- return partial(stepper.step, uncond_denoised=uncond, eta=eta, s_noise=s_noise)
+ return partial(
+ stepper.step,
+ sigmas=schedule.device_tensor,
+ uncond_denoised=uncond,
+ eta=eta,
+ s_noise=s_noise,
+ )
return _ancestral_euler_denoising_loop(
sigmas=sigmas,
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/streaming_model.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/streaming_model.py
new file mode 100644
index 0000000..6f9bfee
--- /dev/null
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/streaming_model.py
@@ -0,0 +1,36 @@
+"""Context manager for block-streaming model builds (paired with :func:`~ltx_pipelines.utils.gpu_model.gpu_model`)."""
+
+from __future__ import annotations
+
+from collections.abc import Iterator
+from contextlib import contextmanager
+
+import torch
+
+from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy
+from ltx_core.block_streaming import StreamingModelBuilder
+from ltx_pipelines.utils.helpers import cleanup_memory
+
+
+@contextmanager
+def streaming_model(
+ builder: StreamingModelBuilder,
+ target_device: torch.device,
+ dtype: torch.dtype,
+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,
+) -> Iterator[object]:
+ """Build a streaming wrapper, yield it, then tear down and free memory.
+ The builder's own ``cpu_slots_count`` selects RAM vs disk streaming.
+ ``teardown()`` always runs -- it releases non-memory resources (forward
+ hooks, the disk I/O worker thread, open file handles) that GC would not
+ reclaim promptly. ``dispose`` always metas parameter storage; ``TRIM``
+ also runs ``cleanup_memory()``, while ``DEFER`` leaves the CUDA cache warm.
+ """
+ wrapped = builder.build(device=target_device, dtype=dtype)
+ try:
+ yield wrapped
+ finally:
+ wrapped.teardown()
+ wrapped.dispose()
+ if alloc_trim_strategy == AllocatorTrimStrategy.TRIM:
+ cleanup_memory()
diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/types.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/types.py
index e7b11a6..6f3f873 100644
--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/types.py
+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/types.py
@@ -1,17 +1,21 @@
from __future__ import annotations
-from collections.abc import Iterator
-from dataclasses import dataclass, field
+from collections.abc import Callable, Iterator
+from dataclasses import dataclass, field, replace
from enum import Enum
from typing import Generic, NamedTuple, Protocol, TypeVar
import torch
from ltx_core.components.patchifiers import AudioPatchifier, VideoLatentPatchifier
-from ltx_core.conditioning import ConditioningItem
+from ltx_core.conditioning import (
+ ConditioningItem,
+ VideoConditionByKeyframeIndex,
+ VideoConditionByLatentIndex,
+)
from ltx_core.model.transformer import X0Model
from ltx_core.model.video_vae import TilingConfig
-from ltx_core.types import Audio, LatentState
+from ltx_core.types import Audio, LatentState, SpatioTemporalScaleFactors
from ltx_pipelines.utils.constants import VIDEO_LATENT_CHANNELS, VIDEO_SCALE_FACTORS
_T = TypeVar("_T")
@@ -32,19 +36,153 @@ class PipelineOutput(NamedTuple):
tiling_config: TilingConfig | None
-class ImageConditioningInput(NamedTuple):
- """An image to condition on, with the H.264 CRF it should be re-compressed at.
- Leaving ``crf`` unset means "use whatever matches the model": the pipeline fills it in
- from the checkpoint it runs (``ImageConditioner.resolve_crf``), since the value the model
- was trained with is a property of the model generation. Pass ``crf`` explicitly to
- override it, including ``0`` to skip re-compression entirely.
+@dataclass(frozen=True)
+class ImageReplace:
+ """Pin a still by replacing the latent that starts at a pixel-frame index.
+ ``frame_idx`` is zero-based on the decoded video timeline, not a latent index. Because the
+ video VAE is causal, valid values are frame 0 and the first pixel frame of each later latent:
+ ``1 + VIDEO_SCALE_FACTORS.time * k`` (1, 9, 17, ... for the default temporal scale of 8).
"""
- path: str
frame_idx: int
+
+ def __post_init__(self) -> None:
+ _assert_replacement_frame_idx(self.frame_idx)
+
+
+@dataclass(frozen=True)
+class ImageGuide:
+ """Guide the clip with a keyframe at a zero-based decoded pixel-frame index."""
+
+ frame_idx: int
+
+
+ImageBinding = ImageReplace | ImageGuide
+
+
+@dataclass(frozen=True)
+class ImageConditioningInput:
+ """A still to condition on, by file, with the H.264 CRF it should be re-compressed at.
+ All frame indices are zero-based pixel-frame indices. Leaving ``crf`` unset uses the model's
+ checkpoint default; pass it explicitly to override that value, including ``0`` to skip
+ re-compression.
+ How the still binds is ``binding``: :class:`ImageReplace` or :class:`ImageGuide`.
+ Construct with :meth:`at` (frame 0 replaces, later frames guide), :meth:`replace`, or
+ :meth:`guide`. Pipelines load it with
+ :meth:`~ltx_pipelines.utils.blocks.ImageConditioner.load_conditionings` into a
+ :class:`LoadedImageConditioning` before encoding.
+ """
+
+ path: str
strength: float
+ binding: ImageBinding
crf: int | None = None
+ def __post_init__(self) -> None:
+ if not isinstance(self.binding, (ImageReplace, ImageGuide)):
+ raise TypeError(f"binding must be ImageReplace or ImageGuide, got {type(self.binding).__name__}")
+
+ @classmethod
+ def at(
+ cls,
+ path: str,
+ frame_idx: int,
+ strength: float,
+ crf: int | None = None,
+ ) -> ImageConditioningInput:
+ """Bind at a pixel frame, replacing frame 0 and guiding at every later frame."""
+ if frame_idx == 0:
+ return cls.replace(path, 0, strength, crf)
+ else:
+ return cls.guide(path, frame_idx, strength, crf)
+
+ @classmethod
+ def replace(
+ cls,
+ path: str,
+ frame_idx: int,
+ strength: float,
+ crf: int | None = None,
+ ) -> ImageConditioningInput:
+ """Replace at a zero-based pixel frame that starts a causal VAE latent."""
+ return cls(path=path, strength=strength, binding=ImageReplace(frame_idx), crf=crf)
+
+ @classmethod
+ def guide(
+ cls,
+ path: str,
+ frame_idx: int,
+ strength: float,
+ crf: int | None = None,
+ ) -> ImageConditioningInput:
+ """Guide generation with a still at a pixel-frame index."""
+ return cls(path=path, strength=strength, binding=ImageGuide(frame_idx), crf=crf)
+
+
+class DecodedStill(Protocol):
+ """A conditioning still decoded once for a run, cut to each resolution it is encoded at."""
+
+ def tensor(self, height: int, width: int, dtype: torch.dtype, device: torch.device) -> torch.Tensor:
+ """The still resized and cropped to ``height`` x ``width``, as a ``(1, C, 1, H, W)`` VAE input."""
+ ...
+
+
+@dataclass(frozen=True)
+class LoadedImageConditioning:
+ """A decoded still bound to the clip: what a run encodes as an image conditioning.
+ Built by :meth:`~ltx_pipelines.utils.blocks.ImageConditioner.load_conditionings` from an
+ :class:`ImageConditioningInput`. Rebasing it onto a stage, tile, window or chunk timeline
+ (:meth:`map_frame_index`) keeps ``still``, so the file is decoded once per run.
+ """
+
+ still: DecodedStill
+ strength: float
+ binding: ImageBinding
+
+ @property
+ def frame_idx(self) -> int:
+ """Pixel-frame index carried by either supported image binding."""
+ match self.binding:
+ case ImageReplace(frame_idx):
+ return frame_idx
+ case ImageGuide(frame_idx):
+ return frame_idx
+ raise TypeError(f"unhandled image binding: {self.binding!r}")
+
+ def assert_in_clip(self, num_frames: int) -> None:
+ """Raise when this conditioning's pixel-frame index lies outside the planned stitched clip."""
+ if self.frame_idx < 0 or self.frame_idx >= num_frames:
+ raise ValueError(f"frame_idx={self.frame_idx} is outside the planned stitched range [0, {num_frames})")
+
+ def map_frame_index(self, mapper: Callable[[int], int | None]) -> LoadedImageConditioning | None:
+ """Map the pixel-frame index through ``mapper``, preserving replace vs guide binding and the still."""
+ local = mapper(self.frame_idx)
+ if local is None:
+ return None
+ match self.binding:
+ case ImageReplace():
+ return replace(self, binding=ImageReplace(local))
+ case ImageGuide():
+ return replace(self, binding=ImageGuide(local))
+ raise TypeError(f"unhandled image binding: {self.binding!r}")
+
+ def to_item(self, encoded: torch.Tensor) -> ConditioningItem:
+ """Lower to a core item, converting a replacement's pixel frame to a latent index."""
+ match self.binding:
+ case ImageReplace(frame_idx):
+ return VideoConditionByLatentIndex(
+ latent=encoded,
+ strength=self.strength,
+ latent_idx=_replacement_frame_to_latent_index(frame_idx),
+ )
+ case ImageGuide(frame_idx):
+ return VideoConditionByKeyframeIndex(
+ keyframes=encoded,
+ strength=self.strength,
+ frame_idx=frame_idx,
+ )
+ raise TypeError(f"unhandled image binding: {self.binding!r}")
+
class PipelineComponents:
"""
@@ -201,3 +339,24 @@ class OffloadMode(Enum):
NONE = "none"
CPU = "cpu"
DISK = "disk"
+
+
+def _assert_replacement_frame_idx(
+ frame_idx: int,
+ scale_factors: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,
+) -> None:
+ if frame_idx == 0:
+ return
+ temporal_scale = scale_factors.time
+ if frame_idx < 1 or (frame_idx - 1) % temporal_scale != 0:
+ raise ValueError(f"ImageReplace frame_idx must be 0 or 1 + {temporal_scale} * k, got {frame_idx}")
+
+
+def _replacement_frame_to_latent_index(
+ frame_idx: int,
+ scale_factors: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,
+) -> int:
+ if frame_idx == 0:
+ return 0
+ temporal_scale = scale_factors.time
+ return 1 + (frame_idx - 1) // temporal_scale
diff --git a/packages/ltx-trainer/AGENTS.md b/packages/ltx-trainer/AGENTS.md
index be64409..2db140b 100644
--- a/packages/ltx-trainer/AGENTS.md
+++ b/packages/ltx-trainer/AGENTS.md
@@ -152,13 +152,13 @@ The `configs/` directory is documented in two places that both contain a trainin
**Model Interface (Modality-based):**
```python
-from ltx_core.model.transformer.modality import Modality
+from ltx_core.model.transformer.modality import Modality, TokenTimesteps
video = Modality(
enabled=True,
latent=video_latents, # [B, seq_len, 128] patchified latent tokens
sigma=sigma, # [B,] current noise level (per-batch)
- timesteps=video_timesteps, # [B, seq_len] per-token timestep embeddings
+ timesteps=TokenTimesteps(video_timesteps), # [B, seq_len] per-token timesteps (dense form)
positions=video_positions, # [B, 3, seq_len, 2] positional coordinates
context=video_embeds, # text conditioning embeddings
context_mask=None, # optional attention mask for text context
@@ -167,7 +167,7 @@ audio = Modality(
enabled=True,
latent=audio_latents,
sigma=sigma,
- timesteps=audio_timesteps,
+ timesteps=TokenTimesteps(audio_timesteps),
positions=audio_positions, # [B, 1, seq_len, 2]
context=audio_embeds,
context_mask=None,
@@ -548,21 +548,23 @@ If you do need a new strategy:
```python
from dataclasses import replace
-from ltx_core.model.transformer.modality import Modality
+from ltx_core.model.transformer.modality import Modality, TokenTimesteps
# Create modality — all fields except enabled and masks are required
video = Modality(
enabled=True,
latent=latents, # [B, seq_len, 128]
sigma=sigma, # [B,] — the per-batch noise level
- timesteps=timesteps, # [B, seq_len] — per-token (sigma * denoise_mask)
+ timesteps=TokenTimesteps(timesteps), # [B, seq_len] — per-token (sigma * denoise_mask)
positions=positions, # [B, 3, seq_len, 2]
context=context, # text embeddings from embeddings_processor
context_mask=None,
)
# Update (immutable — must use replace)
-video = replace(video, latent=new_latent, sigma=new_sigma, timesteps=new_timesteps)
+video = replace(
+ video, latent=new_latent, sigma=new_sigma, timesteps=TokenTimesteps(new_timesteps)
+)
# Disable a modality
audio = replace(audio, enabled=False)
diff --git a/packages/ltx-trainer/docs/custom-training-strategies.md b/packages/ltx-trainer/docs/custom-training-strategies.md
index 907eec9..8c3bacb 100644
--- a/packages/ltx-trainer/docs/custom-training-strategies.md
+++ b/packages/ltx-trainer/docs/custom-training-strategies.md
@@ -145,7 +145,7 @@ import torch
from pydantic import Field
from torch import Tensor
-from ltx_core.model.transformer.modality import Modality
+from ltx_core.model.transformer.modality import Modality, TokenTimesteps
from ltx_trainer.timestep_samplers import TimestepSampler
from ltx_trainer.training_strategies.base_strategy import (
ModelInputs,
@@ -287,7 +287,7 @@ class InpaintingStrategy(TrainingStrategy):
enabled=True,
latent=noisy_latents,
sigma=sigmas,
- timesteps=timesteps,
+ timesteps=TokenTimesteps(timesteps),
positions=positions,
context=video_prompt_embeds,
context_mask=prompt_attention_mask,
diff --git a/packages/ltx-trainer/docs/utility-scripts.md b/packages/ltx-trainer/docs/utility-scripts.md
index d456b35..a2b92db 100644
--- a/packages/ltx-trainer/docs/utility-scripts.md
+++ b/packages/ltx-trainer/docs/utility-scripts.md
@@ -218,10 +218,9 @@ uv run python scripts/decode_latents.py /path/to/latents/dir /path/to/output \
For inference with trained LoRAs, use the [`ltx-pipelines`](../../ltx-pipelines/) package which provides
production-ready pipelines:
-- **Text/Image-to-Video**: `TI2VidOneStagePipeline`, `TI2VidTwoStagesPipeline`
+- **Text/Image-to-Video and keyframe interpolation**: `TI2VidOneStagePipeline`, `TI2VidTwoStagesPipeline`
- **Distilled (fast) inference**: `DistilledPipeline`
- **IC-LoRA video-to-video**: `ICLoraPipeline`
-- **Keyframe interpolation**: `KeyframeInterpolationPipeline`
All pipelines support loading custom LoRAs trained with this trainer.
diff --git a/packages/ltx-trainer/pyproject.toml b/packages/ltx-trainer/pyproject.toml
index 41549f4..dcc76d7 100644
--- a/packages/ltx-trainer/pyproject.toml
+++ b/packages/ltx-trainer/pyproject.toml
@@ -1,6 +1,6 @@
[project]
name = "ltx-trainer"
-version = "1.4.2"
+version = "1.5.0"
description = "LTX-2 training, democratized."
readme = "README.md"
authors = [
diff --git a/packages/ltx-trainer/src/ltx_trainer/training_strategies/flexible.py b/packages/ltx-trainer/src/ltx_trainer/training_strategies/flexible.py
index 0afd6e0..a46cb01 100644
--- a/packages/ltx-trainer/src/ltx_trainer/training_strategies/flexible.py
+++ b/packages/ltx-trainer/src/ltx_trainer/training_strategies/flexible.py
@@ -15,7 +15,7 @@ import torch
from pydantic import BaseModel, ConfigDict, Field, model_validator
from torch import Tensor
-from ltx_core.model.transformer.modality import Modality
+from ltx_core.model.transformer.modality import Modality, TokenTimesteps
from ltx_core.types import SpatioTemporalScaleFactors
from ltx_trainer.timestep_samplers import TimestepSampler
from ltx_trainer.training_strategies.base_strategy import (
@@ -433,7 +433,7 @@ class FlexibleStrategy(TrainingStrategy):
enabled=True,
latent=noisy_latents,
sigma=sigmas,
- timesteps=timesteps,
+ timesteps=TokenTimesteps(timesteps),
positions=positions,
context=prompt_embeds,
context_mask=prompt_attention_mask,
diff --git a/packages/ltx-trainer/src/ltx_trainer/training_strategies/text_to_video.py b/packages/ltx-trainer/src/ltx_trainer/training_strategies/text_to_video.py
index 66cf229..71d582e 100644
--- a/packages/ltx-trainer/src/ltx_trainer/training_strategies/text_to_video.py
+++ b/packages/ltx-trainer/src/ltx_trainer/training_strategies/text_to_video.py
@@ -12,7 +12,7 @@ import torch
from pydantic import Field
from torch import Tensor
-from ltx_core.model.transformer.modality import Modality
+from ltx_core.model.transformer.modality import Modality, TokenTimesteps
from ltx_trainer import logger
from ltx_trainer.timestep_samplers import TimestepSampler
from ltx_trainer.training_strategies.base_strategy import (
@@ -155,7 +155,7 @@ class TextToVideoStrategy(TrainingStrategy):
enabled=True,
sigma=sigmas,
latent=noisy_video,
- timesteps=video_timesteps,
+ timesteps=TokenTimesteps(video_timesteps),
positions=video_positions,
context=video_prompt_embeds,
context_mask=prompt_attention_mask,
@@ -242,7 +242,7 @@ class TextToVideoStrategy(TrainingStrategy):
enabled=True,
latent=noisy_audio,
sigma=sigmas,
- timesteps=audio_timesteps,
+ timesteps=TokenTimesteps(audio_timesteps),
positions=audio_positions,
context=audio_prompt_embeds,
context_mask=prompt_attention_mask,
diff --git a/packages/ltx-trainer/src/ltx_trainer/training_strategies/video_to_video.py b/packages/ltx-trainer/src/ltx_trainer/training_strategies/video_to_video.py
index 2d133b3..f734db2 100644
--- a/packages/ltx-trainer/src/ltx_trainer/training_strategies/video_to_video.py
+++ b/packages/ltx-trainer/src/ltx_trainer/training_strategies/video_to_video.py
@@ -11,7 +11,7 @@ import torch
from pydantic import Field
from torch import Tensor
-from ltx_core.model.transformer.modality import Modality
+from ltx_core.model.transformer.modality import Modality, TokenTimesteps
from ltx_trainer import logger
from ltx_trainer.timestep_samplers import TimestepSampler
from ltx_trainer.training_strategies.base_strategy import (
@@ -208,7 +208,7 @@ class VideoToVideoStrategy(TrainingStrategy):
enabled=True,
latent=combined_latents,
sigma=sigmas,
- timesteps=timesteps,
+ timesteps=TokenTimesteps(timesteps),
positions=positions,
context=prompt_embeds,
context_mask=prompt_attention_mask,
diff --git a/packages/ltx-trainer/src/ltx_trainer/validation_runner.py b/packages/ltx-trainer/src/ltx_trainer/validation_runner.py
index a5f3fb0..cf6fcee 100644
--- a/packages/ltx-trainer/src/ltx_trainer/validation_runner.py
+++ b/packages/ltx-trainer/src/ltx_trainer/validation_runner.py
@@ -36,7 +36,7 @@ from ltx_core.guidance.perturbations import (
)
from ltx_core.loader import SafetensorsModelStateDictLoader
from ltx_core.model.audio_vae.audio_vae import encode_audio as ltx_encode_audio
-from ltx_core.model.transformer.modality import Modality
+from ltx_core.model.transformer.modality import Modality, TokenTimesteps
from ltx_core.model.transformer.model import X0Model
from ltx_core.model.video_vae import DimensionSizeConfig, TileSizeConfig
from ltx_core.model.video_vae.diffusion_tiling import recommended_decode_tiling_config
@@ -50,6 +50,7 @@ from ltx_core.tools import AudioLatentTools, VideoLatentTools
from ltx_core.types import (
Audio,
AudioLatentShape,
+ DenoiseMask,
LatentState,
SpatioTemporalScaleFactors,
VideoLatentShape,
@@ -614,13 +615,14 @@ class ValidationRunner:
)
video_state = replace(
video_state,
- denoise_mask=torch.zeros_like(video_state.denoise_mask),
+ denoise_mask=DenoiseMask(torch.zeros_like(video_state.denoise_mask.tensor)).compact(device),
frozen=True,
)
video_clean = video_state
else:
video_state = video_tools.create_initial_state(device=device, dtype=torch.bfloat16)
video_state = self._apply_video_conditionings(video_state, video_tools, sample, cached_media, device)
+ video_state = replace(video_state, denoise_mask=video_state.denoise_mask.compact(device))
video_clean = video_state
video_state = noiser(video_state, noise_scale=1.0)
@@ -642,7 +644,7 @@ class ValidationRunner:
)
audio_state = replace(
audio_state,
- denoise_mask=torch.zeros_like(audio_state.denoise_mask),
+ denoise_mask=DenoiseMask(torch.zeros_like(audio_state.denoise_mask.tensor)).compact(device),
frozen=True,
)
audio_clean = audio_state
@@ -650,6 +652,7 @@ class ValidationRunner:
audio_tools = self._create_audio_tools(num_frames, self._config.frame_rate)
audio_state = audio_tools.create_initial_state(device=device, dtype=torch.bfloat16)
audio_state = self._apply_audio_conditionings(audio_state, audio_tools, sample, cached_media, device)
+ audio_state = replace(audio_state, denoise_mask=audio_state.denoise_mask.compact(device))
audio_clean = audio_state
audio_state = noiser(audio_state, noise_scale=1.0)
@@ -754,7 +757,7 @@ class ValidationRunner:
stop = start + num_cond_tokens
state.clean_latent[:, start:stop] = tokens
- state.denoise_mask[:, start:stop] = 0.0
+ state.denoise_mask.tensor[:, start:stop] = 0.0
elif cond.type == "reference" and getattr(cond, "audio", None) is not None:
tokens = tools.patchifier.patchify(latent)
@@ -768,13 +771,12 @@ class ValidationRunner:
denoise_mask = torch.zeros(
*tokens.shape[:2],
1,
- device=device,
dtype=torch.float32,
)
state = LatentState(
latent=torch.cat([state.latent, torch.zeros_like(tokens, dtype=state.latent.dtype)], dim=1),
- denoise_mask=torch.cat([state.denoise_mask, denoise_mask], dim=1),
+ denoise_mask=DenoiseMask(torch.cat([state.denoise_mask.tensor, denoise_mask], dim=1)),
positions=torch.cat([state.positions, positions], dim=2),
clean_latent=torch.cat([state.clean_latent, tokens], dim=1),
attention_mask=None,
@@ -802,7 +804,7 @@ class ValidationRunner:
state = LatentState(
latent=state.latent,
- denoise_mask=state.denoise_mask * inv,
+ denoise_mask=DenoiseMask(state.denoise_mask.tensor * inv.cpu()),
positions=state.positions,
clean_latent=state.clean_latent * inv + tokens * m,
attention_mask=state.attention_mask,
@@ -955,13 +957,13 @@ class ValidationRunner:
if denoised_video is not None and video_clean is not None:
denoised_video = self._post_process_latent(
denoised_video,
- video_state.denoise_mask,
+ video_state.denoise_mask.tensor,
video_clean.clean_latent,
)
if denoised_audio is not None and audio_clean is not None:
denoised_audio = self._post_process_latent(
denoised_audio,
- audio_state.denoise_mask,
+ audio_state.denoise_mask.tensor,
audio_clean.clean_latent,
)
@@ -1266,7 +1268,7 @@ class ValidationRunner:
enabled=True,
latent=state.latent,
sigma=sigma,
- timesteps=state.denoise_mask * sigma,
+ timesteps=TokenTimesteps(state.denoise_mask.tensor * sigma),
positions=state.positions,
context=context,
context_mask=None,
diff --git a/pyproject.toml b/pyproject.toml
index 1ca80f9..cb6551a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -14,6 +14,34 @@ no-build-isolation-package = ["ltx-kernels"]
# 9.25.0.15 is yanked. Linux-only: the wheel is CUDA and has no darwin build.
override-dependencies = ["nvidia-cudnn-cu13==9.24.0.43; sys_platform == 'linux'"]
+# Videoseal declares `decord` (no aarch64 wheels) and `opencv-python` (needs libGL).
+# PixelSeal embed/detect does not use decord (try/except in videoseal). Replace OpenCV
+# with the headless wheel so GPU CI / this host can import cv2.
+[[tool.uv.dependency-metadata]]
+name = "videoseal"
+version = "1.0"
+requires-dist = [
+ "PyWavelets",
+ "av",
+ "calflops",
+ "einops",
+ "lpips",
+ "omegaconf",
+ "opencv-python-headless",
+ "pandas",
+ "pycocotools",
+ "pytorch_msssim",
+ "safetensors",
+ "scikit-image",
+ "scipy",
+ "setuptools",
+ "tensorboard",
+ "timm==0.9.16",
+ "torch>=2.3.1",
+ "tqdm",
+ "transformers",
+]
+
[tool.uv.workspace]
members = ["packages/*"]
# ltx-kernels compiles CUDA extensions; keep it out of the workspace so a plain
@@ -90,8 +118,6 @@ ignore = [
[tool.ruff.lint.per-file-ignores]
# CI gates: stand-alone CLIs whose whole job is printing a report / GitHub annotations.
".github/actions/golden-pairing-gate/check_golden_pairing.py" = ["T201"]
-# Emits `export …` lines for eval by kernels-build.
-".github/actions/kernels-build/export_cuda_home.py" = ["T201"]
# setup.py must set CUDA_HOME before importing torch.utils.cpp_extension.
"packages/ltx-kernels/setup.py" = ["E402"]
# Tests for the CuTe DSL kernels. ltx-kernels is an optional, CUDA-host-only install,
@@ -102,6 +128,10 @@ ignore = [
# NVFP4 tests ``importorskip("ltx_kernels")`` before importing convert/linear/fuse
# (which import the extension at module top level).
"packages/ltx-core/tests/ltx_core/quantization/nvfp4/test_*.py" = ["E402"]
+# PixelSeal / AudioSeal tests ``importorskip`` extras before importing seal classes
+# (which import videoseal / audioseal at module top level).
+"packages/ltx-pipelines/tests/ltx_pipelines/test_videoseal.py" = ["E402"]
+"packages/ltx-pipelines/tests/ltx_pipelines/test_audioseal.py" = ["E402"]
# CuTe DSL kernels. Device-side code written in CUTLASS's own idiom: tensor-algebra
# names are conventionally uppercase (M, T, H, W, NH, TILER_N128), a `@cute.kernel`
# entry point takes one parameter per compile-time constant, and its arguments are