Official Python inference and LoRA trainer package for the LTX-2 audio–video generative model.
Automated PR - 2026-10-18
github-actions[bot]e98f6d927d83
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 2c2c98d..5512c2c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,5 +1,36 @@# Changelog+## 1.5.0 - 2026-10-18++### Added++- `ImageConditioningInput` supports explicit latent-replacement and keyframe-guide bindings. On video CLIs, pass `--image PATH FRAME_IDX STRENGTH replace|guide [CRF]`; omit the binding keyword for the prior automatic binding (frame 0 replaces, later frames guide). Replacement frame indices are expressed in pixel space and must be `0` or a causal latent boundary (`1 + 8 * k`).+- `TI2VidTwoStagesPipeline` supports keyframe interpolation through `ImageConditioningInput.guide()` bindings, or+ `--image … guide` on the CLI. Set `refine_audio=True` or pass `--refine-audio` to preserve the former dedicated+ pipeline's stage-2 audio refinement; the default preserves stage-1 audio.+- `ImageConditioner.load_conditionings(images, color_space=None)` resolves each image conditioning's CRF (like `resolve_crf`), decodes each distinct still once (SDR, or EXR with `color_space`) and returns `LoadedImageConditioning`s: the decoded still plus the input's `binding` and `strength`.+- `MGPUController(cpu_threads_per_rank=8)` caps each rank's intra-op CPU threads (`torch.set_num_threads`) so the ranks' small host tensor ops do not stall on many-core hosts. Pass `None` to keep torch's default.+- `LTX_MGPU_DECODE_RECEIVER_TIMEOUT_SECONDS` sets how long a multi-GPU video decode waits for a rank that stopped streaming before it gives up (default 60).++### Changed++- Multi-GPU pipelines encode prompts faster with the default Accelerate-sharded Gemma: the other ranks now wait on the host until the source rank's Gemma has finished on every GPU, instead of spinning in an NCCL receive on a GPU Gemma is still using. `BroadcastCoordinator` moved from `ltx_pipelines.multigpu._broadcast` to `ltx_core.multigpu.broadcast`, and `BroadcastGemmaWrapper` (and so `AccelerateGemmaWrapper`) takes an optional `coordinator`. `BroadcastCoordinator.wait_for_signal_change` now consumes exactly one signal per call (a worker that polls late no longer skips a signal) and takes an optional `timeout`.+- `ImageConditioningInput` now takes an `ImageReplace` or `ImageGuide` `binding` instead of a `frame_idx`. Replace existing `ImageConditioningInput(path, frame_idx, strength, crf)` calls with `ImageConditioningInput.at(path, frame_idx, strength, crf)` to preserve the previous frame-0 replacement and later-frame guidance behavior; use `.replace(path, frame_idx, strength, crf)` or `.guide(path, frame_idx, strength, crf)` to select a binding explicitly. Both bindings use pixel-frame indices.+- `image_conditionings()` and `image_conditionings_for_chunk()` take `LoadedImageConditioning`s (from `ImageConditioner.load_conditionings`) instead of `ImageConditioningInput`s, and no longer take `color_space`.+- Performance improvements, in collaboration with doubleAI (https://doubleai.com): faster transformer passes (especially under multi-GPU sequence parallelism and with perturbation guidance) and LoRA switches, denoising samplers without per-step CPU synchronization, a DFR spatial epilogue (`--spatial-upscalings 2`) on fewer tiles (at most 1024 px per side, overlapping by 64 px, instead of a 2x2 first step and 4x4 at a 320 px overlap) that multiplies its long video projections in fp8 on GPUs with fp8 GEMMs (compute capability 8.9+), carry keyframes rebuilt on the rank that decodes them, a multi-GPU video decode that streams to MP4 encoding as it decodes, faster diffusion VAE decoding with lower reserved GPU memory, a faster-compiling `--diffvae-optimization blackwell_dsl` kernel, lower GPU memory during MP4 encoding, faster image conditioning, and faster stage starts with a warm model registry. Output is not bit-identical to previous releases, mainly because of the DFR spatial epilogue's new tiles and fp8 GEMMs; diffusion VAE decodes also differ by rounding, and under multi-GPU sequence parallelism audio cross-attention no longer attends to the video sequence's padding tokens.++### Removed++- `ImageConditioningInput.frame_idx`, `.assert_in_clip()`, `.map_frame_index()` and `.to_item()`; use the same members on the `LoadedImageConditioning` returned by `ImageConditioner.load_conditionings`, or `input.binding.frame_idx`.+- `KeyframeInterpolationPipeline` and the `ltx_pipelines.keyframe_interpolation` CLI; use `TI2VidTwoStagesPipeline` with `ImageConditioningInput.guide()` and `refine_audio=True`, or pass `--image … guide` with `--refine-audio`.++### Fixed++- DiffVAE 1-tile decode no longer produces gray frame tails from frame 128 on long 1080p clips under torch < 2.13. CUDA RMSNorm wrapped flattened offsets at 2^32 elements; last-dim norms are now T-chunked.+- Compiling a transformer that uses blockwise FP8/FP6 quantization with `seq_dim_dynamic` no longer pins the sequence length. Previously each new length recompiled the blocks, and a batch size divisible by 4 failed with a `ConstraintViolationError`.+- A warm multi-GPU DFR job (a second or later request in one process) could fail with "Cannot fit a DiffVAE decode tile". Cold and warm runs and every rank now plan the same decode tiles, and ranks with unequal GPUs no longer hang. `--diffvae-optimization blackwell_dsl` keyframe decodes plan smaller tiles.+- Diffusion VAE decode with `--diffvae-optimization blackwell_dsl` could hang indefinitely on datacenter Blackwell: the fused stage-5 attention kernel re-armed its per-tile barriers while some warps were still waiting on them.+## 1.4.2 - 2026-10-01### Added@@ -31,13 +62,21 @@- `--exr-colorspace` on `ltx_pipelines.hdr_ic_lora` (`acescg` / `acescct` / `srgb_linear`; default `acescg`). Only the EXR sidecar encoding changes; the HLG master stays BT.2020/HLG.- `--transformer-path` on `ltx_pipelines.hdr_ic_lora`, which runs the pipeline from a split checkpoint pack instead of a distilled monolith. A split transformer carries no VAE weights, so `--video-vae-path` has to accompany it.- `EXRColorSpace` / `VideoInput` in `ltx_pipelines` and an sRGB EOTF in `ltx_core.hdr`, which convert an SDR source into ACEScct.+- `TokenTimesteps` in `ltx_core`, which holds either dense per-token timesteps or a compact distinct-value table plus per-token indices. `TokenTimesteps.per_token()` expands either representation.+- `DenoiseMask` in `ltx_core.types`, which holds a per-token denoise mask and the table of its distinct values. Build the mask on the host; `DenoiseMask.compact(device)` tabulates its values and moves mask and table to the device before denoising.++### Fixed++- `FlashAttention4` accepts current `flash-attn-4` releases that return the output tensor directly while remaining compatible with older tuple-returning releases.### Changed- The `ltx_kernels_inductor` compile backend batches the fp8-cast policy's per-forward weight and bias upcasts into a few grouped kernels instead of one per tensor. Outputs are bit-identical.+- The `ltx_kernels_inductor` and `ltx_kernels_eager` compile backends run unmasked attention on whichever of cuDNN and flash attention is faster for each shape, which moves short-query attention (audio queries over video keys) off cuDNN's non-split fallback. The first call per attention shape times both kernels, so warmup takes slightly longer. This also applies when an `SDPA_*` attention backend is pinned.- Chunk planning owns generated-keyframe placement and appends slots after pipeline conditionings. Generated slots remain transformer conditioning across every stage; `decode_with_keyframes` now controls only whether the final slots anchor video decode.- `ChunkConfig.overlap_blend_frames` defaults to the full carry overlap (crossfade over all carried frames). Pass `0` for a hard cut at chunk seams.- `ChunkConfig.overlap_blend_frames` linearly crossfades a configurable suffix of each decoded overlap instead of cutting directly between windows, buffering only that suffix during streaming.+- Under sequence parallelism, the `ltx_kernels_inductor` compile backend has the q/k RMS-norm + RoPE kernel and the v projection write each head straight into the all2all exchange buffer of the rank that owns it, instead of producing a local tensor for a separate exchange kernel. Outputs are bit-identical.- DFR spatial epilogues run their first denoising step on a 2x2 tile grid before retiling any remaining steps to 4x4. A one-step schedule stays on the 2x2 grid.- `create_initial_audio_latent` now takes a video latent and required `video_scale_factors`. Joint first stages use `create_initial_av_latents`, which also requires `video_scale_factors`. `create_initial_video_latent` requires `scale_factors` (no default). Pass the same video VAE factors used to size the video latent. Audio-only callers use `AudioLatentShape.from_duration`.- Distilled and DFR sample with Euler ancestral (`eta=1.0`, `s_noise=1.0`) on LTX-2.5+ checkpoints -- distilled on both stages, DFR on stage 1, stage 2 and the spatial epilogue -- so generated output for those checkpoints differs from the previous release. Older generations sample with deterministic Euler. DFR's temporal densify tiles are unchanged at `eta=0.5`.@@ -52,6 +91,9 @@- `--distilled-checkpoint-path` on `ltx_pipelines.hdr_ic_lora` is no longer required, since `--transformer-path` can supply the transformer instead. One of the two is still needed.- DFR seam keyframes on `ltx_pipelines.hdr_ic_lora` are on by default. `--with-keyframes` is replaced by `--no-keyframes`. A 2.4 transformer fails the capability check unless `--no-keyframes` is passed; a 2.5 distilled monolith still needs `--video-vae-path` to a keyframe-trained VAE or the keyframe path is a silent no-op.- `ltx-core` requires `colour-science`, which supplies the primaries, YUV matrix and HLG transfer definitions that were previously hand-written constants.+- The transformer projects AdaLN modulation once per distinct timestep value and reads it per token through `Modality.timesteps.indices` when a pipeline supplies a compact timestep table. Outputs are not bit-identical to previous releases. The saving is proportional to the token count while attention grows with its square, so its share of the forward shrinks as resolution rises: about 2-3% on one GPU at 16k tokens (544x960x241), and less per rank under sequence parallelism.+- `Modality.timesteps` takes a `TokenTimesteps` instead of a tensor; wrap per-token timesteps as `TokenTimesteps(timesteps)`.+- `LatentState.denoise_mask` takes a `DenoiseMask` instead of a tensor; wrap the mask as `DenoiseMask(mask)` with `mask` on the host, and call `compact(device)` on it before a denoising loop reads the state.### Removeddiff --git a/LICENSE-2_x b/LICENSE-2_xindex 62bc2ee..010294d 100644--- a/LICENSE-2_x+++ b/LICENSE-2_x@@ -1,5 +1,5 @@- LTX-2.x Community License Agreement- License date: August 11, 2026+ LTX-2.x Community License Agreement+ License date: August 11, 2026By downloading, using, accessing or distributing any portion or@@ -473,10 +473,10 @@be unaffected thereby and remain valid as if such provision hadnot been set forth herein.- END OF TERMS AND CONDITIONS+ END OF TERMS AND CONDITIONS- Attachment A- Use Restrictions+ Attachment A+ Use RestrictionsWhen using the Outputs, LTX-2.x and any Derivatives thereof, youagree to comply with the Acceptable Use Policydiff --git a/README.md b/README.mdindex b69cbd7..15cbfee 100644--- a/README.md+++ b/README.md@@ -139,11 +139,10 @@ See **[LTX-2.3 models](MODELS-LTX-2.3.md)** for the full list.* **[DistilledPipeline](packages/ltx-pipelines/src/ltx_pipelines/distilled.py)** - Fastest text/image-to-video (starting point)* **[DFRPipeline](packages/ltx-pipelines/src/ltx_pipelines/dfr_pipeline.py)** - Production-quality text/image-to-video (slower, more VRAM): same distilled transformer, generated keyframes, spatial detailing, optional 2x/4x fps. How to run: [DFR in Quick Start](#dfr-production-quality) and [Running DFR](packages/ltx-pipelines/docs/pipelines.md#running-dfr)-* **[TI2VidTwoStagesPipeline](packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py)** - Guided two-stage text/image-to-video with CFG/STG and 2x upsampling+* **[TI2VidTwoStagesPipeline](packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py)** - Guided two-stage text/image-to-video and keyframe interpolation with CFG/STG and 2x upsampling* **[TI2VidTwoStagesHQPipeline](packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq.py)** - Same guided two-stage flow with the res_2s sampler (fewer steps)* **[TI2VidOneStagePipeline](packages/ltx-pipelines/src/ltx_pipelines/ti2vid_one_stage.py)** - Single-stage generation for quick prototyping* **[ICLoraPipeline](packages/ltx-pipelines/src/ltx_pipelines/ic_lora.py)** - Video-to-video and image-to-video transformations (uses distilled model.)-* **[KeyframeInterpolationPipeline](packages/ltx-pipelines/src/ltx_pipelines/keyframe_interpolation.py)** - Interpolate between keyframe images* **[A2VidPipelineTwoStage](packages/ltx-pipelines/src/ltx_pipelines/a2vid_two_stage.py)** - Audio-to-video generation conditioned on an input audio file* **[RetakePipeline](packages/ltx-pipelines/src/ltx_pipelines/retake.py)** - Regenerate a specific time region of an existing video* **[HDRICLoraPipeline](packages/ltx-pipelines/src/ltx_pipelines/hdr_ic_lora.py)** - Video-to-video SDR→HDR with an HDR IC-LoRA, writing a BT.2020/HLG master plus an ACEScct EXR sequence for EXR export and gradingdiff --git a/packages/ltx-core/README.md b/packages/ltx-core/README.mdindex 3d290b0..dbb24ac 100644--- a/packages/ltx-core/README.md+++ b/packages/ltx-core/README.md@@ -439,10 +439,9 @@ Here's how all the components work together conceptually ([`src/ltx_core/compone7. **VAE Decoding**: Decode latents to pixel space (with optional upsampling for two-stage)- [`DFRPipeline`](../ltx-pipelines/src/ltx_pipelines/dfr_pipeline.py) - Production-quality text/image-to-video-- [`TI2VidTwoStagesPipeline`](../ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py) - Guided two-stage text-to-video+- [`TI2VidTwoStagesPipeline`](../ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py) - Guided two-stage text/image-to-video and keyframe interpolation- [`ICLoraPipeline`](../ltx-pipelines/src/ltx_pipelines/ic_lora.py) - Video-to-video with IC-LoRA control- [`DistilledPipeline`](../ltx-pipelines/src/ltx_pipelines/distilled.py) - Fast inference with distilled model-- [`KeyframeInterpolationPipeline`](../ltx-pipelines/src/ltx_pipelines/keyframe_interpolation.py) - Keyframe-based interpolationSee the [ltx-pipelines README](../ltx-pipelines/README.md) for usage examples.diff --git a/packages/ltx-core/pyproject.toml b/packages/ltx-core/pyproject.tomlindex f2c3756..42f3314 100644--- a/packages/ltx-core/pyproject.toml+++ b/packages/ltx-core/pyproject.toml@@ -1,6 +1,6 @@[project]name = "ltx-core"-version = "1.4.2"+version = "1.5.0"description = "Core implementation of Lightricks' LTX-2 model"readme = "README.md"requires-python = ">=3.10"diff --git a/packages/ltx-core/src/ltx_core/block_streaming/builder.py b/packages/ltx-core/src/ltx_core/block_streaming/builder.pyindex cb74a27..f6282d7 100644--- a/packages/ltx-core/src/ltx_core/block_streaming/builder.py+++ b/packages/ltx-core/src/ltx_core/block_streaming/builder.py@@ -28,7 +28,7 @@ from ltx_core.block_streaming.utils import (from ltx_core.block_streaming.wrapper import BlockStreamingWrapperfrom ltx_core.devices import synchronize_devicefrom ltx_core.loader.fuse_loras import FuseRule, bf16_fuse_rule, fuse_lora_weights-from ltx_core.loader.helpers import create_meta_model, load_state_dict, read_model_config, read_model_metadata+from ltx_core.loader.helpers import LazyModelMetadata, create_meta_model, load_state_dictfrom ltx_core.loader.module_ops import ModuleOpsfrom ltx_core.loader.primitives import (LoraPathStrengthAndSDOps,@@ -107,6 +107,7 @@ class StreamingModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType])self._blocks_attr = blocks_attrself._blocks_prefix = blocks_prefixself._cpu_slots_count = cpu_slots_count+ self._metadata = LazyModelMetadata(model_path, self._model_loader)@propertydef model_class_configurator(self) -> type[ModelConfigurator[ModelType]]:@@ -161,6 +162,10 @@ class StreamingModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType])# Block weights live in fixed GPU/CPU slots that are reused across builds.return True+ @property+ def builds_per_shape(self) -> bool:+ return False+def with_sd_ops(self, sd_ops: SDOps | None) -> Self:clone = copy.copy(self)clone._model_sd_ops = sd_ops@@ -192,11 +197,11 @@ class StreamingModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType])def model_config(self) -> dict:"""Read model configuration from the checkpoint metadata."""- return read_model_config(self.model_path, self.model_loader)+ return self.model_metadata().get("config", {})def model_metadata(self) -> dict:"""Read the full checkpoint metadata (including sibling keys beyond ``config``)."""- return read_model_metadata(self.model_path, self.model_loader)+ return self._metadata.get()def meta_model(self, metadata: dict, module_ops: tuple[ModuleOps, ...]) -> ModelType:"""Create a model on the meta device and apply module operations."""diff --git a/packages/ltx-core/src/ltx_core/color/yuv.py b/packages/ltx-core/src/ltx_core/color/yuv.pyindex 81b5d9f..5f8deb4 100644--- a/packages/ltx-core/src/ltx_core/color/yuv.py+++ b/packages/ltx-core/src/ltx_core/color/yuv.py@@ -8,10 +8,13 @@ I420 / planar packing stay here (not provided by colour).from __future__ import annotationsimport enum+import functoolsimport colourimport torch+from ltx_core.devices import upload_from_host+class ColorSpace(enum.Enum):"""YUV color space standard (FFmpeg ``AVCOL_SPC_*`` + colour-science weights)."""@@ -82,6 +85,13 @@ def _rgb_to_ycbcr_matrix(weights_name: str) -> torch.Tensor:# Built once; ``ColorSpace.rgb_to_ycbcr_matrix`` reads from here._RGB_TO_YCBCR = {cs: _rgb_to_ycbcr_matrix(cs._weights_name) for cs in ColorSpace}+[email protected]_cache(maxsize=16)+def _device_rgb_to_ycbcr_matrix(color_space: ColorSpace, device: torch.device, dtype: torch.dtype) -> torch.Tensor:+ """``color_space``'s matrix on ``device`` in ``dtype``, uploaded once without waiting for the device."""+ return upload_from_host(color_space.rgb_to_ycbcr_matrix.to(dtype=dtype), device)++# cuBLAS GEMM dims are int32. PyTorch folds ``(F, H*W, 3) @ (3, 3)`` into one# ``mm`` of size ``F*H*W``, reported as ``n`` in column-major. ``n`` must be# strictly less than this value (see ``at::cuda::blas::gemm``).@@ -121,7 +131,7 @@ def rgb_to_yuv(image: torch.Tensor, color_space: ColorSpace) -> torch.Tensor:are concatenated. (``colour.RGB_to_YCbCr`` is NumPy-only and not used here.)"""_require_rgb_chw(image)- mat = color_space.rgb_to_ycbcr_matrix.to(device=image.device, dtype=image.dtype)+ mat = _device_rgb_to_ycbcr_matrix(color_space, image.device, image.dtype)orig_shape = image.shapeheight, width = orig_shape[-2], orig_shape[-1]frames = image.reshape(-1, 3, height, width)diff --git a/packages/ltx-core/src/ltx_core/components/diffusion_steps.py b/packages/ltx-core/src/ltx_core/components/diffusion_steps.pyindex 7d74ea3..5097365 100644--- a/packages/ltx-core/src/ltx_core/components/diffusion_steps.py+++ b/packages/ltx-core/src/ltx_core/components/diffusion_steps.py@@ -150,8 +150,7 @@ class Res2sDiffusionStep(DiffusionStepProtocol):sigma_up = torch.nan_to_num(sigma_up if sigma_up is not None else torch.zeros_like(sigma_next), 0.0)# Replace NaNs in sigma_down with corresponding sigma_next elements (float32)- nan_mask = torch.isnan(sigma_down)- sigma_down[nan_mask] = sigma_next[nan_mask].to(sigma_down.dtype)+ sigma_down = torch.where(torch.isnan(sigma_down), sigma_next.to(sigma_down.dtype), sigma_down)alpha_ratio = torch.nan_to_num(alpha_ratio, 1.0)return alpha_ratio, sigma_down, sigma_up@@ -180,8 +179,7 @@ class Res2sDiffusionStep(DiffusionStepProtocol):sigma_next = sigmas[step_index + 1]alpha_ratio, sigma_down, sigma_up = self.get_sde_coeff(sigma_next, sigma_up=sigma_next * eta)output_dtype = denoised_sample.dtype- if torch.any(sigma_up == 0) or torch.any(sigma_next == 0):- return denoised_sample+ return_denoised = torch.any(sigma_up == 0) | torch.any(sigma_next == 0)# Extract epsilon predictioneps_next = (sample - denoised_sample) / (sigma - sigma_next)@@ -189,7 +187,7 @@ class Res2sDiffusionStep(DiffusionStepProtocol):# Mix deterministic and stochastic componentsx_noised = alpha_ratio * (denoised_next + sigma_down * eps_next) + sigma_up * noise- return x_noised.to(output_dtype)+ return torch.where(return_denoised, denoised_sample, x_noised.to(output_dtype))class EulerCfgPpDiffusionStep(DiffusionStepProtocol):diff --git a/packages/ltx-core/src/ltx_core/components/noisers.py b/packages/ltx-core/src/ltx_core/components/noisers.pyindex 39122b0..7df5a73 100644--- a/packages/ltx-core/src/ltx_core/components/noisers.py+++ b/packages/ltx-core/src/ltx_core/components/noisers.py@@ -30,7 +30,7 @@ class GaussianNoiser(Noiser):def __call__(self, latent_state: LatentState, noise_scale: float = 1.0) -> LatentState:noise = self._sample_noise(latent_state)latent = torch.lerp(latent_state.latent.float(), noise.float(), noise_scale)- latent = torch.lerp(latent_state.clean_latent.float(), latent, latent_state.denoise_mask)+ latent = torch.lerp(latent_state.clean_latent.float(), latent, latent_state.denoise_mask.tensor)return replace(latent_state,latent=latent.to(latent_state.latent.dtype),diff --git a/packages/ltx-core/src/ltx_core/components/patchifiers.py b/packages/ltx-core/src/ltx_core/components/patchifiers.pyindex 77d6521..1c71116 100644--- a/packages/ltx-core/src/ltx_core/components/patchifiers.py+++ b/packages/ltx-core/src/ltx_core/components/patchifiers.py@@ -105,18 +105,12 @@ class VideoLatentPatchifier(Patchifier):# Shape becomes (3, grid_f, grid_h, grid_w)patch_starts = torch.stack(grid_coords, dim=0)- # Create a tensor containing the size of a single patch:- # (frame_patch_size, height_patch_size, width_patch_size).- # Reshape to (3, 1, 1, 1) to enable broadcasting when adding to the start coordinates.- patch_size_delta = torch.tensor(- self._patch_size,- device=patch_starts.device,- dtype=patch_starts.dtype,- ).view(3, 1, 1, 1)-- # Calculate end coordinates: start + patch_size+ # Calculate end coordinates: start + patch_size, per axis with Python-int patch sizes so+ # no host-built tensor is copied to the device.# Shape becomes (3, grid_f, grid_h, grid_w)- patch_ends = patch_starts + patch_size_delta+ patch_ends = torch.stack(+ [start + size for start, size in zip(grid_coords, self._patch_size, strict=True)], dim=0+ )# Stack start and end coordinates together along the last dimension# Shape becomes (3, grid_f, grid_h, grid_w, 2), where the last dimension is [start, end]@@ -150,18 +144,17 @@ def get_pixel_coords(causal_fix: When True, rewrites the temporal axis of the first frame so causal VAEsthat treat frame zero differently still yield non-negative timestamps."""- # Broadcast the VAE scale factors so they align with the `(batch, axis, patch, bound)` layout.- # Axis 1 of `latent_coords` is ordered (frame/time, height, width) — match that explicitly by- # pulling fields from the NamedTuple rather than relying on tuple iteration order.- broadcast_shape = [1] * latent_coords.ndim- broadcast_shape[1] = -1 # axis dimension corresponds to (frame/time, height, width)- scale_tensor = torch.tensor(- [scale_factors.time, scale_factors.height, scale_factors.width],- device=latent_coords.device,- ).view(*broadcast_shape)-- # Apply per-axis scaling to convert latent bounds into pixel-space coordinates.- pixel_coords = latent_coords * scale_tensor+ # Axis 1 of `latent_coords` is ordered (frame/time, height, width) -- match that explicitly by+ # pulling fields from the NamedTuple rather than relying on tuple iteration order. The factors+ # stay Python ints: a host-built factor tensor would be a blocking copy to the device.+ pixel_coords = torch.stack(+ [+ latent_coords[:, 0] * scale_factors.time,+ latent_coords[:, 1] * scale_factors.height,+ latent_coords[:, 2] * scale_factors.width,+ ],+ dim=1,+ )if causal_fix:# VAE temporal stride for the very first frame is 1 instead of `scale_factors.time`.diff --git a/packages/ltx-core/src/ltx_core/conditioning/mask_utils.py b/packages/ltx-core/src/ltx_core/conditioning/mask_utils.pyindex 3a4fc54..d068775 100644--- a/packages/ltx-core/src/ltx_core/conditioning/mask_utils.py+++ b/packages/ltx-core/src/ltx_core/conditioning/mask_utils.py@@ -97,7 +97,7 @@ def extend_keyframes_mask(if existing is None and not marked:return Noneif existing is None:- existing = torch.zeros_like(latent_state.denoise_mask)+ existing = torch.zeros_like(latent_state.denoise_mask.tensor, device=latent_state.latent.device)new = torch.ones if marked else torch.zerospad = new((existing.shape[0], num_new_tokens, existing.shape[2]),diff --git a/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_cond.py b/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_cond.pyindex 2c8c18a..b23a5d0 100644--- a/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_cond.py+++ b/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_cond.py@@ -4,7 +4,7 @@ from ltx_core.components.patchifiers import get_pixel_coordsfrom ltx_core.conditioning.item import ConditioningItemfrom ltx_core.conditioning.mask_utils import extend_keyframes_mask, update_attention_maskfrom ltx_core.tools import VideoLatentTools-from ltx_core.types import LatentState, VideoLatentShape+from ltx_core.types import DenoiseMask, LatentState, VideoLatentShapeclass VideoConditionByKeyframeIndex(ConditioningItem):@@ -61,7 +61,7 @@ class VideoConditionByKeyframeIndex(ConditioningItem):denoise_mask = torch.full(size=(*tokens.shape[:2], 1),fill_value=1.0 - self.strength,- device=self.keyframes.device,+ device="cpu",dtype=self.keyframes.dtype,)@@ -77,7 +77,7 @@ class VideoConditionByKeyframeIndex(ConditioningItem):return LatentState(latent=torch.cat([latent_state.latent, torch.zeros_like(tokens)], dim=1),- denoise_mask=torch.cat([latent_state.denoise_mask, denoise_mask], dim=1),+ denoise_mask=DenoiseMask(torch.cat([latent_state.denoise_mask.tensor, denoise_mask], dim=1)),positions=torch.cat([latent_state.positions, positions], dim=2),clean_latent=torch.cat([latent_state.clean_latent, tokens], dim=1),attention_mask=new_attention_mask,diff --git a/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_slots.py b/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_slots.pyindex 99f4904..ee0b535 100644--- a/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_slots.py+++ b/packages/ltx-core/src/ltx_core/conditioning/types/keyframe_slots.py@@ -21,7 +21,7 @@ from ltx_core.components.patchifiers import get_pixel_coordsfrom ltx_core.conditioning.item import ConditioningItemfrom ltx_core.conditioning.mask_utils import extend_keyframes_mask, update_attention_maskfrom ltx_core.tools import VideoLatentTools-from ltx_core.types import GeneratedKeyframeLayout, LatentState+from ltx_core.types import DenoiseMask, GeneratedKeyframeLayout, LatentStateclass VideoGeneratedKeyframeSlots(ConditioningItem):@@ -116,7 +116,7 @@ class VideoGeneratedKeyframeSlots(ConditioningItem):dim=1,)# denoise_mask 1 => the noiser lerps slot latent toward noise (clean_latent is ignored).- denoise_mask = torch.ones(batch_size, num_new_tokens, 1, device=device, dtype=latent_state.denoise_mask.dtype)+ denoise_mask = torch.ones(batch_size, num_new_tokens, 1, dtype=latent_state.denoise_mask.tensor.dtype)keyframes_mask = extend_keyframes_mask(latent_state, num_new_tokens, marked=True)@@ -135,7 +135,7 @@ class VideoGeneratedKeyframeSlots(ConditioningItem):return LatentState(latent=torch.cat([latent_state.latent, slot_tokens], dim=1),- denoise_mask=torch.cat([latent_state.denoise_mask, denoise_mask], dim=1),+ denoise_mask=DenoiseMask(torch.cat([latent_state.denoise_mask.tensor, denoise_mask], dim=1)),positions=torch.cat([latent_state.positions, positions], dim=2),clean_latent=torch.cat([latent_state.clean_latent, torch.zeros_like(slot_tokens)], dim=1),attention_mask=new_attention_mask,diff --git a/packages/ltx-core/src/ltx_core/conditioning/types/latent_cond.py b/packages/ltx-core/src/ltx_core/conditioning/types/latent_cond.pyindex 381f899..4cfbd14 100644--- a/packages/ltx-core/src/ltx_core/conditioning/types/latent_cond.py+++ b/packages/ltx-core/src/ltx_core/conditioning/types/latent_cond.py@@ -83,5 +83,5 @@ def _apply_condition_by_latent_index(latent_state = latent_state.clone()latent_state.clean_latent[:, start_token:stop_token] = tokens- latent_state.denoise_mask[:, start_token:stop_token] = 1.0 - strength+ latent_state.denoise_mask.tensor[:, start_token:stop_token] = 1.0 - strengthreturn latent_statediff --git a/packages/ltx-core/src/ltx_core/conditioning/types/mask_cond.py b/packages/ltx-core/src/ltx_core/conditioning/types/mask_cond.pyindex 0464198..153e2ef 100644--- a/packages/ltx-core/src/ltx_core/conditioning/types/mask_cond.py+++ b/packages/ltx-core/src/ltx_core/conditioning/types/mask_cond.py@@ -6,7 +6,7 @@ import torchfrom ltx_core.conditioning.item import ConditioningItemfrom ltx_core.tools import LatentTools-from ltx_core.types import LatentState+from ltx_core.types import DenoiseMask, LatentStateclass VideoConditionByMask(ConditioningItem):@@ -45,5 +45,5 @@ class VideoConditionByMask(ConditioningItem):return replace(latent_state,clean_latent=latent_state.clean_latent * inv + tokens * m,- denoise_mask=latent_state.denoise_mask * inv + (1.0 - self.strength) * m,+ denoise_mask=DenoiseMask(latent_state.denoise_mask.tensor * inv.cpu() + (1.0 - self.strength) * m.cpu()),)diff --git a/packages/ltx-core/src/ltx_core/conditioning/types/noise_mask_cond.py b/packages/ltx-core/src/ltx_core/conditioning/types/noise_mask_cond.pyindex 186ac4e..fbcb3b6 100644--- a/packages/ltx-core/src/ltx_core/conditioning/types/noise_mask_cond.py+++ b/packages/ltx-core/src/ltx_core/conditioning/types/noise_mask_cond.py@@ -22,7 +22,7 @@ class TemporalRegionMask(ConditioningItem):def apply_to(self, latent_state: LatentState, latent_tools: LatentTools) -> LatentState:coords = latent_tools.patchifier.get_patch_grid_bounds(- latent_tools.target_shape, device=latent_state.denoise_mask.device+ latent_tools.target_shape, device=latent_state.denoise_mask.tensor.device)if isinstance(latent_tools.target_shape, AudioLatentShape):# Audio: patchifier get_patch_grid_bounds returns seconds@@ -38,8 +38,8 @@ class TemporalRegionMask(ConditioningItem):t_start, t_end = t_boundaries.unbind(dim=-1) # [B, N]in_region = (t_end > self.start_time) & (t_start < self.end_time)state = latent_state.clone()- mask_val = in_region.to(state.denoise_mask.dtype)- if state.denoise_mask.dim() == 3:+ mask_val = in_region.to(state.denoise_mask.tensor.dtype)+ if state.denoise_mask.tensor.dim() == 3:mask_val = mask_val.unsqueeze(-1)- state.denoise_mask.copy_(mask_val)+ state.denoise_mask.tensor.copy_(mask_val)return statediff --git a/packages/ltx-core/src/ltx_core/conditioning/types/reference_audio_cond.py b/packages/ltx-core/src/ltx_core/conditioning/types/reference_audio_cond.pyindex 3427025..66292a2 100644--- a/packages/ltx-core/src/ltx_core/conditioning/types/reference_audio_cond.py+++ b/packages/ltx-core/src/ltx_core/conditioning/types/reference_audio_cond.py@@ -6,7 +6,7 @@ import torchfrom ltx_core.conditioning.mask_utils import extend_keyframes_mask, update_attention_maskfrom ltx_core.tools import LatentTools-from ltx_core.types import LatentState+from ltx_core.types import DenoiseMask, LatentStateclass AudioConditionByReferenceLatent:@@ -36,7 +36,7 @@ class AudioConditionByReferenceLatent:denoise_mask = torch.full(size=(*tokens.shape[:2], 1),fill_value=1.0 - self.strength,- device=tokens.device,+ device="cpu",dtype=tokens.dtype,)@@ -52,7 +52,7 @@ class AudioConditionByReferenceLatent:return LatentState(latent=torch.cat([latent_state.latent, torch.zeros_like(tokens)], dim=1),- denoise_mask=torch.cat([latent_state.denoise_mask, denoise_mask], dim=1),+ denoise_mask=DenoiseMask(torch.cat([latent_state.denoise_mask.tensor, denoise_mask], dim=1)),positions=torch.cat([latent_state.positions, self.positions], dim=2),clean_latent=torch.cat([latent_state.clean_latent, tokens], dim=1),attention_mask=new_attention_mask,diff --git a/packages/ltx-core/src/ltx_core/conditioning/types/reference_video_cond.py b/packages/ltx-core/src/ltx_core/conditioning/types/reference_video_cond.pyindex adf4775..eee5754 100644--- a/packages/ltx-core/src/ltx_core/conditioning/types/reference_video_cond.py+++ b/packages/ltx-core/src/ltx_core/conditioning/types/reference_video_cond.py@@ -6,7 +6,7 @@ from ltx_core.components.patchifiers import get_pixel_coordsfrom ltx_core.conditioning.item import ConditioningItemfrom ltx_core.conditioning.mask_utils import extend_keyframes_mask, update_attention_maskfrom ltx_core.tools import VideoLatentTools-from ltx_core.types import LatentState, VideoLatentShape+from ltx_core.types import DenoiseMask, LatentState, VideoLatentShapeclass VideoConditionByReferenceLatent(ConditioningItem):@@ -87,7 +87,7 @@ class VideoConditionByReferenceLatent(ConditioningItem):denoise_mask = torch.full(size=(*tokens.shape[:2], 1),fill_value=1.0 - self.strength,- device=self.latent.device,+ device="cpu",dtype=self.latent.dtype,)@@ -103,7 +103,7 @@ class VideoConditionByReferenceLatent(ConditioningItem):return LatentState(latent=torch.cat([latent_state.latent, torch.zeros_like(tokens)], dim=1),- denoise_mask=torch.cat([latent_state.denoise_mask, denoise_mask], dim=1),+ denoise_mask=DenoiseMask(torch.cat([latent_state.denoise_mask.tensor, denoise_mask], dim=1)),positions=torch.cat([latent_state.positions, positions], dim=2),clean_latent=torch.cat([latent_state.clean_latent, tokens], dim=1),attention_mask=new_attention_mask,diff --git a/packages/ltx-core/src/ltx_core/devices.py b/packages/ltx-core/src/ltx_core/devices.pyindex f33271b..33e4049 100644--- a/packages/ltx-core/src/ltx_core/devices.py+++ b/packages/ltx-core/src/ltx_core/devices.py@@ -72,6 +72,16 @@ def allow_async_transfer(device: torch.device) -> bool:return device.type == "cuda"+def upload_from_host(host: torch.Tensor, device: torch.device) -> torch.Tensor:+ """Copy a host tensor to *device*, without blocking the host where the backend allows it.+ The transfer reads a pinned copy, so ``host`` may be modified as soon as this returns. A tensor+ that is not on the host is moved with ``.to``.+ """+ if not allow_async_transfer(device) or host.device.type != "cpu":+ return host.to(device)+ return host.pin_memory().to(device, non_blocking=True)++def synchronize_device(device: DeviceSpec = None) -> None:"""Synchronize CUDA or MPS work if the selected backend supports it."""resolved = resolve_device(device)@@ -110,6 +120,27 @@ def cuda_activation_budget_bytes(device: DeviceSpec = None) -> int:return min(int(free_raw), under_fraction)+def cuda_allocatable_bytes(device: DeviceSpec = None) -> int:+ """Bytes a new CUDA allocation in this process can still get, counting the allocator's cache.+ Like :func:`cuda_activation_budget_bytes`, but reserved blocks that hold no tensor count as+ available: the caching allocator serves new allocations from them and releases them to the+ driver before an allocation fails. ``fraction * device_total - memory_allocated``, capped by+ raw free memory plus those cached blocks.+ """+ if not torch.cuda.is_available():+ return 0+ resolved = resolve_device(device)+ if resolved.type != "cuda":+ return 0+ idx = resolved.index if resolved.index is not None else torch.cuda.current_device()+ free_raw, _total_raw = torch.cuda.mem_get_info(idx)+ props_total = torch.cuda.get_device_properties(idx).total_memory+ fraction = torch.cuda.get_per_process_memory_fraction(idx)+ allocated = int(torch.cuda.memory_allocated(idx))+ cached = max(0, int(torch.cuda.memory_reserved(idx)) - allocated)+ return max(0, min(int(free_raw) + cached, int(fraction * props_total) - allocated))++def mps_activation_budget_bytes(device: DeviceSpec = None) -> int:"""Bytes still available for new MPS allocations in this process.``recommended_max_memory`` is Metal's recommended working-set size, which isdiff --git a/packages/ltx-core/src/ltx_core/guidance/perturbations.py b/packages/ltx-core/src/ltx_core/guidance/perturbations.pyindex 394f96e..dd9c8bd 100644--- a/packages/ltx-core/src/ltx_core/guidance/perturbations.py+++ b/packages/ltx-core/src/ltx_core/guidance/perturbations.py@@ -4,6 +4,8 @@ from enum import IntEnumimport torchfrom torch._prims_common import DeviceLikeType+from ltx_core.types import HostDeviceTensorPair+class PerturbationType(IntEnum):"""Types of attention perturbations for STG (Spatio-Temporal Guidance).@@ -52,20 +54,16 @@ class PerturbationConfig:class BatchedPerturbationConfig:"""Per-block attention keep-masks for a batch, built once from a list of per-sample configs.- Construction materializes ``_block_masks`` -- a ``(len(PerturbationType), num_blocks, B)`` tensor- (1 = keep, 0 = perturbed) whose dim-0 row index is the ``PerturbationType`` value -- from the- perturbation structure. The per-sample config list is NOT retained: every consumer reads the- tensor (``mask`` indexes it; ``any_in_batch`` / ``all_in_batch`` read the host mirror).+ Construction materializes ``_block_masks`` -- paired device and host+ ``(len(PerturbationType), num_blocks, B)`` tensors (1 = keep, 0 = perturbed) whose dim-0 row+ index is the ``PerturbationType`` value -- from the perturbation structure. The per-sample+ config list is NOT retained: every consumer reads the tensor pair.The host build (reading the Python structure) happens here, in ``__init__``, so it MUST be runeagerly OUTSIDE any ``torch.compile`` / CUDA-graph-capture region. The compiled block then reads- perturbation purely as the runtime ``_block_masks`` tensor and never recompiles per config.+ perturbation purely as the runtime device tensor and never recompiles per config."""- _block_masks: torch.Tensor # keep-mask on the compute device, indexed [PerturbationType, block, sample]- # Host mirror so any_in_batch / all_in_batch stay sync-free and graph-break-free. Present for- # configs that may hit the eager skip shortcuts; None for compiled-only configs built via- # ``from_masks`` (the compiled processor reads only ``_block_masks``).- _block_masks_cpu: torch.Tensor | None+ _block_masks: HostDeviceTensorPairdef __init__(self,@@ -81,39 +79,46 @@ class BatchedPerturbationConfig:]for direction in range(len(PerturbationType))]- self._block_masks_cpu = torch.tensor(keep, dtype=dtype, device="cpu")- self._block_masks = self._block_masks_cpu if device is None else self._block_masks_cpu.to(device)+ host = torch.tensor(keep, dtype=dtype, device="cpu")+ if device is not None and torch.device(device).type == "cuda":+ # CUDA non-blocking copies require pinned source memory. The host mirror is read-only+ # after construction.+ host = host.pin_memory()+ device_mask = host.to(device, non_blocking=True)+ else:+ device_mask = host if device is None else host.to(device)+ self._block_masks = HostDeviceTensorPair(host_tensor=host, device_tensor=device_mask)@classmethod- def from_masks(- cls, block_masks: torch.Tensor, block_masks_cpu: torch.Tensor | None = None- ) -> "BatchedPerturbationConfig":+ def from_masks(cls, block_masks: HostDeviceTensorPair) -> "BatchedPerturbationConfig":"""Construct from prebuilt mask tensors (e.g. a batch-dim slice), bypassing the host build.- ``block_masks_cpu`` is only consumed by ``any_in_batch`` / ``all_in_batch`` (the eager- processor's skip shortcuts); pass it when the result may take that path. The compiled- processor reads only ``_block_masks``, so callers on that path may omit the mirror.+ The host tensor is consumed by ``any_in_batch`` / ``all_in_batch`` while the compiled+ processor reads only the device tensor."""obj = cls.__new__(cls)obj._block_masks = block_masks- obj._block_masks_cpu = block_masks_cpureturn obj@propertydef block_masks(self) -> torch.Tensor:"""Keep-mask on the compute device (1 = keep, 0 = perturbed), indexed [type, block, sample]."""- return self._block_masks+ return self._block_masks.device_tensor@property- def block_masks_cpu(self) -> torch.Tensor | None:- """Host mirror of the keep-mask, or None for compiled-only configs (see ``from_masks``)."""- return self._block_masks_cpu+ def block_masks_cpu(self) -> torch.Tensor:+ """Host mirror of the keep-mask."""+ return self._block_masks.host_tensordef batch_slice(self, start: int, end: int) -> "BatchedPerturbationConfig":"""A view over samples ``[start:end]`` of the batch, by slicing the mask tensors.Slicing (never rebuilding) keeps the host mask build outside any compiled / capture region."""- cpu_mask = self._block_masks_cpu[:, :, start:end] if self._block_masks_cpu is not None else None- return BatchedPerturbationConfig.from_masks(self._block_masks[:, :, start:end], cpu_mask)+ return BatchedPerturbationConfig.from_masks(+ HostDeviceTensorPair(+ host_tensor=self._block_masks.host_tensor[:, :, start:end],+ device_tensor=self._block_masks.device_tensor[:, :, start:end],+ )+ )def mask(self, perturbation_type: PerturbationType, block: int) -> torch.Tensor:"""This block's ``(B, 1, 1)`` keep-mask for one perturbation type, as an OWNED tensor.@@ -121,15 +126,13 @@ class BatchedPerturbationConfig:(e.g. self- and cross-attention) don't alias the same storage -- aliased graph inputs arefragile under ``torch.compile``."""- return self._block_masks[perturbation_type, block].reshape(-1, 1, 1).clone()+ return self._block_masks.device_tensor[perturbation_type, block].reshape(-1, 1, 1).clone()def any_in_batch(self, perturbation_type: PerturbationType, block: int) -> bool:- assert self._block_masks_cpu is not None, "host mirror required by the skip-shortcut processor path"- return bool((self._block_masks_cpu[perturbation_type, block] == 0).any())+ return bool((self._block_masks.host_tensor[perturbation_type, block] == 0).any())def all_in_batch(self, perturbation_type: PerturbationType, block: int) -> bool:- assert self._block_masks_cpu is not None, "host mirror required by the skip-shortcut processor path"- return bool((self._block_masks_cpu[perturbation_type, block] == 0).all())+ return bool((self._block_masks.host_tensor[perturbation_type, block] == 0).all())@staticmethoddef empty(diff --git a/packages/ltx-core/src/ltx_core/loader/helpers.py b/packages/ltx-core/src/ltx_core/loader/helpers.pyindex af2ea04..716d130 100644--- a/packages/ltx-core/src/ltx_core/loader/helpers.py+++ b/packages/ltx-core/src/ltx_core/loader/helpers.py@@ -2,11 +2,14 @@from __future__ import annotations+import copy+from collections.abc import Iterator, Mappingfrom os import PathLike, fspathfrom typing import TypeVarimport torchfrom torch import nn+from torch.nn.parameter import is_lazyfrom ltx_core.loader.module_ops import ModuleOpsfrom ltx_core.loader.primitives import StateDict, StateDictLoader@@ -18,6 +21,100 @@ _M = TypeVar("_M", bound=nn.Module)_PathArg = str | PathLike[str]+def _customizes_loading(module: nn.Module) -> bool:+ cls = type(module)+ return (+ cls._load_from_state_dict is not nn.Module._load_from_state_dict+ or cls.set_extra_state is not nn.Module.set_extra_state+ or bool(module._load_state_dict_pre_hooks)+ or bool(module._load_state_dict_post_hooks)+ )+++def assign_state_dict(module: nn.Module, state_dict: Mapping[str, torch.Tensor]) -> None:+ """Same result as ``module.load_state_dict(state_dict, strict=False, assign=True)``, in one pass.+ Each parameter / persistent buffer looks its key up directly, where ``nn.Module.load_state_dict``+ re-filters the whole state dict for every child module. Keys without a matching tensor are left+ untouched and unmatched entries are ignored (``strict=False``); a shape mismatch raises.+ Falls back to ``load_state_dict`` when any submodule customizes loading (an overridden+ ``_load_from_state_dict`` or ``set_extra_state``, load hooks) or swap-tensors loading is on,+ because those need the stock per-module protocol.+ """+ modules = list(module.named_modules(remove_duplicate=False))+ if torch.__future__.get_swap_module_params_on_conversion() or any(_customizes_loading(m) for _, m in modules):+ module.load_state_dict(state_dict, strict=False, assign=True)+ return++ error_msgs: list[str] = []+ for prefix, submodule in modules:+ key_prefix = f"{prefix}." if prefix else ""+ for name, param in submodule._parameters.items():+ tensor = None if param is None else state_dict.get(key_prefix + name)+ if tensor is None:+ continue+ tensor = _matching_shape(tensor, param, key_prefix + name, error_msgs)+ if tensor is None:+ continue+ if isinstance(tensor, nn.Parameter):+ tensor.requires_grad_(param.requires_grad)+ else:+ tensor = nn.Parameter(tensor, requires_grad=param.requires_grad)+ submodule._parameters[name] = tensor+ for name, buffer in submodule._buffers.items():+ if buffer is None or name in submodule._non_persistent_buffers_set:+ continue+ tensor = state_dict.get(key_prefix + name)+ if tensor is None:+ continue+ tensor = _matching_shape(tensor, buffer, key_prefix + name, error_msgs)+ if tensor is not None:+ submodule._buffers[name] = tensor+ if error_msgs:+ raise RuntimeError(f"Error(s) in loading state_dict for {type(module).__name__}:\n\t" + "\n\t".join(error_msgs))+++def _matching_shape(+ tensor: torch.Tensor, current: torch.Tensor, key: str, error_msgs: list[str]+) -> torch.Tensor | None:+ """``tensor`` reshaped the way ``load_state_dict`` accepts it for ``current``, or ``None`` on a mismatch.+ A lazy ``current`` (``UninitializedParameter`` / ``UninitializedBuffer``) has no shape yet and takes+ ``tensor`` as is, as in ``load_state_dict``.+ """+ if is_lazy(current):+ return tensor+ if current.ndim == 0 and tensor.shape == (1,):+ tensor = tensor[0]+ if tensor.shape != current.shape:+ error_msgs.append(+ f"size mismatch for {key}: copying a param with shape {tensor.shape} from checkpoint, "+ f"the shape in current model is {current.shape}."+ )+ return None+ return tensor+++def module_tensors(model: nn.Module) -> Iterator[torch.Tensor]:+ """Every parameter and buffer of ``model``, read straight off each module.+ Unlike ``named_parameters`` / ``named_buffers`` there is no de-duplication, so a tensor+ shared between modules is yielded once per owner.+ """+ for module in model.modules():+ yield from (t for t in module._parameters.values() if t is not None)+ yield from (t for t in module._buffers.values() if t is not None)+++def to_device(model: _M, device: torch.device) -> _M:+ """``model.to(device)``, skipped when every parameter and buffer already lives on ``device``.+ ``nn.Module.to`` rebuilds every parameter even when nothing moves, which costs a full walk of+ a large model per call.+ """+ if device.type == "cuda" and device.index is None:+ device = torch.device("cuda", torch.cuda.current_device())+ if all(t.device == device for t in module_tensors(model)):+ return model+ return model.to(device)++def as_path_list(paths: _PathArg | tuple[_PathArg, ...] | list[_PathArg]) -> list[str]:"""Normalize checkpoint path args to the ``list[str]`` shape registries key on."""if isinstance(paths, (str, PathLike)):@@ -51,6 +148,20 @@ def read_model_metadata(return loader.metadata(first)+class LazyModelMetadata:+ """A checkpoint's parsed metadata, read on first access and shared by every builder copy holding it."""++ def __init__(self, model_path: str | tuple[str, ...], loader: StateDictLoader) -> None:+ self._model_path = model_path+ self._loader = loader+ self._metadata: dict | None = None++ def get(self) -> dict:+ if self._metadata is None:+ self._metadata = read_model_metadata(self._model_path, self._loader)+ return copy.deepcopy(self._metadata)++def read_model_config(model_path: str | tuple[str, ...],loader: StateDictLoader,diff --git a/packages/ltx-core/src/ltx_core/loader/primitives.py b/packages/ltx-core/src/ltx_core/loader/primitives.pyindex 8550bf3..9379c99 100644--- a/packages/ltx-core/src/ltx_core/loader/primitives.py+++ b/packages/ltx-core/src/ltx_core/loader/primitives.py@@ -97,6 +97,11 @@ class ModelBuilderProtocol(BuilderProtocol[BuiltType], Protocol[BuiltType]):"""Whether rebuilds rebind the same GPU weight storages (safe for CUDA-graph capture)."""...+ @property+ def builds_per_shape(self) -> bool:+ """Whether a built model serves only the latent shape it was built for (tiled data parallel)."""+ ...+@propertydef model_sd_ops(self) -> SDOps | None: ...diff --git a/packages/ltx-core/src/ltx_core/loader/single_gpu_model_builder.py b/packages/ltx-core/src/ltx_core/loader/single_gpu_model_builder.pyindex 58b577b..0d1d9bc 100644--- a/packages/ltx-core/src/ltx_core/loader/single_gpu_model_builder.py+++ b/packages/ltx-core/src/ltx_core/loader/single_gpu_model_builder.py@@ -9,11 +9,13 @@ from torch import nnfrom ltx_core.loader.fuse_loras import FuseRule, apply_loras, bf16_fuse_rulefrom ltx_core.loader.helpers import (+ LazyModelMetadata,as_path_list,+ assign_state_dict,create_meta_model,load_state_dict,- read_model_config,- read_model_metadata,+ module_tensors,+ to_device,)from ltx_core.loader.module_ops import ModuleOpsfrom ltx_core.loader.primitives import (@@ -37,6 +39,8 @@ logger: logging.Logger = logging.getLogger(__name__)def _check_uninitialized(model: nn.Module) -> list[str]:"""Return names of any parameters/buffers still on meta device."""+ if not any(t.is_meta for t in module_tensors(model)):+ return []names = []for name, param in model.named_parameters():if str(param.device) == "meta":@@ -95,7 +99,7 @@ def _load_model_weights(sd = model_sd.sdif dtype is not None:sd = _cast_floating_sd(sd, dtype)- meta_model.load_state_dict(sd, strict=False, assign=True)+ assign_state_dict(meta_model, sd)returnlora_state_dicts = [load_state_dict([lora.path], loader, registry, lora_load_device, lora.sd_ops) for lora in loras]@@ -116,7 +120,7 @@ def _load_model_weights(fused_sd = final_sd.sdif dtype is not None:fused_sd = _cast_floating_sd(fused_sd, dtype)- meta_model.load_state_dict(fused_sd, strict=False, assign=True)+ assign_state_dict(meta_model, fused_sd)class SingleGPUModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType], LoRAAdaptableProtocol):@@ -164,6 +168,7 @@ class SingleGPUModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType],self._registry = registry if registry is not None else ModelRegistry(cache_models=True, cache_weights=False)self._lora_load_device = lora_load_device if lora_load_device is not None else torch.device("cpu")self._fuse_rule = fuse_rule+ self._metadata = LazyModelMetadata(model_path, self._model_loader)@propertydef model_sd_ops(self) -> SDOps | None:@@ -186,6 +191,10 @@ class SingleGPUModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType],# Registry may retain a CPU SD; each build H2Ds into fresh GPU storages.return False+ @property+ def builds_per_shape(self) -> bool:+ return False+@propertydef model_path(self) -> str | tuple[str, ...]:return self._model_path@@ -242,10 +251,10 @@ class SingleGPUModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType],return clonedef model_config(self) -> dict:- return read_model_config(self._model_path, self._model_loader)+ return self.model_metadata().get("config", {})def model_metadata(self) -> dict:- return read_model_metadata(self._model_path, self._model_loader)+ return self._metadata.get()def meta_model(self, metadata: dict, module_ops: tuple[ModuleOps, ...]) -> ModelType:struct_key = module_registry_key(@@ -291,4 +300,4 @@ class SingleGPUModelBuilder(Generic[ModelType], ModelBuilderProtocol[ModelType],if uninitialized:logger.warning(f"Uninitialized parameters or buffers: {uninitialized}")return meta_model- return meta_model.to(device)+ return to_device(meta_model, device)diff --git a/packages/ltx-core/src/ltx_core/modality_tiling.py b/packages/ltx-core/src/ltx_core/modality_tiling.pyindex 8d988ba..6ad6f28 100644--- a/packages/ltx-core/src/ltx_core/modality_tiling.py+++ b/packages/ltx-core/src/ltx_core/modality_tiling.py@@ -1,7 +1,6 @@"""Video modality tiling helpers.-Provides :class:`VideoModalityTilingHelper` — a stateless helper that-tiles and blends video :class:`Modality` token sequences by-spatial/temporal region. Tile geometry is represented by the existing+Provides :class:`VideoModalityTilingHelper`, which tiles and blends video+:class:`Modality` token sequences by spatial/temporal region. Tile geometry is represented by the existing:class:`Tile` NamedTuple from :mod:`ltx_core.tiling`; no distributedprimitives are required."""@@ -9,12 +8,13 @@ primitives are required.from __future__ import annotationsimport logging-from collections.abc import Sequence-from dataclasses import dataclass, replace+from collections.abc import Callable, Sequence+from dataclasses import dataclass, field, replacefrom functools import partialimport torch+from ltx_core.devices import upload_from_hostfrom ltx_core.model.transformer.modality import Modalityfrom ltx_core.tiling import (DimensionTilingConfig,@@ -32,6 +32,22 @@ from ltx_core.types import VideoLatentShapelogger = logging.getLogger(__name__)+@dataclass(frozen=True)+class _TilePlan:+ """One tile's token layout for one positions tensor; device tables reused at every step.+ The layout is fixed once planned; only the per-``(dtype, device)`` weight caches fill in lazily.+ """++ keep_indices: torch.Tensor+ gen_indices: torch.Tensor+ cond_indices: torch.Tensor | None+ num_tile_gen: int+ cond_blend_weights: torch.Tensor | None+ positions: torch.Tensor+ blend_weights: dict[tuple[torch.dtype, torch.device], torch.Tensor] = field(default_factory=dict)+ cond_weights: dict[tuple[torch.dtype, torch.device], torch.Tensor] = field(default_factory=dict)++@dataclass(frozen=True)class TilingContext:"""Opaque context produced by :meth:`VideoModalityTilingHelper.tile_modality`.@@ -47,6 +63,7 @@ class TilingContext:"""``(num_kept_cond,)`` — weight for each kept conditioning token,equal to ``1 / num_tiles_that_keep_this_token``. ``None`` whenthere are no conditioning tokens."""+ plan: _TilePlandef seam_split(@@ -77,11 +94,17 @@ def seam_split(class VideoModalityTilingHelper:- """Stateless helper that tiles and blends video :class:`Modality` sequences.+ """Tiles and blends video :class:`Modality` sequences.Constructed once with a :class:`TileCountConfig` and:class:`VideoLatentTools`. Tiles are computed at construction andavailable via the :attr:`tiles` property. Use :meth:`tile_modality`and :meth:`blend` with any tile from that list.+ Every tile's token layout depends only on ``modality.positions``. The first+ :meth:`tile_modality` call for a positions tensor plans all tiles at once (its+ host syncs drain the GPU queue a single time, before the sequence's first+ forward) and later calls with the same tensor -- a denoising loop carries it+ unchanged across steps -- read those plans. The helper holds the tensor while+ its plans are live, so the identity check cannot see a recycled object.Passing ``seams`` (interior latent-frame indices) lets a temporal split land on thosecells instead of blended overlaps (:func:`seam_split`); leftover segments go to theleading tiles. Missing or non-interior seams keep the blended split. A regular keyframe@@ -117,6 +140,9 @@ class VideoModalityTilingHelper:splitters=[frames, height, width],mappers=[frames_mapper, identity_mapping_operation, identity_mapping_operation],)+ self._plan_positions: torch.Tensor | None = None+ self._plan_num_total = -1+ self._plans: dict[tuple[int, bool], _TilePlan] = {}@propertydef tiles(self) -> list[Tile]:@@ -139,26 +165,8 @@ class VideoModalityTilingHelper:A ``(tiled_modality, context)`` tuple. Pass *context* to:meth:`blend` together with the model output."""- device = modality.positions.device- gen_indices = self._generated_token_indices(tile, device=device)- num_total = modality.latent.shape[1]-- cond_blend_weights: torch.Tensor | None = None- if num_total > self._num_generated_tokens:- keep_per_tile_cond = self._all_tiles_cond_keep(modality) # (num_tiles, num_cond) bool- tile_idx = next((i for i, t in enumerate(self._tiles) if t.in_coords == tile.in_coords), None)- if tile_idx is None:- raise RuntimeError(- f"Tile with in_coords={tile.in_coords} is not in this helper's tile set; "- f"pass a tile obtained from `helper.tiles`."- )- my_cond_keep = keep_per_tile_cond[tile_idx]- cond_indices = self._num_generated_tokens + my_cond_keep.nonzero(as_tuple=False).squeeze(1)- keep_indices = torch.cat([gen_indices, cond_indices])- total_keepers = keep_per_tile_cond.sum(dim=0).float() # (num_cond,)- cond_blend_weights = 1.0 / total_keepers[my_cond_keep]- else:- keep_indices = gen_indices+ plan = self._plan(modality, tile, normalize_positions=normalize_positions)+ keep_indices = plan.keep_indicestile_attention_mask = Noneif modality.attention_mask is not None:@@ -171,24 +179,20 @@ class VideoModalityTilingHelper:if modality.keyframes_mask is not None:tile_keyframes_mask = modality.keyframes_mask[:, keep_indices]- positions = modality.positions[:, :, keep_indices, :]- if normalize_positions:- num_tile_gen = self._tile_generated_token_count(tile)- gen_pos = positions[:, :, :num_tile_gen, :] # (B, 3, num_tile_gen, 2)- offset = gen_pos[..., 0].amin(dim=2, keepdim=True).unsqueeze(-1) # (B, 3, 1, 1)- positions = positions - offset-tiled = replace(modality,latent=modality.latent[:, keep_indices, :],- timesteps=modality.timesteps[:, keep_indices],- positions=positions,+ timesteps=modality.timesteps.select_tokens(keep_indices),+ positions=plan.positions,attention_mask=tile_attention_mask,keyframes_mask=tile_keyframes_mask,)return tiled, TilingContext(- keep_indices=keep_indices, num_total_tokens=num_total, cond_blend_weights=cond_blend_weights+ keep_indices=keep_indices,+ num_total_tokens=modality.latent.shape[1],+ cond_blend_weights=plan.cond_blend_weights,+ plan=plan,)# -- blend -------------------------------------------------------------@@ -218,10 +222,8 @@ class VideoModalityTilingHelper:The output tensor with the blended tile added at the correctpositions."""+ plan = context.planbatch, _, dim = tile_to_blend.shape- num_tile_gen = self._tile_generated_token_count(tile)- gen_indices = self._generated_token_indices(tile, device=tile_to_blend.device)-num_total_tokens = context.num_total_tokensexpected_shape = (batch, num_total_tokens, dim)@@ -232,23 +234,89 @@ class VideoModalityTilingHelper:else:result = torch.zeros(*expected_shape, device=tile_to_blend.device, dtype=tile_to_blend.dtype)- # Blend mask is (tile_F, tile_H, tile_W) — one weight per token in row-major order.- blend_weights = tile.blend_mask.reshape(-1).to(device=tile_to_blend.device, dtype=tile_to_blend.dtype)- tile_gen = tile_to_blend[:, :num_tile_gen, :] * blend_weights[None, :, None]-- result[:, gen_indices, :] += tile_gen+ # Blend mask is (tile_F, tile_H, tile_W) — one weight per token in row-major order. The dense mask is+ # built on the host only when the plan has no copy on this device and dtype yet.+ blend_weights = _device_weights(plan.blend_weights, lambda: tile.blend_mask.reshape(-1), tile_to_blend)+ result[:, plan.gen_indices, :] += tile_to_blend[:, : plan.num_tile_gen, :] * blend_weights[None, :, None]# Scatter kept conditioning tokens, weighted by 1/N where N is# the number of tiles that keep each token (so they sum to 1).- if num_total_tokens > self._num_generated_tokens and context.cond_blend_weights is not None:- cond_indices = context.keep_indices[context.keep_indices >= self._num_generated_tokens]- weights = context.cond_blend_weights.to(device=tile_to_blend.device, dtype=tile_to_blend.dtype)- result[:, cond_indices, :] += tile_to_blend[:, num_tile_gen:, :] * weights[None, :, None]+ if num_total_tokens > self._num_generated_tokens and plan.cond_blend_weights is not None:+ cond_blend_weights = plan.cond_blend_weights+ weights = _device_weights(plan.cond_weights, lambda: cond_blend_weights, tile_to_blend)+ result[:, plan.cond_indices, :] += tile_to_blend[:, plan.num_tile_gen :, :] * weights[None, :, None]return result# -- private -----------------------------------------------------------+ def _plan(self, modality: Modality, tile: Tile, *, normalize_positions: bool) -> _TilePlan:+ """*tile*'s plan for this sequence; a new positions tensor or token count plans every tile again."""+ num_total = modality.latent.shape[1]+ if modality.positions is not self._plan_positions or num_total != self._plan_num_total:+ self._plan_positions = modality.positions+ self._plan_num_total = num_total+ self._plans = {}+ tile_idx = next((i for i, t in enumerate(self._tiles) if t.in_coords == tile.in_coords), None)+ if tile_idx is None:+ raise RuntimeError(+ f"Tile with in_coords={tile.in_coords} is not in this helper's tile set; "+ f"pass a tile obtained from `helper.tiles`."+ )+ key = (tile_idx, normalize_positions)+ if key not in self._plans:+ # The one host read of a plan build: how many conditioning tokens each tile keeps is data dependent.+ keep_per_tile_cond = None+ cond_keepers = None+ if num_total > self._num_generated_tokens:+ keep_per_tile_cond = self._all_tiles_cond_keep(modality).cpu()+ cond_keepers = keep_per_tile_cond.sum(dim=0).float() # (num_cond,)+ for idx, t in enumerate(self._tiles):+ cond_keep = keep_per_tile_cond[idx] if keep_per_tile_cond is not None else None+ self._plans[(idx, normalize_positions)] = self._build_plan(+ modality, t, normalize_positions, cond_keep, cond_keepers+ )+ return self._plans[key]++ def _build_plan(+ self,+ modality: Modality,+ tile: Tile,+ normalize_positions: bool,+ cond_keep: torch.Tensor | None,+ cond_keepers: torch.Tensor | None,+ ) -> _TilePlan:+ """*tile*'s plan. ``cond_keep`` is the tile's ``(num_cond,)`` keep row, ``cond_keepers`` how many tiles+ keep each conditioning token; both host tensors, ``None`` without conditioning tokens."""+ device = modality.positions.device+ gen_indices = self._generated_token_indices(tile, device=device)+ cond_indices = None+ cond_blend_weights = None+ if cond_keep is not None and cond_keepers is not None:+ # The keep row is on the host: the indices and weights are derived there and uploaded+ # through pinned memory, so no step of the plan waits on the device.+ host_cond_indices = self._num_generated_tokens + cond_keep.nonzero(as_tuple=False).squeeze(1)+ cond_indices = upload_from_host(host_cond_indices, device)+ keep_indices = torch.cat([gen_indices, cond_indices])+ cond_blend_weights = 1.0 / cond_keepers[cond_keep]+ else:+ keep_indices = gen_indices++ num_tile_gen = self._tile_generated_token_count(tile)+ positions = modality.positions[:, :, keep_indices, :]+ if normalize_positions:+ gen_pos = positions[:, :, :num_tile_gen, :] # (B, 3, num_tile_gen, 2)+ offset = gen_pos[..., 0].amin(dim=2, keepdim=True).unsqueeze(-1) # (B, 3, 1, 1)+ positions = positions - offset+ return _TilePlan(+ keep_indices=keep_indices,+ gen_indices=gen_indices,+ cond_indices=cond_indices,+ num_tile_gen=num_tile_gen,+ cond_blend_weights=cond_blend_weights,+ positions=positions,+ )+def _tile_generated_token_count(self, tile: Tile) -> int:"""Number of generated tokens in *tile*."""frame_slice, height_slice, width_slice = tile.in_coords@@ -300,3 +368,18 @@ class VideoModalityTilingHelper:overlaps_all_dims = overlaps.all(dim=2) # (num_tiles, B, num_cond)has_negative_time = (cond_positions[:, 0, :, 0] < 0)[None] # (1, B, num_cond)return (overlaps_all_dims | has_negative_time).any(dim=1) # (num_tiles, num_cond)+++def _device_weights(+ cache: dict[tuple[torch.dtype, torch.device], torch.Tensor],+ weights: Callable[[], torch.Tensor],+ like: torch.Tensor,+) -> torch.Tensor:+ """``weights()`` on ``like``'s device and dtype, built and uploaded once per plan through pinned memory."""+ key = (like.dtype, like.device)+ if key not in cache:+ host = weights().to(dtype=like.dtype)+ if like.device.type == "cuda" and host.device.type == "cpu":+ host = host.pin_memory()+ cache[key] = host.to(device=like.device, non_blocking=True)+ return cache[key]diff --git a/packages/ltx-core/src/ltx_core/model/disposable.py b/packages/ltx-core/src/ltx_core/model/disposable.pyindex 496a734..9cfceab 100644--- a/packages/ltx-core/src/ltx_core/model/disposable.py+++ b/packages/ltx-core/src/ltx_core/model/disposable.py@@ -24,19 +24,12 @@ class Disposable(DisposableProtocol):if not isinstance(self, nn.Module):raise TypeError(f"{type(self).__name__} must be an nn.Module to dispose")- persistent = set(self.state_dict().keys())- for name, param in list(self.named_parameters()):- parent_name, _, attr = name.rpartition(".")- parent = self.get_submodule(parent_name) if parent_name else self- setattr(- parent,- attr,- nn.Parameter(torch.empty_like(param, device="meta"), requires_grad=param.requires_grad),- )-- for name, buf in list(self.named_buffers()):- if name not in persistent:- continue- parent_name, _, attr = name.rpartition(".")- parent = self.get_submodule(parent_name) if parent_name else self- parent.register_buffer(attr, torch.empty_like(buf, device="meta"), persistent=True)+ for module in self.modules():+ for name, param in module._parameters.items():+ if param is not None:+ module._parameters[name] = nn.Parameter(+ torch.empty_like(param, device="meta"), requires_grad=param.requires_grad+ )+ for name, buf in module._buffers.items():+ if buf is not None and name not in module._non_persistent_buffers_set:+ module._buffers[name] = torch.empty_like(buf, device="meta")diff --git a/packages/ltx-core/src/ltx_core/model/transformer/__init__.py b/packages/ltx-core/src/ltx_core/model/transformer/__init__.pyindex 7d2936b..bd811f6 100644--- a/packages/ltx-core/src/ltx_core/model/transformer/__init__.py+++ b/packages/ltx-core/src/ltx_core/model/transformer/__init__.py@@ -1,6 +1,6 @@"""Transformer model components."""-from ltx_core.model.transformer.modality import Modality+from ltx_core.model.transformer.modality import Modality, TokenTimestepsfrom ltx_core.model.transformer.model import LTXModel, X0Modelfrom ltx_core.model.transformer.model_configurator import (LTXV_AUDIO_ONLY_MODEL_COMFY_RENAMING_MAP,@@ -18,5 +18,6 @@ __all__ = ["LTXModelConfigurator","LTXVideoOnlyModelConfigurator","Modality",+ "TokenTimesteps","X0Model",]diff --git a/packages/ltx-core/src/ltx_core/model/transformer/attention.py b/packages/ltx-core/src/ltx_core/model/transformer/attention.pyindex 4d108c7..c8e7f1f 100644--- a/packages/ltx-core/src/ltx_core/model/transformer/attention.py+++ b/packages/ltx-core/src/ltx_core/model/transformer/attention.py@@ -166,6 +166,13 @@ class FlashAttention3(AttentionCallable):return out+def _flash_attention_4_output(result: torch.Tensor | tuple[torch.Tensor, ...]) -> torch.Tensor:+ """Normalize FA4 return value: tensor (current releases) or legacy (out, aux) tuple."""+ if isinstance(result, tuple):+ return result[0]+ return result++class FlashAttention4(AttentionCallable):label = "FlashAttention4"@@ -186,7 +193,8 @@ class FlashAttention4(AttentionCallable):q, k, v = (t.view(b, -1, heads, dim_head) for t in (q, k, v))- out, _ = flash_attn_4_func(q.to(v.dtype), k.to(v.dtype), v)+ result = flash_attn_4_func(q.to(v.dtype), k.to(v.dtype), v)+ out = _flash_attention_4_output(result)out = out.reshape(b, -1, heads * dim_head)return out@@ -360,8 +368,12 @@ class AttentionFunction(Enum):Every non-AUTOMATIC variant raises :class:`RuntimeError` when the backendisn't usable on this machine -- missing package or SDPA backend rejectedon this hardware (e.g. cuDNN under ``torch.use_deterministic_algorithms``).- Opting in means "this kernel or fail loudly". ``AUTOMATIC`` returns the- cached :func:`automatic_attention` instance so every build shares one callable.+ Opting in means "this kernel or fail loudly", with one exception: under the+ ``ltx_kernels_inductor`` and ``ltx_kernels_eager`` compile backends, an unmasked+ call from ``PYTORCH`` or an ``SDPA_*`` variant runs ``ltx_kernels::sdpa_select``,+ which picks cuDNN or flash attention per shape regardless of the pinned backend.+ ``AUTOMATIC`` returns the cached :func:`automatic_attention` instance so every+ build shares one callable."""match self:case AttentionFunction.AUTOMATIC:diff --git a/packages/ltx-core/src/ltx_core/model/transformer/compiling.py b/packages/ltx-core/src/ltx_core/model/transformer/compiling.pyindex aa4d2dd..4c1b863 100644--- a/packages/ltx-core/src/ltx_core/model/transformer/compiling.py+++ b/packages/ltx-core/src/ltx_core/model/transformer/compiling.py@@ -11,7 +11,11 @@ from ltx_core.loader.module_ops import ModuleOpsfrom ltx_core.loader.sd_ops import SDOpsfrom ltx_core.model.transformer.cudagraph_capture import CudaGraphRunnerfrom ltx_core.model.transformer.model import LTXModel-from ltx_core.model.transformer.transformer_args import BlockPerturbationsProcessor, TransformerArgs+from ltx_core.model.transformer.transformer_args import (+ AdaLNTimestepEmbedding,+ BlockPerturbationsProcessor,+ TransformerArgs,+)# Defaults applied inside the patched forward. Overriding via CompilationConfig# replaces these wholesale; it does not merge.@@ -105,20 +109,20 @@ class CompiledBlockPerturbationsProcessor(BlockPerturbationsProcessor):torch._dynamo.mark_dynamic(args.self_attention_mask, 3)if args.context_mask is not None:torch._dynamo.mark_dynamic(args.context_mask, 2)- # `timesteps` / `embedded_timestep` are per-token when conditioning sets a- # per-position denoise mask, in which case their dim 1 equals the seq length- # and must vary with it. When they're a single timestep broadcast across the- # sequence (dim 1 == 1), leaving them static lets Dynamo keep the size-1- # broadcast.- if args.timesteps.shape[1] > 1:- torch._dynamo.mark_dynamic(args.timesteps, 1)- if args.embedded_timestep.shape[1] > 1:- torch._dynamo.mark_dynamic(args.embedded_timestep, 1)+ if args.timestep_embedding.token_indices is not None:+ torch._dynamo.mark_dynamic(args.timestep_embedding.token_indices, 1)+ # The table dimension is the sequence length for dense per-token conditioning+ # and the candidate count for indexed conditioning. Both can vary between+ # requests. A single row broadcasts and must remain static.+ if args.timestep_embedding.modulation.shape[1] > 1:+ torch._dynamo.mark_dynamic(args.timestep_embedding.modulation, 1)+ if args.timestep_embedding.embedding.shape[1] > 1:+ torch._dynamo.mark_dynamic(args.timestep_embedding.embedding, 1)# `cross_scale_shift_timestep` is the cross-attn AdaLN scale/shift input- # derived from the own-modality per-token timesteps (denoise_mask * sigma),- # so its dim 1 equals the seq length when conditioning is per-token.+ # derived from the own-modality timesteps, so its dim 1 is the sequence length+ # for dense conditioning and the candidate count for indexed conditioning.# `cross_gate_timestep` is the cross-modality sigma scalar -- dim 1 is 1- # and broadcasts, leave it static. Same guard pattern as `timesteps`.+ # and broadcasts, leave it static.if args.cross_scale_shift_timestep is not None and args.cross_scale_shift_timestep.shape[1] > 1:torch._dynamo.mark_dynamic(args.cross_scale_shift_timestep, 1)@@ -161,11 +165,12 @@ class CompiledBlockPerturbationsProcessor(BlockPerturbationsProcessor):def _register_transformer_args_pytree() -> None:- """Register ``TransformerArgs`` as a pytree so :class:`CudaGraphRunner` can flatten it to- tensor leaves for copy-in/out. Idempotent.+ """Register ``TransformerArgs`` and its ``AdaLNTimestepEmbedding`` as pytrees so+ :class:`CudaGraphRunner` can flatten them to tensor leaves for its capture key. Idempotent."""- if TransformerArgs not in pytree.SUPPORTED_NODES:- pytree.register_dataclass(TransformerArgs)+ for node in (TransformerArgs, AdaLNTimestepEmbedding):+ if node not in pytree.SUPPORTED_NODES:+ pytree.register_dataclass(node)@contextlib.contextmanagerdiff --git a/packages/ltx-core/src/ltx_core/model/transformer/cudagraph_capture.py b/packages/ltx-core/src/ltx_core/model/transformer/cudagraph_capture.pyindex 25f55c6..98dbd00 100644--- a/packages/ltx-core/src/ltx_core/model/transformer/cudagraph_capture.py+++ b/packages/ltx-core/src/ltx_core/model/transformer/cudagraph_capture.py@@ -12,13 +12,17 @@ from __future__ import annotationsimport dataclassesimport math-from collections.abc import Callable, Hashable+from collections.abc import Callable, Hashable, Iterator+from contextlib import contextmanager+from contextvars import ContextVarfrom typing import TYPE_CHECKINGimport torchimport torch.utils._pytree as pytreefrom ltx_core.guidance.perturbations import BatchedPerturbationConfig+from ltx_core.model.transformer.transformer_args import AdaLNTimestepEmbedding+from ltx_core.types import HostDeviceTensorPairif TYPE_CHECKING:# torch imports this under TYPE_CHECKING too (a runtime top-level import cycles).@@ -31,9 +35,25 @@ _BlockLoopFn = Callable["tuple[TransformerArgs | None, TransformerArgs | None]",]_ShapeKey = tuple[tuple[int, ...], ...]-# One capture identity: input shapes plus the block's perturbation graph signature (what the block-# specialises on), so a perturbed pass keys to -- and recaptures -- its own graph.-_CaptureKey = tuple[_ShapeKey, Hashable]+# One capture identity: input shapes, the block's perturbation graph signature (what the block+# specialises on), so a perturbed pass keys to -- and recaptures -- its own graph, and the+# enclosing `capture_key_scope`'s key.+_CaptureKey = tuple[_ShapeKey, Hashable, Hashable]++_scope_key: ContextVar[Hashable] = ContextVar("cudagraph_capture_scope_key", default=None)+++@contextmanager+def capture_key_scope(key: Hashable) -> Iterator[None]:+ """Add ``key`` to the capture identity of every :class:`CudaGraphRunner` call inside the scope.+ For host state the block loop branches on that its input shapes do not carry, such as the+ sequence-parallel padding layout: calls under different keys capture and replay their own graphs.+ """+ token = _scope_key.set(key)+ try:+ yield+ finally:+ _scope_key.reset(token)@dataclasses.dataclass@@ -132,6 +152,14 @@ class _InputBufferPool:return staticif isinstance(value, tuple):return tuple(self.copy_value_to_buffer(f"{field}.{i}", t, tokens) for i, t in enumerate(value))+ if isinstance(value, AdaLNTimestepEmbedding):+ return dataclasses.replace(+ value,+ **{+ f.name: self.copy_value_to_buffer(f"{field}.{f.name}", getattr(value, f.name), tokens)+ for f in dataclasses.fields(value)+ },+ )return valuedef copy_args_to_buffers(self, args: TransformerArgs) -> TransformerArgs:@@ -153,7 +181,9 @@ class _InputBufferPool:def copy_perturbations_to_buffers(self, perturbations: BatchedPerturbationConfig) -> BatchedPerturbationConfig:static_block_masks = self.copy_value_to_buffer("block_masks", perturbations.block_masks, tokens=None)assert isinstance(static_block_masks, torch.Tensor)- return BatchedPerturbationConfig.from_masks(static_block_masks, perturbations.block_masks_cpu)+ return BatchedPerturbationConfig.from_masks(+ HostDeviceTensorPair(host_tensor=perturbations.block_masks_cpu, device_tensor=static_block_masks)+ )class CudaGraphRunner:@@ -227,11 +257,12 @@ class CudaGraphRunner:def _cache_key(self, video: TransformerArgs | None, audio: TransformerArgs | None, perturbations: BatchedPerturbationConfig) -> _CaptureKey:- """Capture identity: input shapes + the block's perturbation graph signature. Keying on the- signature makes a pass recapture exactly when it would recompile (and no more) -- the perturbed- pass no longer silently replays the clean same-shape graph."""+ """Capture identity: input shapes + the block's perturbation graph signature + the enclosing+ :func:`capture_key_scope` key. Keying on the signature makes a pass recapture exactly when it+ would recompile (and no more) -- the perturbed pass no longer silently replays the clean+ same-shape graph."""shape = _shape_key(video, audio, perturbations.block_masks)- return shape, self._block_input_processor.graph_signature(perturbations)+ return shape, self._block_input_processor.graph_signature(perturbations), _scope_key.get()@staticmethoddef _configure_shared_input_pool(pool: _InputBufferPool | None, max_tokens: int) -> _InputBufferPool:diff --git a/packages/ltx-core/src/ltx_core/model/transformer/modality.py b/packages/ltx-core/src/ltx_core/model/transformer/modality.pyindex 872307d..ec44fcd 100644--- a/packages/ltx-core/src/ltx_core/model/transformer/modality.py+++ b/packages/ltx-core/src/ltx_core/model/transformer/modality.py@@ -6,6 +6,56 @@ from dataclasses import dataclassimport torch+@dataclass(frozen=True)+class TokenTimesteps:+ """Timestep values and the optional per-token index into their compact table.+ ``values`` is ``(B, T)`` when dense or ``(B, K)`` when ``indices`` supplies+ the ``(B, T)`` token layout. ``zero_index`` is the column of a compact table known, on the+ host, to hold timestep zero in every row, or ``None`` when the table has no such column.+ """++ values: torch.Tensor+ indices: torch.Tensor | None = None+ zero_index: int | None = None++ def per_token(self) -> torch.Tensor:+ """The timestep of each token, with a trailing singleton dimension."""+ if self.indices is None:+ return self.values.unsqueeze(-1) if self.values.ndim == 2 else self.values+ return torch.gather(self.values, 1, self.indices.to(torch.int64)).unsqueeze(-1)++ def select_tokens(self, index: slice | torch.Tensor) -> TokenTimesteps:+ if self.indices is None:+ return TokenTimesteps(self.values[:, index])+ return TokenTimesteps(self.values, self.indices[:, index], self.zero_index)++ def pad_tokens(self, pad: int) -> TokenTimesteps:+ """Append ``pad`` tokens with timestep zero.+ Padded tokens stay visible as keys to video-to-audio cross-attention, so their timestep+ reaches real outputs and must match dense padding. An indexed table keeps its shape when+ ``zero_index`` names a zero column, and the padded tokens point at it; otherwise it gains+ one zero column, which becomes ``zero_index``.+ """+ if self.indices is None:+ pad_shape = list(self.values.shape)+ pad_shape[1] = pad+ return TokenTimesteps(torch.cat([self.values, self.values.new_zeros(pad_shape)], dim=1))+ values, zero_index = self.values, self.zero_index+ if zero_index is None:+ values = torch.cat([values, values.new_zeros(values.shape[0], 1)], dim=1)+ zero_index = self.values.shape[1]+ pad_indices = self.indices.new_full((self.indices.shape[0], pad), zero_index)+ return TokenTimesteps(values, torch.cat([self.indices, pad_indices], dim=1), zero_index)++ def split(self, sizes: list[int]) -> list[TokenTimesteps]:+ values = self.values.split(sizes, dim=0)+ indices = self.indices.split(sizes, dim=0) if self.indices is not None else [None] * len(sizes)+ return [+ TokenTimesteps(chunk_values, chunk_indices, self.zero_index)+ for chunk_values, chunk_indices in zip(values, indices, strict=True)+ ]++@dataclass(frozen=True)class Modality:"""@@ -16,7 +66,7 @@ class Modality:latent: Patchified latent tokens, shape ``(B, T, D)`` where *B* isthe batch size, *T* is the total number of tokens (noisy +conditioning), and *D* is the input dimension.- timesteps: Per-token timestep embeddings, shape ``(B, T)``.+ timesteps: Per-token timestep values or a compact table and its token indices.positions: Per-token patch coordinates used to build the RoPEfrequencies. With the default ``use_middle_indices_grid=True``,shape is ``(B, n_pos_dims, T, 2)`` where ``n_pos_dims=3`` for@@ -38,8 +88,8 @@ class Modality:attention. ``None`` means unrestricted (full) attention betweenall tokens. Built incrementally by conditioning items; see:class:`~ltx_core.conditioning.types.attention_strength_wrapper.ConditioningItemAttentionStrengthWrapper`.- keyframes_mask: Optional per-token marker, shape ``(B, T, 1)`` -- the same layout as- ``timesteps`` -- non-zero for tokens whose latent encodes a *single standalone pixel+ keyframes_mask: Optional per-token marker, shape ``(B, T, 1)`` -- aligned with the latent+ token axis -- non-zero for tokens whose latent encodes a *single standalone pixelframe* rather than the usual multi-frame span. That is the target's first latent frame(the video encoder is causal) plus any generated keyframe slots. Selects the tokensthat receive the model's learned keyframe absolute-position embedding. ``None`` means@@ -51,7 +101,7 @@ class Modality:torch.Tensor) # Shape: (B, T, D) where B is the batch size, T is the number of tokens, and D is input dimensionsigma: torch.Tensor # Shape: (B,). Current sigma value, used for cross-attention timestep calculation.- timesteps: torch.Tensor # Shape: (B, T) where T is the number of timesteps+ timesteps: TokenTimesteps# Shape: (B, n_pos_dims, T, 2) by default (use_middle_indices_grid=True);# n_pos_dims=3 for video, 1 for audio; last dim holds [start, end) patch bounds.# Legacy form (B, n_pos_dims, T) when use_middle_indices_grid=False.@@ -65,11 +115,13 @@ class Modality:def split(self, sizes: list[int]) -> list[Modality]:"""Split along the batch dimension into chunks of the given sizes."""n = len(sizes)- split_fields: dict[str, list[torch.Tensor | None] | list[bool]] = {}+ split_fields: dict[str, list[torch.Tensor | TokenTimesteps | None] | list[bool]] = {}for f in dataclasses.fields(self):value = getattr(self, f.name)if isinstance(value, torch.Tensor):split_fields[f.name] = list(value.split(sizes, dim=0))+ elif isinstance(value, TokenTimesteps):+ split_fields[f.name] = value.split(sizes)elif value is None or isinstance(value, bool):split_fields[f.name] = [value] * nelse:diff --git a/packages/ltx-core/src/ltx_core/model/transformer/model.py b/packages/ltx-core/src/ltx_core/model/transformer/model.pyindex 9783d5b..5def3b9 100644--- a/packages/ltx-core/src/ltx_core/model/transformer/model.py+++ b/packages/ltx-core/src/ltx_core/model/transformer/model.py@@ -12,6 +12,7 @@ from ltx_core.model.transformer.modality import Modalityfrom ltx_core.model.transformer.rope import LTXRopeTypefrom ltx_core.model.transformer.transformer import BasicAVTransformerBlock, TransformerConfigfrom ltx_core.model.transformer.transformer_args import (+ AdaLNTimestepEmbedding,BlockPerturbationsProcessor,MultiModalTransformerArgsPreprocessor,TransformerArgs,@@ -471,9 +472,10 @@ class LTXModel(torch.nn.Module, Disposable):norm_out: torch.nn.LayerNorm,proj_out: torch.nn.Linear,x: torch.Tensor,- embedded_timestep: torch.Tensor,+ timestep_embedding: AdaLNTimestepEmbedding,) -> torch.Tensor:"""Process output for LTXV."""+ embedded_timestep = timestep_embedding.per_token_embedding()# Apply scale-shift modulationscale_shift_values = (scale_shift_table[None, None].to(device=x.device, dtype=x.dtype) + embedded_timestep[:, :, None]@@ -515,7 +517,11 @@ class LTXModel(torch.nn.Module, Disposable):# Process outputvx = (self._process_output(- self.scale_shift_table, self.norm_out, self.proj_out, video_out.x, video_out.embedded_timestep+ self.scale_shift_table,+ self.norm_out,+ self.proj_out,+ video_out.x,+ video_out.timestep_embedding,)if video_out is not Noneelse None@@ -526,7 +532,7 @@ class LTXModel(torch.nn.Module, Disposable):self.audio_norm_out,self.audio_proj_out,audio_out.x,- audio_out.embedded_timestep,+ audio_out.timestep_embedding,)if audio_out is not Noneelse None@@ -595,6 +601,6 @@ class X0Model(torch.nn.Module, Disposable):Denoised video and audio"""vx, ax = self.velocity_model(video, audio, perturbations)- denoised_video = to_denoised(video.latent, vx, video.timesteps) if vx is not None else None- denoised_audio = to_denoised(audio.latent, ax, audio.timesteps) if ax is not None else None+ denoised_video = to_denoised(video.latent, vx, video.timesteps.per_token()) if vx is not None else None+ denoised_audio = to_denoised(audio.latent, ax, audio.timesteps.per_token()) if ax is not None else Nonereturn denoised_video, denoised_audiodiff --git a/packages/ltx-core/src/ltx_core/model/transformer/transformer.py b/packages/ltx-core/src/ltx_core/model/transformer/transformer.pyindex 02ab32d..4d5983c 100644--- a/packages/ltx-core/src/ltx_core/model/transformer/transformer.py+++ b/packages/ltx-core/src/ltx_core/model/transformer/transformer.py@@ -6,7 +6,7 @@ from ltx_core.model.transformer.adaln import adaln_embedding_coefficientfrom ltx_core.model.transformer.attention import Attention, AttentionCallable, AttentionOpsfrom ltx_core.model.transformer.feed_forward import FeedForwardfrom ltx_core.model.transformer.rope import LTXRopeType-from ltx_core.model.transformer.transformer_args import TransformerArgs+from ltx_core.model.transformer.transformer_args import AdaLNTimestepEmbedding, TransformerArgsfrom ltx_core.utils import rms_norm@@ -124,31 +124,45 @@ class BasicAVTransformerBlock(torch.nn.Module):self.norm_eps = norm_epsdef get_ada_values(- self, scale_shift_table: torch.Tensor, batch_size: int, timestep: torch.Tensor, indices: slice+ self,+ scale_shift_table: torch.Tensor,+ batch_size: int,+ timestep_embedding: AdaLNTimestepEmbedding,+ indices: slice,+ modulation: torch.Tensor | None = None,) -> tuple[torch.Tensor, ...]:num_ada_params = scale_shift_table.shape[0]-- ada_values = (- scale_shift_table[indices].unsqueeze(0).unsqueeze(0).to(device=timestep.device, dtype=timestep.dtype)- + timestep.reshape(batch_size, timestep.shape[1], num_ada_params, -1)[:, :, indices, :]+ hidden_dim = scale_shift_table.shape[1]+ modulation = timestep_embedding.modulation if modulation is None else modulation+ modulation = modulation.reshape(batch_size, -1, num_ada_params, hidden_dim)[:, :, indices, :]+ modulation = timestep_embedding.gather(modulation)+ return (+ scale_shift_table[indices].unsqueeze(0).unsqueeze(0).to(device=modulation.device, dtype=modulation.dtype)+ + modulation).unbind(dim=2)- return ada_valuesdef get_av_ca_ada_values(self,scale_shift_table: torch.Tensor,batch_size: int,+ timestep_embedding: AdaLNTimestepEmbedding,scale_shift_timestep: torch.Tensor,gate_timestep: torch.Tensor,scale_shift_indices: slice,num_scale_shift_values: int = 4,) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:scale_shift_ada_values = self.get_ada_values(- scale_shift_table[:num_scale_shift_values, :], batch_size, scale_shift_timestep, scale_shift_indices+ scale_shift_table[:num_scale_shift_values, :],+ batch_size,+ timestep_embedding,+ scale_shift_indices,+ scale_shift_timestep,)- gate_ada_values = self.get_ada_values(- scale_shift_table[num_scale_shift_values:, :], batch_size, gate_timestep, slice(None, None)+ gate_table = scale_shift_table[num_scale_shift_values:, :].to(+ device=gate_timestep.device, dtype=gate_timestep.dtype)+ gate_modulation = gate_timestep.reshape(batch_size, -1, 1, gate_table.shape[-1])+ gate_ada_values = (gate_table[None, None] + gate_modulation).unbind(dim=2)scale, shift = (t.squeeze(2) for t in scale_shift_ada_values)(gate,) = (t.squeeze(2) for t in gate_ada_values)@@ -162,7 +176,7 @@ class BasicAVTransformerBlock(torch.nn.Module):attn: AttentionCallable,scale_shift_table: torch.Tensor,prompt_scale_shift_table: torch.Tensor | None,- timestep: torch.Tensor,+ timestep_embedding: AdaLNTimestepEmbedding,prompt_timestep: torch.Tensor | None,context_mask: torch.Tensor | None,cross_attention_adaln: bool = False,@@ -172,7 +186,10 @@ class BasicAVTransformerBlock(torch.nn.Module):the PostSA composite); this method does not normalize again."""if cross_attention_adaln:- shift_q, scale_q, gate = self.get_ada_values(scale_shift_table, x_normed.shape[0], timestep, slice(6, 9))+ shift_q, scale_q = self.get_ada_values(+ scale_shift_table, x_normed.shape[0], timestep_embedding, slice(6, 8)+ )+ (gate,) = self.get_ada_values(scale_shift_table, x_normed.shape[0], timestep_embedding, slice(8, 9))return apply_cross_attention_adaln(x_normed,context,@@ -205,7 +222,7 @@ class BasicAVTransformerBlock(torch.nn.Module):if run_vx:vshift_msa, vscale_msa, vgate_msa = self.get_ada_values(- self.scale_shift_table, vx.shape[0], video.timesteps, slice(0, 3)+ self.scale_shift_table, vx.shape[0], video.timestep_embedding, slice(0, 3))norm_vx = rms_norm(vx, eps=self.norm_eps) * (1 + vscale_msa) + vshift_msadel vshift_msa, vscale_msa@@ -226,7 +243,7 @@ class BasicAVTransformerBlock(torch.nn.Module):self.attn2,self.scale_shift_table,getattr(self, "prompt_scale_shift_table", None),- video.timesteps,+ video.timestep_embedding,video.prompt_timestep,video.context_mask,cross_attention_adaln=self.cross_attention_adaln,@@ -235,7 +252,7 @@ class BasicAVTransformerBlock(torch.nn.Module):if run_ax:ashift_msa, ascale_msa, agate_msa = self.get_ada_values(- self.audio_scale_shift_table, ax.shape[0], audio.timesteps, slice(0, 3)+ self.audio_scale_shift_table, ax.shape[0], audio.timestep_embedding, slice(0, 3))norm_ax = rms_norm(ax, eps=self.norm_eps) * (1 + ascale_msa) + ashift_msa@@ -256,7 +273,7 @@ class BasicAVTransformerBlock(torch.nn.Module):self.audio_attn2,self.audio_scale_shift_table,getattr(self, "audio_prompt_scale_shift_table", None),- audio.timesteps,+ audio.timestep_embedding,audio.prompt_timestep,audio.context_mask,cross_attention_adaln=self.cross_attention_adaln,@@ -273,6 +290,7 @@ class BasicAVTransformerBlock(torch.nn.Module):scale_ca_video_a2v, shift_ca_video_a2v, gate_out_a2v = self.get_av_ca_ada_values(self.scale_shift_table_a2v_ca_video,vx.shape[0],+ video.timestep_embedding,video.cross_scale_shift_timestep,video.cross_gate_timestep,slice(0, 2),@@ -283,6 +301,7 @@ class BasicAVTransformerBlock(torch.nn.Module):scale_ca_audio_a2v, shift_ca_audio_a2v, _ = self.get_av_ca_ada_values(self.scale_shift_table_a2v_ca_audio,ax.shape[0],+ audio.timestep_embedding,audio.cross_scale_shift_timestep,audio.cross_gate_timestep,slice(0, 2),@@ -305,6 +324,7 @@ class BasicAVTransformerBlock(torch.nn.Module):scale_ca_audio_v2a, shift_ca_audio_v2a, gate_out_v2a = self.get_av_ca_ada_values(self.scale_shift_table_a2v_ca_audio,ax.shape[0],+ audio.timestep_embedding,audio.cross_scale_shift_timestep,audio.cross_gate_timestep,slice(2, 4),@@ -314,6 +334,7 @@ class BasicAVTransformerBlock(torch.nn.Module):scale_ca_video_v2a, shift_ca_video_v2a, _ = self.get_av_ca_ada_values(self.scale_shift_table_a2v_ca_video,vx.shape[0],+ video.timestep_embedding,video.cross_scale_shift_timestep,video.cross_gate_timestep,slice(2, 4),@@ -335,7 +356,7 @@ class BasicAVTransformerBlock(torch.nn.Module):if run_vx:vshift_mlp, vscale_mlp, vgate_mlp = self.get_ada_values(- self.scale_shift_table, vx.shape[0], video.timesteps, slice(3, 6)+ self.scale_shift_table, vx.shape[0], video.timestep_embedding, slice(3, 6))vx_scaled = rms_norm(vx, eps=self.norm_eps) * (1 + vscale_mlp) + vshift_mlpvx = vx + self.ff(vx_scaled) * vgate_mlp@@ -344,7 +365,7 @@ class BasicAVTransformerBlock(torch.nn.Module):if run_ax:ashift_mlp, ascale_mlp, agate_mlp = self.get_ada_values(- self.audio_scale_shift_table, ax.shape[0], audio.timesteps, slice(3, 6)+ self.audio_scale_shift_table, ax.shape[0], audio.timestep_embedding, slice(3, 6))ax_scaled = rms_norm(ax, eps=self.norm_eps) * (1 + ascale_mlp) + ashift_mlpax = ax + self.audio_ff(ax_scaled) * agate_mlpdiff --git a/packages/ltx-core/src/ltx_core/model/transformer/transformer_args.py b/packages/ltx-core/src/ltx_core/model/transformer/transformer_args.pyindex 566bbd3..f5ff226 100644--- a/packages/ltx-core/src/ltx_core/model/transformer/transformer_args.py+++ b/packages/ltx-core/src/ltx_core/model/transformer/transformer_args.py@@ -43,13 +43,35 @@ def apply_keyframes_absolute_embedding(return hidden_states + mask * embedding.to(dtype=hidden_states.dtype)+@dataclass(frozen=True)+class AdaLNTimestepEmbedding:+ """Projected AdaLN timesteps and the token index that selects their rows.+ Transformer blocks consume ``modulation`` and carry ``embedding`` to the model+ output head, which applies the final scale and shift.+ Compact-table indices have the batch offset folded in and address the first two+ dimensions of any projected table passed to ``gather``.+ """++ modulation: torch.Tensor+ embedding: torch.Tensor+ token_indices: torch.Tensor | None = None++ def gather(self, values: torch.Tensor) -> torch.Tensor:+ """Expand a compact table to one row per token."""+ if self.token_indices is None:+ return values+ return values.reshape(-1, *values.shape[2:])[self.token_indices]++ def per_token_embedding(self) -> torch.Tensor:+ return self.gather(self.embedding)++@dataclass(frozen=True)class TransformerArgs:x: torch.Tensorcontext: torch.Tensorcontext_mask: torch.Tensor- timesteps: torch.Tensor- embedded_timestep: torch.Tensor+ timestep_embedding: AdaLNTimestepEmbeddingpositional_embeddings: tuple[torch.Tensor, torch.Tensor]cross_positional_embeddings: tuple[torch.Tensor, torch.Tensor] | Nonecross_scale_shift_timestep: torch.Tensor | None@@ -226,14 +248,13 @@ class TransformerArgsPreprocessor:# Convert [0, 1] attention mask to additive log-space bias:# 1.0 -> log(1.0) = 0.0 (no bias, full attention)# 0.0 -> finfo.min (fully masked)+ # Must stay elementwise: boolean indexing would sync the host on every forward.finfo = torch.finfo(x_dtype)- eps = finfo.tiny-- bias = torch.full_like(attention_mask, finfo.min, dtype=x_dtype)- positive = attention_mask > 0- if positive.any():- bias[positive] = torch.log(attention_mask[positive].clamp(min=eps)).to(x_dtype)-+ bias = torch.where(+ attention_mask > 0,+ torch.log(attention_mask.clamp(min=finfo.tiny)).to(x_dtype),+ torch.full_like(attention_mask, finfo.min, dtype=x_dtype),+ )return bias.unsqueeze(1) # (B, 1, T_q, T_k) for head broadcastdef _prepare_positional_embeddings(@@ -269,8 +290,15 @@ class TransformerArgsPreprocessor:x = apply_keyframes_absolute_embedding(x, modality.keyframes_mask, self.keyframes_embedding_provider)batch_size = x.shape[0]timestep, embedded_timestep = self._prepare_timestep(- modality.timesteps, self.adaln, batch_size, modality.latent.dtype+ modality.timesteps.values, self.adaln, batch_size, modality.latent.dtype)+ timestep_indices = modality.timesteps.indices+ if timestep_indices is not None:+ # Fold the batch into the index so a consumer resolves batch and value with one+ # gather over the table flattened to (B * K, ...). Every consumer of+ # `AdaLNTimestepEmbedding` reads them in that flattened form.+ batch_offsets = torch.arange(batch_size, device=x.device, dtype=torch.int32)+ timestep_indices = timestep_indices + batch_offsets[:, None] * timestep.shape[1]prompt_timestep = Noneif self.prompt_adaln is not None:prompt_timestep, _ = self._prepare_timestep(@@ -291,8 +319,7 @@ class TransformerArgsPreprocessor:x=x,context=context,context_mask=attention_mask,- timesteps=timestep,- embedded_timestep=embedded_timestep,+ timestep_embedding=AdaLNTimestepEmbedding(timestep, embedded_timestep, timestep_indices),positional_embeddings=pe,cross_positional_embeddings=None,cross_scale_shift_timestep=None,@@ -356,7 +383,7 @@ class MultiModalTransformerArgsPreprocessor:return transformer_argsif cross_modality.sigma.numel() > 1:- if cross_modality.sigma.shape[0] != modality.timesteps.shape[0]:+ if cross_modality.sigma.shape[0] != modality.timesteps.values.shape[0]:raise ValueError("Cross modality sigma must have the same batch size as the modality")if cross_modality.sigma.ndim != 1:raise ValueError("Cross modality sigma must be a 1D tensor")@@ -371,7 +398,7 @@ class MultiModalTransformerArgsPreprocessor:)cross_scale_shift_timestep, cross_gate_timestep = self._prepare_cross_attention_timestep(- modality_timesteps=modality.timesteps,+ modality_timesteps=modality.timesteps.values,cross_modality_sigma=cross_modality.sigma,timestep_scale_multiplier=self.simple_preprocessor.timestep_scale_multiplier,batch_size=transformer_args.x.shape[0],diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/convolution.py b/packages/ltx-core/src/ltx_core/model/video_vae/convolution.pyindex 7c3957a..6003e3e 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/convolution.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/convolution.py@@ -8,6 +8,25 @@ from torch.nn import functional as Ffrom ltx_core.model.video_vae.enums import PaddingModeType+def memory_format_of(t: torch.Tensor, prefer_channels_last_3d: bool = False) -> torch.memory_format:+ """Pick the memory format for a workspace allocation.+ When ``prefer_channels_last_3d`` is True and ``t`` is 5D, return+ ``channels_last_3d`` regardless of ``t``'s current strides -- the+ workspace's ``.copy_(t)`` will transcribe the data into the new layout.+ This is needed because intermediate tensors inside the memory-efficient decoder (after+ ``rearrange`` + slice + residual add in ``_upsample_forward_efficient``)+ are not NHWC-contiguous, so an auto-detect helper would silently fall+ back to NCHW for every workspace after the first upsample.+ Otherwise fall back to inspecting ``t``: ``channels_last_3d`` if ``t``+ already uses it, else contiguous.+ """+ if prefer_channels_last_3d and t.dim() == 5:+ return torch.channels_last_3d+ if t.dim() == 5 and t.is_contiguous(memory_format=torch.channels_last_3d):+ return torch.channels_last_3d+ return torch.contiguous_format++def make_conv_nd( # noqa: PLR0913dims: Union[int, Tuple[int, int]],in_channels: int,@@ -302,15 +321,23 @@ class CausalConv3d(nn.Module):)def forward(self, x: torch.Tensor, causal: bool = True) -> torch.Tensor:- if causal:- first_frame_pad = x[:, :, :1, :, :].repeat((1, 1, self.time_kernel_size - 1, 1, 1))- x = torch.concatenate((first_frame_pad, x), dim=2)- else:- first_frame_pad = x[:, :, :1, :, :].repeat((1, 1, (self.time_kernel_size - 1) // 2, 1, 1))- last_frame_pad = x[:, :, -1:, :, :].repeat((1, 1, (self.time_kernel_size - 1) // 2, 1, 1))- x = torch.concatenate((first_frame_pad, x, last_frame_pad), dim=2)- x = self.conv(x)- return x+ front = self.time_kernel_size - 1 if causal else (self.time_kernel_size - 1) // 2+ back = 0 if causal else front+ if front == 0 and back == 0:+ return self.conv(x)+ # Edge frames are replicated into one buffer laid out like ``x``, so channels-last+ # activations reach the conv without a layout round trip.+ frames = x.shape[2]+ padded = torch.empty(+ (*x.shape[:2], front + frames + back, *x.shape[3:]),+ dtype=x.dtype,+ device=x.device,+ memory_format=memory_format_of(x),+ )+ padded[:, :, front : front + frames] = x+ padded[:, :, :front] = x[:, :, :1]+ padded[:, :, front + frames :] = x[:, :, -1:]+ return self.conv(padded)@propertydef weight(self) -> torch.Tensor:diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/decode_workspace.py b/packages/ltx-core/src/ltx_core/model/video_vae/decode_workspace.pynew file mode 100644index 0000000..51b20e5--- /dev/null+++ b/packages/ltx-core/src/ltx_core/model/video_vae/decode_workspace.py@@ -0,0 +1,91 @@+"""The DiffVAE decoder's view of the per-decode buffer workspace (:mod:`ltx_kernels.vae.workspace`).+The workspace exists only with ``ltx-kernels`` installed, the fused stage-5 blocks it serves; without it+every helper here is a plain allocation and :func:`tile_workspace` returns ``None``.+"""++from __future__ import annotations++import contextlib+from functools import cache+from types import ModuleType++import torch+++@cache+def _workspace_module() -> ModuleType | None:+ try:+ from ltx_kernels.vae import workspace # noqa: PLC0415+ except ImportError:+ return None+ return workspace+++def available() -> bool:+ """Whether ``ltx-kernels`` (whose fused blocks the workspace serves) is importable."""+ return _workspace_module() is not None+++def out_slack_rows() -> int:+ """Rows a fused-block output buffer carries past its volume (``ltx_kernels.vae.OUT_SLACK_ROWS``)."""+ from ltx_kernels.vae import OUT_SLACK_ROWS # noqa: PLC0415++ return int(OUT_SLACK_ROWS)+++def workspace_active() -> bool:+ """Whether a decode workspace is active on this thread."""+ ws = _workspace_module()+ return ws is not None and ws.active_decode_workspace() is not None+++def volume(shape: tuple[int, ...], *, dtype: torch.dtype, device: torch.device) -> torch.Tensor:+ """``torch.empty(shape)`` leased from the active workspace until its tile ends."""+ ws = _workspace_module()+ if ws is None:+ return torch.empty(shape, dtype=dtype, device=device)+ return ws.workspace_volume(shape, dtype=dtype, device=device)+++def phase_empty(+ shape: tuple[int, ...],+ *,+ dtype: torch.dtype,+ device: torch.device,+ not_aliasing: tuple[torch.Tensor, ...] = (),+) -> torch.Tensor:+ """``torch.empty(shape)`` carved from the active workspace's idle volumes (a plain allocation otherwise)."""+ ws = _workspace_module()+ if ws is None:+ return torch.empty(shape, dtype=dtype, device=device)+ return ws.workspace_phase_empty(shape, dtype=dtype, device=device, not_aliasing=not_aliasing)+++def phase_reset() -> None:+ """Every phase tensor is dead: the idle volumes may be carved again."""+ ws = _workspace_module()+ if ws is not None:+ ws.workspace_phase_reset()+++def release_tile() -> None:+ """A tile's leased volumes are free for the next tile."""+ ws = _workspace_module()+ active = None if ws is None else ws.active_decode_workspace()+ if active is not None:+ active.release_tile()+++def tile_workspace(large_volume_numel: int) -> object | None:+ """A workspace for one decode whose large volume slots hold ``large_volume_numel`` elements, or ``None``+ without the fused kernels it serves."""+ ws = _workspace_module()+ return None if ws is None else ws.DecodeWorkspace(large_volume_numel=large_volume_numel)+++def scope(workspace: object | None) -> contextlib.AbstractContextManager:+ """Activate ``workspace`` (from :func:`tile_workspace`) for the block; a null context for ``None``."""+ ws = _workspace_module()+ if workspace is None or ws is None:+ return contextlib.nullcontext(None)+ return ws.decode_workspace(workspace)diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_tiling.py b/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_tiling.pyindex 3f9523e..5280308 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_tiling.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_tiling.py@@ -6,6 +6,7 @@ geometry/schedule/mask pieces that tiling uses.from __future__ import annotationsimport itertools+import loggingimport mathfrom dataclasses import dataclassfrom typing import List, Literal, Sequence, Tuple@@ -30,6 +31,8 @@ from ltx_core.tiling import ()from ltx_core.types import VIDEO_SCALE_FACTORS, SpatioTemporalScaleFactors+logger = logging.getLogger(__name__)+ResizeAxisMode = Literal["repeat_last", "symmetric"]_GIB: int = 1 << 30@@ -65,12 +68,13 @@ class _StageFiveBudget:# the fusion it holds several live full-volume intermediates instead of one Inductor buffer,# measured as 11 -> 15.# - BLACKWELL_DSL: the CuTe kernel has its own joint entry point, so the fused path serves-# keyframes too and nothing changes.+# keyframes too, but the keyframe decode's eager det blocks and its keyframe planes raise the+# measured working set to 4.0 (2.5 leaves a streamed multi-GPU decode out of memory warm)._BUDGET_BY_MODE: dict[DiffVAEMode, _StageFiveBudget] = {DiffVAEMode.COMBINED_COMPILE: _StageFiveBudget(coef=11, coef_keyframes=15, reserve_bytes=2 * _GIB),DiffVAEMode.CHUNKED_COMPILE: _StageFiveBudget(coef=7, coef_keyframes=5, reserve_bytes=2 * _GIB),DiffVAEMode.CHUNKED_EAGER: _StageFiveBudget(coef=5, coef_keyframes=5, reserve_bytes=1 * _GIB),- DiffVAEMode.BLACKWELL_DSL: _StageFiveBudget(coef=2.5, coef_keyframes=2.5, reserve_bytes=2 * _GIB),+ DiffVAEMode.BLACKWELL_DSL: _StageFiveBudget(coef=2.5, coef_keyframes=4.0, reserve_bytes=2 * _GIB),}_DEFAULT_ELEMENT_SIZE: int = 2 # bf16 features → fp16 accumulator / bf16 stage-5_ACCUMULATOR_CHANNELS: int = 3 # RGB pixel blend buffer (decoder out_channels)@@ -303,7 +307,7 @@ def recommended_decode_tiling_config( # noqa: PLR0913production bf16 features use fp16 accumulators (2), matching:func:`accumulator_element_size`. Stage-5 uses the same element size x``stage5_channels`` x ``coef``, which :func:`stage5_mem_coef` reads off the- mode and ``keyframes`` (11 / 7 / 5 / 2.5 by mode; 15 / 5 / 5 / 2.5 with a+ mode and ``keyframes`` (11 / 7 / 5 / 2.5 by mode; 15 / 5 / 5 / 4.0 with akeyframe stream, which runs eager blocks).Args beyond the geometry:keyframes: this decode carries a keyframe stream (joint attention, eager blocks).@@ -411,6 +415,17 @@ def recommended_decode_tiling_config( # noqa: PLR0913scored.sort()_waste, _vol, _ntiles, tile_t, tile_h, tile_w = scored[0]+ logger.info(+ "DiffVAE AUTO tiling: %df x %dx%d (overlap T=%d HW=%d) mode=%s keyframes=%s free_bytes=%d",+ tile_t,+ tile_h,+ tile_w,+ overlap_t,+ overlap_hw,+ mode.value,+ keyframes,+ int(free_bytes),+ )return TileSizeConfig(frames=DimensionSizeConfig(tile_size=tile_t, overlap=overlap_t),height=DimensionSizeConfig(tile_size=tile_h, overlap=overlap_hw),diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_video_decoder.py b/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_video_decoder.pyindex 1ffd659..98567eb 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_video_decoder.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/diffusion_video_decoder.py@@ -10,18 +10,25 @@ from typing import Iterator, List, Literal, Tupleimport torchfrom torch import nn+from ltx_core.devices import upload_from_hostfrom ltx_core.model.disposable import Disposablefrom ltx_core.model.transformer.timestep_embedding import PixArtAlphaCombinedTimestepSizeEmbeddings-from ltx_core.model.video_vae import diffusion_tiling+from ltx_core.model.video_vae import decode_workspace, diffusion_tilingfrom ltx_core.model.video_vae.keyframes import (DecodeKeyframes,+ KeyframePlanes,KeyframeStream,keyframe_clip_times,planes_for_tile,remaining_time_strides,upsample_keyframe_planes,)-from ltx_core.model.video_vae.ops import PerChannelStatistics, patchify, unpatchify+from ltx_core.model.video_vae.ops import (+ PerChannelStatistics,+ apply_rms_norm_uint32_safe,+ patchify_channels_last,+ unpatchify,+)from ltx_core.model.video_vae.transformer import (AdaLNZero,ChannelLinear,@@ -29,6 +36,8 @@ from ltx_core.model.video_vae.transformer import (LinearPixelShuffleUpsample,NABlock,)+from ltx_core.model.video_vae.transformer.rope_math import abs_rope_table_cache+from ltx_core.model.video_vae.transformer.stage5_tail import stage5_tail, stage5_tail_supportedfrom ltx_core.model.video_vae.video_vae import VideoDecoder, iter_decoded_single_framesfrom ltx_core.tiling import (Tile,@@ -68,6 +77,52 @@ _DIFF_STAGE5_KERNEL_DEFAULT: Tuple[int, int, int] = (3, 7, 7)_DIFF_STAGE5_DEPTH_DEFAULT: int = 8_DIFF_STAGE_DEPTHS_DEFAULT: Tuple[int, ...] = (*_L_STAGE_DEPTHS[:-1], _DIFF_STAGE5_DEPTH_DEFAULT)+# Rows of the stage-5 volume normalised per in-place ``norm_out`` chunk (unfused tail).+_NORM_OUT_CHUNK_ROWS: int = 1 << 20+# Pixel bytes per ``conv_out`` / unpatchify chunk (unfused tail) and per tile-blend chunk.+_TAIL_CHUNK_BYTES: int = 1 << 29+++def _release_cached_blocks(device: torch.device) -> None:+ """Return the caching allocator's free segments, so one decode phase's churn is not reserved on top of+ the next one's."""+ if device.type == "cuda":+ torch.cuda.synchronize(device)+ torch.cuda.empty_cache()+++def _blend_tile_into(+ buffer: torch.Tensor,+ weights: torch.Tensor | None,+ local_coords: tuple,+ pixel_tile: torch.Tensor,+ masks: tuple,+) -> None:+ """The masked accumulate of ``pixel_tile`` into ``buffer`` (and ``weights``), one temporal chunk at a time.+ Per-element masks and accumulate, so chunking reproduces the whole-tile blend bit for bit.+ """+ frames = int(pixel_tile.shape[2])+ t_lo, t_hi, _ = local_coords[2].indices(buffer.shape[2])+ if t_hi - t_lo != frames:+ raise RuntimeError(f"tile covers {frames} frames but its buffer slice spans {t_hi - t_lo}")+ frame_bytes = pixel_tile[:, :, :1].numel() * 4+ chunk_t = max(1, min(frames, _TAIL_CHUNK_BYTES // max(frame_bytes, 1)))+ for f0 in range(0, frames, chunk_t):+ f1 = min(frames, f0 + chunk_t)+ coords = (local_coords[0], local_coords[1], slice(t_lo + f0, t_lo + f1), local_coords[3], local_coords[4])+ chunk_masks = (+ masks[0],+ masks[1],+ masks[2][f0:f1] if masks[2].numel() > 1 else masks[2],+ masks[3],+ masks[4],+ )+ chunk = pixel_tile[:, :, f0:f1].to(buffer.dtype)+ buffer[coords] += scale_by_masks_1d(chunk, chunk_masks)+ if weights is not None:+ strength = torch.ones(chunk.shape, device=buffer.device, dtype=buffer.dtype)+ weights[coords] += scale_by_masks_1d(strength, chunk_masks)+class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):"""Diffusion-based video VAE decoder (Neighborhood-Attention backbone).@@ -264,13 +319,13 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):x = self.conv_in(x)planes = x.shape[1]if valid is None:- valid = torch.ones(planes, dtype=torch.bool, device=x.device)+ valid = torch.ones(planes, dtype=torch.bool)times = keyframe_clip_times(keyframes.pixel_frame_indices,self._keyframe_time_strides[0],keyframes.clip_start_frame,)- return KeyframeStream(x=x, times=times.to(device=x.device), valid=valid.to(device=x.device)).masked()+ return KeyframeStream.from_host(x, times, valid).masked()def _run_det_stage_with_keyframes(self,@@ -305,11 +360,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):clip_start_frame,extra_origin=next_time_origin,)- return x, KeyframeStream(- x=keyframe_x,- times=next_times.to(device=keyframe_x.device),- valid=keyframes.valid,- ).masked()+ return x, dataclasses.replace(keyframes, x=keyframe_x).with_times(next_times).masked()def forward_stages_1_to_3_with_keyframes(self,@@ -383,14 +434,13 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):"""# Rebuild from global indices rather than trusting the caller's stream: stages 1-3 of a# full-clip decode are global, and Dist has already folded clip_start into clip times.- keyframes = dataclasses.replace(- keyframes,- times=keyframe_clip_times(+ keyframes = keyframes.with_times(+ keyframe_clip_times(pixel_frame_indices,self._keyframe_time_strides[3],clip_start_frame,extra_origin=stage4_time_origin,- ).to(device=keyframes.x.device),+ ))if self.deferred_stage4_upsample:return self._forward_stage_4_deferred_with_keyframes(@@ -446,11 +496,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):clip_start_frame,extra_origin=pixel_time_origin,)- return x, KeyframeStream(- x=keyframes.x,- times=stage5_times.to(device=keyframes.x.device),- valid=keyframes.valid,- ).masked()+ return x, keyframes.with_times(stage5_times).masked()def forward_stage_4(self,@@ -492,8 +538,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):def _context_and_x_for_diff_step(self, context: torch.Tensor, x_t: torch.Tensor) -> torch.Tensor:"""Build block-ready ``[context | conv_in_x_t(patched x)]`` for ``forward_diff_step``."""- noised_pixels_patched = patchify(x_t, patch_size_hw=self.patch_size, patch_size_t=1)- x = self.conv_in_x_t(noised_pixels_patched.permute(0, 2, 3, 4, 1))+ x = self.conv_in_x_t(patchify_channels_last(x_t, self.patch_size))return torch.cat([context, x], dim=-1)def _keyframe_context_and_x_for_diff_step(@@ -506,23 +551,21 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):``keyframe_x_t`` is ``(B, C_pix, P, H_pix, W_pix)`` -- the keyframe planes' own noisedpixels, one pixel frame per plane, through the *shared* ``conv_in_x_t``."""- patched = patchify(keyframe_x_t, patch_size_hw=self.patch_size, patch_size_t=1)- x = self.conv_in_x_t(patched.permute(0, 2, 3, 4, 1))+ x = self.conv_in_x_t(patchify_channels_last(keyframe_x_t, self.patch_size))x = x * keyframe_valid[None, :, None, None, None]return torch.cat([keyframe_context, x], dim=-1)def _x_for_diff_step(self, x_t: torch.Tensor) -> torch.Tensor:"""Conv-processed noised pixels only (deferred-context path)."""- noised_pixels_patched = patchify(x_t, patch_size_hw=self.patch_size, patch_size_t=1)- return self.conv_in_x_t(noised_pixels_patched.permute(0, 2, 3, 4, 1))+ return self.conv_in_x_t(patchify_channels_last(x_t, self.patch_size))def _keyframe_x_for_diff_step(self, keyframe_x_t: torch.Tensor, keyframe_valid: torch.Tensor) -> torch.Tensor:"""Mask-zeroed keyframe noised pixels only (deferred-context path).- Contiguous by construction: the chunked pathway mutates this buffer in place.+ Contiguous by construction: the chunked pathway mutates this buffer in place. Masked in+ place: ``x`` is this call's own fresh projection, whose storage the fused block chain recycles."""- patched = patchify(keyframe_x_t, patch_size_hw=self.patch_size, patch_size_t=1)- x = self.conv_in_x_t(patched.permute(0, 2, 3, 4, 1))- return (x * keyframe_valid[None, :, None, None, None]).contiguous()+ x = self.conv_in_x_t(patchify_channels_last(keyframe_x_t, self.patch_size))+ return x.contiguous().mul_(keyframe_valid[None, :, None, None, None].to(x.dtype))def forward_diff_step(self,@@ -553,8 +596,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):context_and_x: torch.Tensor,keyframe_context_and_x: torch.Tensor,t: torch.Tensor,- keyframe_times: torch.Tensor,- keyframe_valid: torch.Tensor,+ keyframe_planes: KeyframePlanes,) -> tuple[torch.Tensor, torch.Tensor]:"""One dual-stream stage-5 step. Returns ``(video_pred, keyframe_pred)`` in pixel space.The keyframe stream at stage 5 is a genuine second *pixel* diffusion stream -- its own@@ -565,7 +607,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):discarded: callers use the video prediction only.Both buffers follow ``forward_diff_step``'s ``[context | x]`` layout and the same``copy_``-into-a-view discipline. Video T/H/W stay dynamic; the keyframe plane axis is- specialized, since ``keyframe_times`` / ``keyframe_valid`` pin it.+ specialized, since ``keyframe_planes`` pin it."""x_half = context_and_x[..., self.context_channels :]keyframe_half = keyframe_context_and_x[..., self.context_channels :]@@ -575,7 +617,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):if self.mark_dynamic_shapes:for dim in (1, 2, 3):torch._dynamo.mark_dynamic(context_and_x, dim)- # Keyframe dim 1 is the plane count, not T, and keyframe_times / keyframe_valid pin it.+ # Keyframe dim 1 is the plane count, not T, and keyframe_planes pin it.# Marking it dynamic and then specializing to P raises ConstraintViolationError.for dim in (2, 3):torch._dynamo.mark_dynamic(keyframe_context_and_x, dim)@@ -585,8 +627,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):context_and_x,keyframe_context_and_x,modulation,- keyframe_times,- keyframe_valid,+ keyframe_planes,)x_half.copy_(x_out)keyframe_half.copy_(keyframe_out)@@ -594,11 +635,32 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):return self._pixels_from_stage5(x_half), self._pixels_from_stage5(keyframe_half)def _pixels_from_stage5(self, x: torch.Tensor) -> torch.Tensor:- """Shared stage-5 tail: ``norm_out`` -> ``conv_out`` -> channels-first -> unpatchify."""- x = self.norm_out(x)- x = self.conv_out(x)- x = x.permute(0, 4, 1, 2, 3).contiguous()- return unpatchify(x, patch_size_hw=self.patch_size, patch_size_t=1)+ """Shared stage-5 tail: ``norm_out`` -> ``conv_out`` -> channels-first -> unpatchify.+ Without the fused tail, ``norm_out`` runs row chunk by row chunk into a contiguous ``x`` (the+ block chain's own output, read by nothing after this) and ``conv_out`` + unpatchify one+ temporal chunk at a time into the pixel volume: same values, no volume-sized transients.+ Each eager RMSNorm launch stays under 2**32 elements (torch < 2.13 CUDA kernels wrap+ flattened offsets).+ """+ if stage5_tail_supported(x, self.norm_out, self.conv_out, self.patch_size):+ return stage5_tail(x, self.norm_out, self.conv_out, self.patch_size)+ if x.is_contiguous():+ rows = x.view(-1, x.shape[-1])+ for r0 in range(0, rows.shape[0], _NORM_OUT_CHUNK_ROWS):+ chunk = rows[r0 : r0 + _NORM_OUT_CHUNK_ROWS]+ chunk.copy_(apply_rms_norm_uint32_safe(self.norm_out, chunk))+ else:+ x = apply_rms_norm_uint32_safe(self.norm_out, x)+ b, t, h5, w5, _ = x.shape+ p = self.patch_size+ pixels = torch.empty((b, self.out_channels, t, h5 * p, w5 * p), device=x.device, dtype=x.dtype)+ frame_bytes = b * h5 * w5 * self.out_channels * p * p * x.element_size()+ chunk_t = max(1, min(t, _TAIL_CHUNK_BYTES // max(frame_bytes, 1)))+ for t0 in range(0, t, chunk_t):+ t1 = min(t, t0 + chunk_t)+ y = self.conv_out(x[:, t0:t1]).permute(0, 4, 1, 2, 3).contiguous()+ pixels[:, :, t0:t1].copy_(unpatchify(y, patch_size_hw=self.patch_size, patch_size_t=1))+ return pixelsdef forward_diff_step_deferred(self,@@ -632,10 +694,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):for block in self.diff_blocks:x = block.forward_x_ctx(x, stage4_feat, modulation, drop_leading_frame=drop_leading_frame)- x = self.norm_out(x)- x = self.conv_out(x)- x = x.permute(0, 4, 1, 2, 3).contiguous()- return unpatchify(x, patch_size_hw=self.patch_size, patch_size_t=1)+ return self._pixels_from_stage5(x)def forward_diff_step_deferred_with_keyframes(self,@@ -644,8 +703,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):keyframe_x: torch.Tensor,keyframe_stage4_feat: torch.Tensor,t: torch.Tensor,- keyframe_times: torch.Tensor,- keyframe_valid: torch.Tensor,+ keyframe_planes: KeyframePlanes,*,drop_leading_frame: bool = True,) -> tuple[torch.Tensor, torch.Tensor]:@@ -680,8 +738,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):keyframe_x,keyframe_stage4_feat,modulation,- keyframe_times,- keyframe_valid,+ keyframe_planes,drop_leading_frame=drop_leading_frame,)else:@@ -692,8 +749,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):keyframe_x,keyframe_stage4_feat,modulation,- keyframe_times,- keyframe_valid,+ keyframe_planes,drop_leading_frame=drop_leading_frame,)@@ -838,25 +894,28 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):x_t = _noise(canvas_t) if x_t_tile_init is None else x_t_tile_initkeyframe_x_t = _noise(keyframes.num_planes)- def _step(t_now: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:+ def _step(t_now: torch.Tensor, *, final: bool = False) -> tuple[torch.Tensor, torch.Tensor]:+ nonlocal x_t, keyframe_x_tif self.deferred_stage4_upsample:+ x_in = self._x_for_diff_step(x_t)+ keyframe_x_in = self._keyframe_x_for_diff_step(keyframe_x_t, keyframes.planes.valid.device_tensor)+ if final and self.model_output_type == "x0":+ # An x0 model's last step never reads the noise again; release both canvases now.+ x_t = keyframe_x_t = None # type: ignore[assignment]return self.forward_diff_step_deferred_with_keyframes(- self._x_for_diff_step(x_t),+ x_in,context_tile,- self._keyframe_x_for_diff_step(keyframe_x_t, keyframes.valid),+ keyframe_x_in,keyframes.x,t_now,- keyframes.times,- keyframes.valid,+ keyframes.planes,drop_leading_frame=is_origin,)context_and_x = self._context_and_x_for_diff_step(context_tile, x_t)keyframe_context_and_x = self._keyframe_context_and_x_for_diff_step(- keyframes.x, keyframe_x_t, keyframes.valid- )- return self.forward_diff_step_with_keyframes(- context_and_x, keyframe_context_and_x, t_now, keyframes.times, keyframes.valid+ keyframes.x, keyframe_x_t, keyframes.planes.valid.device_tensor)+ return self.forward_diff_step_with_keyframes(context_and_x, keyframe_context_and_x, t_now, keyframes.planes)_, num_steps = timestep.shapefor i in range(num_steps - 1):@@ -867,7 +926,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):keyframe_x_t = self._euler_step(keyframe_x_t, keyframe_out.to(torch.float32), t_now, t_next)t_now = timestep[:, -1]- video_out, _ = _step(t_now)+ video_out, _ = _step(t_now, final=True)if self.model_output_type == "x0":return video_outreturn self._euler_step(x_t, video_out.to(torch.float32), t_now, torch.zeros_like(t_now))@@ -900,7 +959,11 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):group_shape = full_video_shape._replace(frames=group_temporal_len)full_torch_shape = full_video_shape.to_torch_shape()accum_dtype = torch.float16 if feat_s4.dtype == torch.bfloat16 else feat_s4.dtype- buffer = torch.zeros(group_shape.to_torch_shape(), device=feat_s4.device, dtype=accum_dtype)+ # One tile with all-ones masks covering the group *is* the blend (0 + x * 1).+ sole_tile = len(tiles) == 1 and complementary and all(bool((mask == 1).all()) for mask in tiles[0].masks_1d)+ buffer = (+ None if sole_tile else torch.zeros(group_shape.to_torch_shape(), device=feat_s4.device, dtype=accum_dtype)+ )weights: torch.Tensor | None = None if complementary else torch.zeros_like(buffer)local_temporal_slice = slice(0, group_temporal_len)@@ -925,9 +988,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):f"{pixel_hi - 1 + clip_start_frame}] selected no keyframe planes "f"out of {int(pixel_frame_indices.shape[0])}; planes_for_tile always keeps at least one")- tile_stream = stream.select_planes(keep.to(stream.valid.device)).crop_spatial(- tile.in_coords[2], tile.in_coords[3]- )+ tile_stream = stream.select_planes(keep).crop_spatial(tile.in_coords[2], tile.in_coords[3])# Not debug-only: the decode below needs this tile's plane positions.tile_indices = pixel_frame_indices[keep.to(pixel_frame_indices.device)]if logger.isEnabledFor(logging.INFO):@@ -978,15 +1039,23 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):pixel_time_origin=float(pixel_lo),clip_start_frame=clip_start_frame,)+ # The tile's pixels are a fresh tensor; its leased stage-5 volumes go back to the workspace.+ decode_workspace.release_tile()content_pixel_shape = diffusion_tiling.pixel_tile_shape(full_torch_shape, tile.out_coords)pixel_tile = diffusion_tiling.crop_pixels_to_content(pixel_tile,content_pixel_shape[2],content_pixel_shape[3],content_pixel_shape[4],- ).to(buffer.dtype)-- masks = tuple(m.to(device=buffer.device, dtype=torch.float32) for m in tile.masks_1d)+ )+ if sole_tile and pixel_tile.shape == group_shape.to_torch_shape():+ pixel_tile = pixel_tile.to(accum_dtype)+ _release_cached_blocks(pixel_tile.device)+ return pixel_tile, None+ if buffer is None:+ buffer = torch.zeros(group_shape.to_torch_shape(), device=feat_s4.device, dtype=accum_dtype)++ masks = tuple(upload_from_host(m.to(dtype=torch.float32), buffer.device) for m in tile.masks_1d)local_coords = (tile.out_coords[0],tile.out_coords[1],@@ -994,13 +1063,28 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):tile.out_coords[3],tile.out_coords[4],)- buffer[local_coords] += scale_by_masks_1d(pixel_tile, masks)- if weights is not None:- strength = torch.ones(pixel_tile.shape, device=buffer.device, dtype=buffer.dtype)- weights[local_coords] += scale_by_masks_1d(strength, masks)+ _blend_tile_into(buffer, weights, local_coords, pixel_tile, masks)+ # The tiles' transients are gone; hand their segments back before the caller emits the group.+ del pixel_tile+ _release_cached_blocks(buffer.device)return buffer, weights+ def _tile_workspace(self, tiles: List[Tile], feat_s4: torch.Tensor, content_s4_frames: int) -> object | None:+ """The per-decode buffer workspace the fused stage-5 blocks draw from; ``None`` without ``ltx-kernels``.+ Large-volume slots hold the biggest stage-5 volume any tile asks for, so they are allocated once.+ """+ st, sh, sw = (int(v) for v in self.upsamples[3].stride)+ rows = 0+ for tile in tiles:+ feat_tile, _, _, _ = diffusion_tiling.slice_stage4_tile(feat_s4, tile, content_frames=content_s4_frames)+ _, t, h, w, _ = feat_tile.shape+ rows = max(rows, t * st * h * sh * w * sw)+ if not decode_workspace.available():+ return None+ channels = int(self.conv_in_x_t.out_features)+ return decode_workspace.tile_workspace((rows + decode_workspace.out_slack_rows()) * channels)+def _decode_groups_with_keyframes( # noqa: PLR0913, PLR0915self,feat_s4: torch.Tensor,@@ -1022,7 +1106,8 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):is retained between iterations, and a group's exclusive frames are yielded before thenext group decodes. That keeps residency at roughly two tile extents rather than awhole video, which matters more here than on the plain path -- a keyframe decode also- carries a second pixel stream through stage 5.+ carries a second pixel stream through stage 5. Every group's tiles draw their large+ stage-5 buffers from one workspace allocated for the decode."""full_video_shape = (VideoLatentShape.from_torch_shape(latent.shape)@@ -1068,6 +1153,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):scaled_w_pad = diffusion_tiling.scale_axis_pad(w_pad, self.video_downscale_factors.width)overlap_stub: torch.Tensor | None = Noneoverlap_stub_weights: torch.Tensor | None = None+ workspace = self._tile_workspace(tiles, feat_s4, s4_t)def _emit(buf: torch.Tensor, wts: torch.Tensor | None, global_start: int) -> torch.Tensor | None:"""Finalize, crop to content, and lay out one emitted run of frames."""@@ -1080,7 +1166,24 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):if wts is not None:floor = diffusion_tiling._weight_floor(wts.dtype)chunk = chunk / wts[:, :, :frames_keep].clamp(min=floor)- chunk = diffusion_tiling.crop_pixels_to_content(+ if as_fhwc:+ # Crop first, then cast and re-lay out in one copy: one frame-run transient instead of two.+ cropped = diffusion_tiling.crop_pixels_to_content(+ chunk,+ frames_keep,+ content_pixel.height,+ content_pixel.width,+ h_pad=scaled_h_pad,+ w_pad=scaled_w_pad,+ )[0]+ out = torch.empty(+ (cropped.shape[1], cropped.shape[2], cropped.shape[3], cropped.shape[0]),+ device=cropped.device,+ dtype=latent.dtype,+ )+ out.copy_(cropped.permute(1, 2, 3, 0))+ return out+ return diffusion_tiling.crop_pixels_to_content(chunk.to(latent.dtype),frames_keep,content_pixel.height,@@ -1088,7 +1191,6 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):h_pad=scaled_h_pad,w_pad=scaled_w_pad,)- return chunk[0].permute(1, 2, 3, 0).contiguous() if as_fhwc else chunkfor group_index, group in enumerate(groups):curr_temporal_slice = group_slices[group_index]@@ -1099,20 +1201,24 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):curr_temporal_slice.start,curr_temporal_slice.stop,)- buffer, weights = self._decode_temporal_group_isolated_with_keyframes(- group,- feat_s4,- stream,- pixel_frame_indices,- s4_t,- x_t_init,- timestep,- full_video_shape,- curr_temporal_slice,- generator,- complementary=complementary,- clip_start_frame=clip_start_frame,- )+ # Each group's tiles have their own transient sizes; hand back the previous group's free segments first.+ _release_cached_blocks(feat_s4.device)+ # Same-extent tiles and blocks build their abs-RoPE tables once per group.+ with decode_workspace.scope(workspace), abs_rope_table_cache(self):+ buffer, weights = self._decode_temporal_group_isolated_with_keyframes(+ group,+ feat_s4,+ stream,+ pixel_frame_indices,+ s4_t,+ x_t_init,+ timestep,+ full_video_shape,+ curr_temporal_slice,+ generator,+ complementary=complementary,+ clip_start_frame=clip_start_frame,+ )if overlap_stub is not None:overlap_len = int(overlap_stub.shape[2])@@ -1137,6 +1243,7 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):)if emitted is not None:yield emitted+ del emitted# Retain only the trailing overlap for the next group's handoff.overlap_stub = buffer[:, :, exclusive_len:].clone()if not complementary:@@ -1273,7 +1380,11 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):group_shape = full_video_shape._replace(frames=group_temporal_len)full_torch_shape = full_video_shape.to_torch_shape()accum_dtype = torch.float16 if feat_s4.dtype == torch.bfloat16 else feat_s4.dtype- buffer = torch.zeros(group_shape.to_torch_shape(), device=feat_s4.device, dtype=accum_dtype)+ # One tile with all-ones masks covering the group *is* the blend (0 + x * 1).+ sole_tile = len(tiles) == 1 and complementary and all(bool((mask == 1).all()) for mask in tiles[0].masks_1d)+ buffer = (+ None if sole_tile else torch.zeros(group_shape.to_torch_shape(), device=feat_s4.device, dtype=accum_dtype)+ )weights: torch.Tensor | None = None if complementary else torch.zeros_like(buffer)local_temporal_slice = slice(0, group_temporal_len)@@ -1325,9 +1436,13 @@ class DiffusionVideoDecoder(nn.Module, Disposable, VideoDecoder):content_pixel_shape[2],content_pixel_shape[3],content_pixel_shape[4],- ).to(buffer.dtype)+ ).to(accum_dtype)+ if sole_tile and pixel_tile.shape == group_shape.to_torch_shape():+ return pixel_tile, None+ if buffer is None:+ buffer = torch.zeros(group_shape.to_torch_shape(), device=feat_s4.device, dtype=accum_dtype)- masks = tuple(m.to(device=buffer.device, dtype=torch.float32) for m in tile.masks_1d)+ masks = tuple(upload_from_host(m.to(dtype=torch.float32), buffer.device) for m in tile.masks_1d)local_coords = (tile.out_coords[0],tile.out_coords[1],diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/keyframes.py b/packages/ltx-core/src/ltx_core/model/video_vae/keyframes.pyindex 565dfcd..fa6ee04 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/keyframes.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/keyframes.py@@ -13,12 +13,16 @@ vector per axis. That keeps the RoPE call and the slot tables batch-independent.from __future__ import annotations+import dataclassesfrom dataclasses import dataclass-from typing import Sequence+from typing import NamedTuple, Sequenceimport torchfrom einops import rearrange+from ltx_core.devices import upload_from_host+from ltx_core.types import HostDeviceTensorPair+#: Keyframe planes visible to one video query (and video frames visible to one keyframe#: query). Fixed at 2 upstream -- its backward kernel hardcodes at most two slots.KEYFRAME_CONTEXT_SLOTS = 2@@ -110,35 +114,71 @@ class DecodeKeyframes:return int(self.latents.shape[2])+class KeyframePlanes(NamedTuple):+ """Where a keyframe plane stack sits in time and which planes are live.+ Each field is a :class:`~ltx_core.types.HostDeviceTensorPair`: the attention kernels read the+ device tensor, and everything that picks planes or builds plane tables reads the host one, so a+ decode never reads plane metadata back from the device.+ Attributes:+ times: ``(P,)`` float32 plane position in *this stage's* temporal units and *local to+ the current tile* -- the same origin the video stream's RoPE uses. Both streams+ must share one origin or the joint softmax sees wrong offsets.+ valid: ``(P,)`` bool. Invalid planes are masked out of every softmax and their+ activations are re-zeroed after each upsample.+ """++ times: HostDeviceTensorPair+ valid: HostDeviceTensorPair++ @classmethod+ def on_device(cls, times: torch.Tensor, valid: torch.Tensor, device: torch.device) -> KeyframePlanes:+ """The planes on ``device`` with their host mirror; host inputs are uploaded without waiting."""+ return cls(HostDeviceTensorPair.on_device(times, device), HostDeviceTensorPair.on_device(valid, device))++@dataclass(frozen=True)class KeyframeStream:"""The keyframe half of the dual stream at one decoder stage.Attributes:x: ``(B, P, H, W, C)`` channels-last activations. ``H``/``W`` always match thevideo stream at the same stage; ``P`` is invariant across the whole decode.- times: ``(P,)`` float32 plane position in *this stage's* temporal units and- *local to the current tile* -- the same origin the video stream's RoPE uses.- Both streams must share one origin or the joint softmax sees wrong offsets.- valid: ``(P,)`` bool. Invalid planes are masked out of every softmax and their- activations are re-zeroed after each upsample.+ planes: the planes' :class:`KeyframePlanes`, on ``x``'s device with their host mirror."""x: torch.Tensor- times: torch.Tensor- valid: torch.Tensor+ planes: KeyframePlanes++ @classmethod+ def from_host(cls, x: torch.Tensor, times: torch.Tensor, valid: torch.Tensor) -> KeyframeStream:+ """A stream on ``x``'s device; host ``times`` / ``valid`` are uploaded without waiting for it.+ Device inputs are read back once here, so later plane selection stays on the host.+ """+ return cls(x=x, planes=KeyframePlanes.on_device(times, valid, x.device))def masked(self) -> KeyframeStream:"""Re-zero invalid planes' activations (channels-last plane axis)."""- return KeyframeStream(x=self.x * self.valid[None, :, None, None, None], times=self.times, valid=self.valid)+ return dataclasses.replace(self, x=self.x * self.planes.valid.device_tensor[None, :, None, None, None])def select_planes(self, keep: torch.Tensor) -> KeyframeStream:- """Subset the plane axis, keeping ``x``/``times``/``valid`` in step.+ """Subset the plane axis, keeping ``x`` and both copies of the planes in step.``keep`` is a ``(P,)`` bool mask. Used by tiled decode, which gives each tile only the- planes near it -- see :func:`planes_for_tile`.+ planes near it -- see :func:`planes_for_tile`. The selection is made on the host and the+ device side follows by uploaded indices, so a host ``keep`` never waits for the device+ (a device one is read back to size the result)."""if keep.shape != (self.num_planes,):raise ValueError(f"keep must be ({self.num_planes},) bool, got {tuple(keep.shape)}")- return KeyframeStream(x=self.x[:, keep], times=self.times[keep], valid=self.valid[keep])+ keep_host = keep.cpu()+ index_host = keep_host.nonzero().flatten()+ index = upload_from_host(index_host, self.x.device)+ times, valid = self.planes+ return KeyframeStream(+ x=self.x.index_select(1, index),+ planes=KeyframePlanes(+ HostDeviceTensorPair(times.host_tensor[keep_host], times.device_tensor.index_select(0, index)),+ HostDeviceTensorPair(valid.host_tensor[keep_host], valid.device_tensor.index_select(0, index)),+ ),+ )def crop_spatial(self, height: slice, width: slice) -> KeyframeStream:"""Crop H/W with the *same* slices the video stream's tile used.@@ -146,7 +186,13 @@ class KeyframeStream:reads as ghosting rather than as an obvious failure -- the same hazard as the spatialpadding rule."""- return KeyframeStream(x=self.x[:, :, height, width, :], times=self.times, valid=self.valid)+ return dataclasses.replace(self, x=self.x[:, :, height, width, :])++ def with_times(self, times: torch.Tensor) -> KeyframeStream:+ """The same planes with ``times`` for another stage or origin (a host tensor is uploaded without waiting)."""+ return dataclasses.replace(+ self, planes=self.planes._replace(times=HostDeviceTensorPair.on_device(times, self.x.device))+ )@propertydef num_planes(self) -> int:diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/memory_efficient_decode.py b/packages/ltx-core/src/ltx_core/model/video_vae/memory_efficient_decode.pyindex eaff636..afea9d0 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/memory_efficient_decode.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/memory_efficient_decode.py@@ -55,7 +55,7 @@ from torch.nn import functional as Ffrom ltx_core.loader.module_ops import ModuleOpsfrom ltx_core.loader.sd_ops import KeyValueOperationResult, SDOpsfrom ltx_core.model.common.normalization import PixelNorm-from ltx_core.model.video_vae.convolution import CausalConv3d+from ltx_core.model.video_vae.convolution import CausalConv3d, memory_format_offrom ltx_core.model.video_vae.ops import unpatchifyfrom ltx_core.model.video_vae.resnet import ResnetBlock3D, UNetMidBlock3Dfrom ltx_core.model.video_vae.sampling import DepthToSpaceUpsample@@ -69,25 +69,6 @@ if TYPE_CHECKING:# ----------------------------------------------------------------------------def _memory_format_of(t: torch.Tensor, prefer_channels_last_3d: bool = False) -> torch.memory_format:- """Pick the memory format for a workspace allocation.- When ``prefer_channels_last_3d`` is True and ``t`` is 5D, return- ``channels_last_3d`` regardless of ``t``'s current strides -- the- workspace's ``.copy_(t)`` will transcribe the data into the new layout.- This is needed because intermediate tensors inside the decoder (after- ``rearrange`` + slice + residual add in ``_upsample_forward_efficient``)- are not NHWC-contiguous, so an auto-detect helper would silently fall- back to NCHW for every workspace after the first upsample.- Otherwise fall back to inspecting ``t``: ``channels_last_3d`` if ``t``- already uses it, else contiguous.- """- if prefer_channels_last_3d and t.dim() == 5:- return torch.channels_last_3d- if t.dim() == 5 and t.is_contiguous(memory_format=torch.channels_last_3d):- return torch.channels_last_3d- return torch.contiguous_format--def _find_temporal_split_size(num_frames: int) -> int:"""Find chunk size for in-place temporal convolution.The chunk size ensures the last chunk has at least 3 frames@@ -164,7 +145,7 @@ def inplace_conv3d_temporal_chunked(workspace: torch.Tensor, conv: nn.Conv3d) ->workspace.shape[4],device=workspace.device,dtype=workspace.dtype,- memory_format=_memory_format_of(workspace),+ memory_format=memory_format_of(workspace),)o_buf = torch.empty_like(x_buf)@@ -223,7 +204,7 @@ def _causal_pad(x: torch.Tensor, pad_size: int) -> torch.Tensor:x.shape[4],device=x.device,dtype=x.dtype,- memory_format=_memory_format_of(x),+ memory_format=memory_format_of(x),)padded[:, :, pad_size:].copy_(x)for i in range(pad_size):@@ -386,7 +367,7 @@ def _midblock_forward_efficient(hidden_states.shape[4],device=hidden_states.device,dtype=hidden_states.dtype,- memory_format=_memory_format_of(hidden_states, prefer_channels_last_3d),+ memory_format=memory_format_of(hidden_states, prefer_channels_last_3d),)for resnet in block.res_blocks:@@ -430,7 +411,7 @@ def _upsample_forward_efficient(if causal:x = _causal_pad_free_and_conv(x, block.conv)else:- mem_fmt = _memory_format_of(x, prefer_channels_last_3d)+ mem_fmt = memory_format_of(x, prefer_channels_last_3d)workspace = torch.empty(x.shape[0],max(in_channels, out_channels),@@ -480,7 +461,7 @@ def _final_norm_and_conv_out(conv_out = conv_out_mod.convfeature_channels = sample.shape[1]- mem_fmt = _memory_format_of(sample, prefer_channels_last_3d)+ mem_fmt = memory_format_of(sample, prefer_channels_last_3d)workspace = torch.empty(sample.shape[0],max(feature_channels, conv_out.out_channels),diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/ops.py b/packages/ltx-core/src/ltx_core/model/video_vae/ops.pyindex 7eccea5..c959cdd 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/ops.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/ops.py@@ -1,4 +1,9 @@+from __future__ import annotations++from collections.abc import Sequence+import torch+import torch.nn.functional as Ffrom einops import rearrangefrom torch import nn@@ -32,6 +37,16 @@ def patchify(x: torch.Tensor, patch_size_hw: int, patch_size_t: int = 1) -> torcreturn x+def patchify_channels_last(x: torch.Tensor, patch_size_hw: int) -> torch.Tensor:+ """``patchify(x, patch_size_hw).permute(0, 2, 3, 4, 1)`` for a 5D ``x``, materialized contiguous.+ Same element mapping as that pair, in one copy instead of a copy plus the one a+ following ``Linear`` makes of the permuted view -- and a contiguous input lets the+ ``Linear`` fold its bias into the matmul. At ``patch_size_hw == 1`` ``rearrange`` alone+ returns a permuted view of ``x``.+ """+ return rearrange(x, "b c f (h q) (w r) -> b f h w (c r q)", q=patch_size_hw, r=patch_size_hw).contiguous()++def unpatchify(x: torch.Tensor, patch_size_hw: int, patch_size_t: int = 1) -> torch.Tensor:"""Rearrange channels back into spatial dimensions. Inverse of patchify - moves pixels from@@ -82,3 +97,77 @@ class PerChannelStatistics(nn.Module):return (x - self.get_buffer("mean-of-means").view(1, -1, 1, 1, 1).to(x)) / self.get_buffer("std-of-means").view(1, -1, 1, 1, 1).to(x)+++def apply_rms_norm_uint32_safe(norm: nn.RMSNorm, x: torch.Tensor) -> torch.Tensor:+ """``norm(x)`` via :func:`rms_norm_uint32_safe`."""+ return rms_norm_uint32_safe(x, tuple(norm.normalized_shape), norm.weight, norm.eps)+++def rms_norm_uint32_safe(+ x: torch.Tensor,+ normalized_shape: Sequence[int],+ weight: torch.Tensor | None = None,+ eps: float = 1e-6,+) -> torch.Tensor:+ """Last-dim RMSNorm that never feeds CUDA a volume of ``2**32``+ elements.+ PyTorch < 2.13 computes ``row * N`` in 32-bit in the layer_norm / RMSNorm+ kernels (pytorch/pytorch#181600). On 2.13+ this is ``F.rms_norm``.+ """+ torch_mm = tuple(int(p) for p in torch.__version__.split("+")[0].split(".")[:2])+ max_elements = (1 << 32) if torch_mm < (2, 13) else (1 << 62)+ return rms_norm_chunked(x, normalized_shape, weight, eps, max_elements=max_elements)+++def rms_norm_chunked(+ x: torch.Tensor,+ normalized_shape: Sequence[int],+ weight: torch.Tensor | None = None,+ eps: float = 1e-6,+ *,+ max_elements: int,+) -> torch.Tensor:+ """Last-dim RMSNorm with each launch strictly below ``max_elements``.+ Prefers slicing T (dim 1) on ``ndim >= 3``. If one T-plane is already at or+ above the limit, flattens to rows and chunks dim 0. Writes into a preallocated+ output so peak extra memory is one window, not every slice plus ``torch.cat``.+ """+ rms_args = (normalized_shape, weight, eps)+ if x.numel() < max_elements:+ return F.rms_norm(x, *rms_args)+ if x.ndim >= 3:+ t = int(x.shape[1])+ per_frame = x.numel() // max(t, 1)+ if per_frame < max_elements:+ chunk_t = _chunk_extent(t, per_frame, max_elements)+ if chunk_t >= t:+ return F.rms_norm(x, *rms_args)+ return _rms_norm_into(torch.empty_like(x), x, 1, chunk_t, *rms_args)+ if x.ndim < 2:+ return F.rms_norm(x, *rms_args)+ flat = x.reshape(-1, x.shape[-1])+ n = int(flat.shape[0])+ per_row = max(int(flat.shape[-1]), 1)+ chunk = _chunk_extent(n, per_row, max_elements)+ if chunk >= n:+ return F.rms_norm(x, *rms_args)+ out = _rms_norm_into(torch.empty_like(flat), flat, 0, chunk, *rms_args)+ return out if x.ndim == 2 else out.view(x.shape)+++def _chunk_extent(length: int, per_item: int, max_elements: int) -> int:+ """Largest slice along ``length`` whose numel stays strictly below ``max_elements``."""+ if length <= 0:+ return 1+ return max(1, (max_elements - 1) // max(per_item, 1))+++def _rms_norm_into(out: torch.Tensor, x: torch.Tensor, dim: int, chunk: int, *rms_args: object) -> torch.Tensor:+ """Last-dim RMSNorm along ``dim``, writing each slice into a preallocated ``out``."""+ n = int(x.shape[dim])+ idx: list[slice | int] = [slice(None)] * x.ndim+ for i in range(0, n, chunk):+ idx[dim] = slice(i, min(i + chunk, n))+ key = tuple(idx)+ out[key] = F.rms_norm(x[key], *rms_args)+ return outdiff --git a/packages/ltx-core/src/ltx_core/model/video_vae/sampling.py b/packages/ltx-core/src/ltx_core/model/video_vae/sampling.pyindex 5d8f742..24643b6 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/sampling.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/sampling.py@@ -5,7 +5,7 @@ import torchfrom einops import rearrangefrom torch import nn-from .convolution import make_conv_nd+from .convolution import make_conv_nd, memory_format_offrom .enums import PaddingModeType@@ -36,6 +36,7 @@ class SpaceToDepthDownsample(nn.Module):x: torch.Tensor,causal: bool = True,) -> torch.Tensor:+ memory_format = memory_format_of(x)if self.stride[0] == 2:x = torch.cat([x[:, :, :1, :, :], x], dim=2) # duplicate first frames for padding@@ -60,9 +61,12 @@ class SpaceToDepthDownsample(nn.Module):p3=self.stride[2],)- x = x + x_in-- return x+ # The space-to-depth rearranges come out NCHW; summing into a buffer laid out like the input+ # keeps channels-last activations channels-last for the next block's convs. ``out=`` has no+ # autograd support, so a graph-recording call pays one extra copy instead.+ if x.requires_grad or x_in.requires_grad:+ return (x + x_in).contiguous(memory_format=memory_format)+ return torch.add(x, x_in, out=torch.empty_like(x, memory_format=memory_format))class DepthToSpaceUpsample(nn.Module):diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/attention.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/attention.pyindex 5a73311..85b5087 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/attention.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/attention.py@@ -13,16 +13,25 @@ from typing import TYPE_CHECKING, Protocolimport torchfrom torch import nn-from ltx_core.model.video_vae.transformer.det_attn_rope import det_qkv_rope, det_qkv_rope_at_times+from ltx_core.model.video_vae.transformer.det_attn_rope import (+ det_project_qkv_eager,+ det_project_qkv_lean,+ det_qkv_rope,+ det_qkv_rope_at_times,+ det_rope_qk_,+ linear_into_phase_buffer,+)+from ltx_core.model.video_vae.transformer.det_rms_norm import DetRMSNormfrom ltx_core.model.video_vae.transformer.qkv import QKVProjectionsfrom ltx_core.model.video_vae.transformer.rope_math import (DEFAULT_ABS_ROPE_NUM_TILES,+ AbsRopeTableCache,default_rope_dim_split,rope_inv_freqs,)if TYPE_CHECKING:- from ltx_core.model.video_vae.keyframes import KeyframeStream+ from ltx_core.model.video_vae.keyframes import KeyframePlanes, KeyframeStreamtry:import natten@@ -59,7 +68,8 @@ class JointNAAttentionCallable(Protocol):Same conventions as :class:`NAAttentionCallable` -- Q/K/V already normed, scaled andabsolutely RoPE'd -- with a second stream shaped ``(B, P, H, W, NH, HD)`` whose planeaxis sits in video's temporal slot. Both streams' RoPE must share one origin, so- ``keyframe_times`` are tile-local. Returns one output per stream.+ ``keyframe_planes.times`` are tile-local. Backends that build plane tables on the host read the+ planes' host tensors; the others read the device ones. Returns one output per stream.NATTEN and the CuTe DSL kernel cannot express a joint window, so this is a separateslot from ``attention_function`` rather than a widening of it: it keeps the shippingkeyframe-less hot path untouched, and it is immune to the install-order hazard that@@ -75,8 +85,7 @@ class JointNAAttentionCallable(Protocol):keyframe_q: torch.Tensor,keyframe_k: torch.Tensor,keyframe_v: torch.Tensor,- keyframe_times: torch.Tensor,- keyframe_valid: torch.Tensor,+ keyframe_planes: KeyframePlanes,) -> tuple[torch.Tensor, torch.Tensor]: ...@@ -146,6 +155,8 @@ class NeighborhoodAttention3D(nn.Module):self.rope_base = rope_baseself.rope_num_tiles = DEFAULT_ABS_ROPE_NUM_TILESself.rope_compute_dtype = torch.float32+ # The decode scope's abs-RoPE table cache while one is open (see ``abs_rope_table_cache``).+ self.rope_table_cache: AbsRopeTableCache | None = None# Kept for the chunked opaque residual (string arg); callable is the swap surface.self.natten_backend: str | None = Noneself.attention_function: NAAttentionCallable = NattenAttention()@@ -159,8 +170,8 @@ class NeighborhoodAttention3D(nn.Module):self.qkv = QKVProjections(dim)self.proj = nn.Linear(dim, dim, bias=True)- self.q_norm = nn.RMSNorm(head_dim, eps=1e-6)- self.k_norm = nn.RMSNorm(head_dim, eps=1e-6)+ self.q_norm = DetRMSNorm(head_dim, eps=1e-6)+ self.k_norm = DetRMSNorm(head_dim, eps=1e-6)# W-chunking configuration (consumed by ``chunked.attn``).self.w_chunks = 1 # 1 = no chunking@@ -186,7 +197,12 @@ class NeighborhoodAttention3D(nn.Module):f"got (T,H,W)=({t},{h},{w}) vs kernel={self.kernel_size}")- q, k, v = det_qkv_rope(self, x)+ if torch.compiler.is_compiling():+ q, k, v = det_qkv_rope(self, x)+ else:+ # Eager: norms (and Q's scale) and RoPE in place on the fresh projections; same values.+ q, k, v = det_project_qkv_eager(self, x)+ q, k = det_rope_qk_(self, q.contiguous(), k.contiguous())# natten's CUTLASS kernel silently produces wrong output if inputs are non-contiguous.q, k, v = q.contiguous(), k.contiguous(), v.contiguous()out = self.attention_function(self, q, k, v)@@ -210,8 +226,17 @@ class NeighborhoodAttention3D(nn.Module):batch, t, h, w, _ = x.shapeplanes = keyframes.x.shape[1]- q, k, v = det_qkv_rope(self, x)- keyframe_q, keyframe_k, keyframe_v = det_qkv_rope_at_times(self, keyframes.x, keyframes.times)+ eager = not torch.compiler.is_compiling()+ if eager:+ # Projections in workspace phase buffers, norms, scale and RoPE in place; same values.+ q, k, v = det_project_qkv_lean(self, x)+ del x+ q, k = det_rope_qk_(self, q, k)+ else:+ q, k, v = det_qkv_rope(self, x)+ keyframe_q, keyframe_k, keyframe_v = det_qkv_rope_at_times(+ self, keyframes.x, keyframes.planes.times.device_tensor+ )q, k, v = q.contiguous(), k.contiguous(), v.contiguous()keyframe_q = keyframe_q.contiguous()keyframe_k = keyframe_k.contiguous()@@ -225,10 +250,12 @@ class NeighborhoodAttention3D(nn.Module):keyframe_q,keyframe_k,keyframe_v,- keyframes.times,- keyframes.valid,+ keyframes.planes,)- out = self.proj(out.reshape(batch, t, h, w, self.dim))+ # Q/K/V are dead once the attention has run; three volumes need not stay live across the projection.+ del q, k, v, keyframe_q, keyframe_k, keyframe_v+ out = out.reshape(batch, t, h, w, self.dim)+ out = linear_into_phase_buffer(self.proj, out) if eager else self.proj(out)keyframe_out = self.proj(keyframe_out.reshape(batch, planes, h, w, self.dim))return out, dataclasses.replace(keyframes, x=keyframe_out)diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/blocks.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/blocks.pyindex 152dd4d..4acf092 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/blocks.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/blocks.py@@ -12,9 +12,11 @@ from typing import TYPE_CHECKINGimport torchfrom torch import nn+from ltx_core.model.video_vae import decode_workspacefrom ltx_core.model.video_vae.transformer.attention import NeighborhoodAttention3D+from ltx_core.model.video_vae.transformer.det_rms_norm import DetRMSNormfrom ltx_core.model.video_vae.transformer.layers import AdaLNZero-from ltx_core.model.video_vae.transformer.swiglu import SwiGLU, plain_mlp+from ltx_core.model.video_vae.transformer.swiglu import SwiGLU, plain_mlp, plain_mlp_accumulateif TYPE_CHECKING:from ltx_core.model.video_vae.keyframes import KeyframeStream@@ -37,17 +39,16 @@ class NABlock(nn.Module):rope_dim_split: tuple[int, int, int] | None = None,) -> None:super().__init__()- self.norm1 = nn.RMSNorm(dim, eps=1e-6)+ self.norm1 = DetRMSNorm(dim, eps=1e-6)self.attn = NeighborhoodAttention3D(dim, kernel_size, head_dim=head_dim, rope_dim_split=rope_dim_split)- self.norm2 = nn.RMSNorm(dim, eps=1e-6)+ self.norm2 = DetRMSNorm(dim, eps=1e-6)hidden = (int(dim * mlp_ratio) + 15) // 16 * 16self.mlp = SwiGLU(dim, hidden)def forward(self, x: torch.Tensor) -> torch.Tensor:- """Channels-last in/out: (B, T, H, W, C)."""+ """Channels-last in/out: (B, T, H, W, C). The norms pick their kernel themselves (:class:`DetRMSNorm`)."""x = x + self.attn(self.norm1(x))- x = plain_mlp(x, self.mlp, self.norm2, self.mlp.tile)- return x+ return plain_mlp(x, self.mlp, self.norm2, self.mlp.tile)def forward_with_keyframes(self,@@ -60,15 +61,28 @@ class NABlock(nn.Module):the decoder re-zeroes after each upsample instead, matching upstream, so a maskedplane's hidden state may drift within a stage. It is masked out of every softmaxregardless, so this is cosmetic; reproducing it keeps us comparable.+ Eager calls carve the video stream's per-block temporaries from the decode workspace's+ idle volumes (none outlives the block, so the arena is reset at both block boundaries)+ and add the residuals into the fresh attention and MLP outputs; same values."""+ if torch.compiler.is_compiling():+ attn_out, keyframe_attn = self.attn.forward_with_keyframes(+ self.norm1(x), dataclasses.replace(keyframes, x=self.norm1(keyframes.x))+ )+ x = x + attn_out+ keyframe_x = keyframes.x + keyframe_attn.x+ x = plain_mlp(x, self.mlp, self.norm2, self.mlp.tile)+ keyframe_x = plain_mlp(keyframe_x, self.mlp, self.norm2, self.mlp.tile)+ return x, dataclasses.replace(keyframes, x=keyframe_x)+ decode_workspace.phase_reset()attn_out, keyframe_attn = self.attn.forward_with_keyframes(- self.norm1(x),- dataclasses.replace(keyframes, x=self.norm1(keyframes.x)),+ self.norm1(x), dataclasses.replace(keyframes, x=self.norm1(keyframes.x)))- x = x + attn_out- keyframe_x = keyframes.x + keyframe_attn.x- x = plain_mlp(x, self.mlp, self.norm2, self.mlp.tile)- keyframe_x = plain_mlp(keyframe_x, self.mlp, self.norm2, self.mlp.tile)+ x = attn_out.add_(x)+ keyframe_x = keyframe_attn.x.add_(keyframes.x)+ x = plain_mlp_accumulate(x, self.mlp, self.norm2, self.mlp.tile)+ keyframe_x = plain_mlp_accumulate(keyframe_x, self.mlp, self.norm2, self.mlp.tile)+ decode_workspace.phase_reset()return x, dataclasses.replace(keyframes, x=keyframe_x)diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/attn.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/attn.pyindex 79da5e0..32cd029 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/attn.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/attn.py@@ -25,6 +25,8 @@ import torchimport torch.nn.functional as Ffrom torch import nn+from ltx_core.model.video_vae.keyframes import KeyframePlanes+from ltx_core.model.video_vae.ops import rms_norm_uint32_safefrom ltx_core.model.video_vae.transformer.attention import (JointNAAttentionCallable,NAAttentionCallable,@@ -37,6 +39,7 @@ from ltx_core.model.video_vae.transformer.rope_math import (rot_abs_axis_impl,t_positions,)+from ltx_core.types import HostDeviceTensorPair# Bound by ``chunked()`` for the duration of the opaque custom_op. Dynamo only sees# the op call site; the Python body reads these at runtime.@@ -155,10 +158,10 @@ def _slab_qkv( # noqa: PLR0913eps = 1e-6head_shape = (batch, t, h, ext_w, num_heads, head_dim)q = F.linear(y, q_w, q_b).view(head_shape)- q = F.rms_norm(q, (head_dim,), q_norm_w, eps) * attn_scale+ q = rms_norm_uint32_safe(q, (head_dim,), q_norm_w, eps) * attn_scaleq = _apply_in_slab_abs_rope(q, rope_split, inv_freqs, w_pos, compute_dtype=rope_dtype, t_pos=t_pos)k = F.linear(y, k_w, k_b).view(head_shape)- k = F.rms_norm(k, (head_dim,), k_norm_w, eps)+ k = rms_norm_uint32_safe(k, (head_dim,), k_norm_w, eps)k = _apply_in_slab_abs_rope(k, rope_split, inv_freqs, w_pos, compute_dtype=rope_dtype, t_pos=t_pos)v = F.linear(y, v_w, v_b).view(head_shape)return q, k, v@@ -208,7 +211,7 @@ def _attn_on_chunk_impl( # noqa: PLR0913inv_freqs = (inv_t, inv_h, inv_w)rope_split = (d_t, d_h, d_w)- y = F.rms_norm(x_chunk, (dim,), norm_weight, 1e-6)+ y = rms_norm_uint32_safe(x_chunk, (dim,), norm_weight, 1e-6)y = y * (1.0 + scale) + shiftq, k, v = _slab_qkv(@@ -244,6 +247,8 @@ def _joint_attn_on_chunk_impl( # noqa: PLR0913w_pos: torch.Tensor,keyframe_times: torch.Tensor,keyframe_valid: torch.Tensor,+ keyframe_times_host: torch.Tensor,+ keyframe_valid_host: torch.Tensor,scale: torch.Tensor,shift: torch.Tensor,norm_weight: torch.Tensor,@@ -286,9 +291,9 @@ def _joint_attn_on_chunk_impl( # noqa: PLR0913weights = (q_w, q_b, k_w, k_b, v_w, v_b, q_norm_w, k_norm_w)meta = (rope_split, inv_freqs, num_heads, head_dim, attn_scale, rope_dtype)- y = F.rms_norm(x_chunk, (dim,), norm_weight, 1e-6)+ y = rms_norm_uint32_safe(x_chunk, (dim,), norm_weight, 1e-6)y = y * (1.0 + scale) + shift- keyframe_y = F.rms_norm(keyframe_chunk, (dim,), norm_weight, 1e-6)+ keyframe_y = rms_norm_uint32_safe(keyframe_chunk, (dim,), norm_weight, 1e-6)keyframe_y = keyframe_y * (1.0 + scale) + shiftq, k, v = _slab_qkv(y, w_pos, *weights, *meta)@@ -305,8 +310,10 @@ def _joint_attn_on_chunk_impl( # noqa: PLR0913keyframe_q.contiguous(),keyframe_k.contiguous(),keyframe_v.contiguous(),- times,- keyframe_valid,+ KeyframePlanes(+ HostDeviceTensorPair(keyframe_times_host.to(dtype=torch.float32), times),+ HostDeviceTensorPair(keyframe_valid_host, keyframe_valid),+ ),)del q, k, v, keyframe_q, keyframe_k, keyframe_vreturn (@@ -661,6 +668,8 @@ def _na_residual_w_chunked_joint_op( # noqa: PLR0913shift: torch.Tensor,keyframe_times: torch.Tensor,keyframe_valid: torch.Tensor,+ keyframe_times_host: torch.Tensor,+ keyframe_valid_host: torch.Tensor,norm_weight: torch.Tensor,q_w: torch.Tensor,q_b: torch.Tensor,@@ -743,6 +752,8 @@ def _na_residual_w_chunked_joint_op( # noqa: PLR0913w_pos,times,valid,+ keyframe_times_host,+ keyframe_valid_host,scale_,shift_,*rest, # type: ignore[arg-type]@@ -770,6 +781,8 @@ def _na_residual_w_chunked_joint_fake( # noqa: PLR0913shift: torch.Tensor,keyframe_times: torch.Tensor,keyframe_valid: torch.Tensor,+ keyframe_times_host: torch.Tensor,+ keyframe_valid_host: torch.Tensor,norm_weight: torch.Tensor,q_w: torch.Tensor,q_b: torch.Tensor,@@ -805,6 +818,8 @@ def _na_residual_w_chunked_joint_fake( # noqa: PLR0913shift,keyframe_times,keyframe_valid,+ keyframe_times_host,+ keyframe_valid_host,norm_weight,q_w,q_b,@@ -842,8 +857,7 @@ def chunked_with_keyframes(norm: nn.RMSNorm,scale: torch.Tensor,shift: torch.Tensor,- keyframe_times: torch.Tensor,- keyframe_valid: torch.Tensor,+ keyframe_planes: KeyframePlanes,) -> tuple[torch.Tensor, torch.Tensor]:"""In-place dual-stream W-chunked residual through one joint softmax per slab.Keyframe counterpart of :func:`chunked`, using ``attn.joint_attention_function``@@ -858,8 +872,10 @@ def chunked_with_keyframes(keyframe_x,scale,shift,- keyframe_times,- keyframe_valid,+ keyframe_planes.times.device_tensor,+ keyframe_planes.valid.device_tensor,+ keyframe_planes.times.host_tensor,+ keyframe_planes.valid.host_tensor,*packed,attn.w_chunks,)diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/block.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/block.pyindex e8e49b7..924a593 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/block.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/block.py@@ -2,6 +2,8 @@from __future__ import annotations+from typing import TYPE_CHECKING+import torchfrom ltx_core.model.video_vae.transformer.blocks import DiffusionNABlock@@ -14,6 +16,9 @@ from ltx_core.model.video_vae.transformer.chunked.context import deferred_keyfrafrom ltx_core.model.video_vae.transformer.chunked.mlp import residual_modulating_mlpfrom ltx_core.model.video_vae.transformer.layers import LinearPixelShuffleUpsample+if TYPE_CHECKING:+ from ltx_core.model.video_vae.keyframes import KeyframePlanes+class ChunkedDiffusionNABlock(DiffusionNABlock):"""Deferred-context diffusion block.@@ -47,8 +52,7 @@ class ChunkedDiffusionNABlock(DiffusionNABlock):shift_msa: torch.Tensor,scale_mlp: torch.Tensor,shift_mlp: torch.Tensor,- keyframe_times: torch.Tensor,- keyframe_valid: torch.Tensor,+ keyframe_planes: KeyframePlanes,) -> tuple[torch.Tensor, torch.Tensor]:"""Dual-stream chunked NA residual + modulating SwiGLU, both in place.Invalid keyframe planes are re-zeroed on the way out, matching@@ -63,12 +67,11 @@ class ChunkedDiffusionNABlock(DiffusionNABlock):self.norm1,scale_msa,shift_msa,- keyframe_times,- keyframe_valid,+ keyframe_planes,)x = residual_modulating_mlp(x, self.mlp, self.norm2, scale_mlp, shift_mlp, self.mlp.tile)keyframe_x = residual_modulating_mlp(keyframe_x, self.mlp, self.norm2, scale_mlp, shift_mlp, self.mlp.tile)- keyframe_x.mul_(keyframe_valid[None, :, None, None, None])+ keyframe_x.mul_(keyframe_planes.valid.device_tensor[None, :, None, None, None])return x, keyframe_xdef forward_x_ctx_with_keyframes(@@ -78,8 +81,7 @@ class ChunkedDiffusionNABlock(DiffusionNABlock):keyframe_x: torch.Tensor,keyframe_stage4_feat: torch.Tensor,modulation: tuple[torch.Tensor, ...],- keyframe_times: torch.Tensor,- keyframe_valid: torch.Tensor,+ keyframe_planes: KeyframePlanes,*,drop_leading_frame: bool = True,) -> tuple[torch.Tensor, torch.Tensor]:@@ -118,8 +120,7 @@ class ChunkedDiffusionNABlock(DiffusionNABlock):shift_msa,scale_mlp,shift_mlp,- keyframe_times,- keyframe_valid,+ keyframe_planes,)def forward_x_ctx(diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/mlp.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/mlp.pyindex 7131697..515c1b1 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/mlp.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/chunked/mlp.py@@ -6,6 +6,7 @@ import torchimport torch.nn.functional as Ffrom torch import nn+from ltx_core.model.video_vae.ops import rms_norm_uint32_safefrom ltx_core.model.video_vae.transformer.swiglu import (SwiGLUTileSpec,_fused_gate_up_swiglu_triton,@@ -87,7 +88,7 @@ def _chunked_residual_modulated_impl(n = end - startxc = x_flat[start:end]y = y_buf[:n]- y.copy_(F.rms_norm(xc, (dim,), norm_weight, eps))+ y.copy_(rms_norm_uint32_safe(xc, (dim,), norm_weight, eps))y.mul_(1.0 + s).add_(sh)ws = workspace[:n]diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/attn.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/attn.pyindex efcb05b..8b1e339 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/attn.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/attn.py@@ -5,9 +5,12 @@ Does not share a residual body with ``chunked`` — only the NA module weights.from __future__ import annotations+from typing import TYPE_CHECKING+import torchfrom torch import nn+from ltx_core.model.video_vae.ops import apply_rms_norm_uint32_safefrom ltx_core.model.video_vae.transformer.attention import NeighborhoodAttention3Dfrom ltx_core.model.video_vae.transformer.rope_math import (h_positions,@@ -15,6 +18,9 @@ from ltx_core.model.video_vae.transformer.rope_math import (t_positions,)+if TYPE_CHECKING:+ from ltx_core.model.video_vae.keyframes import KeyframePlanes+# Transparent path: innermost per-axis rotation is a nested compile region._rot_abs_axis = torch.compiler.nested_compile_region(rot_abs_axis_impl)@@ -93,8 +99,8 @@ def _qkv_nested_rope(its (possibly fractional) plane times there."""q, k, v = attn.project_qkv(x)- q = attn.q_norm(q) * attn.scale- k = attn.k_norm(k)+ q = apply_rms_norm_uint32_safe(attn.q_norm, q) * attn.scale+ k = apply_rms_norm_uint32_safe(attn.k_norm, k)inv_freqs = (attn.rope_inv_t.to(device=x.device),attn.rope_inv_h.to(device=x.device),@@ -127,12 +133,11 @@ def full_with_keyframes(norm: nn.RMSNorm,scale: torch.Tensor,shift: torch.Tensor,- keyframe_times: torch.Tensor,- keyframe_valid: torch.Tensor,+ keyframe_planes: KeyframePlanes,) -> tuple[torch.Tensor, torch.Tensor]:"""Dual-stream AdaLN residual attention: one joint softmax, shared weights.- Plain tensors rather than a ``KeyframeStream`` here: this is the stage-5 hot path, and- rebuilding a dataclass per block inside a compiled region buys nothing.+ The planes' metadata rather than a ``KeyframeStream`` here: this is the stage-5 hot path,+ and rebuilding a dataclass per block inside a compiled region buys nothing.Both streams take the *same* ``scale``/``shift``. Upstream computes a separate keyframemodulation, but the two are identical unless per-frame timestep conditioning supplies aconditioning mask, which we deliberately do not implement yet.@@ -147,11 +152,11 @@ def full_with_keyframes(batch, t, h, w, _ = x.shapeplanes = keyframe_x.shape[1]- y = norm(x) * (1.0 + scale) + shift- keyframe_y = norm(keyframe_x) * (1.0 + scale) + shift+ y = apply_rms_norm_uint32_safe(norm, x) * (1.0 + scale) + shift+ keyframe_y = apply_rms_norm_uint32_safe(norm, keyframe_x) * (1.0 + scale) + shiftq, k, v = _qkv_nested_rope(attn, y)- keyframe_q, keyframe_k, keyframe_v = _qkv_nested_rope(attn, keyframe_y, keyframe_times)+ keyframe_q, keyframe_k, keyframe_v = _qkv_nested_rope(attn, keyframe_y, keyframe_planes.times.device_tensor)q, k, v = q.contiguous(), k.contiguous(), v.contiguous()keyframe_q = keyframe_q.contiguous()keyframe_k = keyframe_k.contiguous()@@ -165,8 +170,7 @@ def full_with_keyframes(keyframe_q,keyframe_k,keyframe_v,- keyframe_times,- keyframe_valid,+ keyframe_planes,)x = x + attn.proj(out.reshape(batch, t, h, w, attn.dim))keyframe_x = keyframe_x + attn.proj(keyframe_out.reshape(batch, planes, h, w, attn.dim))@@ -181,7 +185,7 @@ def full(shift: torch.Tensor,) -> torch.Tensor:"""``x + NA(modulate(norm(x)))`` with nested full-volume abs-RoPE."""- y = norm(x) * (1.0 + scale) + shift+ y = apply_rms_norm_uint32_safe(norm, x) * (1.0 + scale) + shiftbatch, t, h, w, _ = y.shapekt, kh, kw = attn.kernel_sizeif t < kt or h < kh or w < kw:diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/block.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/block.pyindex 89d237f..c73fa90 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/block.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/block.py@@ -2,6 +2,8 @@from __future__ import annotations+from typing import TYPE_CHECKING+import torchfrom ltx_core.model.video_vae.transformer.blocks import DiffusionNABlock@@ -10,6 +12,9 @@ from ltx_core.model.video_vae.transformer.combined.attn import full_with_keyframfrom ltx_core.model.video_vae.transformer.combined.context import combined as inject_contextfrom ltx_core.model.video_vae.transformer.combined.mlp import residual_mlp+if TYPE_CHECKING:+ from ltx_core.model.video_vae.keyframes import KeyframePlanes+class CombinedDiffusionNABlock(DiffusionNABlock):"""Combined-context diffusion block: ``forward`` / ``forward_combined``."""@@ -19,8 +24,7 @@ class CombinedDiffusionNABlock(DiffusionNABlock):context_and_x: torch.Tensor,keyframe_context_and_x: torch.Tensor,modulation: tuple[torch.Tensor, ...],- keyframe_times: torch.Tensor,- keyframe_valid: torch.Tensor,+ keyframe_planes: KeyframePlanes,) -> tuple[torch.Tensor, torch.Tensor]:"""Dual-stream block; returns the two updated x halves (not the concat buffers).Each stream gets its own ``context_proj(context)`` injection from its own@@ -39,12 +43,11 @@ class CombinedDiffusionNABlock(DiffusionNABlock):self.norm1,scale_msa,shift_msa,- keyframe_times,- keyframe_valid,+ keyframe_planes,)x = residual_mlp(x, self.mlp, self.norm2, scale_mlp, shift_mlp, self.mlp.tile)keyframe_x = residual_mlp(keyframe_x, self.mlp, self.norm2, scale_mlp, shift_mlp, self.mlp.tile)- return x, keyframe_x * keyframe_valid[None, :, None, None, None]+ return x, keyframe_x * keyframe_planes.valid.device_tensor[None, :, None, None, None]def forward_combined(self,diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/mlp.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/mlp.pyindex 386d113..e706610 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/mlp.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/combined/mlp.py@@ -5,6 +5,7 @@ from __future__ import annotationsimport torchfrom torch import nn+from ltx_core.model.video_vae.ops import apply_rms_norm_uint32_safefrom ltx_core.model.video_vae.transformer.layers import modulatefrom ltx_core.model.video_vae.transformer.swiglu import SwiGLUTileSpec, swiglu_tiled@@ -18,7 +19,7 @@ def residual_mlp(tile: SwiGLUTileSpec,) -> torch.Tensor:"""Combined*: ``x + swiglu_tiled(modulate(norm(x), scale, shift))``."""- y = modulate(norm(x), scale, shift)+ y = modulate(apply_rms_norm_uint32_safe(norm, x), scale, shift)if y.numel() == 0:return xreturn x + swiglu_tiled(y, mlp.w_gate.weight, mlp.w_up.weight, mlp.w_down.weight, tile)diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/det_attn_rope.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/det_attn_rope.pyindex 22b772b..1d1dcfa 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/det_attn_rope.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/det_attn_rope.py@@ -17,8 +17,20 @@ from __future__ import annotationsimport torch+try:+ from ltx_kernels.vae.rope_dsl import abs_rope_supported, run_abs_rope+except ImportError: # ltx-kernels is optional; the slab path below covers every device+ abs_rope_supported = None+ run_abs_rope = None++from ltx_core.model.video_vae import decode_workspace+from ltx_core.model.video_vae.ops import apply_rms_norm_uint32_safefrom ltx_core.model.video_vae.transformer.rope_math import (DEFAULT_ABS_ROPE_NUM_TILES,+ AbsRopeTableCache,+ RopeAxis,+ RopePositions,+ cached_rope_tables,h_positions,rot_abs_axis_impl,t_positions,@@ -33,25 +45,40 @@ def _apply_opaque_rope_slab(w_pos: torch.Tensor,compute_dtype: torch.dtype,t_pos: torch.Tensor | None = None,+ w_pos_key: RopePositions | None = None,+ cache: AbsRopeTableCache | None = None,) -> torch.Tensor:"""Rotate one W-extent with raw abs-RoPE (runs inside the opaque op).``t_pos`` overrides the default integer ``arange`` on the first axis; the keyframe- stream passes its (possibly fractional) plane times there.+ stream passes its (possibly fractional) plane times there. ``w_pos_key`` names ``w_pos``+ by value for ``cache``."""d_t, d_h, _ = rope_splitinv_t, inv_h, inv_w = inv_freqst = x.shape[1]h = x.shape[2]positions_t = t_positions(t, x.device) if t_pos is None else t_pos- xt = rot_abs_axis_impl(x[..., :d_t], positions_t, inv_t, axis=1, compute_dtype=compute_dtype)+ xt = rot_abs_axis_impl(+ x[..., :d_t],+ positions_t,+ inv_t,+ axis=1,+ compute_dtype=compute_dtype,+ pos_key=RopePositions(RopeAxis.T, t) if t_pos is None else None,+ cache=cache,+ )xh = rot_abs_axis_impl(x[..., d_t : d_t + d_h],h_positions(h, x.device),inv_h,axis=2,compute_dtype=compute_dtype,+ pos_key=RopePositions(RopeAxis.H, h),+ cache=cache,+ )+ xw = rot_abs_axis_impl(+ x[..., d_t + d_h :], w_pos, inv_w, axis=3, compute_dtype=compute_dtype, pos_key=w_pos_key, cache=cache)- xw = rot_abs_axis_impl(x[..., d_t + d_h :], w_pos, inv_w, axis=3, compute_dtype=compute_dtype)return torch.cat([xt, xh, xw], dim=-1)@@ -63,12 +90,18 @@ def _apply_opaque_tiled_rope(num_tiles: int,compute_dtype: torch.dtype,t_pos: torch.Tensor | None = None,+ cache: AbsRopeTableCache | None = None,) -> torch.Tensor:- """Fixed-``num_tiles`` W split + per-slab rotation (body of the opaque op).+ """Fixed-``num_tiles`` W split + per-slab rotation (body of the opaque op); tables from ``cache`` if given.+ With ``ltx-kernels`` installed, a CUDA bf16 volume at the fp32 compute dtype is rotated in+ one launch instead (:func:`ltx_kernels.vae.rope_dsl.run_abs_rope`); the split only bounds+ the slab path's fp32 temporaries, so it does not change the angles.The keyframe stream shares the video stream's W extent at every stage, so passing thesame ``num_tiles`` yields identical slab boundaries and identical ``w_pos`` -- whichis what keeps the two streams' W phases comparable inside the joint softmax."""+ if compute_dtype == torch.float32 and abs_rope_supported is not None and abs_rope_supported(x, rope_split):+ return run_abs_rope(x, rope_split, _rope_tables(x, inv_freqs, t_pos, cache))slabs = torch.chunk(x, num_tiles, dim=3)w_off = 0parts: list[torch.Tensor] = []@@ -83,12 +116,53 @@ def _apply_opaque_tiled_rope(w_pos=w_pos,compute_dtype=compute_dtype,t_pos=t_pos,+ w_pos_key=RopePositions(RopeAxis.W, w_slab, w_off),+ cache=cache,))w_off = w_off + w_slabreturn torch.cat(parts, dim=3)+def _rope_tables(+ x: torch.Tensor,+ inv_freqs: tuple[torch.Tensor, torch.Tensor, torch.Tensor],+ t_pos: torch.Tensor | None,+ cache: AbsRopeTableCache | None,+) -> tuple[torch.Tensor, ...]:+ """``(cos_t, sin_t, cos_h, sin_h, cos_w, sin_w)`` over ``x``'s T/H/W, as the slab path computes them.+ torch's own fp32 ``cos`` / ``sin`` of ``pos * inv``: the slab offsets make the W positions+ the plain ``arange(W)``, so these are exactly the angles :func:`rot_abs_axis_impl` uses.+ """+ _, t, h, w, _, _ = x.shape++ def build() -> tuple[torch.Tensor, ...]:+ positions = (+ t_positions(t, x.device) if t_pos is None else t_pos.to(device=x.device, dtype=torch.float32),+ h_positions(h, x.device),+ torch.arange(w, dtype=torch.float32, device=x.device),+ )+ tables: list[torch.Tensor] = []+ for pos, inv in zip(positions, inv_freqs, strict=True):+ angles = pos[:, None] * inv[None, :]+ tables += [angles.cos(), angles.sin()]+ return tuple(tables)++ # Integer positions are named by their extents; explicit plane times are not cached.+ key = (+ None+ if t_pos is not None+ else (+ RopePositions(RopeAxis.T, t),+ RopePositions(RopeAxis.H, h),+ RopePositions(RopeAxis.W, w),+ *(inv.data_ptr() for inv in inv_freqs),+ x.device,+ )+ )+ return cached_rope_tables(cache, key, build)++@torch.library.custom_op("ltx_core::abs_rope", mutates_args=())def _abs_rope_op(x: torch.Tensor,@@ -197,23 +271,40 @@ def _abs_rope_at_t_fake(return torch.empty(x.shape, device=x.device, dtype=x.dtype)-def _rope_config(attn: object, x: torch.Tensor) -> tuple[tuple[torch.Tensor, ...], int, torch.dtype]:- """``(inv_freqs, num_tiles, compute_dtype)`` read off an attention module."""- inv_freqs = (- attn.rope_inv_t.to(device=x.device), # type: ignore[attr-defined]- attn.rope_inv_h.to(device=x.device), # type: ignore[attr-defined]- attn.rope_inv_w.to(device=x.device), # type: ignore[attr-defined]- )+def _rope_config(+ attn: object, x: torch.Tensor+) -> tuple[tuple[torch.Tensor, ...], int, torch.dtype, AbsRopeTableCache | None]:+ """``(inv_freqs, num_tiles, compute_dtype, cache)`` read off an attention module.+ ``cache`` is the :class:`AbsRopeTableCache` the module holds for an enclosing+ :func:`~ltx_core.model.video_vae.transformer.rope_math.abs_rope_table_cache` scope, if this thread may+ read it (never while tracing or capturing). With one, the inverse frequencies are the cache's tensors of+ the module's ``rope_dim_split`` and ``rope_base``: the module's ``rope_inv_*`` buffers are+ :func:`rope_inv_freqs` of those (non-persistent, so never loaded), cast with the module, so the values+ are the same and every block of one configuration shares one set of cached tables.+ """+ buffers = (attn.rope_inv_t, attn.rope_inv_h, attn.rope_inv_w) # type: ignore[attr-defined]+ # Checked before the slot is read, so a traced graph never guards on the scope's cache.+ cache = None if torch.compiler.is_compiling() else getattr(attn, "rope_table_cache", None)+ if cache is not None and not cache.usable():+ cache = None+ if cache is None:+ inv_freqs = tuple(inv.to(device=x.device) for inv in buffers)+ else:+ base = float(attn.rope_base) # type: ignore[attr-defined]+ inv_freqs = tuple(+ cache.inv_freqs(dim, base, inv.dtype, x.device)+ for dim, inv in zip(attn.rope_dim_split, buffers, strict=True) # type: ignore[attr-defined]+ )num_tiles = getattr(attn, "rope_num_tiles", DEFAULT_ABS_ROPE_NUM_TILES)compute_dtype = getattr(attn, "rope_compute_dtype", torch.float32)- return inv_freqs, num_tiles, compute_dtype+ return inv_freqs, num_tiles, compute_dtype, cachedef _det_project_qkv(attn: object, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:"""Shared Q/K/V proj + norm + scale for :func:`det_qkv_rope` and ``_at_times``."""q, k, v = attn.project_qkv(x) # type: ignore[attr-defined]- q = attn.q_norm(q) # type: ignore[attr-defined]- k = attn.k_norm(k) # type: ignore[attr-defined]+ q = apply_rms_norm_uint32_safe(attn.q_norm, q) # type: ignore[attr-defined]+ k = apply_rms_norm_uint32_safe(attn.k_norm, k) # type: ignore[attr-defined]q = q * attn.scale # type: ignore[attr-defined]return q, k, v@@ -232,11 +323,26 @@ def det_qkv_rope_at_times("""if t_pos.ndim != 1 or t_pos.shape[0] != x.shape[1]:raise ValueError(f"t_pos must be ({x.shape[1]},) to match the plane axis, got {tuple(t_pos.shape)}")- q, k, v = _det_project_qkv(attn, x)+ q, k, v = _det_project_qkv(attn, x) if torch.compiler.is_compiling() else det_project_qkv_eager(attn, x)- inv_freqs, num_tiles, compute_dtype = _rope_config(attn, x)+ inv_freqs, num_tiles, compute_dtype, cache = _rope_config(attn, x)d_t, d_h, d_w = attn.rope_dim_split # type: ignore[attr-defined]positions = t_pos.to(device=x.device, dtype=torch.float32)+ if cache is not None:+ # Eager under a table scope: the op's body itself, so the H/W tables come from the scope's cache.+ rotated = [+ _apply_opaque_tiled_rope(+ tensor,+ (d_t, d_h, d_w),+ inv_freqs,+ num_tiles=num_tiles,+ compute_dtype=compute_dtype,+ t_pos=positions,+ cache=cache,+ )+ for tensor in (q, k)+ ]+ return rotated[0], rotated[1], vrotated = [_abs_rope_at_t_op(tensor,@@ -256,19 +362,84 @@ def det_qkv_rope_at_times(def det_qkv_rope(attn: object, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:"""Q/K/V proj + norm/scale + opaque full-volume abs-RoPE."""q, k, v = _det_project_qkv(attn, x)- inv_freqs, num_tiles, compute_dtype = _rope_config(attn, x)- q = _apply_opaque_abs_rope(- q,- attn.rope_dim_split, # type: ignore[attr-defined]- inv_freqs,- num_tiles=num_tiles,- compute_dtype=compute_dtype,- )- k = _apply_opaque_abs_rope(- k,- attn.rope_dim_split, # type: ignore[attr-defined]- inv_freqs,- num_tiles=num_tiles,- compute_dtype=compute_dtype,- )+ inv_freqs, num_tiles, compute_dtype, cache = _rope_config(attn, x)+ rope_split = attn.rope_dim_split # type: ignore[attr-defined]+ if cache is not None:+ # Eager under a table scope: the op's body itself, with the scope's cached tables.+ q, k = (+ _apply_opaque_tiled_rope(+ t, rope_split, inv_freqs, num_tiles=num_tiles, compute_dtype=compute_dtype, cache=cache+ )+ for t in (q, k)+ )+ return q, k, v+ q = _apply_opaque_abs_rope(q, rope_split, inv_freqs, num_tiles=num_tiles, compute_dtype=compute_dtype)+ k = _apply_opaque_abs_rope(k, rope_split, inv_freqs, num_tiles=num_tiles, compute_dtype=compute_dtype)+ return q, k, v+++def linear_into_phase_buffer(linear: torch.nn.Linear, x: torch.Tensor) -> torch.Tensor:+ """``linear(x)`` for a contiguous channels-last ``x``, into a workspace phase buffer when one is active.+ Exactly ``addmm(bias, x.view(-1, K), W.t())`` with an ``out=``, so the values are those of ``nn.Linear``.+ """+ if not x.is_contiguous() or linear.bias is None:+ return linear(x)+ rows = x.view(-1, linear.in_features)+ out = decode_workspace.phase_empty((rows.shape[0], linear.out_features), dtype=x.dtype, device=x.device)+ torch.addmm(linear.bias, rows, linear.weight.t(), out=out)+ return out.view(*x.shape[:-1], linear.out_features)+++def det_project_qkv_lean(attn: object, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:+ """:func:`_det_project_qkv` for the eager dual-stream det blocks: projections into workspace phase+ buffers, norms (and Q's scale) in place. Same values."""+ batch, t, h, w, _ = x.shape+ qkv = attn.qkv # type: ignore[attr-defined]+ shape = (batch, t, h, w, attn.num_heads, attn.head_dim) # type: ignore[attr-defined]+ q = linear_into_phase_buffer(qkv.to_q, x).view(shape)+ k = linear_into_phase_buffer(qkv.to_k, x).view(shape)+ v = linear_into_phase_buffer(qkv.to_v, x).view(shape)+ q = attn.q_norm.forward_(q, scale=attn.scale) # type: ignore[attr-defined]+ k = attn.k_norm.forward_(k) # type: ignore[attr-defined]return q, k, v+++def det_project_qkv_eager(attn: object, x: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor, torch.Tensor]:+ """:func:`_det_project_qkv` on fresh projections with the norms (and Q's scale) in place. Same values."""+ q, k, v = attn.project_qkv(x) # type: ignore[attr-defined]+ q = attn.q_norm.forward_(q, scale=attn.scale) # type: ignore[attr-defined]+ k = attn.k_norm.forward_(k) # type: ignore[attr-defined]+ return q, k, v+++def det_rope_qk_(attn: object, q: torch.Tensor, k: torch.Tensor) -> tuple[torch.Tensor, torch.Tensor]:+ """The RoPE half of :func:`det_qkv_rope`, rotating contiguous ``q`` and ``k`` in place.+ One :func:`~ltx_kernels.vae.rope_dsl.run_abs_rope` launch per tensor where it applies, else W slab by+ W slab, each slab read in full before it is overwritten; both equal the out-of-place rotation bit for+ bit. Eager callers only: the compiled det path keeps the pure ``ltx_core::abs_rope`` op.+ """+ inv_freqs, num_tiles, compute_dtype, cache = _rope_config(attn, q)+ rope_split = attn.rope_dim_split # type: ignore[attr-defined]+ for x in (q, k):+ if not x.is_contiguous():+ raise ValueError("det_rope_qk_ needs contiguous Q/K (fresh projections)")+ if compute_dtype == torch.float32 and abs_rope_supported is not None and abs_rope_supported(x, rope_split):+ run_abs_rope(x, rope_split, _rope_tables(x, inv_freqs, None, cache), out=x)+ continue+ w_off = 0+ for slab in torch.chunk(x, num_tiles, dim=3):+ w_slab = slab.shape[3]+ w_pos = torch.arange(w_slab, dtype=torch.float32, device=x.device) + w_off+ slab.copy_(+ _apply_opaque_rope_slab(+ slab,+ rope_split,+ inv_freqs,+ w_pos=w_pos,+ compute_dtype=compute_dtype,+ w_pos_key=RopePositions(RopeAxis.W, w_slab, w_off),+ cache=cache,+ )+ )+ w_off = w_off + w_slab+ return q, kdiff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/det_rms_norm.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/det_rms_norm.pynew file mode 100644index 0000000..0082e65--- /dev/null+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/det_rms_norm.py@@ -0,0 +1,68 @@+"""RMSNorm of the DiffVAE deterministic blocks: the fused row kernel or the uint32-safe norm, chosen per call."""++from __future__ import annotations++import torch+from torch import nn++from ltx_core.model.video_vae import decode_workspace+from ltx_core.model.video_vae.ops import apply_rms_norm_uint32_safe+from ltx_core.model.video_vae.transformer.row_rms_norm import row_rms_norm_, row_rms_norm_supported++# Rows of a Q/K volume (or of a block input) normalised per chunk when the row kernel does not apply.+_NORM_CHUNK_ROWS: int = 1 << 22+++class DetRMSNorm(nn.RMSNorm):+ """``nn.RMSNorm`` (same parameters and state dict) for the det blocks' block and Q/K norms.+ Traced, it is the uint32-safe functional norm (:func:`~ltx_core.model.video_vae.ops.rms_norm_uint32_safe`),+ so compiled graphs keep the functional op. Eager, it goes through the fused row kernel+ (:mod:`~ltx_core.model.video_vae.transformer.row_rms_norm`; ATen's arithmetic, the fp32 sum of squares may+ round differently, 1 bf16 ulp on ~2e-6 of the elements) where that applies, else the uint32-safe norm+ row chunk by row chunk; :meth:`forward` writes into a decode-workspace phase buffer when one is active,+ :meth:`forward_` back into its input.+ """++ def _fused(self, rows: torch.Tensor) -> bool:+ """Whether the fused row kernel serves this norm on 2-D ``rows``."""+ if self.weight is None or self.eps is None or tuple(self.normalized_shape) != (rows.shape[-1],):+ return False+ return bool(row_rms_norm_supported(rows, self.weight))++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ """``norm(x)`` over the last dim, into a workspace phase buffer when one is active, else a fresh tensor."""+ if torch.compiler.is_compiling() or not x.is_contiguous():+ return apply_rms_norm_uint32_safe(self, x)+ rows_in = x.view(-1, x.shape[-1])+ fused = self._fused(rows_in)+ if not decode_workspace.workspace_active():+ if not fused:+ return apply_rms_norm_uint32_safe(self, x)+ out = torch.empty_like(x)+ row_rms_norm_(rows_in, self.weight, float(self.eps), out=out.view(-1, x.shape[-1]))+ return out+ out = decode_workspace.phase_empty(tuple(x.shape), dtype=x.dtype, device=x.device)+ rows_out = out.view(-1, x.shape[-1])+ if fused:+ row_rms_norm_(rows_in, self.weight, float(self.eps), out=rows_out)+ return out+ for r0 in range(0, rows_in.shape[0], _NORM_CHUNK_ROWS):+ rows_out[r0 : r0 + _NORM_CHUNK_ROWS].copy_(+ apply_rms_norm_uint32_safe(self, rows_in[r0 : r0 + _NORM_CHUNK_ROWS])+ )+ return out++ def forward_(self, x: torch.Tensor, *, scale: float | None = None) -> torch.Tensor:+ """``norm(x)`` (then ``* scale``, a second bf16 rounding as ``x.mul_(scale)`` after the norm is) written+ back into a contiguous ``x``, RMSNorm being per row; a new tensor for a non-contiguous ``x``. Eager only."""+ if not x.is_contiguous():+ y = apply_rms_norm_uint32_safe(self, x)+ return y if scale is None else y.mul_(scale)+ rows = x.view(-1, x.shape[-1])+ if self._fused(rows):+ row_rms_norm_(rows, self.weight, float(self.eps), scale=scale)+ return x+ for r0 in range(0, rows.shape[0], _NORM_CHUNK_ROWS):+ chunk = rows[r0 : r0 + _NORM_CHUNK_ROWS]+ chunk.copy_(apply_rms_norm_uint32_safe(self, chunk))+ return x if scale is None else x.mul_(scale)diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/attn.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/attn.pyindex 99cc85e..2f17c94 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/attn.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/attn.py@@ -35,16 +35,17 @@ def require_softmax_bound(attn: NeighborhoodAttention3D) -> torch.Tensor:Apply registers a NaN 0-d buffer; SDOps overwrite it at load. Rejecting meta, empty,and non-finite values catches both a missed key under ``assign=True`` (meta leftbehind) and an apply-without-SDOps path that left the NaN sentinel on CUDA.- The non-finite probe is skipped while ``torch.compile`` is tracing: ``isfinite().all()``- is data-dependent and forces specialization that conflicts with the decoder's hard- ``mark_dynamic`` on T/H/W. Meta/empty checks still run under compile; the NaN sentinel- is caught on eager first-call warmups and non-compiled paths.+ The non-finite probe is an asynchronous assert: it raises at once for a CPU buffer, and+ as a device-side assert for a CUDA one, so the per-block call never blocks the host on+ the GPU. It is skipped while ``torch.compile`` is tracing: ``isfinite().all()`` is+ data-dependent and forces specialization that conflicts with the decoder's hard+ ``mark_dynamic`` on T/H/W. Meta/empty checks still run under compile."""bound = getattr(attn, NA_SOFTMAX_BOUND_BUFFER, None)if bound is None or bound.is_meta or bound.numel() == 0:raise RuntimeError(_SOFTMAX_BOUND_UNSET)- if not torch.compiler.is_compiling() and not torch.isfinite(bound).all():- raise RuntimeError(_SOFTMAX_BOUND_UNSET)+ if not torch.compiler.is_compiling():+ torch._assert_async(torch.isfinite(bound).all(), _SOFTMAX_BOUND_UNSET)return bounddiff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/block.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/block.pyindex 223805a..2d71a45 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/block.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/block.py@@ -2,11 +2,15 @@from __future__ import annotations+from typing import TYPE_CHECKING, NamedTuple+import torch+from ltx_core.devices import upload_from_hostfrom ltx_core.model.video_vae.transformer.blocks import DiffusionNABlockfrom ltx_core.model.video_vae.transformer.dsl_kernels.attn import require_softmax_boundfrom ltx_core.model.video_vae.transformer.layers import AdaLNZero, LinearPixelShuffleUpsample+from ltx_core.types import HostDeviceTensorPair#: The stage-4 upsample fused into ``context_proj``, packed by SDOps as one Linear per#: pixel-shuffle subpixel: ``(P * C, Ctx)`` / ``(P * C,)``. Registered empty by@@ -14,6 +18,19 @@ from ltx_core.model.video_vae.transformer.layers import AdaLNZero, LinearPixelShFUSED_CTX_WEIGHT_BUFFER = "fused_ctx_weight"FUSED_CTX_BIAS_BUFFER = "fused_ctx_bias"+if TYPE_CHECKING:+ from ltx_core.model.video_vae.keyframes import KeyframePlanes+++class LiveKeyframes(NamedTuple):+ """The anchor stream of one fused launch: only its live planes, since the kernel has no notion of validity."""++ x: torch.Tensor+ stage4_feat: torch.Tensor+ """The *pre*-upsample stage-4 feature, like the video's."""+ times: HostDeviceTensorPair+ """The planes' times; the kernel builds its plane tables from the host tensor, without a device read."""+def fused_ctx_weights(block: DiffusionNABlock) -> "tuple[torch.Tensor, torch.Tensor] | None":"""This block's packed fused-context weights, or ``None`` if they never landed.@@ -94,8 +111,7 @@ class DSLDiffusionNABlock(DiffusionNABlock):keyframe_x: torch.Tensor,keyframe_stage4_feat: torch.Tensor,modulation: tuple[torch.Tensor, ...],- keyframe_times: torch.Tensor,- keyframe_valid: torch.Tensor,+ keyframe_planes: KeyframePlanes,*,drop_leading_frame: bool = True,out: torch.Tensor | None = None,@@ -107,35 +123,43 @@ class DSLDiffusionNABlock(DiffusionNABlock):``drop_leading_frame`` is the video stream's alone: each plane is a ``T=1`` clip whosetemporal stride collapses, so the kernel always takes the subpixel a one-frame inputshuffles out to. That is a tiling property and must not leak into the anchor arm.- ``keyframe_valid`` is applied by dropping the invalid planes before the launch and+ ``keyframe_planes.valid`` is applied by dropping the invalid planes before the launch andrestoring their rows as zeros -- the kernel has no notion of validity, and dropping isorder-preserving, so the surviving planes rank identically under the ``(|dt|, index)``- tie-break. Zero rows are the contract: an invalid plane's every key is masked.+ tie-break. Zero rows are the contract: an invalid plane's every key is masked. The live planes+ and the kernel's plane tables come from the planes' host tensors, so the launch never reads+ the device."""- keep = keyframe_valid.nonzero().flatten()- if int(keep.numel()) == int(keyframe_x.shape[1]):+ keyframe_times, keyframe_valid = keyframe_planes+ keep_host = keyframe_valid.host_tensor.nonzero().flatten()+ all_live = int(keep_host.numel()) == int(keyframe_x.shape[1])+ if all_live:return self._fused_forward_x_ctx(x,stage4_feat,modulation,drop_leading_frame=drop_leading_frame,out=out,- keyframes=(keyframe_x, keyframe_stage4_feat, keyframe_times),+ keyframes=LiveKeyframes(keyframe_x, keyframe_stage4_feat, keyframe_times),out_keyframes=out_keyframes,)# Dropping planes shortens the anchor stream, so a buffer sized for the full plane# count no longer fits it -- and the result is scattered into a fresh tensor anyway.# The anchor stream forgoes recycling for this launch; the video stream does not.+ keep = upload_from_host(keep_host, keyframe_x.device)video_out, live = self._fused_forward_x_ctx(x,stage4_feat,modulation,drop_leading_frame=drop_leading_frame,out=out,- keyframes=(+ keyframes=LiveKeyframes(keyframe_x.index_select(1, keep),keyframe_stage4_feat.index_select(1, keep),- keyframe_times.index_select(0, keep),+ HostDeviceTensorPair(+ keyframe_times.host_tensor.index_select(0, keep_host),+ keyframe_times.device_tensor.index_select(0, keep),+ ),),)restored = keyframe_x.new_zeros(keyframe_x.shape)@@ -152,15 +176,14 @@ class DSLDiffusionNABlock(DiffusionNABlock):*,drop_leading_frame: bool,out: torch.Tensor | None,- keyframes: "tuple[torch.Tensor, torch.Tensor, torch.Tensor] | None" = None,+ keyframes: LiveKeyframes | None = None,out_keyframes: torch.Tensor | None = None,) -> "torch.Tensor | tuple[torch.Tensor, torch.Tensor]":"""Marshal weights into :func:`ltx_kernels.vae.run_block_fna_dsl`.- ``keyframes`` is ``(x, stage4_feat, times)`` for the anchor stream, and passing it- makes the launch dual-stream and the return a pair. ``out_keyframes`` is that- stream's recycled output buffer, the anchor-side counterpart of ``out``. Its stage-4 feature is the- *pre*-upsample one, like the video's: the kernel runs the deferred hop over each- plane in isolation, taking the temporal subpixel a one-frame input shuffles out to.+ ``keyframes`` is the anchor stream, and passing it makes the launch dual-stream and the return a+ pair. ``out_keyframes`` is that stream's recycled output buffer, the anchor-side counterpart of+ ``out``. The kernel runs the deferred hop over each plane in isolation, taking the temporal+ subpixel a one-frame input shuffles out to."""from ltx_kernels.vae import run_block_fna_dsl # noqa: PLC0415@@ -214,9 +237,10 @@ class DSLDiffusionNABlock(DiffusionNABlock):{}if keyframes is Noneelse {- "x_keyframes": keyframes[0],- "stage4_keyframes": keyframes[1],- "keyframe_times": keyframes[2],+ "x_keyframes": keyframes.x,+ "stage4_keyframes": keyframes.stage4_feat,+ "keyframe_times": keyframes.times.device_tensor,+ "keyframe_times_host": keyframes.times.host_tensor,"out_keyframes": out_keyframes,}),@@ -227,4 +251,4 @@ class DSLDiffusionNABlock(DiffusionNABlock):if keyframes is None:return result.to(dtype=x.dtype)video_out, keyframe_out = result- return video_out.to(dtype=x.dtype), keyframe_out.to(dtype=keyframes[0].dtype)+ return video_out.to(dtype=x.dtype), keyframe_out.to(dtype=keyframes.x.dtype)diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/chain.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/chain.pyindex c3bcd19..eb177d1 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/chain.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/chain.py@@ -2,22 +2,23 @@from __future__ import annotations+from typing import TYPE_CHECKING+import torchfrom torch import nn+from ltx_core.model.video_vae import decode_workspacefrom ltx_core.model.video_vae.transformer.layers import ChannelLinear--def _out_slack_rows() -> int:- from ltx_kernels.vae import OUT_SLACK_ROWS # noqa: PLC0415-- return int(OUT_SLACK_ROWS)+if TYPE_CHECKING:+ from ltx_core.model.video_vae.keyframes import KeyframePlanesdef alloc_block_volume(shape: tuple[int, ...], *, device: torch.device, dtype: torch.dtype) -> torch.Tensor:- """A ``(1,T,H,W,C)`` tensor whose storage carries the fused kernel's lane slack."""+ """A ``(1,T,H,W,C)`` tensor whose storage carries the fused kernel's lane slack, leased from the active+ decode workspace until its tile ends (``torch.empty`` style without one)."""_, t, h, w, c = shape- rows = torch.empty((t * h * w + _out_slack_rows(), c), device=device, dtype=dtype)+ rows = decode_workspace.volume((t * h * w + decode_workspace.out_slack_rows(), c), device=device, dtype=dtype)return rows[: t * h * w].view(1, t, h, w, c)@@ -38,7 +39,7 @@ def _recyclable_rows(volume: torch.Tensor) -> torch.Tensor | None:return Nonechannels = volume.shape[-1]n_rows = volume.numel() // channels- want = n_rows + _out_slack_rows()+ want = n_rows + decode_workspace.out_slack_rows()have = volume.untyped_storage().nbytes() // (volume.element_size() * channels)if not volume.is_contiguous() or have < want:return None@@ -91,8 +92,7 @@ class DSLDiffusionBlockChain(nn.ModuleList):keyframe_x: torch.Tensor,keyframe_stage4_feat: torch.Tensor,modulation: tuple[torch.Tensor, ...],- keyframe_times: torch.Tensor,- keyframe_valid: torch.Tensor,+ keyframe_planes: KeyframePlanes,*,drop_leading_frame: bool = True,) -> tuple[torch.Tensor, torch.Tensor]:@@ -112,8 +112,7 @@ class DSLDiffusionBlockChain(nn.ModuleList):keyframe_x,keyframe_stage4_feat,modulation,- keyframe_times,- keyframe_valid,+ keyframe_planes,drop_leading_frame=drop_leading_frame,out=spare,out_keyframes=keyframe_spare,diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/joint_attn.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/joint_attn.pyindex 69d090d..a751cc5 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/joint_attn.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/dsl_kernels/joint_attn.py@@ -22,6 +22,8 @@ in the kernel:from __future__ import annotations+from typing import TYPE_CHECKING+import torchfrom ltx_core.model.video_vae.transformer.attention import (@@ -34,6 +36,9 @@ from ltx_core.model.video_vae.transformer.dsl_kernels.attn import (require_softmax_bound,)+if TYPE_CHECKING:+ from ltx_core.model.video_vae.keyframes import KeyframePlanes+@torch.library.custom_op("ltx_core::na_attention_joint_dsl", mutates_args=())def na_attention_joint_dsl( # noqa: PLR0913 -- a custom_op schema takes only flat tensors/ints@@ -57,9 +62,14 @@ def na_attention_joint_dsl( # noqa: PLR0913 -- a custom_op schema takes only flfrom ltx_kernels.vae import run_na_attention_bound # noqa: PLC0415from ltx_kernels.vae.keyframe_slots import nearest_keyframe_slots # noqa: PLC0415+ from ltx_core.devices import upload_from_host # noqa: PLC0415+planes = int(keyframe_q.shape[1])- keep = keyframe_valid.nonzero().flatten()- live_times = keyframe_times.index_select(0, keep)+ # Plane selection and slot tables are host work; free when the times and validity arrive on+ # the host, one read of each otherwise.+ keep_host = keyframe_valid.detach().cpu().nonzero().flatten()+ live_times = keyframe_times.detach().cpu().index_select(0, keep_host)+ keep = upload_from_host(keep_host, q.device)video_out, keyframe_out = run_na_attention_bound(q,k,@@ -143,13 +153,14 @@ class DSLJointAttention(JointNAAttentionCallable):keyframe_q: torch.Tensor,keyframe_k: torch.Tensor,keyframe_v: torch.Tensor,- keyframe_times: torch.Tensor,- keyframe_valid: torch.Tensor,+ keyframe_planes: KeyframePlanes,) -> tuple[torch.Tensor, torch.Tensor]:from ltx_core.model.video_vae.keyframes import KEYFRAME_CONTEXT_SLOTS # noqa: PLC0415if not self._available or not dsl_joint_supported(attn, q):- return self._fallback(attn, q, k, v, keyframe_q, keyframe_k, keyframe_v, keyframe_times, keyframe_valid)+ return self._fallback(attn, q, k, v, keyframe_q, keyframe_k, keyframe_v, keyframe_planes)+ # The op builds its plane tables on the host, so it gets the planes' host tensors.+ keyframe_times, keyframe_valid = keyframe_planes.times.host_tensor, keyframe_planes.valid.host_tensordtype = keyframe_v.dtypereturn na_attention_joint_dsl(q.to(dtype=dtype),diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/fallback_na/__init__.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/fallback_na/__init__.pyindex 68f8225..923cf35 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/fallback_na/__init__.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/fallback_na/__init__.py@@ -19,6 +19,7 @@ from ltx_core.model.video_vae.transformer.fallback_na.joint_eager import ()if TYPE_CHECKING:+ from ltx_core.model.video_vae.keyframes import KeyframePlanesfrom ltx_core.model.video_vae.transformer.attention import NAAttentionCallable, NeighborhoodAttention3Dlogger = logging.getLogger(__name__)@@ -227,8 +228,7 @@ class TritonJointAttention:keyframe_q: torch.Tensor,keyframe_k: torch.Tensor,keyframe_v: torch.Tensor,- keyframe_times: torch.Tensor,- keyframe_valid: torch.Tensor,+ keyframe_planes: KeyframePlanes,) -> tuple[torch.Tensor, torch.Tensor]:from ltx_core.model.video_vae.keyframes import KEYFRAME_CONTEXT_SLOTS # noqa: PLC0415@@ -239,8 +239,8 @@ class TritonJointAttention:)return na_attention_joint_triton(*_joint_dtypes(q, k, v, keyframe_q, keyframe_k, keyframe_v),- keyframe_times,- keyframe_valid,+ keyframe_planes.times.device_tensor,+ keyframe_planes.valid.device_tensor,list(attn.kernel_size),KEYFRAME_CONTEXT_SLOTS,)@@ -261,15 +261,14 @@ class EagerJointAttention:keyframe_q: torch.Tensor,keyframe_k: torch.Tensor,keyframe_v: torch.Tensor,- keyframe_times: torch.Tensor,- keyframe_valid: torch.Tensor,+ keyframe_planes: KeyframePlanes,) -> tuple[torch.Tensor, torch.Tensor]:from ltx_core.model.video_vae.keyframes import KEYFRAME_CONTEXT_SLOTS # noqa: PLC0415return na_attention_joint_eager(*_joint_dtypes(q, k, v, keyframe_q, keyframe_k, keyframe_v),- keyframe_times,- keyframe_valid,+ keyframe_planes.times.device_tensor,+ keyframe_planes.valid.device_tensor,list(attn.kernel_size),KEYFRAME_CONTEXT_SLOTS,)@@ -296,11 +295,10 @@ class AutoJointAttention:keyframe_q: torch.Tensor,keyframe_k: torch.Tensor,keyframe_v: torch.Tensor,- keyframe_times: torch.Tensor,- keyframe_valid: torch.Tensor,+ keyframe_planes: KeyframePlanes,) -> tuple[torch.Tensor, torch.Tensor]:backend = self._triton if self._triton is not None and q.is_cuda else self._eager- return backend(attn, q, k, v, keyframe_q, keyframe_k, keyframe_v, keyframe_times, keyframe_valid)+ return backend(attn, q, k, v, keyframe_q, keyframe_k, keyframe_v, keyframe_planes)class EagerSdpaAttention:diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/layers.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/layers.pyindex f41a1b3..e44e01b 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/layers.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/layers.py@@ -9,6 +9,9 @@ import torch.nn.functional as Ffrom einops import rearrangefrom torch import nn+# Bytes of projection output per chunk of LinearPixelShuffleUpsample's eager forward.+_SHUFFLE_CHUNK_BYTES: int = 1 << 30+class ChannelLinear(nn.Linear):"""``nn.Linear`` exposing ``in_channels``/``out_channels`` for config introspection."""@@ -47,18 +50,41 @@ class LinearPixelShuffleUpsample(nn.Module):frame of its own to drop, since the one duplicate frame in the full(untiled) tensor belongs solely to the origin chunk."""- x = self.proj(x)- x = rearrange(- x,- "b t h w (c p1 p2 p3) -> b (t p1) (h p2) (w p3) c",- p1=self.stride[0],- p2=self.stride[1],- p3=self.stride[2],- )+ if torch.compiler.is_compiling():+ x = self.proj(x)+ x = rearrange(+ x,+ "b t h w (c p1 p2 p3) -> b (t p1) (h p2) (w p3) c",+ p1=self.stride[0],+ p2=self.stride[1],+ p3=self.stride[2],+ )+ else:+ x = self._proj_shuffle(x)if self.stride[0] == 2 and drop_leading_frame:x = x[:, 1:, :, :, :]return x+ def _proj_shuffle(self, x: torch.Tensor) -> torch.Tensor:+ """``rearrange(proj(x), "b t h w (c p1 p2 p3) -> b (t p1) (h p2) (w p3) c")``, chunked over ``t``.+ A per-row Linear plus a permutation, so the chunked result is the one-shot result; one output+ volume is live plus at most ``_SHUFFLE_CHUNK_BYTES`` of projection.+ """+ b, t, h, w, _ = x.shape+ p1, p2, p3 = self.stride+ c = self.out_channels+ out = torch.empty((b, t * p1, h * p2, w * p3, c), device=x.device, dtype=x.dtype)+ # (b, t, p1, h, p2, w, p3, c): the output with every shuffled axis split back out.+ out_split = out.view(b, t, p1, h, p2, w, p3, c)+ frame_bytes = b * h * w * self.proj_out_channels * x.element_size()+ chunk = max(1, min(t, _SHUFFLE_CHUNK_BYTES // max(frame_bytes, 1)))+ for t0 in range(0, t, chunk):+ t1 = min(t, t0 + chunk)+ y = self.proj(x[:, t0:t1])+ y = y.view(b, t1 - t0, h, w, c, p1, p2, p3).permute(0, 1, 5, 2, 6, 3, 7, 4)+ out_split[:, t0:t1].copy_(y)+ return out+class AdaLNZero(nn.Module):"""Per-block AdaLN-Zero modulation: ``t_emb`` -> 7 (scale/shift/gate) chunks.diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/rope_math.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/rope_math.pyindex a671a8c..482e12e 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/rope_math.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/rope_math.py@@ -2,12 +2,111 @@from __future__ import annotations+import enum+import threading+from collections.abc import Callable, Hashable, Iterator+from contextlib import contextmanager+from dataclasses import dataclass+import numpy as npimport torchDEFAULT_ABS_ROPE_NUM_TILES = 4+class RopeAxis(enum.Enum):+ """An axis of the channels-last ``(B, T, H, W, ...)`` volume abs-RoPE rotates."""++ T = "t"+ H = "h"+ W = "w"+++@dataclass(frozen=True)+class RopePositions:+ """The integer positions ``offset + arange(extent)`` on ``axis``, named by value for the table cache."""++ axis: RopeAxis+ extent: int+ offset: int = 0+++class AbsRopeTableCache:+ """The abs-RoPE inputs one decode scope reuses: cos/sin tables by key and inverse frequencies by configuration.+ Opened by :func:`abs_rope_table_cache`, which hands it to the decoder's attention modules for the scope.+ Only the thread that opened it reads it, so every table is used on the stream order it was built in. Keys+ name positions by value (:class:`RopePositions`) and inverse frequencies by this cache's own tensors, so a+ cached table is exactly the one a fresh build would produce.+ """++ def __init__(self) -> None:+ self._owner = threading.get_ident()+ self._tables: dict[Hashable, tuple[torch.Tensor, ...]] = {}+ self._inv_freqs: dict[tuple[int, float, torch.dtype, torch.device], torch.Tensor] = {}++ def usable(self) -> bool:+ """Whether this thread may read the cache: the opening thread, outside CUDA graph capture. Traced code+ never reaches the cache (see ``det_attn_rope._rope_config``)."""+ if threading.get_ident() != self._owner:+ return False+ return not (torch.cuda.is_available() and torch.cuda.is_current_stream_capturing())++ def tables(self, key: Hashable, build: Callable[[], tuple[torch.Tensor, ...]]) -> tuple[torch.Tensor, ...]:+ """``build()``, kept under ``key`` for the rest of the scope."""+ tables = self._tables.get(key)+ if tables is None:+ tables = build()+ self._tables[key] = tables+ return tables++ def inv_freqs(self, dim: int, base: float, dtype: torch.dtype, device: torch.device) -> torch.Tensor:+ """:func:`rope_inv_freqs` as ``dtype`` on ``device``, one tensor per configuration for the scope.+ Attention blocks of one configuration rotate with this tensor instead of their own equal-valued+ buffers, so table keys (which name the frequencies by ``data_ptr``) match across blocks.+ """+ key = (dim, base, dtype, device)+ inv = self._inv_freqs.get(key)+ if inv is None:+ inv = rope_inv_freqs(dim, base).to(device=device, dtype=dtype)+ self._inv_freqs[key] = inv+ return inv+++@contextmanager+def abs_rope_table_cache(module: object) -> Iterator[AbsRopeTableCache]:+ """Give ``module``'s attention blocks (submodules with a ``rope_table_cache`` slot) one+ :class:`AbsRopeTableCache` for the scope, so same-extent tiles and blocks build their tables once.+ A decoder that is not an ``nn.Module`` has no such blocks and gets a cache nothing reads.+ Re-entrant on one thread (an inner scope uses the outer cache); the cache is dropped when the scope+ that opened it exits. Blocks already holding another thread's cache keep it, and this thread builds+ its tables per call there.+ """+ modules = module.modules() if isinstance(module, torch.nn.Module) else ()+ holders = [m for m in modules if hasattr(m, "rope_table_cache")]+ for holder in holders:+ if holder.rope_table_cache is not None and holder.rope_table_cache.usable():+ yield holder.rope_table_cache+ return+ cache = AbsRopeTableCache()+ claimed = [m for m in holders if m.rope_table_cache is None]+ for holder in claimed:+ holder.rope_table_cache = cache+ try:+ yield cache+ finally:+ for holder in claimed:+ holder.rope_table_cache = None+++def cached_rope_tables(+ cache: AbsRopeTableCache | None, key: Hashable | None, build: Callable[[], tuple[torch.Tensor, ...]]+) -> tuple[torch.Tensor, ...]:+ """``build()``, reused under ``key`` from ``cache``; always rebuilt without a cache or a key."""+ if cache is None or key is None:+ return build()+ return cache.tables(key, build)++def t_positions(t: int, device: torch.device) -> torch.Tensor:return torch.arange(t, dtype=torch.float32, device=device)@@ -44,17 +143,25 @@ def rot_abs_axis_impl(axis: int,*,compute_dtype: torch.dtype,+ pos_key: RopePositions | None = None,+ cache: AbsRopeTableCache | None = None,) -> torch.Tensor:- """Absolute RoPE on one axis chunk ``xc[..., D]`` (D even) → new tensor."""+ """Absolute RoPE on one axis chunk ``xc[..., D]`` (D even) → new tensor.+ ``pos_key`` names ``pos`` by value, so the tables come from ``cache`` when one is given.+ """out_dtype = xc.dtypepairs = xc.reshape(*xc.shape[:-1], xc.shape[-1] // 2, 2)xe = pairs[..., 0].to(compute_dtype)xo = pairs[..., 1].to(compute_dtype)- shape = [1, 1, 1, 1, 1, inv.shape[0]]- shape[axis] = pos.shape[0]- ang = (pos[:, None] * inv[None, :]).reshape(shape)- c = ang.cos().to(compute_dtype)- s = ang.sin().to(compute_dtype)++ def build() -> tuple[torch.Tensor, torch.Tensor]:+ shape = [1, 1, 1, 1, 1, inv.shape[0]]+ shape[axis] = pos.shape[0]+ ang = (pos[:, None] * inv[None, :]).reshape(shape)+ return ang.cos().to(compute_dtype), ang.sin().to(compute_dtype)++ key = None if pos_key is None else ("axis", axis, pos_key, compute_dtype, inv.data_ptr(), inv.numel(), pos.device)+ c, s = cached_rope_tables(cache, key, build)re = xe * c - xo * sro = xe * s + xo * cout = torch.stack([re, ro], dim=-1).reshape(xc.shape)diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/row_rms_norm.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/row_rms_norm.pynew file mode 100644index 0000000..d6048a0--- /dev/null+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/row_rms_norm.py@@ -0,0 +1,106 @@+# ruff: noqa: ANN001, ANN201, ANN202, N803++"""Row RMSNorm for the DiffVAE in Triton: the in-place det-stage norms, and the norm the stage-5 tail fuses.+ATen's arithmetic (fp32 ``rstd = rsqrt(mean(x^2) + eps)``, ``y = x * rstd * w``, one rounding to the activation+dtype) at full bandwidth; the fp32 sum of squares may round differently (<1 bf16 ulp on a small fraction of rows).+:func:`rms_rstd` is its row reduction, shared by :func:`row_rms_norm_` and+:mod:`~ltx_core.model.video_vae.transformer.stage5_tail`.+"""++from __future__ import annotations++import torch++try:+ import triton+ import triton.language as tl+except (ImportError, OSError): # pragma: no cover - platform without Triton+ triton = None # type: ignore[assignment]+ tl = None # type: ignore[assignment]++_MAX_WIDTH = 1024+_BLOCK_ELEMS = 8192+++def row_rms_norm_supported(x: torch.Tensor, weight: torch.Tensor) -> bool:+ """Contiguous 2-D bf16 CUDA rows of width <= 1024, with a contiguous 1-D floating weight of that width on+ the same device, and Triton."""+ if triton is None or not x.is_cuda or x.dim() != 2 or not x.is_contiguous() or x.dtype != torch.bfloat16:+ return False+ if weight.dim() != 1 or weight.shape[0] != x.shape[1] or weight.device != x.device:+ return False+ if not weight.is_floating_point() or not weight.is_contiguous():+ return False+ return 1 <= x.shape[1] <= _MAX_WIDTH+++if triton is not None:++ @triton.jit+ def rms_rstd(x, N: tl.constexpr, eps):+ """``rsqrt(mean(x^2) + eps)`` per row of fp32 ``(ROWS, BLOCK_N)`` rows of width ``N`` (zero-padded past it).+ The caller scales: ``x * rstd[:, None] * w[None, :]``, then rounds.+ """+ return tl.rsqrt(tl.sum(x * x, axis=1) / N + eps)++ @triton.jit+ def _row_rms_norm_kernel(+ x_ptr,+ y_ptr,+ w_ptr,+ n_rows,+ eps,+ scale,+ N: tl.constexpr,+ BLOCK_N: tl.constexpr,+ ROWS: tl.constexpr,+ HAS_SCALE: tl.constexpr,+ ):+ pid = tl.program_id(0)+ row0 = pid * ROWS+ rows = row0 + tl.arange(0, ROWS)+ cols = tl.arange(0, BLOCK_N)+ row_ok = rows < n_rows+ col_ok = cols < N+ mask = row_ok[:, None] & col_ok[None, :]+ offs = rows[:, None].to(tl.int64) * N + cols[None, :]+ x = tl.load(x_ptr + offs, mask=mask, other=0.0).to(tl.float32)+ w = tl.load(w_ptr + cols, mask=col_ok, other=0.0).to(tl.float32)+ y = x * rms_rstd(x, N, eps)[:, None] * w[None, :]+ if HAS_SCALE:+ # The caller's ``q.mul_(scale)`` on the bf16 norm output: round first, then scale, then round.+ y = y.to(tl.bfloat16).to(tl.float32) * scale+ tl.store(y_ptr + offs, y.to(tl.bfloat16), mask=mask)+++def row_rms_norm_(+ x: torch.Tensor, weight: torch.Tensor, eps: float, *, scale: float | None = None, out: torch.Tensor | None = None+) -> torch.Tensor:+ """``rms_norm(x, weight, eps)`` (``* scale`` afterwards, as a second bf16 rounding) row by row.+ ``x`` is ``(rows, N)`` contiguous bf16 (see :func:`row_rms_norm_supported`). Written over ``x`` itself,+ or into ``out`` (same shape/dtype/contiguity, must not partially overlap ``x``). Returns the written tensor.+ """+ rows, n = x.shape+ y = x if out is None else out+ if out is not None and (out.shape != x.shape or out.dtype != x.dtype or not out.is_contiguous()):+ raise ValueError("out must be a contiguous tensor of x's shape and dtype")+ if rows == 0:+ return y+ block_n = triton.next_power_of_2(n)+ rows_per_cta = max(1, min(64, _BLOCK_ELEMS // block_n))+ grid = (triton.cdiv(rows, rows_per_cta),)+ w = weight if weight.dtype in (torch.bfloat16, torch.float32) else weight.to(torch.float32)+ _row_rms_norm_kernel[grid](+ x,+ y,+ w,+ rows,+ float(eps),+ float(scale) if scale is not None else 1.0,+ N=n,+ BLOCK_N=block_n,+ ROWS=rows_per_cta,+ HAS_SCALE=scale is not None,+ num_warps=4 if block_n <= 256 else 8,+ )+ return ydiff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/stage5_tail.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/stage5_tail.pynew file mode 100644index 0000000..9c97fbc--- /dev/null+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/stage5_tail.py@@ -0,0 +1,152 @@+# ruff: noqa: ANN001, N803, N806, PLR0913++"""DiffVAE stage-5 tail in one pass: ``norm_out`` -> ``conv_out`` -> unpatchify.+The eager tail makes four passes over the stage-5 volume (the normalised copy, the+projection, a channels-first copy, the depth-to-space copy) and holds two extra+full-size tensors. Here each block of tokens is normalised in fp32, rounded to the+activation dtype where ``nn.RMSNorm`` rounds, projected on the tensor cores with the+bias in the epilogue, and every output channel is stored straight to its pixel.+"""++from __future__ import annotations++import torch++try:+ import triton+ import triton.language as tl+except ImportError: # pragma: no cover+ triton = None # type: ignore[assignment]+ tl = None # type: ignore[assignment]++if triton is not None:+ from ltx_core.model.video_vae.transformer.row_rms_norm import rms_rstd++ @triton.jit+ def _stage5_tail_kernel(+ x_ptr,+ norm_w_ptr,+ proj_w_ptr,+ proj_b_ptr,+ out_ptr,+ n_tokens,+ F,+ Hh,+ Ww,+ eps,+ C: tl.constexpr,+ OUT: tl.constexpr,+ OUT_PAD: tl.constexpr,+ PATCH: tl.constexpr,+ BLOCK_T: tl.constexpr,+ ) -> None:+ """``out[b, c, f, h*P+q, w*P+r] = conv_out(norm_out(x))[b, f, h, w, c*P*P + r*P + q]``.+ int64 offsets: the stage-5 volume passes 2**31 elements at 1080p.+ """+ tokens = tl.program_id(0).to(tl.int64) * BLOCK_T + tl.arange(0, BLOCK_T).to(tl.int64)+ token_mask = tokens < n_tokens+ channels = tl.arange(0, C)+ x = tl.load(x_ptr + tokens[:, None] * C + channels[None, :], mask=token_mask[:, None], other=0.0)+ x = x.to(tl.float32)+ rstd = rms_rstd(x, C, eps)+ normed = (x * rstd[:, None] * tl.load(norm_w_ptr + channels).to(tl.float32)[None, :]).to(x_ptr.dtype.element_ty)++ outs = tl.arange(0, OUT_PAD)+ out_mask = outs < OUT+ w = tl.load(proj_w_ptr + outs[None, :] * C + channels[:, None], mask=out_mask[None, :], other=0.0)+ acc = tl.dot(normed, w)+ acc += tl.load(proj_b_ptr + outs, mask=out_mask, other=0.0).to(tl.float32)[None, :]++ # (T, c*P*P + r*P + q) -> (c*P + q, t*P + r): each row is a run of consecutive pixels+ # along W, so the store coalesces.+ C_PAD: tl.constexpr = OUT_PAD // (PATCH * PATCH)+ vals = tl.reshape(acc.to(out_ptr.dtype.element_ty), (BLOCK_T, C_PAD, PATCH, PATCH))+ vals = tl.reshape(tl.permute(vals, (1, 3, 0, 2)), (C_PAD * PATCH, BLOCK_T * PATCH))+ rows = tl.arange(0, C_PAD * PATCH)+ cols = tl.arange(0, BLOCK_T * PATCH)+ c = rows // PATCH+ q = rows % PATCH+ t = tl.program_id(0).to(tl.int64) * BLOCK_T + (cols // PATCH).to(tl.int64)+ r = cols % PATCH+ ww = t % Ww+ hh = (t // Ww) % Hh+ f = (t // Ww // Hh) % F+ b = t // Ww // Hh // F+ H = Hh * PATCH+ W = Ww * PATCH+ offsets = (+ (+ ((b[None, :] * (OUT // (PATCH * PATCH)) + c[:, None]) * F + f[None, :]) * H+ + hh[None, :] * PATCH+ + q[:, None]+ )+ * W+ + ww[None, :] * PATCH+ + r[None, :]+ )+ tl.store(out_ptr + offsets, vals, mask=(c < OUT // (PATCH * PATCH))[:, None] & (t < n_tokens)[None, :])+++def stage5_tail_supported(x: torch.Tensor, norm: torch.nn.RMSNorm, proj: torch.nn.Linear, patch_size: int) -> bool:+ """Whether :func:`stage5_tail` takes this ``(B, T, H, W, C)`` channels-last volume.+ Not while ``torch.compile`` traces: the compiled DiffVAE modes leave the tail to Inductor.+ """+ c = x.shape[-1]+ return (+ triton is not None+ and not torch.compiler.is_compiling()+ and x.is_cuda+ and x.dim() == 5+ and x.dtype in (torch.bfloat16, torch.float16)+ and x.is_contiguous()+ and c >= 16+ and c & (c - 1) == 0+ and norm.weight is not None+ and proj.bias is not None+ and proj.weight.dtype == x.dtype+ and proj.out_features % (patch_size * patch_size) == 0+ )+++def _block_tokens(device: torch.device) -> int:+ """Tokens per program of the stage-5 tail kernel for ``device``'s architecture."""+ major, _ = torch.cuda.get_device_capability(device)+ if major == 10:+ # Datacenter Blackwell. Swept at a 1080p stage-5 volume on sm_100.+ return 128+ return 32+++def stage5_tail(x: torch.Tensor, norm: torch.nn.RMSNorm, proj: torch.nn.Linear, patch_size: int) -> torch.Tensor:+ """``unpatchify(proj(norm(x)).permute(0, 4, 1, 2, 3))`` as one kernel; ``(B, C_out, T, H*P, W*P)``.+ Check :func:`stage5_tail_supported` first.+ """+ batch, frames, hh, ww, c = x.shape+ out_features = proj.out_features+ out = torch.empty(+ (batch, out_features // (patch_size * patch_size), frames, hh * patch_size, ww * patch_size),+ dtype=x.dtype,+ device=x.device,+ )+ n_tokens = batch * frames * hh * ww+ eps = norm.eps if norm.eps is not None else torch.finfo(x.dtype).eps+ block_tokens = _block_tokens(x.device)+ with torch.cuda.device(x.device):+ _stage5_tail_kernel[(triton.cdiv(n_tokens, block_tokens),)](+ x,+ norm.weight,+ proj.weight,+ proj.bias,+ out,+ n_tokens,+ frames,+ hh,+ ww,+ eps,+ C=c,+ OUT=out_features,+ OUT_PAD=max(16, triton.next_power_of_2(out_features)),+ PATCH=patch_size,+ BLOCK_T=block_tokens,+ )+ return outdiff --git a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/swiglu.py b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/swiglu.pyindex 21b92a2..fc235c9 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/transformer/swiglu.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/transformer/swiglu.py@@ -228,6 +228,28 @@ if _TRITON_AVAILABLE:)+@dataclass(frozen=True)+class _GateUpLaunch:+ """Launch config of the fused gate+up kernel.+ Every config accumulates each output in the same K order, so all are bit-identical.+ """++ blocks: tuple[int, int, int] # (BLOCK_M, BLOCK_N, BLOCK_K)+ num_warps: int+ num_stages: int+++def _gate_up_launch(device: torch.device) -> _GateUpLaunch:+ """Fused gate+up kernel launch config for ``device``'s architecture."""+ major, _ = torch.cuda.get_device_capability(device)+ if major == 10:+ # Datacenter Blackwell. Swept at the DiffVAE det-stage MLP shapes (dim 512..2048,+ # 16k-row tiles) on sm_100.+ return _GateUpLaunch(blocks=(256, 128, 64), num_warps=8, num_stages=3)+ # Triton's default warps and stages.+ return _GateUpLaunch(blocks=(64, 64, 32), num_warps=4, num_stages=3)++_DUAL_GATE_UP_DIM_MIN: Final[int] = 256_DUAL_GATE_UP_DIM_MAX: Final[int] = 2048@@ -270,7 +292,8 @@ def _fused_gate_up_swiglu_triton()m, k = x.shapen = w_gate.shape[0]- block_m, block_n, block_k = 64, 64, 32+ launch = _gate_up_launch(x.device)+ block_m, block_n, block_k = launch.blocksgrid = (triton.cdiv(m, block_m), triton.cdiv(n, block_n))with torch.cuda.device(x.device):_fused_gate_up_swiglu_kernel[grid](@@ -292,6 +315,8 @@ def _fused_gate_up_swiglu_triton(BLOCK_M=block_m,BLOCK_N=block_n,BLOCK_K=block_k,+ num_warps=launch.num_warps,+ num_stages=launch.num_stages,)@@ -462,14 +487,24 @@ def swiglu_tiled(return _swiglu_tiled_op(x, w_gate, w_up, w_down, tile_size, num_tiles, bool(use_triton))-def plain_mlp(x: torch.Tensor, mlp: nn.Module, norm: nn.RMSNorm, tile: SwiGLUTileSpec) -> torch.Tensor:- """Det ``NABlock``: ``x + swiglu_tiled(norm(x))`` (no AdaLN)."""+def plain_mlp(x: torch.Tensor, mlp: nn.Module, norm: nn.Module, tile: SwiGLUTileSpec) -> torch.Tensor:+ """Det ``NABlock``: ``x + swiglu_tiled(norm(x))`` (no AdaLN); ``norm`` is the block's+ :class:`~ltx_core.model.video_vae.transformer.det_rms_norm.DetRMSNorm`."""y = norm(x)if y.numel() == 0:return xreturn x + swiglu_tiled(y, mlp.w_gate.weight, mlp.w_up.weight, mlp.w_down.weight, tile)+def plain_mlp_accumulate(x: torch.Tensor, mlp: nn.Module, norm: nn.Module, tile: SwiGLUTileSpec) -> torch.Tensor:+ """:func:`plain_mlp` with the residual added into the MLP's own fresh output: same values (addition+ commutes exactly), one volume fewer live."""+ y = norm(x)+ if y.numel() == 0:+ return x+ return swiglu_tiled(y, mlp.w_gate.weight, mlp.w_up.weight, mlp.w_down.weight, tile).add_(x)++class SwiGLU(nn.Module):"""Gated MLP weights: ``w_down(silu(w_gate(x)) * w_up(x))``."""diff --git a/packages/ltx-core/src/ltx_core/model/video_vae/video_vae.py b/packages/ltx-core/src/ltx_core/model/video_vae/video_vae.pyindex c3f8d8d..565495d 100644--- a/packages/ltx-core/src/ltx_core/model/video_vae/video_vae.py+++ b/packages/ltx-core/src/ltx_core/model/video_vae/video_vae.py@@ -10,7 +10,7 @@ from ltx_core.model.common.normalization import PixelNormfrom ltx_core.model.disposable import Disposablefrom ltx_core.model.transformer.attention import AttentionCallable, AttentionFunctionfrom ltx_core.model.video_vae.attention import AttnBlock3D-from ltx_core.model.video_vae.convolution import make_conv_nd+from ltx_core.model.video_vae.convolution import make_conv_nd, memory_format_offrom ltx_core.model.video_vae.enums import LogVarianceType, NormLayerType, PaddingModeTypefrom ltx_core.model.video_vae.keyframes import DecodeKeyframesfrom ltx_core.model.video_vae.ops import PerChannelStatistics, patchify@@ -291,6 +291,9 @@ class VideoEncoder(nn.Module, Disposable):# This reduces H,W by patch_size and increases channels, making convolutions more efficient# Example: (B, 3, F, 512, 512) -> (B, 48, F, 128, 128) with patch_size=4sample = patchify(sample, patch_size_hw=self.patch_size, patch_size_t=1)+ # Activations follow the weights' layout: channels-last weights (CHANNELS_LAST_3D_WEIGHTS)+ # keep the whole encoder on NHWC conv kernels.+ sample = sample.contiguous(memory_format=memory_format_of(self.conv_in.weight))sample = self.conv_in(sample)for down_block in self.down_blocks:@@ -335,7 +338,7 @@ class VideoEncoder(nn.Module, Disposable):# Split into means and logvar, then normalize meansmeans, _ = torch.chunk(sample, 2, dim=1)- return self.per_channel_statistics.normalize(means)+ return self.per_channel_statistics.normalize(means).contiguous()def tiled_encode(self,diff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/_broadcast.py b/packages/ltx-core/src/ltx_core/multigpu/broadcast.pysimilarity index 55%rename from packages/ltx-pipelines/src/ltx_pipelines/multigpu/_broadcast.pyrename to packages/ltx-core/src/ltx_core/multigpu/broadcast.pyindex dd13306..f5b2777 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/_broadcast.py+++ b/packages/ltx-core/src/ltx_core/multigpu/broadcast.py@@ -1,7 +1,8 @@-"""Store-based broadcast signaling between the relay rank and workers.+"""Store-based broadcast signaling between a driver rank and workers.Workers poll an incrementing counter in the process group's store instead of-blocking directly on a collective, so an idle fleet doesn't trip the NCCL-watchdog timeout while waiting for the next job.+blocking directly on a collective. An idle worker therefore neither trips the+NCCL watchdog timeout nor keeps an NCCL receive kernel spinning on its GPU+while the driver is still producing what it will broadcast."""from __future__ import annotations@@ -14,7 +15,7 @@ import torch.distributed as distclass BroadcastCoordinator:- """Coordinates broadcast signaling between the relay and workers via a distributed store.+ """Coordinates broadcast signaling between the driver and workers via a distributed store.Uses an incrementing counter to signal when a broadcast is ready. Workerspoll for counter changes to avoid NCCL timeout issues during idle periods.The counter wraps at a large value to prevent overflow.@@ -24,7 +25,7 @@ class BroadcastCoordinator:"""Initialize the broadcast coordinator.Args:store: Distributed store (TCPStore, PrefixStore, etc.) for coordination across ranks.- is_driver: Whether this rank is the relay (signals broadcasts).+ is_driver: Whether this rank is the driver (signals broadcasts)."""self.store = storeself.is_driver = is_driver@@ -45,23 +46,36 @@ class BroadcastCoordinator:Raises:RuntimeError: If called on a non-driver rank."""+ self.notify()+ yield++ def notify(self) -> None:+ """Increment the signal counter so workers waiting on it proceed (driver only).+ Raises:+ RuntimeError: If called on a non-driver rank.+ """if not self.is_driver:- raise RuntimeError("broadcast_context can only be called on driver rank")+ raise RuntimeError("only the driver rank can signal a broadcast")self.signal = (self.signal + 1) % self.signal_maxself.store.set(self.broadcast_key, str(self.signal))- yield- def wait_for_signal_change(self, poll_interval: float = 0.01) -> None:- """Wait until the signal changes from the last seen value (worker only).+ def wait_for_signal_change(self, poll_interval: float = 0.01, timeout: float | None = None) -> None:+ """Wait for the driver's next signal (worker only).+ Each call consumes exactly one signal. If the driver has signaled several times since+ the previous call, this returns at once and the next call consumes the next signal, so+ a worker that polls late never skips one.Args:poll_interval: Seconds to sleep between polls (default: 0.01).+ timeout: Seconds to wait before raising; ``None`` waits indefinitely.+ Raises:+ TimeoutError: If no new signal arrives within ``timeout``."""- while True:- current = self.get_current_signal()- if current != self.last_seen_signal:- self.last_seen_signal = current- return+ deadline = None if timeout is None else time.monotonic() + timeout+ while self.get_current_signal() == self.last_seen_signal:+ if deadline is not None and time.monotonic() > deadline:+ raise TimeoutError(f"no broadcast signal from the driver within {timeout} s")time.sleep(poll_interval)+ self.last_seen_signal = (self.last_seen_signal + 1) % self.signal_maxdef get_current_signal(self) -> int:"""Get the current signal value from the store."""diff --git a/packages/ltx-core/src/ltx_core/multigpu/gemma/broadcast_wrapper.py b/packages/ltx-core/src/ltx_core/multigpu/gemma/broadcast_wrapper.pyindex e8ce742..999ad43 100644--- a/packages/ltx-core/src/ltx_core/multigpu/gemma/broadcast_wrapper.py+++ b/packages/ltx-core/src/ltx_core/multigpu/gemma/broadcast_wrapper.py@@ -6,8 +6,13 @@ import torchimport torch.distributed as distfrom ltx_core.model.disposable import Disposable+from ltx_core.multigpu.broadcast import BroadcastCoordinatorfrom ltx_core.text_encoders.gemma.encoders.base_encoder import LTXGemmaTextEncoder+# Bounds a wait on the source rank, which includes its Gemma load on a cold job and any prompt enhancement.+# Matches NCCL's default collective timeout, which bounded the receive these waits precede.+_SRC_WAIT_TIMEOUT_S = 600.0+class BroadcastGemmaWrapper(torch.nn.Module, Disposable):"""Encoder/group plumbing, prompt enhancement, and result broadcast.@@ -21,6 +26,10 @@ class BroadcastGemmaWrapper(torch.nn.Module, Disposable):``dist.get_group_rank``.dtype: Target dtype for output tensors.device: Target device for output tensors.+ coordinator: Store signal whose driver is ``src_rank``. When set, the other ranks wait for it on+ the host before each broadcast from ``src_rank``, so their NCCL receive kernels do not spin+ on a GPU the source rank's encoder is still running on. A wait raises ``TimeoutError``+ after ``_SRC_WAIT_TIMEOUT_S``, e.g. when the source rank failed before broadcasting."""def __init__(@@ -30,6 +39,7 @@ class BroadcastGemmaWrapper(torch.nn.Module, Disposable):src_rank: int,dtype: torch.dtype = torch.bfloat16,device: torch.device | None = None,+ coordinator: BroadcastCoordinator | None = None,) -> None:super().__init__()if device is None and torch.cuda.is_available():@@ -40,6 +50,7 @@ class BroadcastGemmaWrapper(torch.nn.Module, Disposable):self._rank = dist.get_rank(broadcast_group)self._dtype = dtypeself._device = device+ self._coordinator = coordinatordef encode(self, prompts: list[str]) -> list[tuple[tuple[torch.Tensor, ...], torch.Tensor]]:"""Encode a batch of prompts to per-prompt hidden states; implemented by subclasses."""@@ -86,6 +97,7 @@ class BroadcastGemmaWrapper(torch.nn.Module, Disposable):return self._broadcast_str(result)def _broadcast_str(self, value: str | None) -> str:+ self._wait_for_src()obj_list: list[str | None] = [value]dist.broadcast_object_list(obj_list, src=self._src_rank, group=self._group)result = obj_list[0]@@ -99,6 +111,7 @@ class BroadcastGemmaWrapper(torch.nn.Module, Disposable):src_rank: int,) -> tuple[tuple[torch.Tensor, ...], torch.Tensor]:"""Broadcast hidden states + attention mask via NCCL from ``src_rank``."""+ self._wait_for_src(*(hidden_states or ()), attention_mask)if self._rank == src_rank:meta = [{"hs_shapes": [h.shape for h in hidden_states], "mask_shape": attention_mask.shape}]else:@@ -118,3 +131,14 @@ class BroadcastGemmaWrapper(torch.nn.Module, Disposable):dist.broadcast(attention_mask, src=src_rank, group=self._group)return hidden_states, attention_mask++ def _wait_for_src(self, *src_outputs: torch.Tensor | None) -> None:+ """Hold the other ranks on the host until the source rank has computed ``src_outputs``."""+ if self._coordinator is None:+ return+ if self._rank != self._src_rank:+ self._coordinator.wait_for_signal_change(poll_interval=0.001, timeout=_SRC_WAIT_TIMEOUT_S)+ return+ for device in {t.device for t in src_outputs if t is not None and t.is_cuda}:+ torch.cuda.synchronize(device)+ self._coordinator.notify()diff --git a/packages/ltx-core/src/ltx_core/multigpu/sharded_sd.py b/packages/ltx-core/src/ltx_core/multigpu/sharded_sd.pyindex 6eb55ee..6dd5c6a 100644--- a/packages/ltx-core/src/ltx_core/multigpu/sharded_sd.py+++ b/packages/ltx-core/src/ltx_core/multigpu/sharded_sd.py@@ -1,5 +1,5 @@"""Sharded state dict with distributed weight backup.-Each rank stores ~1/N of the model weights. Bucketed broadcasts+Each rank stores ~1/N of the model weights. Bucketed all-gather roundsrestore weights into a target state dict using only a small,caller-provided staging buffer."""@@ -7,7 +7,8 @@ caller-provided staging buffer.from __future__ import annotationsimport hashlib-from dataclasses import dataclass+from collections.abc import Iterable+from dataclasses import dataclass, replaceimport torchimport torch.distributed as dist@@ -23,6 +24,16 @@ def _nbytes(t: torch.Tensor) -> int:return t.numel() * t.element_size()+# Staging slots start on a multiple of this many bytes, which keeps the multi-tensor copies' loads aligned.+_SLOT_ALIGN = 256+++def _as_bytes(t: torch.Tensor, key: str) -> torch.Tensor:+ if not t.is_contiguous():+ raise RuntimeError(f"ShardedSD: tensor '{key}' is not contiguous")+ return t.view(torch.uint8).view(-1)++@dataclassclass ShardedSD:"""Sharded state dict with distributed weight backup.@@ -32,7 +43,7 @@ class ShardedSD:rollback, checkpoint recovery).- Deterministic ownership: ``MD5(key) % world_size``- Local storage only for owned keys (VRAM ≈ 1/world_size of model)- - Bucketed broadcast using a single small staging buffer+ - Bucketed all-gather rounds through a single small staging bufferUsage::backup = ShardedSD.from_state_dict(model.state_dict(), group)staging = torch.empty(64 * 1024 * 1024, dtype=torch.uint8, device=device)@@ -101,63 +112,71 @@ class ShardedSD:_owner_groups=owner_groups,)+ def subset(self, keys: Iterable[str]) -> ShardedSD:+ """The same shards restricted to ``keys`` (no copies): restoring it writes only those keys.+ Every rank must pass the same keys, as for :meth:`broadcast_shards_into`.+ """+ wanted = frozenset(keys)+ return replace(+ self,+ keys=tuple(k for k in self.keys if k in wanted),+ key_sizes={k: v for k, v in self.key_sizes.items() if k in wanted},+ owner_of={k: v for k, v in self.owner_of.items() if k in wanted},+ local_shard={k: v for k, v in self.local_shard.items() if k in wanted},+ _owner_groups={r: [k for k in ks if k in wanted] for r, ks in self._owner_groups.items()},+ )+def broadcast_shards_into(self,target_sd: dict[str, torch.Tensor],staging: torch.Tensor,) -> None:- """Broadcast stored weights from sharded backup into *target_sd*.+ """Restore the stored weights from the sharded backup into *target_sd*.This is a **cooperative operation** — all ranks in the process groupmust call it simultaneously.- *staging* is a caller-owned ``uint8`` scratch buffer; its size sets the- broadcast granularity (tensors larger than it split across rounds). It- may be shared by instances that never broadcast at the same time. Writes- directly into existing tensors in *target_sd*.+ *staging* is a caller-owned ``uint8`` scratch buffer: one send slot plus ``world``+ receive slots are carved from it, and tensors larger than a slot split across rounds.+ It may be shared by instances that never broadcast at the same time. In round ``i``+ every rank packs its ``i``-th bucket and one ``all_gather_into_tensor`` ships all of+ them, so every link direction carries data at once; the pack and the unpack of a round+ are single multi-tensor copies. Writes directly into existing tensors in *target_sd*."""if staging.dtype != torch.uint8 or staging.numel() == 0:raise ValueError("staging must be a non-empty uint8 buffer")- for owner, klist in self._owner_groups.items():- if klist:- self._broadcast_group(owner, klist, target_sd, staging)-- def _broadcast_group(- self,- owner: int,- keys: list[str],- target_sd: dict[str, torch.Tensor],- staging: torch.Tensor,- ) -> None:- """Pack & broadcast params from *owner*, splitting tensors across rounds."""- rounds = self._plan_rounds(keys, staging.numel())-- for round_chunks in rounds:- filled = 0- if self.rank == owner:- for k, offset, chunk_size in round_chunks:- src = self.local_shard[k]- if not src.is_contiguous():- raise RuntimeError(f"ShardedSD: local shard tensor '{k}' is not contiguous")- src_bytes = src.view(torch.uint8).view(-1)- staging[filled : filled + chunk_size].copy_(- src_bytes[offset : offset + chunk_size], non_blocking=True- )- filled += chunk_size- else:- filled = sum(chunk_size for (_, _, chunk_size) in round_chunks)-- if filled == 0:- continue- view = staging[:filled]- dist.broadcast(view, src=owner, group=self.group)-- cursor = 0- for k, offset, chunk_size in round_chunks:- dst = target_sd[k]- if not dst.is_contiguous():- raise RuntimeError(f"ShardedSD: target tensor '{k}' is not contiguous")- dst_bytes = dst.view(torch.uint8).view(-1)- dst_bytes[offset : offset + chunk_size].copy_(staging[cursor : cursor + chunk_size], non_blocking=True)- cursor += chunk_size+ slot = (staging.numel() // (self.world + 1)) // _SLOT_ALIGN * _SLOT_ALIGN+ if slot <= 0:+ raise ValueError(f"staging buffer of {staging.numel()} bytes is too small for {self.world} ranks")+ plans = {owner: self._plan_rounds(keys, slot) for owner, keys in self._owner_groups.items()}+ n_rounds = max((len(rounds) for rounds in plans.values()), default=0)+ send = staging[:slot]+ recv = staging[slot : slot + self.world * slot]+ for i in range(n_rounds):+ mine = plans[self.rank][i] if i < len(plans[self.rank]) else []+ if mine:+ pack_dst: list[torch.Tensor] = []+ pack_src: list[torch.Tensor] = []+ filled = 0+ for key, offset, chunk in mine:+ pack_dst.append(send[filled : filled + chunk])+ pack_src.append(_as_bytes(self.local_shard[key], key)[offset : offset + chunk])+ filled += chunk+ torch._foreach_copy_(pack_dst, pack_src)+ dist.all_gather_into_tensor(recv, send, group=self.group)+ unpack_dst: list[torch.Tensor] = []+ unpack_src: list[torch.Tensor] = []+ for owner, rounds in plans.items():+ if i >= len(rounds):+ continue+ cursor = owner * slot+ for key, offset, chunk in rounds[i]:+ dst = target_sd[key]+ # A zero-copy shard is the target tensor itself: nothing to write back.+ if owner != self.rank or dst.data_ptr() != self.local_shard[key].data_ptr():+ unpack_dst.append(_as_bytes(dst, key)[offset : offset + chunk])+ unpack_src.append(recv[cursor : cursor + chunk])+ cursor += chunk+ if unpack_dst:+ torch._foreach_copy_(unpack_dst, unpack_src)def _plan_rounds(self, keys: list[str], capacity: int) -> list[list[tuple[str, int, int]]]:"""Build rounds that pack a *capacity*-byte buffer, splitting tensors if needed.diff --git a/packages/ltx-core/src/ltx_core/multigpu/transformer/attention.py b/packages/ltx-core/src/ltx_core/multigpu/transformer/attention.pyindex 2f9e7e2..5da68fd 100644--- a/packages/ltx-core/src/ltx_core/multigpu/transformer/attention.py+++ b/packages/ltx-core/src/ltx_core/multigpu/transformer/attention.py@@ -60,6 +60,11 @@ class AttentionManager:)self.group = groupself._all2all_timeout_seconds = _DEFAULT_ALL2ALL_TIMEOUT_SECONDS+ # Padded tokens in the current forward's video sequence, one at the end of each of the last+ # `key_padding` shards (see `tile_modality_for_rank`), which the redistributions cut from the+ # gathered keys. Set by the SP model wrapper per forward; host state the block loop branches+ # on, so Dynamo guards on it and the wrapper adds it to the CUDA-graph capture key.+ self.key_padding = 0@propertydef all2all_timeout_seconds(self) -> float:@@ -90,6 +95,24 @@ class AttentionManager:return out+def drop_padding_keys(t: torch.Tensor, world_size: int, padding: int, dim: int = 1) -> torch.Tensor:+ """Drop the padded tokens from a gathered sequence in shard order along ``dim``.+ Each of the last ``padding`` of the ``world_size`` equal shards ends with one padded token (the+ layout of :func:`~ltx_core.multigpu.transformer.sequence_parallel.tile_modality_for_rank`). With+ one, it is the last token and the result is a view.+ """+ if padding == 0:+ return t+ dim %= t.ndim+ tokens = t.shape[dim]+ if padding == 1:+ return t.narrow(dim, 0, tokens - 1)+ shard = tokens // world_size+ full = (world_size - padding) * shard+ short = t.narrow(dim, full, tokens - full).unflatten(dim, (padding, shard)).narrow(dim + 1, 0, shard - 1)+ return torch.cat([t.narrow(dim, 0, full), short.flatten(dim, dim + 1)], dim=dim)++class _All2AllRedistribute:"""Shared redistribute/gather pipeline for self-attention SP wrappers.Folds the head dim view-and-shuffle so the masked and unmasked variants only@@ -113,6 +136,10 @@ class _All2AllRedistribute:t_q, t_k, t_v = self.manager.send_recv_qkv(q, k, v)local_heads = heads // self.manager.world_size+ # The gathered keys hold the sharding's padded tokens; cut them here, so attention only ever+ # sees real keys. Queries keep them: their rows are dropped when the outputs are gathered.+ t_k = drop_padding_keys(t_k, self.manager.world_size, self.manager.key_padding)+ t_v = drop_padding_keys(t_v, self.manager.world_size, self.manager.key_padding)# `flatten` / `unflatten` collapse only the head dims, avoiding a `-1` in the# seq position -- that would otherwise be ambiguous if the seq is 0 for a@@ -146,6 +173,12 @@ class All2AllAttention(AttentionCallable):class MaskedAll2AllAttention(MaskedAttentionCallable):+ """Masked SP attention, for a caller's own mask.+ The mask is in the shard order the keys are gathered in+ (:func:`~ltx_core.multigpu.transformer.sequence_parallel.tile_modality_for_rank`); its columns for+ the padded keys the redistribution cuts are cut with them.+ """+def __init__(self, manager: AttentionManager, original_attention: MaskedAttentionCallable):self._sp = _All2AllRedistribute(manager)self.original_attention = original_attention@@ -159,6 +192,7 @@ class MaskedAll2AllAttention(MaskedAttentionCallable):mask: torch.Tensor,) -> torch.Tensor:t_q, t_k, t_v, local_heads, head_dim = self._sp.redistribute(q, k, v, heads)+ mask = drop_padding_keys(mask, self._sp.manager.world_size, self._sp.manager.key_padding, dim=-1)hidden_states = self.original_attention(q=t_q, k=t_k, v=t_v, heads=local_heads, mask=mask)return self._sp.gather(hidden_states, local_heads, head_dim)@@ -193,6 +227,9 @@ class _AudioAll2AllRedistribute:t_q = q[:, :, heads_per_rank * rank : heads_per_rank * (rank + 1), :].clone()t_k = self.manager.all2all_k.send_recv_heads(k, copy_out=self.manager.copy_out)t_v = self.manager.all2all_v.send_recv_heads(v, copy_out=self.manager.copy_out)+ # The keys are the sharded video sequence: cut its padded tokens, as for self-attention.+ t_k = drop_padding_keys(t_k, self.manager.world_size, self.manager.key_padding)+ t_v = drop_padding_keys(t_v, self.manager.world_size, self.manager.key_padding)# `flatten` / `unflatten` collapse only the head dims, avoiding a `-1` in the# seq position -- that would otherwise be ambiguous if the seq is 0 for adiff --git a/packages/ltx-core/src/ltx_core/multigpu/transformer/sequence_parallel.py b/packages/ltx-core/src/ltx_core/multigpu/transformer/sequence_parallel.pyindex 6624263..81498fd 100644--- a/packages/ltx-core/src/ltx_core/multigpu/transformer/sequence_parallel.py+++ b/packages/ltx-core/src/ltx_core/multigpu/transformer/sequence_parallel.py@@ -16,6 +16,7 @@ import torchfrom ltx_core.guidance.perturbations import BatchedPerturbationConfigfrom ltx_core.loader.module_ops import ModuleOpsfrom ltx_core.model.transformer.attention import Attention+from ltx_core.model.transformer.cudagraph_capture import capture_key_scopefrom ltx_core.model.transformer.modality import Modalityfrom ltx_core.model.transformer.model import LTXModelfrom ltx_core.model.transformer.transformer import BasicAVTransformerBlock@@ -72,11 +73,6 @@ def pad_modality_for_uniform_sharding(latent_pad = torch.zeros(b, pad, modality.latent.shape[2], dtype=dtype, device=device)latent = torch.cat([modality.latent, latent_pad], dim=1)- timesteps_pad_shape = list(modality.timesteps.shape)- timesteps_pad_shape[1] = pad- timesteps_pad = torch.zeros(timesteps_pad_shape, dtype=modality.timesteps.dtype, device=modality.timesteps.device)- timesteps = torch.cat([modality.timesteps, timesteps_pad], dim=1)-positions_pad_shape = list(modality.positions.shape)positions_pad_shape[2] = padpositions_pad = torch.zeros(positions_pad_shape, dtype=modality.positions.dtype, device=modality.positions.device)@@ -113,7 +109,7 @@ def pad_modality_for_uniform_sharding(padded = replace(modality,latent=latent,- timesteps=timesteps,+ timesteps=modality.timesteps.pad_tokens(pad),positions=positions,attention_mask=attention_mask,keyframes_mask=keyframes_mask,@@ -132,51 +128,87 @@ def compute_sequence_offsets(token_counts: list[int]) -> list[int]:return [0, *accumulate(token_counts[:-1])]+def spread_padding_order(num_tokens: int, padded_tokens: int, world_size: int, device: torch.device) -> torch.Tensor:+ """Source row of each position of a padded sequence laid out shard by shard, with the padding spread.+ ``padded_tokens`` is ``num_tokens`` rounded up to a multiple of ``world_size``, with the padded+ rows at its end (:func:`pad_modality_for_uniform_sharding`). Every shard holds+ ``padded_tokens // world_size`` rows: the first shards only real tokens, each of the last+ ``padded_tokens - num_tokens`` shards one real token fewer and then one padded row. Real tokens+ keep their order, so the real rows of the shards, concatenated, are the original sequence.+ Built on ``device`` from ``arange``, without a host sync.+ """+ shard = padded_tokens // world_size+ full = world_size - (padded_tokens - num_tokens)+ position = torch.arange(padded_tokens, device=device)+ rank, offset = position // shard, position % shard+ short = (rank - full).clamp(min=0)+ real = rank * shard - short + offset+ return torch.where((rank < full) | (offset < shard - 1), real, num_tokens + short)++def tile_modality_for_rank(modality: Modality,rank: int,world_size: int,+ num_tokens: int | None = None,) -> tuple[Modality, list[int]]:"""Tile a modality's tensors for a specific GPU rank.Splits the sequence dimension (dim 1 for latent/timesteps, dim 2 for positions)across GPUs, returning the local tile for the given rank.+ ``num_tokens`` is the real token count of a modality :func:`pad_modality_for_uniform_sharding`+ padded. The padding is then spread one row per shard over the last shards+ (:func:`spread_padding_order`), so no rank holds more than one padded token, and the+ attention mask is reordered the same way: the sequence every rank gathers in the all-to-all+ is in shard order.Args:modality: The modality to tile.rank: Current GPU rank.world_size: Total number of GPUs.+ num_tokens: Real (unpadded) token count; ``None`` when the modality is not padded.Returns:- Tuple of (tiled_modality, token_counts_per_rank).+ Tuple of (tiled_modality, real token count per rank)."""total_tokens = modality.latent.shape[1]token_counts = compute_sequence_partition(total_tokens, world_size)- offsets = compute_sequence_offsets(token_counts)-- start = offsets[rank]- end = start + token_counts[rank]+ shard = token_counts[rank]+ start = rank * shard+ padding = 0 if num_tokens is None else total_tokens - num_tokens+ if padding == 0:+ rows: slice | torch.Tensor = slice(start, start + shard)+ attention_mask = modality.attention_mask+ else:+ order = spread_padding_order(num_tokens, total_tokens, world_size, modality.latent.device)+ rows = order[start : start + shard]+ attention_mask = modality.attention_mask+ if attention_mask is not None:+ attention_mask = attention_mask.index_select(-1, order)+ if attention_mask.shape[-2] > 1:+ attention_mask = attention_mask.index_select(-2, order)+ token_counts = [shard] * (world_size - padding) + [shard - 1] * padding# Tile latent: (B, T, D) -> (B, T_local, D)- tiled_latent = modality.latent[:, start:end, :]-- # Tile timesteps: (B, T) -> (B, T_local)- tiled_timesteps = modality.timesteps[:, start:end]+ tiled_latent = modality.latent[:, rows, :]# Tile positions: (B, 3, T, 2) -> (B, 3, T_local, 2)- tiled_positions = modality.positions[:, :, start:end, :]+ tiled_positions = modality.positions[:, :, rows]# Tile keyframes_mask: (B, T, 1) -> (B, T_local, 1). Sharding splits the sequence, so each# rank must carry the marker for its own slice -- the embedding is applied per rank, before# the all-to-all.tiled_keyframes_mask = Noneif modality.keyframes_mask is not None:- tiled_keyframes_mask = modality.keyframes_mask[:, start:end]+ tiled_keyframes_mask = modality.keyframes_mask[:, rows]+ # Sharding narrows the per-token timesteps; `select_tokens` applies the slice to whichever+ # tensor carries the token dimension.tiled_modality = replace(modality,latent=tiled_latent,- timesteps=tiled_timesteps,+ timesteps=modality.timesteps.select_tokens(rows),positions=tiled_positions,keyframes_mask=tiled_keyframes_mask,+ attention_mask=attention_mask,)return tiled_modality, token_counts@@ -293,13 +325,17 @@ class SequenceParallelModelWrapper(torch.nn.Module):if video is None:return self.model(video, audio, perturbations)- # Pad the video seq dim up to a multiple of world_size so all ranks get- # equal shards. The attention mask we attach makes the padded keys- # invisible to attention; padded rows are sliced off after the gather.+ # Pad the video seq dim up to a multiple of world_size so all ranks get equal shards. The+ # attention redistributions cut the padded keys (`key_padding`), so a modality without a mask+ # of its own goes in without the padding mask: attention only ever sees real keys. The+ # padded rows are dropped when the outputs are gathered.+ has_mask = video.attention_mask is not Nonevideo, t_orig = pad_modality_for_uniform_sharding(video, self.attention_manager.world_size)+ if not has_mask:+ video = replace(video, attention_mask=None)video_tile, token_counts = tile_modality_for_rank(- video, self.attention_manager.rank, self.attention_manager.world_size+ video, self.attention_manager.rank, self.attention_manager.world_size, num_tokens=t_orig)total_tokens = sum(token_counts)if total_tokens > self.attention_manager.max_tokens:@@ -307,11 +343,18 @@ class SequenceParallelModelWrapper(torch.nn.Module):f"Total video token count ({total_tokens}) exceeds attention_manager max_tokens "f"({self.attention_manager.max_tokens}). Use a smaller resolution or fewer frames.")- torch.distributed.barrier(self.attention_manager.group)- video, audio = self.model(video_tile, audio, perturbations)+ # This must remain a device-side collective: the first all2all requires aligned ranks,+ # while the host must remain free to queue the next forward.+ torch.distributed.all_reduce(+ torch.zeros(1, device=video_tile.latent.device), group=self.attention_manager.group+ )+ # The padding layout is host state the redistributions branch on (`key_padding`) but no input+ # shape carries: Dynamo guards on it, and the scope keys the CUDA-graph captures on it.+ padding = video.latent.shape[1] - t_orig+ self.attention_manager.key_padding = padding+ with capture_key_scope(("sequence_parallel_padding", padding)):+ video, audio = self.model(video_tile, audio, perturbations)+ # Each rank's real tokens lead its shard, so gathering the real counts drops the padding and+ # restores the caller's order and length.video = gather_output_tokens(video, token_counts, self.attention_manager.group)- # Unpad: drop the rows we added in `pad_modality_for_uniform_sharding` to make- # the seq dim divisible by world_size, restoring the caller's original length.- if video.shape[1] != t_orig:- video = video[:, :t_orig, :]return video, audiodiff --git a/packages/ltx-core/src/ltx_core/multigpu/vae/distributed_decoder.py b/packages/ltx-core/src/ltx_core/multigpu/vae/distributed_decoder.pyindex 20b73f6..7377521 100644--- a/packages/ltx-core/src/ltx_core/multigpu/vae/distributed_decoder.py+++ b/packages/ltx-core/src/ltx_core/multigpu/vae/distributed_decoder.py@@ -16,17 +16,24 @@ management — they are independent concerns.from __future__ import annotations+import contextlibimport logging+import os+import queue+import threadingfrom collections.abc import Callable, Iterator, Sequencefrom dataclasses import dataclass+from typing import Any, NamedTupleimport torchimport torch.distributed as distfrom einops import rearrangefrom torch.multiprocessing import Queue+from ltx_core.devices import upload_from_hostfrom ltx_core.model.disposable import Disposablefrom ltx_core.model.video_vae.keyframes import DecodeKeyframes+from ltx_core.model.video_vae.transformer.rope_math import abs_rope_table_cachefrom ltx_core.model.video_vae.video_vae import (VideoDecoder,clip_generators,@@ -80,7 +87,7 @@ def dist_rank_tiles(rank: int,world_size: int,) -> list[Tile]:- """Round-robin Dist tiles for ``rank`` (same assignment as :meth:`DistributedVideoDecoder._decode_tiles`)."""+ """Round-robin Dist tiles for ``rank`` (the assignment :meth:`DistributedVideoDecoder.decode_video` decodes)."""if world_size < 1:raise ValueError(f"world_size must be >= 1, got {world_size}")if not 0 <= rank < world_size:@@ -89,6 +96,231 @@ def dist_rank_tiles(return [t for i, t in enumerate(all_tiles) if i % world_size == rank]+# How long the driver waits for the ranks' end markers after the last batch, and for its queue+# receiver thread once the decode ended; only a worker that died without its end marker runs it out.+# LTX_MGPU_DECODE_RECEIVER_TIMEOUT_SECONDS overrides the 60 s default.+_RECEIVER_JOIN_TIMEOUT_SECONDS = float(os.environ.get("LTX_MGPU_DECODE_RECEIVER_TIMEOUT_SECONDS", "60"))+++def _planes_with_shared_rope_tables(+ decoder: torch.nn.Module, latents: list[torch.Tensor], gens: list[torch.Generator | None]+) -> Iterator[torch.Tensor]:+ """``iter_decoded_single_frames`` under one abs-RoPE table scope: every plane has the same extent, so+ its tables are built once. Consume it on one thread."""+ with abs_rope_table_cache(decoder):+ yield from iter_decoded_single_frames(decoder, latents, gens)+++class _StreamAssembler:+ """The driver's assembly of streamed chunks into temporal batches, consumed lazily.+ Worker chunks arrive over the queue on a receiver thread and the driver's own tiles are+ decoded on a producer thread, each chunk handed over with an event recorded on the stream+ that made it. :meth:`batches` blends a batch on a side stream that waits only on those+ events, as soon as every tile overlapping it has delivered its frames, and synchronizes the+ side stream before yielding it: the batch is complete on every stream, so its consumer (the+ MP4 encoder) can work on it while the decode keeps running on the default stream. A batch that+ ``device_fn`` places on another GPU is blended on a side stream of that GPU, which reads the+ chunks through the driver's side stream, and both are synchronized. Chunks are+ blended in (rank, tile, frame) order -- what the whole-tile path summed in -- so the pixels+ do not depend on arrival order.+ """++ def __init__(+ self,+ tiles_by_rank: dict[int, list[Tile]],+ full_shape: VideoPixelShape,+ num_temporal_batches: int,+ weights: torch.Tensor | None,+ device_fn: Callable[[int], str | torch.device],+ *,+ mp_queue: Queue, # type: ignore[type-arg]+ driver_rank: int,+ ) -> None:+ self.tiles_by_rank = tiles_by_rank+ self.keys = sorted((rank, index) for rank, tiles in tiles_by_rank.items() for index in range(len(tiles)))+ self.chunks: dict[tuple[int, int], list[DecodedTile]] = {key: [] for key in self.keys}+ self.delivered = dict.fromkeys(self.keys, 0)+ self.full_shape = full_shape+ self.grid = _temporal_batches(full_shape.frames, num_temporal_batches)+ self.weights = weights+ self.device_fn = device_fn+ self.mp_queue = mp_queue+ self.driver_rank = driver_rank+ self.remote_ranks = sorted(rank for rank in tiles_by_rank if rank != driver_rank)+ self.device = torch.device("cuda", torch.cuda.current_device()) if torch.cuda.is_available() else None+ self.inference = torch.is_inference_mode_enabled()+ self.handoff: queue.SimpleQueue = queue.SimpleQueue()+ self.threads: list[threading.Thread] = []+ self.batch_streams: dict[int, torch.cuda.Stream] = {}++ def start(self, own: Iterator[tuple[int, DecodedTile]]) -> None:+ """Start the receiver thread (worker chunks) and the producer thread (``own``, this rank's chunks)."""+ if self.remote_ranks:+ # A daemon, so a worker that died without its end marker cannot pin the process at exit.+ receiver = threading.Thread(target=self._receive, name="dist-vae-recv", daemon=True)+ receiver.start()+ self.threads.append(receiver)+ producer = threading.Thread(target=self._produce, args=(own,), name="dist-vae-decode")+ producer.start()+ self.threads.append(producer)++ def _device_ctx(self) -> contextlib.AbstractContextManager:+ return torch.cuda.device(self.device) if self.device is not None else contextlib.nullcontext()++ def _produce(self, own: Iterator[tuple[int, DecodedTile]]) -> None:+ try:+ with torch.inference_mode(self.inference), self._device_ctx():+ for index, chunk in own:+ event = None+ if chunk.pixels.is_cuda:+ event = torch.cuda.Event()+ event.record()+ self.handoff.put(("chunk", self.driver_rank, index, chunk, event))+ self.handoff.put(("end", self.driver_rank, None, None, None))+ except BaseException as exc: # re-raised on the consuming thread+ self.handoff.put(("error", self.driver_rank, None, exc, None))++ def _receive(self) -> None:+ pending = set(self.remote_ranks)+ try:+ # Unpickling a CUDA IPC tensor makes the current stream wait on its producer's event; a+ # stream of this thread's own keeps that wait off the default stream the decode runs on.+ stream = torch.cuda.Stream(device=self.device) if self.device is not None else None+ stream_ctx = torch.cuda.stream(stream) if stream is not None else contextlib.nullcontext()+ with self._device_ctx(), stream_ctx:+ while pending:+ rank, index, frames_start, pixels = self.mp_queue.get()+ if index is None:+ pending.discard(rank)+ kind = "error" if frames_start == _DECODE_FAILED else "end"+ self.handoff.put((kind, rank, None, None, None))+ continue+ event = None+ if stream is not None and pixels.is_cuda:+ event = torch.cuda.Event()+ event.record(stream)+ chunk = DecodedTile(pixels, self.tiles_by_rank[rank][index], frames_start)+ self.handoff.put(("chunk", rank, index, chunk, event))+ except BaseException as exc: # re-raised on the consuming thread+ self.handoff.put(("error", -1, None, exc, None))++ def _missing(self, batch: slice) -> bool:+ for key in self.keys:+ frames = self.tiles_by_rank[key[0]][key[1]].out_coords[0]+ if frames.start < batch.stop and frames.start + self.delivered[key] < min(batch.stop, frames.stop):+ return True+ return False++ def batches(self) -> Iterator[torch.Tensor]:+ """The temporal batches in playback order, each complete on every stream when yielded."""+ stream = torch.cuda.Stream(device=self.device) if self.device is not None else None+ pending = {self.driver_rank, *self.remote_ranks}+ try:+ for b, batch in enumerate(self.grid):+ while self._missing(batch):+ if not pending:+ raise RuntimeError("Dist decode: every rank finished before the batches were covered")+ kind, rank, index, payload, event = self.handoff.get()+ if kind != "chunk":+ self._finish(pending, kind, rank, payload)+ continue+ if event is not None and stream is not None:+ stream.wait_event(event)+ payload.pixels.record_stream(stream)+ self.chunks[(rank, index)].append(payload)+ self.delivered[(rank, index)] = payload.frames_start + payload.pixels.shape[0]+ yield self._blend(b, batch, stream)+ self._drain(pending)+ finally:+ self.join()++ @staticmethod+ def _finish(pending: set[int], kind: str, rank: int, payload: object) -> None:+ """Take ``rank``'s end marker off ``pending``; raise if it is a failure."""+ pending.discard(rank)+ if kind == "error":+ raise RuntimeError(f"Dist decode: rank {rank} failed decoding its tiles") from (+ payload if isinstance(payload, BaseException) else None+ )++ def _drain(self, pending: set[int]) -> None:+ """Wait for the end markers still due after the last batch, raising on a failure marker.+ A rank can deliver every frame the batches need and fail afterwards (its decode yielded+ the wrong frame count, or raised on the way out); its failure marker comes after its+ chunks, so the decode is complete only once every rank reported.+ """+ while pending:+ try:+ kind, rank, _index, payload, _event = self.handoff.get(timeout=_RECEIVER_JOIN_TIMEOUT_SECONDS)+ except queue.Empty:+ logger.warning(+ "Dist decode: ranks %s sent no end marker %.0f s after the last batch; check their logs",+ sorted(pending),+ _RECEIVER_JOIN_TIMEOUT_SECONDS,+ )+ return+ if kind != "chunk":+ self._finish(pending, kind, rank, payload)++ def _batch_stream(self, device: torch.device) -> torch.cuda.Stream | None:+ """Side stream of the GPU ``device`` names when that is not the driver's GPU, else ``None``."""+ if self.device is None or device.type != "cuda":+ return None+ index = self.device.index if device.index is None else device.index+ if index == self.device.index:+ return None+ if index not in self.batch_streams:+ self.batch_streams[index] = torch.cuda.Stream(device=index)+ return self.batch_streams[index]++ def _blend(self, b: int, batch: slice, stream: torch.cuda.Stream | None) -> torch.Tensor:+ chunks = [chunk for key in self.keys for chunk in self.chunks[key]]+ device = torch.device(self.device_fn(b))+ batch_stream = self._batch_stream(device)+ stream_ctx = torch.cuda.stream(stream) if stream is not None else contextlib.nullcontext()+ # Entered last, so it is the current device and stream for the batch's own work, while the+ # driver's side stream (waiting on the chunks' events) stays current for its GPU: copies of+ # the chunks off the driver GPU run there, and the batch stream waits for them.+ batch_ctx = torch.cuda.stream(batch_stream) if batch_stream is not None else contextlib.nullcontext()+ with torch.inference_mode(self.inference), self._device_ctx(), stream_ctx, batch_ctx:+ out = _assemble_batch(chunks, batch, self.full_shape.height, self.full_shape.width, device, self.weights)+ # The batch must be complete when handed out, and its chunks read before their memory is reused.+ if batch_stream is not None:+ batch_stream.synchronize()+ if stream is not None:+ stream.synchronize()+ for key in self.keys:+ first = self.tiles_by_rank[key[0]][key[1]].out_coords[0].start+ self.chunks[key] = [c for c in self.chunks[key] if first + c.frames_start + c.pixels.shape[0] > batch.stop]+ return out++ def join(self) -> None:+ """Wait for the threads, which end on their own once the last chunk went out."""+ for thread in self.threads:+ thread.join(timeout=_RECEIVER_JOIN_TIMEOUT_SECONDS if thread.daemon else None)+ if thread.is_alive():+ logger.warning(+ "Dist decode thread %s is still waiting after %.0f s: a worker rank ended without its "+ "end marker; check that rank's log",+ thread.name,+ _RECEIVER_JOIN_TIMEOUT_SECONDS,+ )+ self.threads = []+++# A worker's last queue message after its chunks: its decode finished, or raised.+_DECODE_DONE = 0+_DECODE_FAILED = -1+++class OwnedSingleFrames(NamedTuple):+ """One rank's share of a split single-frame decode (see ``DistributedVideoDecoder.owned_single_frames``)."""++ indices: list[int]+ planes: Iterator[torch.Tensor]+ gather: Callable[[dict[int, Any]], dict[int, Any]]++def dist_rank_plane_indices(count: int, rank: int, world_size: int) -> list[int]:"""Round-robin share of ``count`` independent single-frame clips owned by ``rank``.The same assignment rule as :func:`dist_rank_tiles`, on a list of whole clips instead of@@ -140,12 +372,15 @@ def dist_rank_tile_pixel_shape(class DecodedTile:"""A VAE-decoded tile with pixel-space placement.Attributes:- pixels: ``[F_tile, H_tile, W_tile, C]`` in the decoder's native dtype.+ pixels: ``[F, H_tile, W_tile, C]`` in the decoder's native dtype: the tile's frames+ ``frames_start .. frames_start + F`` (all of them for a whole tile).pixel_tile: Carries ``out_coords`` (f, h, w slices) and separable ``masks_1d``.+ frames_start: Offset of ``pixels[0]`` within the tile's frames."""pixels: torch.Tensorpixel_tile: Tile+ frames_start: int = 0# ------------------------------------------------------------------@@ -161,6 +396,48 @@ def _to_decoded_tile(return DecodedTile(pixels=pixels, pixel_tile=tile)+def _temporal_batches(total_frames: int, num_temporal_batches: int) -> list[slice]:+ batch_size = (total_frames + num_temporal_batches - 1) // num_temporal_batches+ batches = [slice(b * batch_size, min((b + 1) * batch_size, total_frames)) for b in range(num_temporal_batches)]+ return [b for b in batches if b.stop > b.start]+++def _assemble_batch(+ tiles: Sequence[DecodedTile],+ batch_range: slice,+ output_height: int,+ output_width: int,+ device: str | torch.device,+ weights: torch.Tensor | None,+) -> torch.Tensor:+ """Blend the frames of ``tiles`` (whole tiles or chunks of them) that fall in ``batch_range``."""+ dtype = tiles[0].pixels.dtype+ output = torch.zeros(+ batch_range.stop - batch_range.start, output_height, output_width, 3, device=device, dtype=dtype+ )+ for tile in tiles:+ f_slice, h_slice, w_slice = tile.pixel_tile.out_coords+ first = f_slice.start + tile.frames_start+ overlap = slice(max(batch_range.start, first), min(batch_range.stop, first + tile.pixels.shape[0]))+ if overlap.start >= overlap.stop:+ continue+ pixel_frames = slice(overlap.start - first, overlap.stop - first)+ mask_frames = slice(overlap.start - f_slice.start, overlap.stop - f_slice.start)+ out_frames = slice(overlap.start - batch_range.start, overlap.stop - batch_range.start)+ # Blend weights stay float32 so the multiply promotes bf16/fp16 pixels.+ mf, mh, mw = (+ upload_from_host(m.to(dtype=torch.float32), torch.device(device)) for m in tile.pixel_tile.masks_1d+ )+ pix = tile.pixels[pixel_frames].to(device=device, non_blocking=True)+ # Channel axis is not on the pixel Tile; length-1 ones broadcasts over C.+ output[out_frames, h_slice, w_slice, :] += scale_by_masks_1d(+ pix, (mf[mask_frames], mh, mw, torch.ones(1, device=device, dtype=torch.float32))+ )+ if weights is not None:+ output.div_(weights[batch_range.start : batch_range.stop].to(device=device)[:, :, :, None])+ return output++def gather_frames(tiles: list[DecodedTile],total_frames: int,@@ -180,42 +457,8 @@ def gather_frames("""if device_fn is None:device_fn = lambda b: f"cuda:{b % world_size}" # noqa: E731-- batch_size = (total_frames + num_temporal_batches - 1) // num_temporal_batches-- for b in range(num_temporal_batches):- batch_range = slice(b * batch_size, min((b + 1) * batch_size, total_frames))- batch_len = batch_range.stop - batch_range.start- if batch_len <= 0:- break-- device = device_fn(b)- dtype = tiles[0].pixels.dtype- output = torch.zeros(batch_len, output_height, output_width, 3, device=device, dtype=dtype)-- for tile in tiles:- f_slice, h_slice, w_slice = tile.pixel_tile.out_coords-- overlap = slice(max(batch_range.start, f_slice.start), min(batch_range.stop, f_slice.stop))- if overlap.start >= overlap.stop:- continue-- tile_frames = slice(overlap.start - f_slice.start, overlap.stop - f_slice.start)- out_frames = slice(overlap.start - batch_range.start, overlap.stop - batch_range.start)-- # Blend weights stay float32 so the multiply promotes bf16/fp16 pixels.- mf, mh, mw = (m.to(device=device, dtype=torch.float32) for m in tile.pixel_tile.masks_1d)- mf = mf[tile_frames]- pix = tile.pixels[tile_frames].to(device=device, non_blocking=True)- # Channel axis is not on the pixel Tile; length-1 ones broadcasts over C.- output[out_frames, h_slice, w_slice, :] += scale_by_masks_1d(- pix, (mf, mh, mw, torch.ones(1, device=device, dtype=torch.float32))- )-- if weights is not None:- batch_weights = weights[batch_range.start : batch_range.stop].to(device=device)- output.div_(batch_weights[:, :, :, None])- yield output+ for b, batch_range in enumerate(_temporal_batches(total_frames, num_temporal_batches)):+ yield _assemble_batch(tiles, batch_range, output_height, output_width, device_fn(b), weights)# ------------------------------------------------------------------@@ -274,6 +517,7 @@ class DistributedVideoDecoder(torch.nn.Module, Disposable):if num_temporal_batches is not None and num_temporal_batches < 1:raise ValueError(f"num_temporal_batches must be >= 1, got {num_temporal_batches}")self.num_temporal_batches = self.world_size if num_temporal_batches is None else num_temporal_batches+ self._assembler: _StreamAssembler | None = None@propertydef video_downscale_factors(self) -> SpatioTemporalScaleFactors:@@ -302,53 +546,119 @@ class DistributedVideoDecoder(torch.nn.Module, Disposable):device_fn: Callable[[int], str | torch.device] | None = None,**_kwargs: object,) -> Iterator[torch.Tensor]:- """Distributed decode — all ranks decode, driver assembles.- Not a generator so that worker side-effects (decode + queue.put)- execute eagerly regardless of whether the caller iterates.- 1. Each rank decodes its latent tile (with optional intra-GPU tiling).- 2. Workers send their :class:`DecodedTile` to the driver via the queue.- 3. The driver collects all tiles, blends overlaps, and returns- temporal batches distributed across GPUs.+ """Distributed decode -- all ranks decode, the driver assembles while the decode runs.+ Not a generator, so that a worker's side effects (decode + queue.put) run eagerly whether or+ not the caller iterates.+ 1. Each rank decodes its latent tiles (with optional intra-GPU tiling), chunk by chunk.+ 2. Workers put each chunk on the queue as soon as it is decoded, then a done marker -- or a+ failure marker if their decode raises, which the driver re-raises instead of waiting.+ 3. The driver decodes its own tiles on a thread and returns an iterator over the temporal+ batches, each yielded once every tile overlapping it has delivered those frames and+ complete on every stream (see :class:`_StreamAssembler`), so whatever consumes them (the+ MP4 encoder) overlaps the rest of the decode. Chunks no later batch overlaps are released."""latent_shape = VideoLatentShape.from_torch_shape(latent.shape)scale = self.decoder.video_downscale_factorsfull_shape = latent_shape.upscale(scale)-- # Phase 1: each rank decodes its assigned tiles.- my_tiles = self._decode_tiles(latent, latent_shape, scale, generator, tiling_config, keyframes)-- # Phase 2: workers send tiles to driver.+ tiles_by_rank = {+ rank: dist_rank_tiles(self.vae_tiling, latent_shape, scale, rank, self.world_size)+ for rank in range(self.world_size)+ }+ own = self._decoded_chunks(latent, tiles_by_rank[self.rank], scale, generator, tiling_config, keyframes)if self.rank != self.driver_rank:- self.queue.put((self.rank, my_tiles))+ try:+ for local_index, chunk in own:+ self.queue.put((self.rank, local_index, chunk.frames_start, chunk.pixels))+ except BaseException:+ # The driver waits on this queue for every tile; tell it this rank's never comes.+ self.queue.put((self.rank, None, _DECODE_FAILED, None))+ raise+ self.queue.put((self.rank, None, _DECODE_DONE, None))return iter([])- # Phase 3: driver collects and assembles.- all_tiles = self._collect_tiles(my_tiles)- if masks_are_complementary(- [t.pixel_tile for t in all_tiles],- (full_shape.frames, full_shape.height, full_shape.width),- ):+ pixel_tiles = [tile for rank in sorted(tiles_by_rank) for tile in tiles_by_rank[rank]]+ if masks_are_complementary(pixel_tiles, (full_shape.frames, full_shape.height, full_shape.width)):weights = Noneelse:logger.warning("VAE blend masks are not complementary; falling back to dense [F,H,W] ""weight normalization (expensive for large videos).")- weights = compute_summed_weights(- [t.pixel_tile for t in all_tiles],- (full_shape.frames, full_shape.height, full_shape.width),- )- batches = gather_frames(- all_tiles,- full_shape.frames,- full_shape.height,- full_shape.width,+ weights = compute_summed_weights(pixel_tiles, (full_shape.frames, full_shape.height, full_shape.width))+ if device_fn is None:+ device_fn = lambda b: f"cuda:{b % self.world_size}" # noqa: E731+ assembler = _StreamAssembler(+ tiles_by_rank,+ full_shape,self.num_temporal_batches,- self.world_size,weights,- device_fn=device_fn,+ device_fn,+ mp_queue=self.queue,+ driver_rank=self.driver_rank,)- return batches+ self._join_assembler()+ self._assembler = assembler+ assembler.start(own)+ return assembler.batches()++ def _join_assembler(self) -> None:+ if self._assembler is not None:+ self._assembler.join()+ self._assembler = None++ def dispose(self) -> None:+ """Join a streaming decode still running (it ends once its last chunk went out), then dispose."""+ self._join_assembler()+ super().dispose()++ def _decoded_chunks(+ self,+ latent: torch.Tensor,+ tiles: list[Tile],+ scale: SpatioTemporalScaleFactors,+ generator: torch.Generator | None,+ tiling_config: TilingConfig | None,+ keyframes: DecodeKeyframes | None,+ ) -> Iterator[tuple[int, DecodedTile]]:+ """Decode this rank's ``tiles`` and yield ``(local tile index, chunk)`` in temporal order.+ Each tile gets its keyframe planes narrowed to its own window:+ * spatially, with the same slices as its video latent -- planes are full-frame, so+ handing them over uncropped offsets every one of them by the tile origin;+ * temporally, via :meth:`DecodeKeyframes.for_frame_span`, which keeps the planes inside+ the tile **and the nearest one on each side of it**, leaves ``pixel_frame_indices``+ **global**, and sets ``clip_start_frame`` to the tile's first pixel. DiffVAE then+ uses ``t_s(48) - t_s(56)`` for the 8-frame gap instead of rewriting 48 to -8.+ """+ if self.vae_tiling.frames.num_tiles > 1 and tiling_config is not None:+ for tile in tiles:+ latent_f = tile.in_coords[0].stop - tile.in_coords[0].start+ pixel_f = (latent_f - 1) * scale.time + 1+ if _sgpu_has_temporal_tiling(tiling_config, pixel_f):+ raise ValueError(+ "Cannot combine multi-GPU temporal tiling (vae_tiling.frames.num_tiles > 1) "+ "with single-GPU temporal tiling that would split this rank's tile "+ f"({pixel_f} frames → {tiling_config.video_chunks_number(pixel_f)} chunks). "+ "Use only one to avoid causal decoding conflicts."+ )+ for local_index, tile in enumerate(tiles):+ latent_slice = latent[:, :, tile.in_coords[0], tile.in_coords[1], tile.in_coords[2]]+ tile_keyframes = (+ None+ if keyframes is None+ else keyframes.for_frame_span(tile.out_coords[0].start, tile.out_coords[0].stop - 1).crop_spatial(+ tile.in_coords[1], tile.in_coords[2]+ )+ )+ # Only pass the keyword when there is something to pass: the wrapped decoder may be+ # any VideoDecoder, including one written against the pre-keyframes signature.+ extra = {} if tile_keyframes is None else {"keyframes": tile_keyframes}+ start = 0+ for pixels in self.decoder.decode_video(latent_slice, tiling_config, generator=generator, **extra):+ yield local_index, DecodedTile(pixels=pixels, pixel_tile=tile, frames_start=start)+ start += pixels.shape[0]+ expected = tile.out_coords[0].stop - tile.out_coords[0].start+ if start != expected:+ raise RuntimeError(f"Dist tile decode yielded {start} frames for a {expected}-frame tile")def decode_single_frames(self,@@ -360,28 +670,44 @@ class DistributedVideoDecoder(torch.nn.Module, Disposable):rank ``r`` decodes planes ``r, r + W, r + 2W, ...`` on the inner SGPU decoder and everyrank ends up holding the whole list, in the order the latents were given. A DiffVAEplane is expensive enough that decoding all of them on every rank -- what this used to- do -- wastes ``W - 1`` GPUs per plane.+ do -- wastes ``W - 1`` GPUs per plane. A caller that only needs something derived from+ the pixels should use :meth:`owned_single_frames` and gather that instead.``decode_video`` is still the wrong tool here: it splits one volume, and its workersyield nothing after queueing their tile.Not a generator. The all-gather has to run on every rank whether or not the calleriterates the result, and a rank whose share is empty (fewer planes than ranks, or aragged tail) still takes part with an empty payload instead of skipping the collective-- one absent participant blocks the group until the NCCL timeout.- Pixels cross the group as CPU tensors: ``all_gather_object`` pickles a CUDA tensor with- its source device index, so a plane decoded on ``cuda:3`` would be rebuilt on ``cuda:3``- on every rank. Each plane is moved onto the device of its own latent on the way out, and- a rank keeps the GPU copy of the planes it decoded itself.+ Each plane is moved onto the device of its own latent on the way out, and a rank keeps+ the GPU copy of the planes it decoded itself.+ """+ if not latents:+ return iter(())+ owned = self.owned_single_frames(latents, generator)+ mine: dict[int, torch.Tensor] = dict(zip(owned.indices, owned.planes, strict=True))+ merged = owned.gather(mine)+ # Lazy on the way out: the collective already ran, and the caller usually consumes one+ # plane at a time, so only the ranks' own planes stay resident on the GPU.+ return (merged[index].to(latents[index].device) for index in range(len(latents)))++ def owned_single_frames(+ self,+ latents: Sequence[torch.Tensor],+ generator: torch.Generator | Sequence[torch.Generator | None] | None = None,+ ) -> OwnedSingleFrames:+ """This rank's round-robin share of the one-frame clips, decoded, and the group's gather.+ The pixels stay on this rank; ``gather`` is a collective every rank calls exactly once,+ with ``{index: value}`` for its own indices, to receive every rank's values. Values cross+ the group through ``all_gather_object``, as CPU tensors where they are tensors (a pickled+ CUDA tensor is rebuilt on its source device index on every rank), so callers gather what+ they derive from the pixels -- e.g. re-encoded latents -- rather than the pixels.With a per-plane ``generator`` sequence the pixels do not depend on the world size. Asingle shared generator (or ``None``) is drawn from independently on each rank, so withthose the noise a given plane gets does."""- if not latents:- return iter(())validate_single_frame_latents(latents)gens = clip_generators(len(latents), generator)- if self.world_size == 1:- return iter_decoded_single_frames(self.decoder, latents, gens)- if len(latents) > 1 and not isinstance(generator, Sequence):+ if self.world_size > 1 and len(latents) > 1 and not isinstance(generator, Sequence):logger.warning("Dist single-frame decode is splitting %d planes over %d ranks with a shared ""generator; each rank draws its own noise, so the pixels depend on the world "@@ -389,7 +715,6 @@ class DistributedVideoDecoder(torch.nn.Module, Disposable):len(latents),self.world_size,)-my_indices = dist_rank_plane_indices(len(latents), self.rank, self.world_size)logger.info("Dist single-frame decode: rank %d of %d takes %d of %d planes",@@ -398,92 +723,30 @@ class DistributedVideoDecoder(torch.nn.Module, Disposable):len(my_indices),len(latents),)- decoded = iter_decoded_single_frames(+ planes = _planes_with_shared_rope_tables(self.decoder, [latents[i] for i in my_indices], [gens[i] for i in my_indices])- mine: dict[int, torch.Tensor] = dict(zip(my_indices, decoded, strict=True))-- payload = [(index, pixels.to("cpu")) for index, pixels in mine.items()]- shares: list[list[tuple[int, torch.Tensor]] | None] = [None] * self.world_size- dist.all_gather_object(shares, payload, group=self.vae_group)- merged: dict[int, torch.Tensor] = {}- for share in shares:- merged.update(dict(share or ()))- # This rank's own planes are already on the right device; prefer them over the round trip.- merged.update(mine)- missing = [index for index in range(len(latents)) if index not in merged]- if missing:- raise RuntimeError(f"Dist single-frame decode lost planes {missing} in the all-gather")- # Lazy on the way out: the collective already ran, and the caller usually consumes one- # plane at a time, so only the ranks' own planes stay resident on the GPU.- return (merged[index].to(latents[index].device) for index in range(len(latents)))++ def gather(owned: dict[int, Any]) -> dict[int, Any]:+ if self.world_size == 1:+ return dict(owned)+ payload = [+ (index, value.to("cpu") if isinstance(value, torch.Tensor) else value) for index, value in owned.items()+ ]+ shares: list[list[tuple[int, Any]] | None] = [None] * self.world_size+ dist.all_gather_object(shares, payload, group=self.vae_group)+ merged: dict[int, Any] = {}+ for share in shares:+ merged.update(dict(share or ()))+ # This rank's own values are already where the caller put them; prefer them over the round trip.+ merged.update(owned)+ missing = [index for index in range(len(latents)) if index not in merged]+ if missing:+ raise RuntimeError(f"Dist single-frame decode lost planes {missing} in the all-gather")+ return merged++ return OwnedSingleFrames(my_indices, planes, gather)# ------------------------------------------------------------------# Private helpers# -------------------------------------------------------------------- def _decode_tiles(- self,- latent: torch.Tensor,- latent_shape: VideoLatentShape,- scale: SpatioTemporalScaleFactors,- generator: torch.Generator | None,- tiling_config: TilingConfig | None = None,- keyframes: DecodeKeyframes | None = None,- ) -> list[DecodedTile]:- """Decode this rank's assigned latent tiles and convert to :class:`DecodedTile` list.- Each tile gets its keyframe planes narrowed to its own window:- * spatially, with the same slices as its video latent -- planes are full-frame, so- handing them over uncropped offsets every one of them by the tile origin;- * temporally, via :meth:`DecodeKeyframes.for_frame_span`, which keeps the planes inside- the tile **and the nearest one on each side of it**, leaves ``pixel_frame_indices``- **global**, and sets ``clip_start_frame`` to the tile's first pixel. DiffVAE then- uses ``t_s(48) - t_s(56)`` for the 8-frame gap instead of rewriting 48 to -8.- """- my_tiles = dist_rank_tiles(self.vae_tiling, latent_shape, scale, self.rank, self.world_size)- if self.vae_tiling.frames.num_tiles > 1 and tiling_config is not None:- for tile in my_tiles:- latent_f = tile.in_coords[0].stop - tile.in_coords[0].start- pixel_f = (latent_f - 1) * scale.time + 1- if _sgpu_has_temporal_tiling(tiling_config, pixel_f):- raise ValueError(- "Cannot combine multi-GPU temporal tiling (vae_tiling.frames.num_tiles > 1) "- "with single-GPU temporal tiling that would split this rank's tile "- f"({pixel_f} frames → {tiling_config.video_chunks_number(pixel_f)} chunks). "- "Use only one to avoid causal decoding conflicts."- )- decoded = []- for tile in my_tiles:- # One MGPU tile only — cat stitches SGPU temporal yields of this tile,- # not the full video (full assembly is gather_frames on the driver).- latent_slice = latent[:, :, tile.in_coords[0], tile.in_coords[1], tile.in_coords[2]]- tile_keyframes = (- None- if keyframes is None- else keyframes.for_frame_span(tile.out_coords[0].start, tile.out_coords[0].stop - 1).crop_spatial(- tile.in_coords[1], tile.in_coords[2]- )- )- # Only pass the keyword when there is something to pass: the wrapped decoder may be- # any VideoDecoder, including one written against the pre-keyframes signature.- extra = {} if tile_keyframes is None else {"keyframes": tile_keyframes}- chunks = list(self.decoder.decode_video(latent_slice, tiling_config, generator=generator, **extra))- pixels = torch.cat(chunks, dim=0) # [F, H, W, C] in [0, 1]- decoded.append(_to_decoded_tile(pixels, tile))- return decoded-- def _collect_tiles(self, driver_tiles: list[DecodedTile]) -> list[DecodedTile]:- """Collect tiles from all workers via the queue. Returns flat list of all tiles.- Sorted by rank so the downstream reduction in ``gather_frames``- (in-place ``+=`` over overlapping pixel regions) processes tiles in a- fixed order. Queue-arrival order would otherwise vary run-to-run and- yield 1-ulp bf16 drift from non-associative floating-point summation.- """- per_rank: dict[int, list[DecodedTile]] = {self.driver_rank: driver_tiles}- for _ in range(self.world_size - 1):- worker_rank, worker_tiles = self.queue.get()- per_rank[worker_rank] = worker_tiles- result: list[DecodedTile] = []- for rank in sorted(per_rank):- result.extend(per_rank[rank])- return resultdiff --git a/packages/ltx-core/src/ltx_core/quantization/fp8_cast.py b/packages/ltx-core/src/ltx_core/quantization/fp8_cast.pyindex dfa8789..d829719 100644--- a/packages/ltx-core/src/ltx_core/quantization/fp8_cast.py+++ b/packages/ltx-core/src/ltx_core/quantization/fp8_cast.py@@ -1,3 +1,5 @@+import dataclasses+import functoolsfrom pathlib import Pathimport safetensors@@ -263,6 +265,82 @@ def _fp8_cast_fuse(fp8_cast_fuse_rule = FuseRule(aggregation_dtype=torch.bfloat16, fuse_fn=_fp8_cast_fuse)+E4M3_MAX = float(torch.finfo(torch.float8_e4m3fn).max)+# fp8 x fp8 GEMMs (torch._scaled_mm) need compute capability 8.9+ (Ada, Hopper, Blackwell).+_FP8_GEMM_MIN_CAPABILITY = (8, 9)+++def supports_fp8_gemm(device: torch.device) -> bool:+ """Whether *device* runs the e4m3 x e4m3 GEMMs of :class:`Fp8RowwiseLinear`."""+ return device.type == "cuda" and torch.cuda.get_device_capability(device) >= _FP8_GEMM_MIN_CAPABILITY+++class Fp8RowwiseLinear(Fp8CastLinear):+ """:class:`Fp8CastLinear` that multiplies long activations in e4m3 instead of upcasting the weight.+ A bf16 input of at least ``_min_rows`` rows on a device with fp8 GEMMs (:func:`supports_fp8_gemm`)+ is rounded to e4m3 with one scale per row (the row's absolute maximum maps to the e4m3 maximum)+ and multiplied with the stored e4m3 weight at scale 1 through ``torch._scaled_mm`` (unit scales,+ fp32 accumulation, bf16 product); the row scales and the bias are then applied in fp32. Other+ inputs, including any on a device without fp8 GEMMs, take the upcast path. Lossy:+ the activations keep e4m3's 3 mantissa bits. The cast is plain tensor ops so a compile+ backend can fuse it into the kernel producing the activation.+ """++ _min_rows: int++ def forward(self, input: torch.Tensor) -> torch.Tensor: # noqa: A002, type: ignore[override]+ in_features = input.shape[-1]+ rows = input.numel() // in_features+ if (+ rows < self._min_rows+ or input.dtype != torch.bfloat16+ or not supports_fp8_gemm(input.device)+ or self._with_stochastic_rounding+ ):+ return super().forward(input)+ x = input.reshape(rows, in_features)+ amax = x.abs().amax(dim=-1, keepdim=True).float()+ row_scale = torch.where(amax > 0, amax / E4M3_MAX, torch.ones_like(amax))+ x8 = (x.float() / row_scale).to(torch.float8_e4m3fn)+ # Unit (tensor-wise) scales select the cuBLASLt fp8 GEMM; per-row scales would select a slower CUTLASS+ # kernel with per-launch host setup. The row scale and the bias are applied to the bf16 product as plain+ # tensor ops, which a compile backend fuses into the kernel consuming this Linear's output.+ unit = torch.ones((), device=input.device, dtype=torch.float32)+ out = torch._scaled_mm(+ x8, self.weight.t(), scale_a=unit, scale_b=unit, out_dtype=torch.bfloat16, use_fast_accum=False+ )+ out = out.float() * row_scale+ if self.bias is not None:+ out = out + self.bias.float()+ return out.to(torch.bfloat16).view(*input.shape[:-1], self.out_features)+++def _amend_forward_with_rowwise_fp8_gemm(model: torch.nn.Module, min_rows: int) -> torch.nn.Module:+ """Retype every :class:`Fp8CastLinear` of *model* to :class:`Fp8RowwiseLinear`."""+ for m in model.modules():+ if type(m) is Fp8CastLinear:+ m.__class__ = Fp8RowwiseLinear+ m._min_rows = min_rows+ return model+++def with_rowwise_fp8_gemm(policy: QuantizationPolicy, min_rows: int = 8192) -> QuantizationPolicy:+ """*policy* (an fp8-cast policy) with its Linears' long inputs multiplied in e4m3 (:class:`Fp8RowwiseLinear`).+ Weights and their loading are unchanged, so a stage on this policy shares its weights with one+ on *policy*. Only for devices where :func:`supports_fp8_gemm` holds. ``min_rows`` is the per-rank+ activation row count from which the e4m3 GEMM is taken: the video token sequences of long passes,+ not the audio or text-context projections.+ """+ if policy.fuse_rule is not fp8_cast_fuse_rule:+ raise ValueError("with_rowwise_fp8_gemm needs an fp8-cast policy (weights stored in e4m3)")+ op = ModuleOps(+ name=f"fp8_rowwise_gemm_from_{min_rows}_rows",+ matcher=lambda model: isinstance(model, LTXModel),+ mutator=functools.partial(_amend_forward_with_rowwise_fp8_gemm, min_rows=min_rows),+ )+ return dataclasses.replace(policy, module_ops=(*policy.module_ops, op))++# Raw safetensors storage prefix shared by every diffusion-transformer# parameter (and every prequant `*_scale` sibling). Verified against# ltx-2.3-22b-{dev,distilled}-fp8.safetensors: 2924/2924 and 2992/2992 ofdiff --git a/packages/ltx-core/src/ltx_core/tools.py b/packages/ltx-core/src/ltx_core/tools.pyindex d3ae087..602da21 100644--- a/packages/ltx-core/src/ltx_core/tools.py+++ b/packages/ltx-core/src/ltx_core/tools.py@@ -12,7 +12,7 @@ from ltx_core.components.patchifiers import (get_pixel_coords,)from ltx_core.components.protocols import Patchifier-from ltx_core.types import LatentState, SpatioTemporalScaleFactors+from ltx_core.types import DenoiseMask, LatentState, SpatioTemporalScaleFactorsDEFAULT_SCALE_FACTORS = SpatioTemporalScaleFactors.default()@@ -48,7 +48,9 @@ class LatentTools(Protocol):latent_state = latent_state.clone()latent = self.patchifier.patchify(latent_state.latent)clean_latent = self.patchifier.patchify(latent_state.clean_latent)- denoise_mask = self.patchifier.patchify(latent_state.denoise_mask)+ denoise_mask = latent_state.denoise_mask.with_same_values(+ self.patchifier.patchify(latent_state.denoise_mask.tensor)+ )# keyframes_mask shares denoise_mask's layout, so it converts alongside it; leaving it# behind would produce a state mixing patchified and unpatchified per-token tensors.keyframes_mask = (@@ -70,8 +72,9 @@ class LatentTools(Protocol):latent = self.patchifier.unpatchify(latent_state.latent, output_shape=self.target_shape)clean_latent = self.patchifier.unpatchify(latent_state.clean_latent, output_shape=self.target_shape)denoise_mask = self.patchifier.unpatchify(- latent_state.denoise_mask, output_shape=self.target_shape.mask_shape()+ latent_state.denoise_mask.tensor, output_shape=self.target_shape.mask_shape())+ denoise_mask = latent_state.denoise_mask.with_same_values(denoise_mask)keyframes_mask = (self.patchifier.unpatchify(latent_state.keyframes_mask, output_shape=self.target_shape.mask_shape())if latent_state.keyframes_mask is not None@@ -101,12 +104,12 @@ class LatentTools(Protocol):num_tokens = self.patchifier.get_token_count(self.target_shape)latent = latent_state.latent[:, :num_tokens]clean_latent = latent_state.clean_latent[:, :num_tokens]- denoise_mask = torch.ones_like(latent_state.denoise_mask)[:, :num_tokens]+ denoise_mask = torch.ones_like(latent_state.denoise_mask.tensor)[:, :num_tokens]positions = latent_state.positions[:, :, :num_tokens]return LatentState(latent=latent,- denoise_mask=denoise_mask,+ denoise_mask=DenoiseMask(denoise_mask, denoise_mask.new_ones((1,))),positions=positions,clean_latent=clean_latent,attention_mask=None,@@ -155,11 +158,7 @@ class VideoLatentTools(LatentTools):clean_latent = initial_latent.clone()- denoise_mask = torch.ones(- *self.target_shape.mask_shape().to_torch_shape(),- device=device,- dtype=torch.float32,- )+ denoise_mask = torch.ones(*self.target_shape.mask_shape().to_torch_shape(), dtype=torch.float32)latent_coords = self.patchifier.get_patch_grid_bounds(output_shape=self.target_shape,@@ -176,7 +175,7 @@ class VideoLatentTools(LatentTools):state = self.patchify(LatentState(latent=initial_latent,- denoise_mask=denoise_mask,+ denoise_mask=DenoiseMask(denoise_mask),positions=positions,clean_latent=clean_latent,)@@ -191,7 +190,7 @@ class VideoLatentTools(LatentTools):independently of whether any keyframe slots exist. This is plain data about the latent, soit is always populated; models without the keyframe embedding ignore it."""- mask = torch.zeros_like(state.denoise_mask)+ mask = torch.zeros_like(state.denoise_mask.tensor, device=state.latent.device)mask[:, : self.tokens_per_latent_frame] = 1.0return mask@@ -262,11 +261,7 @@ class AudioLatentTools(LatentTools):clean_latent = initial_latent.clone()- denoise_mask = torch.ones(- *self.target_shape.mask_shape().to_torch_shape(),- device=device,- dtype=torch.float32,- )+ denoise_mask = torch.ones(*self.target_shape.mask_shape().to_torch_shape(), dtype=torch.float32)latent_coords = self.patchifier.get_patch_grid_bounds(output_shape=self.target_shape,@@ -275,6 +270,9 @@ class AudioLatentTools(LatentTools):return self.patchify(LatentState(- latent=initial_latent, denoise_mask=denoise_mask, positions=latent_coords, clean_latent=clean_latent+ latent=initial_latent,+ denoise_mask=DenoiseMask(denoise_mask),+ positions=latent_coords,+ clean_latent=clean_latent,))diff --git a/packages/ltx-core/src/ltx_core/types.py b/packages/ltx-core/src/ltx_core/types.pyindex 115b1e3..f50eee2 100644--- a/packages/ltx-core/src/ltx_core/types.py+++ b/packages/ltx-core/src/ltx_core/types.py@@ -3,6 +3,8 @@ from typing import NamedTupleimport torch+from ltx_core.devices import upload_from_host+class VideoPixelShape(NamedTuple):"""@@ -200,6 +202,24 @@ class AudioLatentShape(NamedTuple):)+@dataclass(frozen=True)+class HostDeviceTensorPair:+ """A tensor together with its host mirror."""++ host_tensor: torch.Tensor+ device_tensor: torch.Tensor++ @classmethod+ def on_device(cls, tensor: torch.Tensor, device: torch.device) -> "HostDeviceTensorPair":+ """``tensor``'s host mirror and a copy on ``device``.+ Reads the device back only when ``tensor`` lives there; a host tensor is uploaded without+ waiting for the device.+ """+ host = tensor.cpu()+ device_tensor = tensor if tensor.device == device else upload_from_host(host, device)+ return cls(host_tensor=host, device_tensor=device_tensor)++@dataclass(frozen=True)class Audio:"""@@ -247,6 +267,42 @@ class GeneratedKeyframeLayout:return slice(self.first_token, self.first_token + self.num_tokens)+@dataclass(frozen=True)+class DenoiseMask:+ """A per-token denoising mask and the table of its distinct values.+ Conditioning builds ``tensor`` on the host, editing it in place or replacing it, with+ ``values`` unset. ``compact`` tabulates the distinct values there, where ``torch.unique`` needs+ no device synchronization, and moves both to the device; the denoising loop only sees the+ compacted form, and nothing may edit ``tensor`` after it. ``values`` must hold exactly the+ values present: a missing one silently modulates its tokens with a neighbouring value's+ conditioning, and an extra one changes the shape the modulation is projected at.+ ``zero_index`` is the position of ``0.0`` in ``values``, recorded on the host by ``compact``+ (``None`` when no token is fully conditioned), so padding can reuse that entry.+ """++ tensor: torch.Tensor+ values: torch.Tensor | None = None+ zero_index: int | None = None++ def compact(self, device: torch.device) -> "DenoiseMask":+ """Tabulate the distinct mask values on the host and move mask and table to ``device``."""+ if self.tensor.device.type != "cpu":+ raise ValueError(f"denoise mask must be built on the host, got {self.tensor.device}")+ values = torch.unique(self.tensor, sorted=True)+ zeros = (values == 0).nonzero()+ zero_index = int(zeros[0, 0]) if zeros.numel() else None+ return DenoiseMask(upload_from_host(self.tensor, device), upload_from_host(values, device), zero_index)++ def with_same_values(self, tensor: torch.Tensor) -> "DenoiseMask":+ """Replace the tensor with a reordering or batch repeat of the same token values."""+ return DenoiseMask(tensor, self.values, self.zero_index)++ def clone(self) -> "DenoiseMask":+ return DenoiseMask(+ self.tensor.clone(), self.values.clone() if self.values is not None else None, self.zero_index+ )++@dataclass(frozen=True)class LatentState:"""@@ -277,7 +333,7 @@ class LatentState:"""latent: torch.Tensor- denoise_mask: torch.Tensor+ denoise_mask: DenoiseMaskpositions: torch.Tensorclean_latent: torch.Tensorattention_mask: torch.Tensor | None = Nonediff --git a/packages/ltx-kernels/csrc/all2all/all2all.cpp b/packages/ltx-kernels/csrc/all2all/all2all.cppindex d3ff803..b9a028d 100644--- a/packages/ltx-kernels/csrc/all2all/all2all.cpp+++ b/packages/ltx-kernels/csrc/all2all/all2all.cpp@@ -61,7 +61,7 @@ All2All::All2All(int rank, int world_size, int num_tokens, int hidden_dim, int n// Calculate sizes for each region of the shared memory blockint64_t ptrs_bytes = MAX_NUM_PEERS * sizeof(void *);- int64_t barrier_signal_bytes = MAX_NUM_PEERS * sizeof(int);+ int64_t barrier_signal_bytes = kNumSignalSlots * sizeof(int);int64_t barrier_signal_ptrs_bytes = MAX_NUM_PEERS * sizeof(int *);// Allocate the main shared memory block and create IPC handle@@ -320,6 +320,97 @@ at::Tensor All2All::allgather(at::Tensor &x, bool copy_output) {return get_local_buffer_tensor(x, batch_size, out_tokens, out_heads, head_size, copy_output, stream);}+void All2All::rope_send_recv_heads(at::Tensor &x, at::Tensor &sin_freqs, at::Tensor &cos_freqs, at::Tensor &weights,+ double eps, bool barrier) {+ EP_HOST_ASSERT(x.dim() == 3 and x.scalar_type() == at::ScalarType::BFloat16);+ EP_HOST_ASSERT(tensor_dtype == at::ScalarType::BFloat16);+ EP_HOST_ASSERT(x.device().is_cuda() and x.device().index() == rank);+ EP_HOST_ASSERT(weights.scalar_type() == at::ScalarType::BFloat16);+ EP_HOST_ASSERT(cos_freqs.dim() == 4 and sin_freqs.dim() == 4);+ // The kernel walks x and weights from one base pointer each, so a strided input is made contiguous,+ // as ops/rms_norm_split_rope.cpp does for x.+ if (!x.is_contiguous()) {+ x = x.contiguous();+ }+ if (!weights.is_contiguous()) {+ weights = weights.contiguous();+ }+ int b = x.size(0);+ int s = x.size(1);+ int h = x.size(2);+ int n = cos_freqs.size(1);+ EP_HOST_ASSERT(int64_t(b) * s * world_size * (h / world_size) * int64_t(elementSize(tensor_dtype)) <= tensor_bytes);++ // Same freq-table handling as ops/rms_norm_split_rope.cpp: innermost dim contiguous, outer strides+ // forwarded, a batch-shared table expanded (stride 0) rather than copied.+ if (cos_freqs.stride(-1) != 1) {+ cos_freqs = cos_freqs.contiguous();+ }+ if (sin_freqs.stride(-1) != 1) {+ sin_freqs = sin_freqs.contiguous();+ }+ if (cos_freqs.size(0) == 1 && b > 1) {+ cos_freqs = cos_freqs.expand({b, n, cos_freqs.size(2), cos_freqs.size(3)});+ }+ if (sin_freqs.size(0) == 1 && b > 1) {+ sin_freqs = sin_freqs.expand({b, n, sin_freqs.size(2), sin_freqs.size(3)});+ }++ at::cuda::CUDAGuard device_guard{x.device()};+ auto stream = at::cuda::getCurrentCUDAStream().stream();+ all2all_cuda::rope_scatter_heads_launch(x.data_ptr(), sin_freqs.data_ptr(), cos_freqs.data_ptr(), weights.data_ptr(),+ static_cast<float>(eps), b, s, n, h, cos_freqs.stride(0), cos_freqs.stride(1),+ cos_freqs.stride(2), sin_freqs.stride(0), sin_freqs.stride(1),+ sin_freqs.stride(2), buffer_ptrs_gpu, barrier_signal_ptrs_gpu, rank,+ world_size, timeout_cycles_, barrier, stream);+}++void All2All::linear_send_recv_heads(at::Tensor &x, at::Tensor &weight, at::Tensor &bias, bool barrier) {+ // A blockwise-fp8 exchange quantizes in its own copy kernel; this path writes the GEMM's bf16 output.+ EP_HOST_ASSERT(tensor_dtype == at::ScalarType::BFloat16);+ EP_HOST_ASSERT(x.dim() == 3 and x.scalar_type() == tensor_dtype);+ EP_HOST_ASSERT(weight.dim() == 2 and weight.scalar_type() == tensor_dtype);+ EP_HOST_ASSERT(bias.dim() == 1 and bias.scalar_type() == tensor_dtype);+ EP_HOST_ASSERT(x.device().is_cuda() and x.device().index() == rank);+ int b = x.size(0);+ int s = x.size(1);+ int h = weight.size(0);+ EP_HOST_ASSERT(h % world_size == 0);+ int64_t cols = h / world_size;+ int64_t total_tokens = int64_t(s) * world_size;+ EP_HOST_ASSERT(int64_t(b) * total_tokens * cols * int64_t(elementSize(tensor_dtype)) <= tensor_bytes);++ at::cuda::CUDAGuard device_guard{x.device()};+ auto stream = at::cuda::getCurrentCUDAStream().stream();+ auto options = x.options();+ // Peers first, so the NVLink-bound writes start as early as possible.+ for (int step = 1; step <= world_size; step++) {+ int target = (rank + step) % world_size;+ auto w = weight.narrow(0, target * cols, cols);+ auto bias_slice = bias.narrow(0, target * cols, cols);+ for (int batch = 0; batch < b; batch++) {+ int64_t offset = (int64_t(batch) * total_tokens + int64_t(rank) * s) * cols;+ // A peer buffer is IPC-mapped into this device's address space; pinning the target device skips the+ // pointer-owner lookup, which would report the peer.+ auto out =+ at::for_blob(static_cast<uint8_t *>(buffer_ptrs[target]) + offset * elementSize(tensor_dtype), {s, cols})+ .options(options)+ .target_device(x.device())+ .make_tensor();+ at::addmm_out(out, bias_slice, x.select(0, batch), w.t());+ }+ }+ if (barrier) {+ all2all_cuda::fused_barrier_launch(barrier_signal_ptrs_gpu, rank, world_size, timeout_cycles_, stream);+ }+}++void All2All::exchange_barrier() {+ at::cuda::CUDAGuard device_guard{static_cast<c10::DeviceIndex>(rank)};+ auto stream = at::cuda::getCurrentCUDAStream().stream();+ all2all_cuda::fused_barrier_launch(barrier_signal_ptrs_gpu, rank, world_size, timeout_cycles_, stream);+}+} // namespace all2all} // namespace ltx_kernels@@ -376,6 +467,12 @@ PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {.def("gather_heads", <x_kernels::all2all::All2All::gather_heads,"Inverse All2All to gather heads back to original distribution.").def("allgather", <x_kernels::all2all::All2All::allgather, "Gathers sequence tokens from all ranks.")+ .def("rope_send_recv_heads", <x_kernels::all2all::All2All::rope_send_recv_heads,+ "Fused RMS-norm + split RoPE written straight into every rank's head-exchange buffer.")+ .def("linear_send_recv_heads", <x_kernels::all2all::All2All::linear_send_recv_heads,+ "Linear projection written straight into every rank's head-exchange buffer.")+ .def("exchange_barrier", <x_kernels::all2all::All2All::exchange_barrier,+ "Waits until every rank's earlier writes on this stream are visible to every rank.").def("set_timeout_seconds", <x_kernels::all2all::All2All::set_timeout_seconds,"Sets the barrier timeout in seconds (converted to cycles via the device peak SM clock).");}diff --git a/packages/ltx-kernels/csrc/all2all/all2all.hpp b/packages/ltx-kernels/csrc/all2all/all2all.hppindex 34b0efc..97addd9 100644--- a/packages/ltx-kernels/csrc/all2all/all2all.hpp+++ b/packages/ltx-kernels/csrc/all2all/all2all.hpp@@ -234,6 +234,20 @@ public:*/at::Tensor allgather(at::Tensor &x, bool copy_output);+ /// Fused RMS-norm + split RoPE of ``x`` ``[b, s, h]``, written straight into every rank's buffer in the+ /// ``send_recv_heads`` layout ``[b, world_size * s, h / world_size]``. With ``barrier``, returns once all+ /// ranks' writes into the local buffer are visible; without it, a later exchange_barrier() does that.+ void rope_send_recv_heads(at::Tensor &x, at::Tensor &sin_freqs, at::Tensor &cos_freqs, at::Tensor &weights,+ double eps, bool barrier);++ /// ``x @ weight.T + bias`` with each rank's output columns written straight into that rank's buffer in+ /// the ``send_recv_heads`` layout. ``barrier`` as for rope_send_recv_heads().+ void linear_send_recv_heads(at::Tensor &x, at::Tensor &weight, at::Tensor &bias, bool barrier);++ /// Returns once every rank's writes issued earlier on this stream -- into any instance's buffer --+ /// are visible to every rank. Completes the exchanges run with ``barrier`` false.+ void exchange_barrier();+/*** @brief Sets the all2all barrier timeout in seconds.*diff --git a/packages/ltx-kernels/csrc/all2all/cuda/api.cuh b/packages/ltx-kernels/csrc/all2all/cuda/api.cuhindex 3d1bb35..c4cf9d0 100644--- a/packages/ltx-kernels/csrc/all2all/cuda/api.cuh+++ b/packages/ltx-kernels/csrc/all2all/cuda/api.cuh@@ -89,6 +89,14 @@ void allgather_launch(void **buffer_ptrs, int **barrier_signal_ptrs, void *x, inint seqlen, int hidden_dim, int total_tokens, cudaStream_t stream, int num_sms,at::ScalarType tensor_dtype, uint64_t timeout_cycles);+void rope_scatter_heads_launch(void *x, void *sin_freqs, void *cos_freqs, void *weights, float eps, int b, int s, int n,+ int h, long cos_sb, long cos_sn, long cos_ss, long sin_sb, long sin_sn, long sin_ss,+ void **buffer_ptrs, int **barrier_signal_ptrs, int rank, int world_size,+ uint64_t timeout_cycles, bool barrier, cudaStream_t stream);++void fused_barrier_launch(int **barrier_signal_ptrs, int rank, int world_size, uint64_t timeout_cycles,+ cudaStream_t stream);+} // namespace all2all_cuda} // namespace all2all} // namespace ltx_kernelsdiff --git a/packages/ltx-kernels/csrc/all2all/cuda/fused_send.cu b/packages/ltx-kernels/csrc/all2all/cuda/fused_send.cunew file mode 100644index 0000000..23c111d--- /dev/null+++ b/packages/ltx-kernels/csrc/all2all/cuda/fused_send.cu@@ -0,0 +1,161 @@+/**+ * Producers that write straight into the All2All head-exchange layout.+ *+ * ``send_recv_heads`` reads a finished ``[batch, tokens, heads, head_size]`` tensor and copies each+ * rank's head slice into that rank's IPC buffer. The kernels here fold that copy into the kernel that+ * produces the tensor, so the tensor is never materialized locally and its peer slice travels over+ * NVLink while the producer runs:+ *+ * - ``rope_scatter_heads``: fused RMS-norm + split RoPE (the math of ops/rms_norm_split_rope) whose+ * output rows land in the owning rank's buffer at ``[batch, rank * tokens + token, head % hpr, :]``.+ * - ``fused_barrier``: signals every rank that this rank's writes into its buffer are complete, then+ * waits for the same from every rank. Used after producers that are not kernels of ours (cuBLAS).+ *+ * Both complete only once every rank's writes into the local buffer are visible, as+ * ``send_recv_heads`` does.+ */++#include <algorithm>+#include <cstdio>++#include "../../include/cuda/configs.cuh"+#include "../../include/cuda/rms_norm_split_rope.cuh"+#include "../../include/cuda/utils.cuh"+#include "api.cuh"++#include <ATen/cuda/CUDAContext.h>+#include <c10/cuda/CUDAException.h>+#include <cuda_bf16.h>++namespace ltx_kernels {+namespace all2all {+namespace all2all_cuda {++using bf16 = __nv_bfloat16;++// Rotation pairs per thread, as in ops/rms_norm_split_rope_cuda.cu.+constexpr int kScatterPairsPerThread = 16;+constexpr int kScatterCtasPerSm = 8;++template <int PAIRS, int THREADS_PER_ROW>+__global__ void __launch_bounds__(THREADS_PER_ROW)+ rope_scatter_heads(const bf16 *__restrict__ x, const bf16 *__restrict__ sin_freqs,+ const bf16 *__restrict__ cos_freqs, const bf16 *__restrict__ weights, float eps, int b, int s,+ int n, int h, long cos_sb, long cos_sn, long cos_ss, long sin_sb, long sin_sn, long sin_ss,+ void **buffer_ptrs, int **barrier_signal_ptrs, int rank, int world_size, uint64_t timeout_cycles,+ bool barrier) {+ constexpr int kWarpsPerRow = THREADS_PER_ROW / 32;+ constexpr int kRowElems = PAIRS * THREADS_PER_ROW * 2;+ __shared__ float smem[kWarpsPerRow];+ // The finished row, staged so it leaves as contiguous 16-byte-per-lane runs: the RoPE pair layout+ // alone would store 16-byte chunks 32 bytes apart, half-filling every sector it sends over NVLink.+ __shared__ __align__(16) bf16 row_out[kRowElems];++ const int lane_in_row = threadIdx.x;+ const int d = h / n;+ const int half = d / 2;+ const int pair0 = lane_in_row * PAIRS;+ const int head = pair0 / half;+ const int freq0 = pair0 - head * half;+ const int first_channel = head * d + freq0;+ const int second_channel = first_channel + half;+ const int hpr = n / world_size;+ const int slice = hpr * d; // elements of one row that go to one rank+ const int64_t total_tokens = int64_t(s) * world_size;++ bf16 first_weight[PAIRS], second_weight[PAIRS];+ load_run<PAIRS>(weights + first_channel, first_weight);+ load_run<PAIRS>(weights + second_channel, second_weight);++ for (int token_idx = blockIdx.x; token_idx < b * s; token_idx += gridDim.x) {+ const int batch = token_idx / s;+ const int token = token_idx % s;+ const size_t row = static_cast<size_t>(token_idx) * static_cast<size_t>(h);+ const long cos_offset = static_cast<long>(batch) * cos_sb + static_cast<long>(head) * cos_sn ++ static_cast<long>(token) * cos_ss + freq0;+ const long sin_offset = static_cast<long>(batch) * sin_sb + static_cast<long>(head) * sin_sn ++ static_cast<long>(token) * sin_ss + freq0;++ float first_rot[PAIRS], second_rot[PAIRS];+ rms_norm_split_rope_pairs<PAIRS, kWarpsPerRow>(x + row + first_channel, x + row + second_channel,+ cos_freqs + cos_offset, sin_freqs + sin_offset, first_weight,+ second_weight, eps, h, lane_in_row, smem, first_rot, second_rot);+ bf16 first_out[PAIRS], second_out[PAIRS];+#pragma unroll+ for (int i = 0; i < PAIRS; i++) {+ first_out[i] = __float2bfloat16(first_rot[i]);+ second_out[i] = __float2bfloat16(second_rot[i]);+ }+#pragma unroll+ for (int i = 0; i < PAIRS / 8; i++) {+ reinterpret_cast<int4 *>(row_out + first_channel)[i] = reinterpret_cast<int4 *>(first_out)[i];+ reinterpret_cast<int4 *>(row_out + second_channel)[i] = reinterpret_cast<int4 *>(second_out)[i];+ }+ __syncthreads();++ // Rank `target` owns heads [target * hpr, (target + 1) * hpr), i.e. row elements+ // [target * slice, (target + 1) * slice); its buffer is [batch, world_size * s, hpr, d] with this+ // rank's tokens at [rank * s, (rank + 1) * s).+ const int64_t dst_row = int64_t(batch) * total_tokens + int64_t(rank) * s + token;+ for (int chunk = lane_in_row; chunk < kRowElems / 8; chunk += THREADS_PER_ROW) {+ const int elem = chunk * 8;+ const int target = elem / slice;+ bf16 *dst = static_cast<bf16 *>(buffer_ptrs[target]) + dst_row * slice + (elem - target * slice);+ st_na_global(reinterpret_cast<int4 *>(dst), reinterpret_cast<int4 *>(row_out)[chunk]);+ }+ // row_out and smem are rewritten by the next row.+ __syncthreads();+ }++ // Every block wrote to every rank, so completion is counted across blocks before anyone signals.+ // Without the barrier a later fused_barrier on the stream covers these writes.+ if (barrier) {+ barrier_last_block_wait_and_reset(barrier_signal_ptrs, barrier_signal_ptrs[rank] + kBlocksDoneSlot, rank,+ world_size, threadIdx.x, timeout_cycles);+ }+}++// One block per target rank: the writes it covers are complete already (they came from earlier+// kernels on this stream), so each block only has to signal its rank.+__global__ void fused_barrier(int **barrier_signal_ptrs, int rank, int world_size, uint64_t timeout_cycles) {+ barrier_wait_and_reset(barrier_signal_ptrs, blockIdx.x, rank, world_size, 1, blockIdx.x, threadIdx.x, timeout_cycles);+}++void rope_scatter_heads_launch(void *x, void *sin_freqs, void *cos_freqs, void *weights, float eps, int b, int s, int n,+ int h, long cos_sb, long cos_sn, long cos_ss, long sin_sb, long sin_sn, long sin_ss,+ void **buffer_ptrs, int **barrier_signal_ptrs, int rank, int world_size,+ uint64_t timeout_cycles, bool barrier, cudaStream_t stream) {+ TORCH_CHECK(n > 0 && h % n == 0 && n % world_size == 0, "heads must divide the hidden size and the world size");+ TORCH_CHECK((h / n) % (2 * kScatterPairsPerThread) == 0, "head_dim/2 must be a multiple of ", kScatterPairsPerThread);+ // Persistent CTAs walk the rows, so the completion counter sees one atomic per CTA, not per row.+ int device = 0;+ int num_sms = 0;+ C10_CUDA_CHECK(cudaGetDevice(&device));+ C10_CUDA_CHECK(cudaDeviceGetAttribute(&num_sms, cudaDevAttrMultiProcessorCount, device));+ const int grid = std::min(b * s, num_sms * kScatterCtasPerSm);+#define LAUNCH_ROPE_SCATTER(THREADS_PER_ROW) \+ rope_scatter_heads<kScatterPairsPerThread, (THREADS_PER_ROW)><<<grid, (THREADS_PER_ROW), 0, stream>>>( \+ static_cast<bf16 *>(x), static_cast<bf16 *>(sin_freqs), static_cast<bf16 *>(cos_freqs), \+ static_cast<bf16 *>(weights), eps, b, s, n, h, cos_sb, cos_sn, cos_ss, sin_sb, sin_sn, sin_ss, buffer_ptrs, \+ barrier_signal_ptrs, rank, world_size, timeout_cycles, barrier)+ if (h == 2048) {+ LAUNCH_ROPE_SCATTER(64);+ } else if (h == 4096) {+ LAUNCH_ROPE_SCATTER(128);+ } else {+ TORCH_CHECK(h == 8192, "Hidden dimension must be 2048, 4096, or 8192");+ LAUNCH_ROPE_SCATTER(256);+ }+#undef LAUNCH_ROPE_SCATTER+ C10_CUDA_KERNEL_LAUNCH_CHECK();+}++void fused_barrier_launch(int **barrier_signal_ptrs, int rank, int world_size, uint64_t timeout_cycles,+ cudaStream_t stream) {+ fused_barrier<<<world_size, 32, 0, stream>>>(barrier_signal_ptrs, rank, world_size, timeout_cycles);+ C10_CUDA_KERNEL_LAUNCH_CHECK();+}++} // namespace all2all_cuda+} // namespace all2all+} // namespace ltx_kernelsdiff --git a/packages/ltx-kernels/csrc/include/cuda/configs.cuh b/packages/ltx-kernels/csrc/include/cuda/configs.cuhindex e7381bc..1f60f5b 100644--- a/packages/ltx-kernels/csrc/include/cuda/configs.cuh+++ b/packages/ltx-kernels/csrc/include/cuda/configs.cuh@@ -39,6 +39,16 @@ constexpr double DEFAULT_BARRIER_TIMEOUT_SECONDS = 10.0;*/constexpr int MAX_NUM_PEERS = 8;+/**+ * @brief Layout of each rank's barrier signal slots.+ *+ * [0, MAX_NUM_PEERS): per-source signal counters. kBlocksDoneSlot: the local count of finished+ * blocks that barrier_last_block_wait_and_reset keeps. The count is even so the pointer arrays that+ * follow the slots in the IPC block stay 8-byte aligned.+ */+constexpr int kBlocksDoneSlot = MAX_NUM_PEERS;+constexpr int kNumSignalSlots = MAX_NUM_PEERS + 2;+// =============================================================================// Kernel Configuration// =============================================================================diff --git a/packages/ltx-kernels/csrc/include/cuda/rms_norm_split_rope.cuh b/packages/ltx-kernels/csrc/include/cuda/rms_norm_split_rope.cuhnew file mode 100644index 0000000..8a938d6--- /dev/null+++ b/packages/ltx-kernels/csrc/include/cuda/rms_norm_split_rope.cuh@@ -0,0 +1,92 @@+#pragma once++/**+ * @file rms_norm_split_rope.cuh+ * @brief Per-thread core of fused RMS-norm + split RoPE, shared by the kernels that apply it.+ *+ * A row is one token's hidden vector, ``h = heads * d``. The split RoPE rotates pair ``(i, i + d/2)``+ * of every head, and each thread owns PAIRS consecutive pairs of a single head -- both halves of each,+ * so rotating needs no exchange with another thread and every cos/sin value is read once.+ *+ * The kernels differ in how they walk rows and where the result goes (in place, fp8, or scattered+ * into all2all buffers); they share this function so their rounding is identical by construction.+ */++#include <cuda_bf16.h>++namespace ltx_kernels {++/// Loads N consecutive bf16 values as N / 8 16-byte vectors; `source` must be 16-byte aligned.+template <int N> __device__ __forceinline__ void load_run(const __nv_bfloat16 *source, __nv_bfloat16 values[N]) {+#pragma unroll+ for (int i = 0; i < N / 8; i++) {+ reinterpret_cast<int4 *>(values)[i] = reinterpret_cast<const int4 *>(source)[i];+ }+}++/**+ * @brief One thread's PAIRS rotation pairs of a row, RMS-normalized, weighted and rotated.+ *+ * Every thread of the row must call it (it contains a __syncthreads()), including threads of an+ * inactive row, which pass any valid addresses and discard the result. Returns in fp32 so the+ * caller picks the output type.+ *+ * @param x_first First-half run of this thread's pairs in the input row+ * @param x_second Second-half run (x_first + d/2)+ * @param cos_run, sin_run This thread's PAIRS frequency values+ * @param first_weight, second_weight RMS-norm weights of the two runs+ * @param eps Added to the mean square before the rsqrt+ * @param h Row length+ * @param lane_in_row This thread's index among the row's WARPS_PER_ROW * 32 threads+ * @param row_smem WARPS_PER_ROW floats of shared memory owned by this row+ * @param first_out, second_out The rotated runs+ */+template <int PAIRS, int WARPS_PER_ROW>+__device__ __forceinline__ void+rms_norm_split_rope_pairs(const __nv_bfloat16 *x_first, const __nv_bfloat16 *x_second, const __nv_bfloat16 *cos_run,+ const __nv_bfloat16 *sin_run, const __nv_bfloat16 (&first_weight)[PAIRS],+ const __nv_bfloat16 (&second_weight)[PAIRS], float eps, int h, int lane_in_row,+ float *row_smem, float (&first_out)[PAIRS], float (&second_out)[PAIRS]) {+ __nv_bfloat16 first_raw[PAIRS], second_raw[PAIRS], cos_raw[PAIRS], sin_raw[PAIRS];+ load_run<PAIRS>(x_first, first_raw);+ load_run<PAIRS>(x_second, second_raw);+ load_run<PAIRS>(cos_run, cos_raw);+ load_run<PAIRS>(sin_run, sin_raw);++ float first[PAIRS], second[PAIRS], sum_sq = 0.0f;+#pragma unroll+ for (int i = 0; i < PAIRS; i++) {+ first[i] = __bfloat162float(first_raw[i]);+ second[i] = __bfloat162float(second_raw[i]);+ sum_sq += first[i] * first[i] + second[i] * second[i];+ }+#pragma unroll+ for (int offset = 16; offset > 0; offset >>= 1) {+ sum_sq += __shfl_xor_sync(0xffffffff, sum_sq, offset);+ }++ // One barrier: every thread re-sums the per-warp partials itself. Electing one thread to sum them+ // and publish the result back needs a second barrier.+ if ((lane_in_row & 31) == 0) {+ row_smem[lane_in_row / 32] = sum_sq;+ }+ __syncthreads();+ float total_sum = 0.0f;+#pragma unroll+ for (int i = 0; i < WARPS_PER_ROW; i++) {+ total_sum += row_smem[i];+ }+ const float inv_rms = rsqrtf(total_sum / h + eps);++#pragma unroll+ for (int i = 0; i < PAIRS; i++) {+ const float a = first[i] * inv_rms * __bfloat162float(first_weight[i]);+ const float c = second[i] * inv_rms * __bfloat162float(second_weight[i]);+ const float cs = __bfloat162float(cos_raw[i]);+ const float sn = __bfloat162float(sin_raw[i]);+ first_out[i] = a * cs - c * sn;+ second_out[i] = c * cs + a * sn;+ }+}++} // namespace ltx_kernelsdiff --git a/packages/ltx-kernels/csrc/include/cuda/utils.cuh b/packages/ltx-kernels/csrc/include/cuda/utils.cuhindex 4a65502..2d1c83b 100644--- a/packages/ltx-kernels/csrc/include/cuda/utils.cuh+++ b/packages/ltx-kernels/csrc/include/cuda/utils.cuh@@ -357,4 +357,65 @@ __device__ __forceinline__ void barrier_wait_and_reset_roundrobin(int **barrier_}}+/**+ * @brief Barrier for a kernel in which every block writes to every rank.+ *+ * barrier_wait_and_reset and its round-robin variant have each block signal the one rank it wrote+ * to, which assumes a block's writes go to a single rank. When every block writes to every rank,+ * no single block's signal covers the others' writes. Here each block instead counts itself done+ * in `blocks_done` (a counter on this rank), and the last block to finish signals every rank once,+ * then waits for one signal from every rank -- the same per-source counters and counts as+ * barrier_wait_and_reset with one block per target. Earlier blocks return without waiting, so no+ * block spins on an SM that unfinished blocks still need.+ *+ * Call from every block after its last write. Needs blockDim.x >= world_size.+ *+ * @param barrier_signal_ptrs Array of pointers to barrier signal buffers for each rank+ * @param blocks_done This rank's count of finished blocks; zero on entry, zero again on return+ * @param rank This GPU's rank+ * @param world_size Total number of GPUs/ranks+ * @param thread_id The thread ID within the block (threadIdx.x)+ * @param timeout_cycles Number of cycles to wait before timeout+ */+__device__ __forceinline__ void barrier_last_block_wait_and_reset(int **barrier_signal_ptrs, int *blocks_done, int rank,+ int world_size, int thread_id,+ uint64_t timeout_cycles) {+ __shared__ bool is_last_block;++ // Release: every block fences its own writes before counting itself done.+ __threadfence_system();+ __syncthreads();+ if (thread_id == 0) {+ is_last_block = atomicAdd(blocks_done, 1) == static_cast<int>(gridDim.x) - 1;+ }+ __syncthreads();+ if (!is_last_block) {+ return;+ }+ if (thread_id == 0) {+ *blocks_done = 0;+ }+ __threadfence_system();++ // Each thread signals one rank and waits for that rank's signal.+ if (thread_id < world_size) {+ atomicAdd_system(barrier_signal_ptrs[thread_id] + rank, 1);+ auto start_time = clock64();+ while (true) {+ // Acquire: seeing the signal guarantees the peer's data is visible.+ int recv_count = ld_acquire_sys_global(barrier_signal_ptrs[rank] + thread_id);+ if (recv_count == 1) {+ break;+ }+ if (clock64() - start_time >= timeout_cycles) {+ printf("All2All barrier timeout (last block): rank=%d, waiting_for_source=%d, expected=1, got=%d\n", rank,+ thread_id, recv_count);+ trap();+ }+ }+ // Reset barrier for next use+ atomicSub_system(barrier_signal_ptrs[rank] + thread_id, 1);+ }+}+} // namespace ltx_kernelsdiff --git a/packages/ltx-kernels/csrc/ops/rms_norm_split_rope_cuda.cu b/packages/ltx-kernels/csrc/ops/rms_norm_split_rope_cuda.cuindex 928065b..e247070 100644--- a/packages/ltx-kernels/csrc/ops/rms_norm_split_rope_cuda.cu+++ b/packages/ltx-kernels/csrc/ops/rms_norm_split_rope_cuda.cu@@ -5,6 +5,8 @@#include <cuda_bf16.h>#include <cuda_fp8.h>+#include "cuda/rms_norm_split_rope.cuh"+// CUDA kernel template for RMS norm + split RoPE// out_t can be at::Float8_e4m3fn or at::BFloat16@@ -21,14 +23,6 @@ using fp8 = __nv_fp8_e4m3;// kPairsPerThread * threads_per_row * 2 must equal h.constexpr int kPairsPerThread = 16;-template<int N>-__device__ __forceinline__ void _load_run(const bf16* source, bf16 values[N]) {- #pragma unroll- for (int i = 0; i < N / 8; i++) {- reinterpret_cast<int4*>(values)[i] = reinterpret_cast<const int4*>(source)[i];- }-}-// Frequency tables are laid out logically as [b, n, s, d/2] (what// apply_split_rotary_emb produces -- a swapaxes view whose physical layout is// [b, s, n, d/2]). The strides (sb, sn, ss; inner d/2 stride is 1) are forwarded from@@ -70,40 +64,14 @@ __global__ void _rms_norm_split_rope_kernel(+ static_cast<long>(head) * sin_sn+ static_cast<long>(safe_token % s) * sin_ss + freq0;- bf16 first_raw[PAIRS], second_raw[PAIRS];- bf16 cos_raw[PAIRS], sin_raw[PAIRS];bf16 first_weight[PAIRS], second_weight[PAIRS];- _load_run<PAIRS>(x + row + first_channel, first_raw);- _load_run<PAIRS>(x + row + second_channel, second_raw);- _load_run<PAIRS>(cos_freqs + freq_offset, cos_raw);- _load_run<PAIRS>(sin_freqs + sin_offset, sin_raw);- _load_run<PAIRS>(weights + first_channel, first_weight);- _load_run<PAIRS>(weights + second_channel, second_weight);-- float first[PAIRS], second[PAIRS], sum_sq = 0.0f;- #pragma unroll- for (int i = 0; i < PAIRS; i++) {- first[i] = float(first_raw[i]);- second[i] = float(second_raw[i]);- sum_sq += first[i] * first[i] + second[i] * second[i];- }- #pragma unroll- for (int offset = 16; offset > 0; offset >>= 1) {- sum_sq += __shfl_xor_sync(0xffffffff, sum_sq, offset);- }-- // One barrier: every thread re-sums the per-warp partials itself. Electing one- // thread to sum them and publish the result back needs a second barrier.- if ((lane_in_row & 31) == 0) {- smem[row_in_block * kWarpsPerRow + lane_in_row / 32] = sum_sq;- }- __syncthreads();- float total_sum = 0.0f;- #pragma unroll- for (int i = 0; i < kWarpsPerRow; i++) {- total_sum += smem[row_in_block * kWarpsPerRow + i];- }- const float inv_rms = rsqrtf(total_sum / h + eps);+ ltx_kernels::load_run<PAIRS>(weights + first_channel, first_weight);+ ltx_kernels::load_run<PAIRS>(weights + second_channel, second_weight);+ float first_rot[PAIRS], second_rot[PAIRS];+ ltx_kernels::rms_norm_split_rope_pairs<PAIRS, kWarpsPerRow>(+ x + row + first_channel, x + row + second_channel, cos_freqs + freq_offset, sin_freqs + sin_offset,+ first_weight, second_weight, eps, h, lane_in_row, smem + row_in_block * kWarpsPerRow, first_rot,+ second_rot);// at::Float8_e4m3fn / at::BFloat16 do not convert from the native CUDA types, so// the staging arrays and the output pointer both use the native one.@@ -111,16 +79,12 @@ __global__ void _rms_norm_split_rope_kernel(store_t first_out[PAIRS], second_out[PAIRS];#pragma unrollfor (int i = 0; i < PAIRS; i++) {- const float a = first[i] * inv_rms * float(first_weight[i]);- const float c = second[i] * inv_rms * float(second_weight[i]);- const float cs = float(cos_raw[i]);- const float sn = float(sin_raw[i]);if constexpr (std::is_same_v<out_t, at::Float8_e4m3fn>) {- first_out[i] = fp8(a * cs - c * sn);- second_out[i] = fp8(c * cs + a * sn);+ first_out[i] = fp8(first_rot[i]);+ second_out[i] = fp8(second_rot[i]);} else {- first_out[i] = __float2bfloat16(a * cs - c * sn);- second_out[i] = __float2bfloat16(c * cs + a * sn);+ first_out[i] = __float2bfloat16(first_rot[i]);+ second_out[i] = __float2bfloat16(second_rot[i]);}}if (active) {diff --git a/packages/ltx-kernels/pyproject.toml b/packages/ltx-kernels/pyproject.tomlindex 0f15d73..208137c 100644--- a/packages/ltx-kernels/pyproject.toml+++ b/packages/ltx-kernels/pyproject.toml@@ -8,7 +8,7 @@ build-backend = "setuptools.build_meta"[project]name = "ltx-kernels"-version = "1.4.2"+version = "1.5.0"dependencies = ["torch","triton>=3.0",diff --git a/packages/ltx-kernels/setup.py b/packages/ltx-kernels/setup.pyindex 8d548d9..3bdcf0c 100644--- a/packages/ltx-kernels/setup.py+++ b/packages/ltx-kernels/setup.py@@ -261,6 +261,7 @@ if __name__ == "__main__":"csrc/all2all/all2all.cpp","csrc/all2all/cuda/all2all_heads.cu","csrc/all2all/cuda/allgather.cu",+ "csrc/all2all/cuda/fused_send.cu",],library_dirs=cuda_link_dirs,extra_compile_args={"cxx": all2all_args, "nvcc": ["-O3"]},@@ -281,7 +282,7 @@ if __name__ == "__main__":"csrc/ops/rms_norm_split_rope.cpp","csrc/ops/rms_norm_split_rope_cuda.cu",],- include_dirs=[str(ROOT / "csrc/ops/include"), *_nvidia_include_dirs()],+ include_dirs=[str(ROOT / "csrc/ops/include"), str(ROOT / "csrc/include"), *_nvidia_include_dirs()],library_dirs=cuda_link_dirs,extra_compile_args={"cxx": ["-O3", "-std=c++17"],diff --git a/packages/ltx-kernels/src/ltx_kernels/all_to_all.py b/packages/ltx-kernels/src/ltx_kernels/all_to_all.pyindex 8fa5677..75aaf11 100644--- a/packages/ltx-kernels/src/ltx_kernels/all_to_all.py+++ b/packages/ltx-kernels/src/ltx_kernels/all_to_all.py@@ -44,6 +44,70 @@ def _gather_heads_fake(x: torch.Tensor, buffer: torch.Tensor, comm_id: int) -> Nreturn None+@custom_op("ltx_kernels::rope_send_recv_heads", mutates_args=("buffer",), device_types="cuda")+def _rope_send_recv_heads_op(+ x: torch.Tensor,+ weights: torch.Tensor,+ cos_freqs: torch.Tensor,+ sin_freqs: torch.Tensor,+ eps: float,+ buffer: torch.Tensor, # noqa: ARG001+ comm_id: int,+ barrier: bool = True,+) -> None:+ All2All._runtime_registry[comm_id].rope_send_recv_heads(x, sin_freqs, cos_freqs, weights, eps, barrier)+++@_rope_send_recv_heads_op.register_fake+def _rope_send_recv_heads_fake(+ x: torch.Tensor, # noqa: ARG001+ weights: torch.Tensor, # noqa: ARG001+ cos_freqs: torch.Tensor, # noqa: ARG001+ sin_freqs: torch.Tensor, # noqa: ARG001+ eps: float, # noqa: ARG001+ buffer: torch.Tensor, # noqa: ARG001+ comm_id: int, # noqa: ARG001+ barrier: bool = True, # noqa: ARG001+) -> None:+ return None+++@custom_op("ltx_kernels::linear_send_recv_heads", mutates_args=("buffer",), device_types="cuda")+def _linear_send_recv_heads_op(+ x: torch.Tensor,+ weight: torch.Tensor,+ bias: torch.Tensor,+ buffer: torch.Tensor, # noqa: ARG001+ comm_id: int,+ barrier: bool = True,+) -> None:+ All2All._runtime_registry[comm_id].linear_send_recv_heads(x, weight, bias, barrier)+++@_linear_send_recv_heads_op.register_fake+def _linear_send_recv_heads_fake(+ x: torch.Tensor, # noqa: ARG001+ weight: torch.Tensor, # noqa: ARG001+ bias: torch.Tensor, # noqa: ARG001+ buffer: torch.Tensor, # noqa: ARG001+ comm_id: int, # noqa: ARG001+ barrier: bool = True, # noqa: ARG001+) -> None:+ return None+++@custom_op("ltx_kernels::exchange_barrier", mutates_args=("buffers",), device_types="cuda")+def _exchange_barrier_op(buffers: list[torch.Tensor], comm_id: int) -> None: # noqa: ARG001+ # ``buffers`` are the exchanges this barrier completes: declaring them mutated orders every read of+ # them after it.+ All2All._runtime_registry[comm_id].exchange_barrier()+++@_exchange_barrier_op.register_fake+def _exchange_barrier_fake(buffers: list[torch.Tensor], comm_id: int) -> None: # noqa: ARG001+ return None++class All2All:"""IPC-based All2All communication for distributed head-parallel attention.This class manages GPU memory buffers and IPC handles to enable efficientdiff --git a/packages/ltx-kernels/src/ltx_kernels/blockwise/_common.py b/packages/ltx-kernels/src/ltx_kernels/blockwise/_common.pyindex 478bb68..68d03c5 100644--- a/packages/ltx-kernels/src/ltx_kernels/blockwise/_common.py+++ b/packages/ltx-kernels/src/ltx_kernels/blockwise/_common.py@@ -24,11 +24,13 @@ def _get_tma_aligned_size(n: int, element_size: int) -> int:return ((n + num_elems_tma - 1) // num_elems_tma) * num_elems_tma-def _empty_tma_aligned_scales(num_rows: int, num_blocks: int, device: torch.device) -> tuple[torch.Tensor, int]:- """Allocate MN-major FP32 scales with TMA-padded row storage."""- tma_aligned_mn = _get_tma_aligned_size(num_rows, 4)- scales = torch.empty((num_blocks, tma_aligned_mn), dtype=torch.float, device=device).t()[:num_rows]- return scales, tma_aligned_mn+def _empty_tma_aligned_scales(num_rows: int, num_blocks: int, device: torch.device) -> torch.Tensor:+ """Allocate ``(num_blocks, tma_aligned_rows)`` FP32 scale storage.+ Kernels write this tensor itself; the op returns ``.t()[:num_rows]``, the MN-major+ ``(num_rows, num_blocks)`` view. Writing through that view instead makes inductor guard+ on whether ``num_rows`` is TMA-aligned, so a dynamic row count recompiles per alignment.+ """+ return torch.empty((num_blocks, _get_tma_aligned_size(num_rows, 4)), dtype=torch.float, device=device)@jitdiff --git a/packages/ltx-kernels/src/ltx_kernels/blockwise/ops/quantize.py b/packages/ltx-kernels/src/ltx_kernels/blockwise/ops/quantize.pyindex 86dfd0e..de73d76 100644--- a/packages/ltx-kernels/src/ltx_kernels/blockwise/ops/quantize.py+++ b/packages/ltx-kernels/src/ltx_kernels/blockwise/ops/quantize.py@@ -33,10 +33,10 @@ def _block_quantize_kernel(X,Out_scales,X_out,+ tma_aligned_mn: int,H: tl.constexpr,NUM_BLOCKS: tl.constexpr,BLOCK_SIZE: tl.constexpr,- TMA_ALIGNED_MN: tl.constexpr,SCALE_MAX: tl.constexpr,):token_idx = tl.program_id(0).to(tl.int64)@@ -48,7 +48,7 @@ def _block_quantize_kernel(X_out + token_idx * H + BLOCK_SIZE * tl.arange(0, NUM_BLOCKS)[:, None] + tl.arange(0, BLOCK_SIZE)[None, :])tl.store(x_out_ptr, o_quant)- out_scales_ptr = Out_scales + token_idx + TMA_ALIGNED_MN * tl.arange(0, NUM_BLOCKS)[:, None]+ out_scales_ptr = Out_scales + token_idx + tma_aligned_mn * tl.arange(0, NUM_BLOCKS)[:, None]tl.store(out_scales_ptr, o_scales)@@ -65,7 +65,7 @@ def blockwise_quantize(x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:num_rows = b * snum_blocks = h // _BLOCK_SIZEout = torch.empty((num_rows, h), device=x.device, dtype=torch.float8_e4m3fn)- scales, tma_aligned_mn = _empty_tma_aligned_scales(num_rows, num_blocks, x.device)+ scales = _empty_tma_aligned_scales(num_rows, num_blocks, x.device)wrap_triton(_block_quantize_kernel)[(num_rows,)](x,scales,@@ -73,10 +73,10 @@ def blockwise_quantize(x: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor]:H=h,BLOCK_SIZE=_BLOCK_SIZE,NUM_BLOCKS=num_blocks,- TMA_ALIGNED_MN=tma_aligned_mn,+ tma_aligned_mn=scales.shape[1],SCALE_MAX=_FP8_SCALE_MAX,)- return out.view(b, s, h), scales+ return out.view(b, s, h), scales.t()[:num_rows]@jitdiff --git a/packages/ltx-kernels/src/ltx_kernels/compile/passes/__init__.py b/packages/ltx-kernels/src/ltx_kernels/compile/passes/__init__.pyindex e224521..df86570 100644--- a/packages/ltx-kernels/src/ltx_kernels/compile/passes/__init__.py+++ b/packages/ltx-kernels/src/ltx_kernels/compile/passes/__init__.py@@ -8,11 +8,13 @@ from ltx_kernels.compile.passes.fuse_trailing_quantize import (AdaZeroTrailingQuantizePass,GatedAttentionTrailingQuantizePass,)+from ltx_kernels.compile.passes.fused_send_recv import CoalescedExchangeBarrierPass, FusedSendRecvPassfrom ltx_kernels.compile.passes.gated_attention import GatedAttentionPassfrom ltx_kernels.compile.passes.grouped_upcast import GroupedUpcastPassfrom ltx_kernels.compile.passes.linear_gelu import LinearGELUPassfrom ltx_kernels.compile.passes.pre_attention import PreAttentionPassfrom ltx_kernels.compile.passes.rms_fma import RMSFmaAdaLNPass, RMSFmaPass+from ltx_kernels.compile.passes.sdpa_select import SDPASelectPassdef inductor_passes() -> list[PassBase]:@@ -22,8 +24,10 @@ def inductor_passes() -> list[PassBase]:adds cost more there than they save, alone and together, by cutting inductor's ownfusion across the block. The quantize passes are inert unless``ltx_core.quantization.blockwise`` installs the linears that emit- ``ltx_kernels::blockwise_quantize``. :class:`GroupedUpcastPass` is inert outside the- fp8-cast policy.+ ``ltx_kernels::blockwise_quantize``. :class:`FusedSendRecvPass` is inert outside+ sequence parallelism and binds the op :class:`PreAttentionPass` leaves behind;+ :class:`CoalescedExchangeBarrierPass` then gives the exchanges it fuses one barrier per attention.+ :class:`GroupedUpcastPass` is inert outside the fp8-cast policy.One instance per stream shape (LTX-2.3: video 4096, audio 2048), since a pass pinsthe ``hidden_dim`` and ``heads`` literals its pattern carries."""@@ -32,11 +36,14 @@ def inductor_passes() -> list[PassBase]:PreAttentionPass(hidden_dim=2048, heads=32, eps=1e-6),GatedAttentionPass(),LinearGELUPass(),+ FusedSendRecvPass(),+ CoalescedExchangeBarrierPass(),BlockwiseQuantizeCSE(),AdaZeroTrailingQuantizePass(hidden_dim=4096, eps=1e-6),AdaZeroTrailingQuantizePass(hidden_dim=2048, eps=1e-6),GatedAttentionTrailingQuantizePass(),GroupedUpcastPass(),+ SDPASelectPass(),]@@ -64,6 +71,8 @@ __all__ = ["AdaZeroPass","AdaZeroTrailingQuantizePass","BlockwiseQuantizeCSE",+ "CoalescedExchangeBarrierPass",+ "FusedSendRecvPass","GatedAttentionPass","GatedAttentionTrailingQuantizePass","GroupedUpcastPass",@@ -71,6 +80,7 @@ __all__ = ["PreAttentionPass","RMSFmaAdaLNPass","RMSFmaPass",+ "SDPASelectPass","eager_passes","inductor_passes",]diff --git a/packages/ltx-kernels/src/ltx_kernels/compile/passes/fused_send_recv.py b/packages/ltx-kernels/src/ltx_kernels/compile/passes/fused_send_recv.pynew file mode 100644index 0000000..9b58722--- /dev/null+++ b/packages/ltx-kernels/src/ltx_kernels/compile/passes/fused_send_recv.py@@ -0,0 +1,186 @@+"""Pass that folds the all2all head exchange into the kernel producing the exchanged tensor."""++import logging+import operator++import torch+from torch.fx import GraphModule, Node+from torch.fx.experimental.symbolic_shapes import statically_known_true+from torch.fx.passes.infra.pass_base import PassBase, PassResult++logger = logging.getLogger(__name__)+++def _is_target(node: object, *targets: object) -> bool:+ return isinstance(node, Node) and node.op == "call_function" and node.target in targets+++def _sole_user(node: Node) -> Node | None:+ return next(iter(node.users)) if len(node.users) == 1 else None+++class FusedSendRecvPass(PassBase):+ """Rewrite ``send_recv_heads(producer(...).view(b, t, heads, d))`` to a fused producer-exchange op.+ Producers taken, when the exchange is their only consumer:+ - ``ltx_kernels::rms_norm_split_rope`` (left by :class:`PreAttentionPass`) ->+ ``ltx_kernels::rope_send_recv_heads``, which normalizes, rotates and writes each head straight+ into its owning rank's buffer.+ - ``linear`` -> ``ltx_kernels::linear_send_recv_heads``, which runs the projection per rank-owned+ column block with the output in that rank's buffer.+ Both write the same bytes ``send_recv_heads`` would, into the same buffer, so everything reading+ the buffer afterwards is unchanged. Must run after :class:`PreAttentionPass`.+ A plain pass rather than a :class:`SubgraphRewritePass`: ``send_recv_heads`` returns nothing,+ so a pattern ending in it has no returning node for the matcher to anchor on, and capturing it+ through Dynamo would run the exchange.+ """++ def __init__(self) -> None:+ self.match_count = 0++ def call(self, graph_module: GraphModule) -> PassResult:+ from ltx_kernels import all_to_all as _all_to_all_ops # noqa: F401, PLC0415+ from ltx_kernels.ops import rms_norm_split_rope as _rope_ops # noqa: F401, PLC0415++ send_recv = (torch.ops.ltx_kernels.send_recv_heads, torch.ops.ltx_kernels.send_recv_heads.default)+ rope = torch.ops.ltx_kernels.rms_norm_split_rope.default+ graph = graph_module.graph+ modified = False+ for node in list(graph.nodes):+ if not _is_target(node, *send_recv) or node.kwargs:+ continue+ view, buffer, comm_id = node.args+ if not (isinstance(view, Node) and view.op == "call_method" and view.target == "view"):+ continue+ producer = view.args[0]+ if _sole_user(view) is not node or not isinstance(producer, Node) or _sole_user(producer) is not view:+ continue+ if _is_target(producer, rope) and not producer.kwargs:+ x, weights, cos, sin, eps = producer.args+ target, args = torch.ops.ltx_kernels.rope_send_recv_heads.default, (x, weights, cos, sin, eps)+ elif _is_target(producer, torch._C._nn.linear) and len(producer.args) == 3 and not producer.kwargs:+ x, weight, bias = producer.args+ if bias is None:+ continue+ target, args = torch.ops.ltx_kernels.linear_send_recv_heads.default, (x, weight, bias)+ else:+ continue+ value = args[0].meta.get("example_value") if isinstance(args[0], Node) else None+ buffer_value = buffer.meta.get("example_value") if isinstance(buffer, Node) else None+ if not isinstance(value, torch.Tensor) or value.dtype != torch.bfloat16 or value.ndim != 3:+ continue+ # The fused op takes batch and tokens from the producer input, so the exchanged view must+ # split only the hidden dim; a view regrouping batch and tokens has another buffer layout.+ heads_view = view.meta.get("example_value")+ if not (+ isinstance(heads_view, torch.Tensor)+ and heads_view.ndim == 4+ and all(statically_known_true(heads_view.shape[i] == value.shape[i]) for i in (0, 1))+ ):+ continue+ # The fused producers write bf16; a blockwise-fp8 exchange keeps its own copy kernel.+ if not isinstance(buffer_value, torch.Tensor) or buffer_value.dtype != torch.bfloat16:+ continue+ with graph.inserting_before(node):+ fused = graph.call_function(target, (*args, buffer, comm_id))+ node.replace_all_uses_with(fused)+ graph.erase_node(node)+ graph.erase_node(view)+ graph.erase_node(producer)+ self.match_count += 1+ modified = True+ if modified:+ graph.lint()+ graph_module.recompile()+ logger.info("FusedSendRecvPass fused %d exchange(s) into their producers", self.match_count)+ return PassResult(graph_module, modified)+++# Methods that always return a view of their input.+_VIEW_METHODS = frozenset({"view", "unflatten", "transpose", "permute", "unsqueeze", "squeeze"})+# Methods that return a view only when the input's strides allow one, and a copy otherwise.+_MAYBE_VIEW_METHODS = frozenset({"flatten", "reshape"})+++def _is_basic_index(index: object) -> bool:+ """Whether ``tensor[index]`` is basic indexing (a view), not advanced indexing (a copy)."""+ if isinstance(index, tuple):+ return all(_is_basic_index(i) for i in index)+ if isinstance(index, slice):+ return all(_is_basic_index(bound) for bound in (index.start, index.stop, index.step))+ if isinstance(index, Node):+ # A traced size; a tensor index is advanced.+ index = index.meta.get("example_value")+ return (+ index is None or index is Ellipsis or (isinstance(index, (int, torch.SymInt)) and not isinstance(index, bool))+ )+++def _shares_storage(node: Node, source: Node) -> bool:+ """Whether the traced values prove ``node`` is a view of ``source``."""+ from torch.multiprocessing.reductions import StorageWeakRef # noqa: PLC0415++ value, source_value = node.meta.get("example_value"), source.meta.get("example_value")+ if not (isinstance(value, torch.Tensor) and isinstance(source_value, torch.Tensor)):+ return False+ return StorageWeakRef(value.untyped_storage()) == StorageWeakRef(source_value.untyped_storage())+++def _is_view(node: Node) -> bool:+ """Whether ``node`` only re-views its first argument, so reaching it does not read the data."""+ if node.op == "call_method" and node.target in _VIEW_METHODS:+ return True+ if node.op == "call_method" and node.target in _MAYBE_VIEW_METHODS:+ return isinstance(node.args[0], Node) and _shares_storage(node, node.args[0])+ return _is_target(node, operator.getitem) and len(node.args) == 2 and _is_basic_index(node.args[1])+++class CoalescedExchangeBarrierPass(PassBase):+ """Complete consecutive fused exchanges with one barrier instead of one each.+ Attention reads its q, k and v buffers only after all three exchanges, so their per-exchange+ barriers -- a cross-rank round trip each -- can be one ``ltx_kernels::exchange_barrier``, placed+ before the first node that reads any of their buffers. Views of a buffer are not reads; ops that+ may copy (advanced indexing, a ``flatten`` the strides do not allow as a view) are, as is a later+ exchange taking a pending buffer, which then starts the next group. The barrier names the buffers+ as mutated, so everything that reads them is ordered after it. A group that is not read inside the+ graph keeps its barriers. Must run after :class:`FusedSendRecvPass`.+ """++ def __init__(self) -> None:+ self.match_count = 0++ def call(self, graph_module: GraphModule) -> PassResult:+ from ltx_kernels import all_to_all as _all_to_all_ops # noqa: F401, PLC0415++ fused = (+ torch.ops.ltx_kernels.rope_send_recv_heads.default,+ torch.ops.ltx_kernels.linear_send_recv_heads.default,+ )+ graph = graph_module.graph+ removed = 0+ pending: list[Node] = []+ aliases: set[Node] = set()+ for node in list(graph.nodes):+ if pending and _is_view(node) and node.args[0] in aliases:+ aliases.add(node)+ continue+ if pending and any(input_node in aliases for input_node in node.all_input_nodes):+ if len(pending) > 1:+ for exchange in pending:+ exchange.update_kwarg("barrier", False)+ with graph.inserting_before(node):+ buffers = [exchange.args[-2] for exchange in pending]+ graph.call_function(+ torch.ops.ltx_kernels.exchange_barrier.default, (buffers, pending[-1].args[-1])+ )+ removed += len(pending) - 1+ pending, aliases = [], set()+ # Checked after the reads: an exchange whose inputs are a pending buffer completes that group first.+ if _is_target(node, *fused) and node.kwargs.get("barrier", True):+ pending.append(node)+ aliases.add(node.args[-2])+ if removed:+ graph.lint()+ graph_module.recompile()+ logger.info("CoalescedExchangeBarrierPass removed %d exchange barrier(s)", removed)+ self.match_count += removed+ return PassResult(graph_module, bool(removed))diff --git a/packages/ltx-kernels/src/ltx_kernels/compile/passes/sdpa_select.py b/packages/ltx-kernels/src/ltx_kernels/compile/passes/sdpa_select.pynew file mode 100644index 0000000..0189891--- /dev/null+++ b/packages/ltx-kernels/src/ltx_kernels/compile/passes/sdpa_select.py@@ -0,0 +1,39 @@+"""Pass that routes unmasked SDPA calls to ``ltx_kernels::sdpa_select``."""++from typing import ClassVar++import torch+from torch.fx.passes.utils.matcher_utils import InternalMatch++from ltx_kernels.compile.subgraph_rewrite_pass import SubgraphRewritePass, is_cuda_bf16+++class SDPASelectPass(SubgraphRewritePass):+ """Rewrite the unmasked ``scaled_dot_product_attention(q, k, v)`` to ``ltx_kernels::sdpa_select``.+ The pattern carries the keyword arguments ``Attention``'s SDPA backends pass, and+ ``ignore_literals=False`` pins them: only a call with no mask, no dropout and+ ``is_causal=False`` is the op's to serve. The surrounding ``sdpa_kernel`` priority context+ is left in place and has no effect on the op, which pins its own backend per shape.+ """++ ignore_literals: ClassVar[bool] = False++ @staticmethod+ def pattern_fn(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor:+ return torch.nn.functional.scaled_dot_product_attention(q, k, v, attn_mask=None, dropout_p=0.0, is_causal=False)++ @staticmethod+ def replacement_fn(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor:+ return torch.ops.ltx_kernels.sdpa_select.default(q, k, v)++ def _is_match_eligible(self, match: InternalMatch) -> bool:+ inputs = self._matched_inputs(match)+ if not super()._is_match_eligible(match) or not inputs:+ return False+ return is_cuda_bf16(*inputs) and all(t.ndim == 4 for t in inputs)++ def example_inputs(self) -> tuple:+ return tuple(torch.empty(1, 4, 64, 64, dtype=torch.bfloat16, device="cuda") for _ in range(3))++ def _register_ops(self) -> None:+ from ltx_kernels.ops import sdpa_select as _sdpa_select_ops # noqa: F401, PLC0415diff --git a/packages/ltx-kernels/src/ltx_kernels/devices.py b/packages/ltx-kernels/src/ltx_kernels/devices.pynew file mode 100644index 0000000..86a5ec9--- /dev/null+++ b/packages/ltx-kernels/src/ltx_kernels/devices.py@@ -0,0 +1,19 @@+"""Host-to-device transfer helpers shared by the kernels.+``ltx_kernels`` does not depend on ``ltx_core``, so it keeps its own counterpart of+``ltx_core.devices.upload_from_host``.+"""++from __future__ import annotations++import torch+++def upload_from_host(host: torch.Tensor, device: torch.device) -> torch.Tensor:+ """Copy a host tensor to ``device`` without waiting for the device.+ A pageable host-to-device copy synchronizes the stream; a pinned one is ordered on it instead,+ so ``host`` may be modified as soon as this returns. A tensor that is already on a device, or a+ target that is not CUDA, is moved with ``.to``.+ """+ if host.device.type != "cpu" or device.type != "cuda":+ return host.to(device)+ return host.pin_memory().to(device, non_blocking=True)diff --git a/packages/ltx-kernels/src/ltx_kernels/ops/ada_zero.py b/packages/ltx-kernels/src/ltx_kernels/ops/ada_zero.pyindex d4f0670..aa31ed6 100644--- a/packages/ltx-kernels/src/ltx_kernels/ops/ada_zero.py+++ b/packages/ltx-kernels/src/ltx_kernels/ops/ada_zero.py@@ -37,11 +37,11 @@ def _ada_zero_kernel( # noqa: PLR0913norm_scale_token_stride: int,norm_shift_token_stride: int,seqlen: int,+ tma_aligned_mn: int,eps: float,H: tl.constexpr,NUM_BLOCKS: tl.constexpr,BLOCK_SIZE: tl.constexpr,- TMA_ALIGNED_MN: tl.constexpr,QUANTIZE: tl.constexpr,SCALE_MAX: tl.constexpr,):@@ -80,7 +80,7 @@ def _ada_zero_kernel( # noqa: PLR0913X_out + token_idx * H + BLOCK_SIZE * tl.arange(0, NUM_BLOCKS)[:, None] + tl.arange(0, BLOCK_SIZE)[None, :])tl.store(x_out_ptr, o_quant)- out_scales_ptr = Out_scales + token_idx + TMA_ALIGNED_MN * tl.arange(0, NUM_BLOCKS)[:, None]+ out_scales_ptr = Out_scales + token_idx + tma_aligned_mn * tl.arange(0, NUM_BLOCKS)[:, None]tl.store(out_scales_ptr, o_scales)else:x_out_ptr = X_out + token_idx * H + tl.arange(0, H)@@ -132,7 +132,7 @@ def ada_zero(H=h,NUM_BLOCKS=0,BLOCK_SIZE=_BLOCK_SIZE,- TMA_ALIGNED_MN=0,+ tma_aligned_mn=0,QUANTIZE=False,SCALE_MAX=_FP8_SCALE_MAX,)@@ -160,7 +160,7 @@ def ada_zero_quantized(scale = scale.expand(b, s, h)shift = shift.expand(b, s, h)out = torch.empty((num_rows, h), device=x.device, dtype=torch.float8_e4m3fn)- scales, tma_aligned_mn = _empty_tma_aligned_scales(num_rows, num_blocks, x.device)+ scales = _empty_tma_aligned_scales(num_rows, num_blocks, x.device)wrap_triton(_ada_zero_kernel)[(num_rows,)](x,scale,@@ -176,8 +176,8 @@ def ada_zero_quantized(H=h,NUM_BLOCKS=num_blocks,BLOCK_SIZE=_BLOCK_SIZE,- TMA_ALIGNED_MN=tma_aligned_mn,+ tma_aligned_mn=scales.shape[1],QUANTIZE=True,SCALE_MAX=_FP8_SCALE_MAX,)- return out.view(b, s, h), scales+ return out.view(b, s, h), scales.t()[:num_rows]diff --git a/packages/ltx-kernels/src/ltx_kernels/ops/gated_attention.py b/packages/ltx-kernels/src/ltx_kernels/ops/gated_attention.pyindex 32f95a6..a78b33e 100644--- a/packages/ltx-kernels/src/ltx_kernels/ops/gated_attention.py+++ b/packages/ltx-kernels/src/ltx_kernels/ops/gated_attention.py@@ -40,12 +40,12 @@ def _gated_attention_kernel( # noqa: PLR0913gate_batch_stride: int,gate_token_stride: int,seqlen: int,+ tma_aligned_mn: int,H: tl.constexpr,NUM_HEADS: tl.constexpr,DIM_HEAD: tl.constexpr,NUM_BLOCKS: tl.constexpr,BLOCK_SIZE: tl.constexpr,- TMA_ALIGNED_MN: tl.constexpr,QUANTIZE: tl.constexpr,SCALE_MAX: tl.constexpr,):@@ -74,7 +74,7 @@ def _gated_attention_kernel( # noqa: PLR0913Out + token_idx * H + BLOCK_SIZE * tl.arange(0, NUM_BLOCKS)[:, None] + tl.arange(0, BLOCK_SIZE)[None, :])tl.store(out_ptr, o_quant)- scales_ptr = Out_scales + token_idx + TMA_ALIGNED_MN * tl.arange(0, NUM_BLOCKS)[:, None]+ scales_ptr = Out_scales + token_idx + tma_aligned_mn * tl.arange(0, NUM_BLOCKS)[:, None]tl.store(scales_ptr, o_scales)else:out_ptr = Out + token_idx * H + offsets@@ -120,7 +120,7 @@ def gated_attention(x: torch.Tensor, gate_logits: torch.Tensor) -> torch.Tensor:DIM_HEAD=dim_head,NUM_BLOCKS=num_blocks,BLOCK_SIZE=_BLOCK_SIZE,- TMA_ALIGNED_MN=0,+ tma_aligned_mn=0,QUANTIZE=False,SCALE_MAX=_FP8_SCALE_MAX,)@@ -149,7 +149,7 @@ def gated_attention_quantized(x: torch.Tensor, gate_logits: torch.Tensor) -> Tupgate_logits = gate_logits.expand(b, t, num_heads)out = torch.empty((num_rows, h), device=x.device, dtype=torch.float8_e4m3fn)- scales, tma_aligned_mn = _empty_tma_aligned_scales(num_rows, num_blocks, x.device)+ scales = _empty_tma_aligned_scales(num_rows, num_blocks, x.device)wrap_triton(_gated_attention_kernel)[(num_rows,)](x,@@ -164,9 +164,9 @@ def gated_attention_quantized(x: torch.Tensor, gate_logits: torch.Tensor) -> TupDIM_HEAD=dim_head,NUM_BLOCKS=num_blocks,BLOCK_SIZE=_BLOCK_SIZE,- TMA_ALIGNED_MN=tma_aligned_mn,+ tma_aligned_mn=scales.shape[1],QUANTIZE=True,SCALE_MAX=_FP8_SCALE_MAX,)- return out.view(b, t, h), scales+ return out.view(b, t, h), scales.t()[:num_rows]diff --git a/packages/ltx-kernels/src/ltx_kernels/ops/sdpa_select.py b/packages/ltx-kernels/src/ltx_kernels/ops/sdpa_select.pynew file mode 100644index 0000000..f2eb7c2--- /dev/null+++ b/packages/ltx-kernels/src/ltx_kernels/ops/sdpa_select.py@@ -0,0 +1,88 @@+"""Unmasked SDPA that runs whichever of cuDNN and flash attention is faster for each shape.+cuDNN's fused attention is the fastest kernel for the transformer's long self-attention, but+for a short query against a long key sequence (audio queries over video keys) it falls back to+an sm80 kernel with no split over the keys, which leaves most SMs idle and runs several times+slower than flash attention's split-KV kernel.+The first call per input geometry times both backends and caches the winner; later calls, and+calls made while a CUDA graph is being captured, reuse it. A geometry first seen during capture+runs cuDNN, since timing inside a capture is impossible. A geometry neither backend takes runs+unpinned, so SDPA falls back to its other kernels as the rewritten call would. The choice is+cached per deterministic-algorithms mode, which decides whether cuDNN is usable at all.+Candidates are timed as replays of a CUDA graph, as the transformer runs them. Timing eager+launches measures host launch cost for short attentions, where cuDNN's is higher, and picks+flash for shapes where cuDNN's kernel is several times faster.+"""++import torch+import torch.nn.functional as F+from torch.nn.attention import SDPBackend, sdpa_kernel++_CANDIDATES: tuple[SDPBackend, ...] = (SDPBackend.CUDNN_ATTENTION, SDPBackend.FLASH_ATTENTION)+_TIMING_ITERS = 10++_Key = tuple[tuple[int, ...], tuple[int, ...], tuple[int, ...], torch.dtype, int, bool]+# ``None`` is a geometry neither candidate takes: it runs unpinned.+_choices: dict[_Key, SDPBackend | None] = {}+++def _key(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> _Key:+ deterministic = torch.are_deterministic_algorithms_enabled()+ return (tuple(q.shape), tuple(k.shape), tuple(v.shape), q.dtype, q.device.index, deterministic)+++def _run(backend: SDPBackend | None, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor:+ if backend is None:+ return F.scaled_dot_product_attention(q, k, v)+ with sdpa_kernel([backend]):+ return F.scaled_dot_product_attention(q, k, v)+++def _time(backend: SDPBackend, q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> float:+ try:+ # Eager first, so plan building and workspace allocation stay out of the timed graph.+ _run(backend, q, k, v)+ except RuntimeError:+ return float("inf")+ graph = torch.cuda.CUDAGraph()+ with torch.cuda.graph(graph):+ for _ in range(_TIMING_ITERS):+ _run(backend, q, k, v)+ graph.replay()+ start, end = torch.cuda.Event(enable_timing=True), torch.cuda.Event(enable_timing=True)+ start.record()+ graph.replay()+ end.record()+ end.synchronize()+ return start.elapsed_time(end)+++def _select(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> SDPBackend | None:+ key = _key(q, k, v)+ if key in _choices:+ return _choices[key]+ if torch.cuda.is_current_stream_capturing():+ return SDPBackend.CUDNN_ATTENTION+ times = {backend: _time(backend, q, k, v) for backend in _CANDIDATES}+ fastest = min(times, key=times.__getitem__)+ choice = fastest if times[fastest] != float("inf") else None+ _choices[key] = choice+ return choice++[email protected]_op("ltx_kernels::sdpa_select", mutates_args=(), device_types="cuda")+def sdpa_select(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor:+ """Unmasked, non-causal ``scaled_dot_product_attention`` over ``(B, H, S, D)`` inputs."""+ out = _run(_select(q, k, v), q, k, v)+ if not out.transpose(1, 2).is_contiguous():+ out = out.transpose(1, 2).contiguous().transpose(1, 2)+ return out+++def _sdpa_select_fake(q: torch.Tensor, k: torch.Tensor, v: torch.Tensor) -> torch.Tensor: # noqa: ARG001+ # A (B, S, H, D)-contiguous buffer viewed as (B, H, S, D): what both backends return for the+ # head-interleaved inputs attention builds, so the head merge that follows is a free view.+ b, h, s, _ = q.shape+ return q.new_empty((b, s, h, v.shape[-1])).transpose(1, 2)+++sdpa_select.register_fake(_sdpa_select_fake)diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/aot.py b/packages/ltx-kernels/src/ltx_kernels/vae/aot.pyindex 2784a34..6314c6c 100644--- a/packages/ltx-kernels/src/ltx_kernels/vae/aot.py+++ b/packages/ltx-kernels/src/ltx_kernels/vae/aot.py@@ -140,10 +140,11 @@ def _load_function(file_name: str, function_prefix: str) -> CompiledFunction:@functools.lru_cache(maxsize=8)def production_kernel_keys(gpu_arch: str) -> tuple[KernelKey, ...]:"""Return the finite production specialization set for ``gpu_arch``."""- from ltx_kernels.vae import block_fna_dsl, na_attn_dsl # noqa: PLC0415+ from ltx_kernels.vae import block_fna_dsl, na_attn_dsl, rope_dsl # noqa: PLC0415na = na_attn_dsl._compile_kernelblock = block_fna_dsl._compile_kernel+ rope = rope_dsl._compile_kernelreturn (KernelKey.create(na,@@ -205,6 +206,11 @@ def production_kernel_keys(gpu_arch: str) -> tuple[KernelKey, ...]:slab_box_max=block_fna_dsl.SLAB_BOX_MAX,keyframes=True,),+ KernelKey.create(+ rope,+ gpu_arch,+ rope_dims=(16, 24, 24),+ ),)diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/block_fna_dsl.py b/packages/ltx-kernels/src/ltx_kernels/vae/block_fna_dsl.pyindex 7efbc6a..a11ecae 100644--- a/packages/ltx-kernels/src/ltx_kernels/vae/block_fna_dsl.py+++ b/packages/ltx-kernels/src/ltx_kernels/vae/block_fna_dsl.py@@ -25,6 +25,11 @@ Structure, in the order the kernel runs it:same query tile as the video stream's, with :func:`ltx_kernels.vae.fna_smem.select_tensor`picking the buffer at runtime. Tracing either of them twice cost ~70 s of backend compileon a kernel whose plain build is already 25 s.+* **One body per row pass.** Heads and C-wide chunks are walked by device loops, and each+ warpgroup reaches its own column slice through an SMEM operand view shifted by whole+ K-stages (:func:`ltx_kernels.vae.fna_smem._alias`) rather than through a constexpr arm+ per warpgroup. Only what has to stay static -- slice coordinates within a stage, the+ KV pipeline's buffer parity -- is still unrolled.* **The attention half is not here.** The window mask, the pipelined KV loop and thesoftmax are :func:`ltx_kernels.vae.fna_attn_core.attention_tile`, shared verbatimwith the standalone NA kernel.@@ -50,6 +55,7 @@ import torchfrom cutlass.cute.nvgpu import cpasync, tcgen05from cutlass.utils import SmemAllocator+from ltx_kernels.devices import upload_from_hostfrom ltx_kernels.vae.aot import (KERNEL_CACHE,fake_dynamic_matrix,@@ -75,10 +81,12 @@ from ltx_kernels.vae.fna_attn_core import (row_sum,)from ltx_kernels.vae.fna_gemm import (+ GemmJob,UmmaSharedStorage,_mma_op,_reg_shape,_tmem_load,+ _tmem_shift,acquire_tmem,gemm_tma,release_tmem,@@ -129,6 +137,7 @@ from ltx_kernels.vae.keyframe_slots import (nearest_video_frames,tile_plane_union,)+from ltx_kernels.vae.workspace import workspace_volume# Channel counts the kernel is compiled and validated for. The shipped DiffVAE only# ever asks for 256 (stage 5); 128 is kept because it is the cheap shape to validate@@ -181,6 +190,8 @@ def _check_out(out: torch.Tensor, rows: int, channels: int, x_: torch.Tensor, na#: Bytes in one N=128 B-operand stage, which is what ``gemm_tma``'s TMA transaction#: count has to match._B_STAGE_BYTES = TILE_N * TILE_K * 2+# Weight tiles the MLP keeps in flight; four N=128 x K=64 bf16 stages fill a C=256 s_q exactly.+MLP_B_STAGES = 4# --------------------------------------------------------------------------@@ -306,15 +317,23 @@ class _Workspace:# ---- SMEM operands, all aliased onto one pool; see the map in _fna_kernel ----s_a: cute.Tensor # the activation / A operands_b: cute.Tensor # the streamed weight tile / B operand- s_q: cute.Tensor # queries, then the MLP's hidden chunks+ s_q: cute.Tensor # queries; dead after attention+ s_b4: cute.Tensor # the MLP's weight queue, aliasing s_qs_k: cute.Tensors_v: cute.Tensor- s_p: cute.Tensor+ s_p_own: cute.Tensor # this warpgroup's slice of P (s_p), at stage 0+ s_h_own: cute.Tensor # this warpgroup's slice of the MLP hidden chunk (s_h), at stage 0+ # Raw storage and operand layouts, for views shifted by a runtime number of K-stages.+ pool: cute.Tensor # under s_a, s_p, s_k, s_b, s_v and s_h+ q_pool: cute.Tensor # under s_q and s_b4+ a_layout: Any+ q_layout: Anystorage: Any # barrier storage for the pipelines gemm_tma stands up# ---- MMA operand fragments over those buffers ----fr_a: Anyfr_q: Any+ fr_h: Anyfr_k: Anyfr_p: Anyfr_v: Any@@ -345,6 +364,8 @@ class _Workspace:rowix: cute.Tensor # in-bounds source row per lane; feeds every gatherrowix_lo: cute.Tensor # the stage-4 cell row per lane, for the fused-context gatherrowix_w: cute.Tensor # destination row per lane; scratch when out of range+ kv_rowtab: cute.Tensor # (2, M) KV-tile source rows, for the cooperative K/V copies+ kv_copy_bars: cute.Tensor # K-full x2, P/V-full: the KV loop's operand-ready counterscopy_slot: tuple # this thread's slot in the cooperative copiesred_c: cute.Tensor # row reduction across all warpgroupsred_head: cute.Tensor # row reduction across the warpgroups sharing a head@@ -498,6 +519,8 @@ def _fna_kernel(q_layout: cute.ComposedLayout,k_layout: cute.ComposedLayout,v_layout: cute.ComposedLayout,+ b4_layout: cute.ComposedLayout,+ h_layout: cute.ComposedLayout,):tidx, _, _ = cute.arch.thread_idx()cta_index, _, _ = cute.arch.block_idx()@@ -541,7 +564,19 @@ def _fna_kernel(s_k = _alias(pool, k_layout, 2 * M * TILE_K)s_b = _alias(pool, b_layout, 4 * M * TILE_K)s_v = _alias(pool, v_layout, 4 * M * TILE_K)- s_q = smem.allocate_tensor(IO, q_layout.outer, byte_alignment=128, swizzle=q_layout.inner)+ # s_q holds the queries until attention ends; the MLP then reuses it as its weight queue.+ # Sized for the larger view: at C=128 the queries take half of what the queue needs.+ q_pool_elems = max(cute.cosize(q_layout), cute.cosize(b4_layout))+ q_pool = smem.allocate_tensor(IO, cute.make_layout((q_pool_elems,)), byte_alignment=128)+ s_q = _alias(q_pool, q_layout)+ s_b4 = _alias(q_pool, b4_layout)+ s_h = _alias(pool, h_layout, 4 * M * TILE_K)+ # Warpgroup ``g``'s ``ACC_COLS`` chunk of a 128-wide row is K-stage ``g`` of it, so these+ # views -- shifted ``wg`` stages -- give every warpgroup its own chunk at static stage 0,+ # and a row pass over them is one code copy rather than one per warpgroup.+ wg_stage = ids.wg * (M * TILE_K)+ s_p_own = _alias(pool, p_layout, wg_stage)+ s_h_own = _alias(pool, h_layout, 4 * M * TILE_K + wg_stage)# Completion barriers: one per Q@K accumulator plus one the P@V chain signals.# ``attention_tile`` re-initialises them per query tile so the wait phases stay# compile-time constants.@@ -550,6 +585,8 @@ def _fna_kernel(rowix = smem.allocate_tensor(cutlass.Int32, cute.make_layout((M,)), byte_alignment=16)rowix_lo = smem.allocate_tensor(cutlass.Int32, cute.make_layout((M,)), byte_alignment=16)rowix_w = smem.allocate_tensor(cutlass.Int32, cute.make_layout((M,)), byte_alignment=16)+ kv_rowtab = smem.allocate_tensor(cutlass.Int32, cute.make_layout((2, M)), byte_alignment=16)+ kv_copy_bars = smem.allocate_tensor(cutlass.Int64, cute.make_layout((3,)), byte_alignment=16)m_sm = smem.allocate_tensor(ACC, cute.make_layout((M, NH)), byte_alignment=16)l_sm = smem.allocate_tensor(ACC, cute.make_layout((M, NUM_WARPGROUPS, NH)), byte_alignment=16)red_c = smem.allocate_tensor(ACC, cute.make_layout((M, NUM_WARPGROUPS)), byte_alignment=16)@@ -616,12 +653,19 @@ def _fna_kernel(s_a=s_a,s_b=s_b,s_q=s_q,+ s_b4=s_b4,s_k=s_k,s_v=s_v,- s_p=s_p,+ s_p_own=s_p_own,+ s_h_own=s_h_own,+ pool=pool,+ q_pool=q_pool,+ a_layout=a_layout,+ q_layout=q_layout,storage=storage,fr_a=mma128.make_fragment_A(s_a),fr_q=mma128.make_fragment_A(s_q),+ fr_h=mma128.make_fragment_A(s_h),fr_k=mma128.make_fragment_B(s_k),fr_p=mma64.make_fragment_A(s_p),fr_v=mma64.make_fragment_B(s_v),@@ -644,6 +688,8 @@ def _fna_kernel(rowix=rowix,rowix_lo=rowix_lo,rowix_w=rowix_w,+ kv_rowtab=kv_rowtab,+ kv_copy_bars=kv_copy_bars,copy_slot=cooperative_copy_slot(tidx),red_c=red_c,red_head=red_head,@@ -692,11 +738,20 @@ def _fna_kernel(# keyframe build traces only the guarded call and a plain build only the bare# one. Written as a single call under a runtime-or-constant guard instead, the# plain build would carry a branch it can never take.- while tile_id < n_tiles + ws.n_kf * tile_plane:+ # A keyframe tile carries ``TT`` planes (group ``g`` = planes ``g*TT .. g*TT+TT-1``), so the+ # projections and MLP run once per group of distinct queries. The group is live in this slab+ # if any of its planes is.+ n_groups = (ws.n_kf + shape.tile[0] - 1) // shape.tile[0]+ while tile_id < n_tiles + n_groups * tile_plane:live = cutlass.Int32(1)if tile_id >= n_tiles:- plane = unflatten_3d(tile_id - n_tiles, tile_plane, tile_row)[0]- live = cutlass.Int32(1) if ws.kf_slab[plane] == slab.t_index else cutlass.Int32(0)+ group = unflatten_3d(tile_id - n_tiles, tile_plane, tile_row)[0]+ live = cutlass.Int32(0)+ for j in cutlass.range_constexpr(shape.tile[0]):+ plane_j = group * shape.tile[0] + j+ if plane_j < ws.n_kf: # noqa: SIM102+ if ws.kf_slab[plane_j] == slab.t_index:+ live = cutlass.Int32(1)if live != 0:_run_tile(ws, shape, slab, tile_id, n_tiles, tile_plane, tile_row, KF, SLOTS, P_MAX, NKV_KF)tile_id += n_ctas@@ -907,17 +962,22 @@ def _fill_slab_caches(ws, shape: cutlass.Constexpr, slab, KF: cutlass.Constexprif cutlass.const_expr(shape.ctx_groups > 1):_fill_a_gather(ws.s_q, src_ctx, ws.rowix_lo, ws.copy_slot, channels, channels)cute.arch.sync_threads()- for n in cutlass.range_constexpr(shape.n_chunks):- _project(- ws,- shape,- ws.w_ctx,- ws.acc_main[n],- subpixel * shape.n_chunks + n,- n_k_tiles=shape.ctx_groups * shape.n_k_tiles,- frag2=ws.fr_q,- a_split=shape.n_k_tiles if shape.ctx_groups > 1 else 0,- )+ _run_jobs(+ ws,+ tuple(+ _job(+ ws,+ shape,+ ws.w_ctx,+ ws.acc_main[n],+ subpixel * shape.n_chunks + n,+ n_k_tiles=shape.ctx_groups * shape.n_k_tiles,+ frag2=ws.fr_q,+ a_split=shape.n_k_tiles if shape.ctx_groups > 1 else 0,+ )+ for n in range(shape.n_chunks)+ ),+ )cute.arch.sync_threads()_fill_a_gather(ws.s_a, src_x, ws.rowix, ws.copy_slot, channels)cute.arch.sync_threads()@@ -938,11 +998,26 @@ def _fill_slab_caches(ws, shape: cutlass.Constexpr, slab, KF: cutlass.Constexpr_apply_adaln(ws, shape, rms_inv, param["msa_weight"], param["msa_shift"])cute.arch.sync_threads()- for head in cutlass.range_constexpr(shape.n_heads):- if cutlass.const_expr(KF):- _project_head_kv(ws, shape, head, token, rope_offset, cache, cache_stride, kf_time, kf_row)- else:- _project_head_kv(ws, shape, head, token, rope_offset)+ # Heads go in pairs, one pipeline and one wait per pair, into the two accumulators the+ # residual above has finished with: ``acc_main[0]`` and ``acc_scratch``, ``TILE_N``+ # columns apart. Both loops are device loops, so the per-head pass is traced once;+ # the head reaches only the weight N-tile, a TMEM address, bias offsets and cache+ # rows. ``n_heads`` is even (``attention_tile`` needs it too).+ for pair in cutlass.range(shape.n_heads // 2):+ _run_jobs(+ ws,+ (+ _job(ws, shape, ws.w_kv, ws.acc_main[0], 2 * pair),+ _job(ws, shape, ws.w_kv, ws.acc_scratch, 2 * pair + 1),+ ),+ )+ for j in cutlass.range(2):+ head = 2 * pair + j+ acc = _tmem_shift(ws.acc_main[0], j * TILE_N)+ if cutlass.const_expr(KF):+ _project_head_kv(ws, shape, head, acc, token, rope_offset, cache, cache_stride, kf_time, kf_row)+ else:+ _project_head_kv(ws, shape, head, acc, token, rope_offset)step += ws.n_ctas@@ -1084,7 +1159,8 @@ def _subpixel_step(ws, shape: cutlass.Constexpr, slab, step, cells, n_cells):def _project_head_kv(ws,shape: cutlass.Constexpr,- head: cutlass.Constexpr,+ head,+ acc,cache_row,box_offset,dst=None,@@ -1092,7 +1168,7 @@ def _project_head_kv(kf_time=None,kf_row=None,):- """Project one head's K and V out of the normed row and store both to the cache.+ """Finish one head's K and V projection out of ``acc`` and store both to the cache.The packed K|V weight puts K in the accumulator's first ``HD`` columns and V inthe second, which lines the warpgroup split up with the two paths: the first``WARPGROUPS_PER_HEAD`` warpgroups take K (bias, RMSNorm, RoPE), the rest take V (bias only).@@ -1103,15 +1179,15 @@ def _project_head_kv(``kf_row`` decides that swap at *runtime*, for the merged fill that walks both streams'rows in one loop. Given, both temporal ropes are traced and one is selected; left ``None``only the lattice one is, which is what a plain build gets.+ ``acc`` holds the head's projection, which the caller already ran. ``head`` may be a+ runtime value: it reaches only bias offsets and cache rows."""lane, wg = ws.ids.lane, ws.ids.wgareg, params, param = ws.areg, ws.params, shape.paramn_heads, rope_dims = shape.n_heads, shape.rope_dims- acc = ws.acc_scratchdst = ws.kv_cache if dst is None else dstrow_stride = ws.cache_row_stride if row_stride is None else row_stride- _project(ws, shape, ws.w_kv, acc, head)k_sq = ACC(0.0)for g in cutlass.range_constexpr(NUM_WARPGROUPS):if wg == g:@@ -1162,9 +1238,11 @@ class _TileGeometry(NamedTuple):why these are runtime selections rather than one branch's two exits."""- kf_tile: Any # this tile is a keyframe plane's; None in a binary without that stream+ kf_tile: Any # this tile is a keyframe plane group's; None in a binary without that streamvideo_tile: Any # its complement, which the deep temporal panel keys off- plane: Any # the anchor plane a keyframe tile carries, clamped into range+ plane: Any # this lane's anchor plane (``kf_group * TT + slot_t``), clamped into range+ kf_group: Any # the plane group a keyframe tile carries (planes ``kf_group*TT ..``)+ kf_live: Any # per lane: its plane exists and this slab owns it (0/1 Int32)query_pos: tuple # this lane's (t, h, w) in the volumeread_row: Any # the row ``post`` is gathered fromwrite_row: Any # where the result lands; a scratch row for an out-of-range lane@@ -1202,7 +1280,7 @@ def _tile_geometry(_, vol_h, vol_w = ws.volumeindex = tile_id- kf_tile, video_tile, plane = None, None, None+ kf_tile, video_tile, plane, kf_group, kf_live = None, None, None, None, Noneif cutlass.const_expr(KF):kf_tile = tile_id >= n_core_tilesvideo_tile = tile_id < n_core_tiles@@ -1245,17 +1323,23 @@ def _tile_geometry(# clamped because a video tile evaluates this arithmetic too -- it is pure arithmetic# plus small table lookups, and selecting the results is cheaper than branching# around them.- plane = at_most(at_least(local[0], zero), at_least(ws.n_kf - 1, zero))+ # ``TT`` planes per keyframe tile: slot ``slot_t`` of group ``local[0]`` is plane ``group * TT + slot_t``;+ # :func:`_run_tile` walks the planes one pass set at a time, each lane gated into its own plane's passes.+ kf_group = local[0]slot_t, slot_h, slot_w = ws.query_slot+ plane_raw = kf_group * shape.tile[0] + slot_t+ plane = at_most(at_least(plane_raw, zero), at_least(ws.n_kf - 1, zero))+ kf_live = cutlass.Int32(0)+ if plane_raw < ws.n_kf: # noqa: SIM102+ if ws.kf_slab[plane] == slab.t_index:+ kf_live = cutlass.Int32(1)q_h = tile_origin[1] + slot_hq_w = tile_origin[2] + slot_wr_h = q_h if q_h < vol_h else vol_h - 1r_w = q_w if q_w < vol_w else vol_w - 1kf_read = flatten_3d(plane, r_h, r_w, vol_h, vol_w)kf_write = kf_read- # One plane per tile, so the borrowed tile's other temporal slots hold duplicates- # of ``slot_t == 0``; they compute and park on a scratch row.- if slot_t != 0 or q_h >= vol_h or q_w >= vol_w:+ if kf_live == 0 or q_h >= vol_h or q_w >= vol_w:kf_write = ws.n_kf * vol_h * vol_w + lane# The *panel* still shifts inward -- it only has to contain every key the tile can# reach, and the inward-shifted box is the larger of the two -- but the per-query@@ -1299,6 +1383,8 @@ def _tile_geometry(kf_tile=kf_tile,video_tile=video_tile,plane=plane,+ kf_group=kf_group,+ kf_live=kf_live,query_pos=query_pos,read_row=read_row,write_row=write_row,@@ -1385,7 +1471,7 @@ def _run_tile(if deep:attention_tile((ws.fr_q, ws.fr_k, ws.fr_p, ws.fr_v),- (ws.s_k, ws.s_v, ws.s_p),+ (ws.s_k, ws.s_v, ws.s_p_own),ws.acc_qk,ws.acc_o,(ws.qkbar, ws.pvbar),@@ -1406,6 +1492,8 @@ def _run_tile(shape.n_kv_tiles,None,geom.window_hi,+ ws.kv_rowtab,+ ws.kv_copy_bars,)# Single-plane passes, into the same (m_sm, l_sm, acc_o). The fixed softmax offset means@@ -1416,80 +1504,97 @@ def _run_tile(# keyframe query's own plane, and its <=2 nearest video frames. Which cache a pass reads# is an address, and the pass count is runtime.if cutlass.const_expr(KF):- pass_i = zero- n_passes = ws.kf_counts[geom.t_idx]+ # Each plane's key sets (its own plane, then its <=2 nearest video frames) are one pass set, walked+ # plane by plane with the other lanes gated out; a gated lane's exponentials are exact zeros, so its+ # softmax is untouched.+ n_outer = cutlass.Int32(1)if geom.kf_tile:- n_passes = 1 + ws.kf_video_counts[geom.plane]- while pass_i < n_passes:- # Video tail: this tile's plane union, with each lane gated out of the planes its- # own ``t`` did not pick -- the plane is a per-tile gather but the nearest-S set- # is per ``t``, and a tile spans ``TT`` of them.- source = ws.kf_planes[geom.t_idx * P_MAX + at_most(pass_i, P_MAX - 1)]- sel = zero- for s in cutlass.range_constexpr(SLOTS):- if ws.kf_slots[geom.query_pos[0] * SLOTS + s] == source:- sel = cutlass.Int32(1)- video_kv = zero+ n_outer = cutlass.Int32(shape.tile[0])+ outer = zero+ while outer < n_outer:+ plane_j = zero+ n_passes = ws.kf_counts[geom.t_idx]if geom.kf_tile:- # A keyframe query's passes are tile-uniform -- one plane per tile, and its- # video frames are a property of the plane -- so no lane gating.- sel = cutlass.Int32(1)- if pass_i == 0:- source = geom.plane- else:- source = ws.kf_video[geom.plane * SLOTS + at_most(pass_i - 1, SLOTS - 1)]- video_kv = cutlass.Int32(1)- cache = select_tensor(video_kv != 0, ws.kv_cache, ws.kv_cache_kf)- row_stride = ws.cache_row_stride if video_kv != 0 else ws.kf_cache_row_stride- # The keyframe cache is (n_kf, box_h, box_w): a plane index is absolute, so its- # box origin is 0, while a video frame index is relative to the box.- attention_tile(- (ws.fr_q, ws.fr_k, ws.fr_p, ws.fr_v),- (ws.s_k, ws.s_v, ws.s_p),- ws.acc_qk,- ws.acc_o,- (ws.qkbar, ws.pvbar),- ws.m_sm,- ws.l_sm,- ws.areg,- ws.kvfrag,- (cache, cache),- (zero, shape.n_heads * row_stride),- row_stride,- (- source,- geom.panel_h0,- geom.panel_w0,- slab.box_origin[0] if video_kv != 0 else zero,- slab.box_origin[1],- slab.box_origin[2],- slab.box_shape[1],- slab.box_shape[2],- ),- (zero, geom.window[1], geom.window[2]),- (ids.tidx, lane, ids.wg, ids.warp),- shape.n_heads,- 1,- 1,- shape.panel[1],- shape.panel[2],- 1,- shape.kernel[1],- shape.kernel[2],- NKV_KF,- sel,- # One plane deep; the spatial pair is the query's own window, keeping both- # key sets aligned for the query sharing a softmax over them.- (cutlass.Int32(1), geom.window_hi[1], geom.window_hi[2]),- )- pass_i += 1+ plane_raw_j = geom.kf_group * shape.tile[0] + outer+ plane_j = at_most(plane_raw_j, at_least(ws.n_kf - 1, zero))+ n_passes = 1 + ws.kf_video_counts[plane_j]+ # A plane past ``n_kf`` or owned by another slab is not ours: no passes, its lanes park on scratch.+ if plane_raw_j >= ws.n_kf: # noqa: SIM114+ n_passes = zero+ elif ws.kf_slab[plane_j] != slab.t_index:+ n_passes = zero+ pass_i = zero+ while pass_i < n_passes:+ # Video tail: this tile's plane union, with each lane gated out of the planes its+ # own ``t`` did not pick -- the plane is a per-tile gather but the nearest-S set+ # is per ``t``, and a tile spans ``TT`` of them.+ source = ws.kf_planes[geom.t_idx * P_MAX + at_most(pass_i, P_MAX - 1)]+ sel = zero+ for s in cutlass.range_constexpr(SLOTS):+ if ws.kf_slots[geom.query_pos[0] * SLOTS + s] == source:+ sel = cutlass.Int32(1)+ video_kv = zero+ if geom.kf_tile:+ # Keyframe tile: this pass set belongs to plane ``plane_j`` (the lanes of slot ``outer``).+ sel = cutlass.Int32(1) if ws.query_slot[0] == outer else zero+ if pass_i == 0:+ source = plane_j+ else:+ source = ws.kf_video[plane_j * SLOTS + at_most(pass_i - 1, SLOTS - 1)]+ video_kv = cutlass.Int32(1)+ cache = select_tensor(video_kv != 0, ws.kv_cache, ws.kv_cache_kf)+ row_stride = ws.cache_row_stride if video_kv != 0 else ws.kf_cache_row_stride+ # The keyframe cache is (n_kf, box_h, box_w): a plane index is absolute, so its+ # box origin is 0, while a video frame index is relative to the box.+ attention_tile(+ (ws.fr_q, ws.fr_k, ws.fr_p, ws.fr_v),+ (ws.s_k, ws.s_v, ws.s_p_own),+ ws.acc_qk,+ ws.acc_o,+ (ws.qkbar, ws.pvbar),+ ws.m_sm,+ ws.l_sm,+ ws.areg,+ ws.kvfrag,+ (cache, cache),+ (zero, shape.n_heads * row_stride),+ row_stride,+ (+ source,+ geom.panel_h0,+ geom.panel_w0,+ slab.box_origin[0] if video_kv != 0 else zero,+ slab.box_origin[1],+ slab.box_origin[2],+ slab.box_shape[1],+ slab.box_shape[2],+ ),+ (zero, geom.window[1], geom.window[2]),+ (ids.tidx, lane, ids.wg, ids.warp),+ shape.n_heads,+ 1,+ 1,+ shape.panel[1],+ shape.panel[2],+ 1,+ shape.kernel[1],+ shape.kernel[2],+ NKV_KF,+ sel,+ # One plane deep; the spatial pair is the query's own window, keeping both+ # key sets aligned for the query sharing a softmax over them.+ (cutlass.Int32(1), geom.window_hi[1], geom.window_hi[2]),+ ws.kv_rowtab,+ ws.kv_copy_bars,+ )+ pass_i += 1+ outer += 1# ---------------- out projection, residual, AdaLN --------------------cute.arch.sync_threads()- attn_epilogue(ws.acc_o, ws.l_sm, ws.oreg, ws.s_a, lane, ids.wg, shape.n_heads)+ attn_epilogue(ws.acc_o, ws.l_sm, ws.oreg, ws.pool, ws.a_layout, lane, ids.wg, shape.n_heads)cute.arch.sync_threads()- for n in cutlass.range_constexpr(shape.n_chunks):- _project(ws, shape, ws.w_o, ws.acc_main[n], n)+ _run_jobs(ws, tuple(_job(ws, shape, ws.w_o, ws.acc_main[n], n) for n in range(shape.n_chunks)))# s_a is free once the out-projection has consumed it, so ``post`` comes back# through it coalesced rather than one cache line per lane.cute.arch.sync_threads()@@ -1537,10 +1642,20 @@ def _project_queries(ws, shape: cutlass.Constexpr, rope_pos, kf_time=None, kf_ti"""lane, wg = ws.ids.lane, ws.ids.wgareg, params, param = ws.areg, ws.params, shape.param- acc = ws.acc_scratch- for hp in cutlass.range_constexpr(shape.n_heads // HEADS_PER_TILE):- _project(ws, shape, ws.w_q, acc, hp)+ # Every head pair's GEMM goes through one pipeline and one wait, into the two+ # accumulators attention has not claimed yet: ``acc_main[0]`` and ``acc_scratch``,+ # ``TILE_N`` columns apart.+ n_pairs = shape.n_heads // HEADS_PER_TILE+ assert n_pairs <= 2, "the query projection has two free accumulators"+ _run_jobs(ws, tuple(_job(ws, shape, ws.w_q, _tmem_shift(ws.acc_main[0], hp * TILE_N), hp) for hp in range(n_pairs)))+ # A device loop over the pairs, and each warpgroup handles its own head of the pair: one+ # code copy of the passes below. The runtime head reaches only SMEM data indices, the+ # TMEM column and the query stage, which is a view shifted ``head`` stages.+ assert WARPGROUPS_PER_HEAD == 1, "each warpgroup owns one whole head of a pair"+ for hp in cutlass.range(n_pairs):+ head = hp * HEADS_PER_TILE + wg+ _tmem_load(_tmem_shift(ws.acc_main[0], hp * TILE_N), lane, areg, wg)# Pass one: bias, and the two row sums the normalisation needs. ``sum z^2``# gives the RMSNorm scale; ``sum (z * q_norm_w)^2`` gives ``||q||`` after it,# since the scale factors out. Both come off one pass and one barrier -- as two@@ -1548,54 +1663,40 @@ def _project_queries(ws, shape: cutlass.Constexpr, rope_pos, kf_time=None, kf_ti# reduction and spilled.q_sq = ACC(0.0)q_weighted_sq = ACC(0.0)- for g in cutlass.range_constexpr(NUM_WARPGROUPS):- if wg == g:- c0 = (g % WARPGROUPS_PER_HEAD) * ACC_COLS- head = hp * HEADS_PER_TILE + g // WARPGROUPS_PER_HEAD- _tmem_load(acc, lane, areg, g)- s = ACC(0.0)- t = ACC(0.0)- for d in cutlass.range(ACC_COLS, unroll_full=True):- z = areg[d] + params[param["q_bias"] + head * HD + c0 + d]- areg[d] = z- s += z * z- zw = z * params[param["q_norm"] + c0 + d]- t += zw * zw- q_sq = s- q_weighted_sq = t+ for d in cutlass.range(ACC_COLS, unroll_full=True):+ z = areg[d] + params[param["q_bias"] + head * HD + d]+ areg[d] = z+ q_sq += z * z+ zw = z * params[param["q_norm"] + d]+ q_weighted_sq += zw * zwq_sum, q_norm_sq = head_sum2(ws.red_head, lane, wg, q_sq, q_weighted_sq)q_rms = cute.rsqrt(q_sum / ACC(HD) + ACC(EPS))- for g in cutlass.range_constexpr(NUM_WARPGROUPS):- if wg == g:- c0 = (g % WARPGROUPS_PER_HEAD) * ACC_COLS- head = hp * HEADS_PER_TILE + g // WARPGROUPS_PER_HEAD- # This row and head's fixed softmax offset, ``||q||`` times the ``||k||``- # bound -- see :mod:`ltx_kernels.vae.softmax_bound`. ``||q||`` is the- # reduced sum scaled after the fact, since the RMSNorm factor is uniform- # over the head and so comes out of the square root. The ``q_norm`` run- # already carries the attention scale and the base change, folded in by- # :func:`_pack_channel_params`.- if cutlass.const_expr(g % WARPGROUPS_PER_HEAD == 0):- ws.m_sm[lane, head] = cute.sqrt(q_norm_sq) * q_rms * params[param["k_bound"]] * LOG2E- for d in cutlass.range(ACC_COLS, unroll_full=True):- areg[d] = areg[d] * q_rms * params[param["q_norm"] + c0 + d]- if cutlass.const_expr(kf_time is None):- apply_rope(areg, *shape.rope_dims, ws.rope, *rope_pos, c0, ACC_COLS)- elif kf_tile:- apply_rope_float_t(- areg,- *shape.rope_dims,- ws.rope,- kf_time,- rope_pos[1],- rope_pos[2],- ws.rope_inv[0],- c0,- ACC_COLS,- )- else:- apply_rope(areg, *shape.rope_dims, ws.rope, *rope_pos, c0, ACC_COLS)- _fill_a_chunk(ws.s_q, lane, areg, head * HD + c0, ACC_COLS)+ # This row and head's fixed softmax offset, ``||q||`` times the ``||k||``+ # bound -- see :mod:`ltx_kernels.vae.softmax_bound`. ``||q||`` is the+ # reduced sum scaled after the fact, since the RMSNorm factor is uniform+ # over the head and so comes out of the square root. The ``q_norm`` run+ # already carries the attention scale and the base change, folded in by+ # :func:`_pack_channel_params`.+ ws.m_sm[lane, head] = cute.sqrt(q_norm_sq) * q_rms * params[param["k_bound"]] * LOG2E+ for d in cutlass.range(ACC_COLS, unroll_full=True):+ areg[d] = areg[d] * q_rms * params[param["q_norm"] + d]+ if cutlass.const_expr(kf_time is None):+ apply_rope(areg, *shape.rope_dims, ws.rope, *rope_pos, 0, ACC_COLS)+ elif kf_tile:+ apply_rope_float_t(+ areg,+ *shape.rope_dims,+ ws.rope,+ kf_time,+ rope_pos[1],+ rope_pos[2],+ ws.rope_inv[0],+ 0,+ ACC_COLS,+ )+ else:+ apply_rope(areg, *shape.rope_dims, ws.rope, *rope_pos, 0, ACC_COLS)+ _fill_a_chunk(_alias(ws.q_pool, ws.q_layout, head * (M * TILE_K)), lane, areg, 0, ACC_COLS)@cute.jit@@ -1606,57 +1707,63 @@ def _swiglu_mlp(ws, shape: cutlass.Constexpr):``gemm_tma`` shape serves every C-wide linear. N-tiling reuses that path and letsSiLU read gate/up from TMEM; token-chunking would fight lane ownership and force asecond tiling scheme just for the MLP.- ``s_q`` is reused as the hidden-chunk staging buffer -- it is dead from the moment- the query projection's last GEMM consumed it.+ The weights stream through ``s_b4``, ``MLP_B_STAGES`` tiles deep over ``s_q`` (dead once+ attention has read the queries); the hidden chunk is staged in ``s_h``, the two-stage weight+ slot the MLP no longer uses. Gate and up share one pipeline, as do the two down halves."""lane, wg = ws.ids.lane, ws.ids.wgareg, breg = ws.areg, ws.breg- for nt in cutlass.range_constexpr(shape.hidden // TILE_N):+ # A device loop, so the slice body is in the instruction stream once rather than+ # ``hidden / TILE_N`` times; the down projection's accumulate flag is a runtime predicate.+ for nt in cutlass.range(shape.hidden // TILE_N):# gate and up land in separate accumulators, so SiLU reads both straight from# TMEM instead of parking M gate values in registers.- _project(ws, shape, ws.w_gate, ws.acc_main[0], nt)- _project(ws, shape, ws.w_up, ws.acc_scratch, nt)- for g in cutlass.range_constexpr(NUM_WARPGROUPS):- if wg == g:- _tmem_load(ws.acc_main[0], lane, areg, g)- _tmem_load(ws.acc_scratch, lane, breg, g)- for e in cutlass.range(ACC_COLS, unroll_full=True):- gate = areg[e]- # Two MUFU per element (ex2 and rcp.approx). The one-MUFU identity- # ``silu(x) = x/2 (1 + tanh(x/2))`` is tempting and does not hold up:- # ``tanh.approx.f32`` carries 2^-11 *absolute* error and the down- # projection sums ``hidden`` of them, which cost 20 dB of PSNR- # against torch.- breg[e] = gate * cute.arch.rcp_approx(ACC(1.0) + cute.exp(-gate)) * breg[e]- _fill_a_chunk(ws.s_q, lane, breg, g * ACC_COLS, ACC_COLS)+ _run_jobs(+ ws,+ (_job(ws, shape, ws.w_gate, ws.acc_main[0], nt), _job(ws, shape, ws.w_up, ws.acc_scratch, nt)),+ ws.s_b4,+ MLP_B_STAGES,+ )+ _tmem_load(ws.acc_main[0], lane, areg, wg)+ _tmem_load(ws.acc_scratch, lane, breg, wg)+ for e in cutlass.range(ACC_COLS, unroll_full=True):+ gate = areg[e]+ # Two MUFU per element (ex2 and rcp.approx). The one-MUFU identity+ # ``silu(x) = x/2 (1 + tanh(x/2))`` is tempting and does not hold up:+ # ``tanh.approx.f32`` carries 2^-11 *absolute* error and the down+ # projection sums ``hidden`` of them, which cost 20 dB of PSNR+ # against torch.+ breg[e] = gate * cute.arch.rcp_approx(ACC(1.0) + cute.exp(-gate)) * breg[e]+ _fill_a_chunk(ws.s_h_own, lane, breg, 0, ACC_COLS)cute.arch.sync_threads()# Down projection: A is this hidden chunk, B is w_down's matching K-slice,- # accumulated across chunks into one accumulator per N-tile. It reads ``s_q``,- # so it takes ``fr_q`` rather than the ``fr_a`` every other projection uses.+ # accumulated across chunks into one accumulator per N-tile. It reads ``s_h``,+ # so it takes ``fr_h`` rather than the ``fr_a`` every other projection uses.atom, tensor = ws.w_down- for n in cutlass.range_constexpr(shape.n_chunks):- gemm_tma(- "n128",- ws.storage,- atom,- tensor,- ws.s_b,- ws.fr_q,- ws.acc_down[n],- n,- nt * 2,- 2,- 0,- nt != 0,- _B_STAGE_BYTES,- )+ _run_jobs(+ ws,+ tuple(+ GemmJob(atom, tensor, ws.fr_h, ws.acc_down[n], n, nt * 2, 2, 0, nt != 0) for n in range(shape.n_chunks)+ ),+ ws.s_b4,+ MLP_B_STAGES,+ )# --------------------------------------------------------------------------# Per-row channel passes+# Each walks the C-wide row in ``TILE_N`` chunks with a device loop, and each warpgroup+# handles its own ``ACC_COLS`` slice of a chunk, so the body is one code copy. A chunk's+# accumulator is ``acc[0]`` shifted ``n * TILE_N`` columns, which requires the+# accumulators to be evenly spaced -- ``acc_main`` and ``acc_down`` both are.# --------------------------------------------------------------------------+def _own_chunk(ws, n):+ """``s_a`` shifted so stage 0 is this warpgroup's slice of chunk ``n``, K-stage ``n * NUM_WARPGROUPS + wg``."""+ return _alias(ws.pool, ws.a_layout, (n * NUM_WARPGROUPS + ws.ids.wg) * (M * TILE_K))++@cute.jitdef _add_residual(ws, shape: cutlass.Constexpr, acc, bias_off: cutlass.Constexpr, bias_row=0):"""``s_a += acc + bias`` in place, returning the new row's sum of squares.@@ -1667,20 +1774,17 @@ def _add_residual(ws, shape: cutlass.Constexpr, acc, bias_off: cutlass.Constexprcontext's per-subpixel bias is reached; it is a runtime value and folds away at 0."""lane, wg = ws.ids.lane, ws.ids.wg- s_a, areg, breg, params = ws.s_a, ws.areg, ws.breg, ws.params+ areg, breg, params = ws.areg, ws.breg, ws.paramstotal = ACC(0.0)- for g in cutlass.range_constexpr(NUM_WARPGROUPS):- if wg == g:- s = ACC(0.0)- for n in cutlass.range_constexpr(shape.n_chunks):- _tmem_load(acc[n], lane, breg, g)- _read_a_chunk(s_a, lane, areg, n * TILE_N + g * ACC_COLS, ACC_COLS)- for e in cutlass.range(ACC_COLS, unroll_full=True):- v = areg[e] + breg[e] + params[bias_off + bias_row + n * TILE_N + g * ACC_COLS + e]- s += v * v- areg[e] = v- _fill_a_chunk(s_a, lane, areg, n * TILE_N + g * ACC_COLS, ACC_COLS)- total = s+ for n in cutlass.range(shape.n_chunks):+ s_a = _own_chunk(ws, n)+ _tmem_load(_tmem_shift(acc[0], n * TILE_N), lane, breg, wg)+ _read_a_chunk(s_a, lane, areg, 0, ACC_COLS)+ for e in cutlass.range(ACC_COLS, unroll_full=True):+ v = areg[e] + breg[e] + params[bias_off + bias_row + n * TILE_N + wg * ACC_COLS + e]+ total += v * v+ areg[e] = v+ _fill_a_chunk(s_a, lane, areg, 0, ACC_COLS)return total@@ -1691,57 +1795,47 @@ def _apply_adaln(ws, shape: cutlass.Constexpr, rms_inv, weight_off: cutlass.Cons``norm_w * (1 + scale)`` into the weight run."""lane, wg = ws.ids.lane, ws.ids.wg- s_a, areg, params = ws.s_a, ws.areg, ws.params- for g in cutlass.range_constexpr(NUM_WARPGROUPS):- if wg == g:- for n in cutlass.range_constexpr(shape.n_chunks):- _read_a_chunk(s_a, lane, areg, n * TILE_N + g * ACC_COLS, ACC_COLS)- for e in cutlass.range(ACC_COLS, unroll_full=True):- c = n * TILE_N + g * ACC_COLS + e- areg[e] = areg[e] * rms_inv * params[weight_off + c] + params[shift_off + c]- _fill_a_chunk(s_a, lane, areg, n * TILE_N + g * ACC_COLS, ACC_COLS)+ areg, params = ws.areg, ws.params+ for n in cutlass.range(shape.n_chunks):+ s_a = _own_chunk(ws, n)+ _read_a_chunk(s_a, lane, areg, 0, ACC_COLS)+ for e in cutlass.range(ACC_COLS, unroll_full=True):+ c = n * TILE_N + wg * ACC_COLS + e+ areg[e] = areg[e] * rms_inv * params[weight_off + c] + params[shift_off + c]+ _fill_a_chunk(s_a, lane, areg, 0, ACC_COLS)@cute.jitdef _add_accumulator(ws, shape: cutlass.Constexpr, acc):"""``s_a += acc`` in place -- the MLP's closing residual, no bias and no norm."""lane, wg = ws.ids.lane, ws.ids.wg- s_a, areg, breg = ws.s_a, ws.areg, ws.breg- for g in cutlass.range_constexpr(NUM_WARPGROUPS):- if wg == g:- for n in cutlass.range_constexpr(shape.n_chunks):- _tmem_load(acc[n], lane, areg, g)- _read_a_chunk(s_a, lane, breg, n * TILE_N + g * ACC_COLS, ACC_COLS)- for e in cutlass.range(ACC_COLS, unroll_full=True):- areg[e] = breg[e] + areg[e]- _fill_a_chunk(s_a, lane, areg, n * TILE_N + g * ACC_COLS, ACC_COLS)+ areg, breg = ws.areg, ws.breg+ for n in cutlass.range(shape.n_chunks):+ s_a = _own_chunk(ws, n)+ _tmem_load(_tmem_shift(acc[0], n * TILE_N), lane, areg, wg)+ _read_a_chunk(s_a, lane, breg, 0, ACC_COLS)+ for e in cutlass.range(ACC_COLS, unroll_full=True):+ areg[e] = breg[e] + areg[e]+ _fill_a_chunk(s_a, lane, areg, 0, ACC_COLS)-def _project(ws, shape, weight, acc, n_idx, accumulate: bool = False, n_k_tiles=None, frag2=None, a_split=0):+def _job(ws, shape, weight, acc, n_idx, accumulate: bool = False, n_k_tiles=None, frag2=None, a_split=0):"""One ``M``-row GEMM of the activation in ``s_a`` against a streamed weight.- Thin wrapper over :func:`gemm_tma` that unpacks the ``(atom, tensor)`` pair and- fills in the operands every projection in this kernel shares. ``frag2`` continues the- reduction in a second operand past K-tile ``a_split``, which is how the fused- context's ``Ctx``-wide row -- too wide for one operand -- stays a single call.+ Unpacks the ``(atom, tensor)`` pair and fills in the operands every projection in this+ kernel shares. ``frag2`` continues the reduction in a second operand past K-tile+ ``a_split``, which is how the fused context's ``Ctx``-wide row -- too wide for one+ operand -- stays a single GEMM."""atom, tensor = weight- gemm_tma(- "n128",- ws.storage,- atom,- tensor,- ws.s_b,- ws.fr_a,- acc,- n_idx,- 0,- shape.n_k_tiles if n_k_tiles is None else n_k_tiles,- 0,- accumulate,- _B_STAGE_BYTES,- frag2,- a_split,- )+ n_kt = shape.n_k_tiles if n_k_tiles is None else n_k_tiles+ return GemmJob(atom, tensor, ws.fr_a, acc, n_idx, 0, n_kt, 0, accumulate, frag2, a_split)+++def _run_jobs(ws, jobs, s_b=None, n_stages=2):+ """Run independent GEMMs through one :func:`gemm_tma` call: one pipeline, one wait.+ ``s_b`` / ``n_stages`` select the weight queue; the default is the two-stage ``ws.s_b``.+ """+ gemm_tma("n128", ws.storage, ws.s_b if s_b is None else s_b, _B_STAGE_BYTES, jobs, n_stages)# --------------------------------------------------------------------------@@ -1945,6 +2039,7 @@ def _keyframe_workspace(plan: _SlabPlan,device: torch.device,out: torch.Tensor | None = None,+ keyframe_times_host: torch.Tensor | None = None,) -> _KeyframeWorkspace:"""Allocate the keyframe caches and precompute the two lookups, or hand back placeholders.The keyframe cache is the video cache with the box's temporal extent replaced by ``n_kf``:@@ -2005,12 +2100,15 @@ def _keyframe_workspace(if int(keyframe_times.numel()) != n_kf:raise ValueError(f"keyframe_times must have {n_kf} entries, got {int(keyframe_times.numel())}")- slots = nearest_keyframe_slots(keyframe_times, t)+ # The plane tables are built on the host: from the caller's host copy of the times when given,+ # else from one read of ``keyframe_times``.+ host_times = keyframe_times.detach().cpu() if keyframe_times_host is None else keyframe_times_host+ slots = nearest_keyframe_slots(host_times, t)# ``SLOTS * tile[0]``, not ``min(n_kf, ...)``: the stride into ``planes`` is a compile-time# constant of the tile and must not move with the anchor layout. Slack columns are -1.planes, _ = tile_plane_union(slots, tile_t=tile[0], max_planes=KEYFRAME_CONTEXT_SLOTS * tile[0])counts = (planes >= 0).sum(dim=-1).to(torch.int32)- video = nearest_video_frames(keyframe_times, t)+ video = nearest_video_frames(host_times, t)# Which slab owns each plane's query tiles: the one whose core holds the plane's own# position. Its box is that core grown by the NA radius (>=1 on every stage) and the <=2# nearest video frames are within a step, so the box always covers them.@@ -2018,7 +2116,7 @@ def _keyframe_workspace(# times the tile depth. Dividing by the tile count instead sends most planes to a slab# index that never comes up, and their query tiles silently never run.core_tokens = plan.core[0] * tile[0]- positions = keyframe_times.detach().cpu().round().clamp_(0, t - 1).to(torch.int64)+ positions = host_times.detach().to(torch.float32).round().clamp_(0, t - 1).to(torch.int64)owner = (positions // core_tokens).clamp_(0, plan.grid[0] - 1).to(torch.int32)kf_rows = n_kf * h * w@@ -2028,11 +2126,11 @@ def _keyframe_workspace(return _as_rows(x[0].detach().to(dtype=torch.bfloat16)).reshape(-1, ch)def i32(x: torch.Tensor) -> torch.Tensor:- return x.reshape(-1).contiguous().to(device=device, dtype=torch.int32)+ return upload_from_host(x.reshape(-1).to(dtype=torch.int32).contiguous(), device)x_rows = rows(x_keyframes, channels)y = (- torch.empty((kf_rows + M, channels), device=device, dtype=torch.bfloat16)+ workspace_volume((kf_rows + M, channels), device=device, dtype=torch.bfloat16)if out is None# Against ``x_rows`` rather than ``x_keyframes``: what the kernel reads is the row# view, and for a non-contiguous input that is a copy the caller cannot alias anyway.@@ -2047,7 +2145,7 @@ def _keyframe_workspace(_dyn_act(rows(stage4_keyframes, ctx_channels)),_dyn_act(y),_dyn_act(torch.empty((2 * num_heads * cache_row_stride, head_dim), device=device, dtype=torch.bfloat16)),- _dyn(torch.empty(cache_row_stride + M, device=device, dtype=torch.float32)),+ _dyn(torch.empty((cache_row_stride + M,), device=device, dtype=torch.float32)),_dyn(keyframe_times.detach().to(device=device, dtype=torch.float32).reshape(-1)),_dyn(i32(slots)),_dyn(i32(planes)),@@ -2100,6 +2198,7 @@ def run_block_fna_dsl(x_keyframes: torch.Tensor | None = None,stage4_keyframes: torch.Tensor | None = None,keyframe_times: torch.Tensor | None = None,+ keyframe_times_host: torch.Tensor | None = None,out_keyframes: torch.Tensor | None = None,) -> "torch.Tensor | tuple[torch.Tensor, torch.Tensor]":"""Run one fused ``DiffusionNABlock``. ``x`` is ``(1, T, H, W, C)``.@@ -2125,6 +2224,8 @@ def run_block_fna_dsl(``out_keyframes`` is the same thing for the anchor stream, sized``(n_kf*H*W + OUT_SLACK_ROWS, C)``; without it a joint launch allocates that stream'soutput itself and the chain's alternation stops one stream short.+ ``keyframe_times_host`` is a host copy of ``keyframe_times``. The anchor stream's plane+ tables are built on the host; with the copy that needs no read of the device."""if not block_fna_available(x.device.index or 0):raise RuntimeError(UNSUPPORTED_MESSAGE)@@ -2169,8 +2270,9 @@ def run_block_fna_dsl(# M extra rows absorb the stores of out-of-range lanes in partial tiles -- which is# why a caller-supplied ``out`` has to carry the same slack.y_rows = T * H * W + OUT_SLACK_ROWS+ # Under an active ltx_kernels.vae.workspace the volume is leased per tile.y_buf = (- torch.empty((y_rows, C), device=x_.device, dtype=torch.bfloat16)+ workspace_volume((y_rows, C), device=x_.device, dtype=torch.bfloat16)if out is Noneelse _check_out(out, y_rows, C, x_))@@ -2181,7 +2283,7 @@ def run_block_fna_dsl(kv_cache = torch.empty((2 * num_heads * plan.cache_row_stride, hd), device=x_.device, dtype=torch.bfloat16)# The cache fill's RMSNorm factor per box token; M extra entries absorb# out-of-range lanes.- norm_cache = torch.empty(plan.cache_row_stride + M, device=x_.device, dtype=torch.float32)+ norm_cache = torch.empty((plan.cache_row_stride + M,), device=x_.device, dtype=torch.float32)barrier = torch.zeros(2, device=x_.device, dtype=torch.int32)kf = _keyframe_workspace(@@ -2199,6 +2301,7 @@ def run_block_fna_dsl(plan,x_.device,out_keyframes,+ keyframe_times_host,)def f32(t: torch.Tensor):@@ -2420,6 +2523,10 @@ def _launch(q_layout = sm100_utils.make_smem_layout_a(mma128, TILER_N128, IO, NH)k_layout = sm100_utils.make_smem_layout_b(mma128, TILER_N128, IO, 2)v_layout = sm100_utils.make_smem_layout_b(mma64, TILER_N64, IO, 4)+ # The MLP's weights stream through a deeper queue over s_q, which is dead after attention,+ # and its hidden chunk is staged in the freed two-stage weight slot instead.+ b4_layout = sm100_utils.make_smem_layout_b(mma128, TILER_N128, IO, MLP_B_STAGES)+ h_layout = sm100_utils.make_smem_layout_a(mma128, TILER_N128, IO, TILE_N // TILE_K)# One TMA atom per weight; ``local_tile`` inside ``gemm_tma`` picks the N-tile and# K-tile, so one atom serves every call site for that weight.@@ -2529,6 +2636,8 @@ def _launch(q_layout,k_layout,v_layout,+ b4_layout,+ h_layout,).launch(grid=[n_ctas, 1, 1], block=[THREADS, 1, 1])diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/fna_attn_core.py b/packages/ltx-kernels/src/ltx_kernels/vae/fna_attn_core.pyindex 59e0c70..08f37c2 100644--- a/packages/ltx-kernels/src/ltx_kernels/vae/fna_attn_core.py+++ b/packages/ltx-kernels/src/ltx_kernels/vae/fna_attn_core.py@@ -24,9 +24,11 @@ import torchfrom cutlass.cute.nvgpu import tcgen05from cutlass.cute.runtime import from_dlpack-from ltx_kernels.vae.fna_gemm import _mma_issue, _tmem_load, _tmem_store+from ltx_kernels.vae.fna_gemm import _mma_issue, _tmem_load, _tmem_shift, _tmem_storefrom ltx_kernels.vae.fna_geometry import at_most, flatten_3d, unflatten_3dfrom ltx_kernels.vae.fna_smem import (+ _alias,+ _cp_kv_rows,_fill_a_chunk,_load_head_row,_publish_smem,@@ -41,6 +43,7 @@ from ltx_kernels.vae.fna_types import (LOG2E,NUM_WARPGROUPS,THREADS,+ TILE_K,WARPGROUPS_PER_HEAD,M,)@@ -338,13 +341,20 @@ def attention_tile(NKV: cutlass.Constexpr,sel=None,window_hi=None,+ rowtab=None,+ copy_bars=None,):"""Windowed NA over one query tile's halo panel, for ``NHG`` heads.Args:frags: ``(frQ, frK, frP, frV)`` -- the four MMA operand fragments.- smem: ``(sK, sV, sP)`` -- K and V double-buffered, P single.+ smem: ``(sK, sV, sP_own)`` -- K and V double-buffered, and this warpgroup's view+ of the single P buffer: ``sP`` shifted by ``wg`` K-stages+ (:func:`~ltx_kernels.vae.fna_smem._alias`), so the P fill is one copy for every+ warpgroup.acc_qk: two TMEM Q@K accumulators; the pipeline alternates between them.- acc_o: ``NHG`` TMEM attention-output accumulators, live for the whole loop.+ acc_o: ``NHG`` TMEM attention-output accumulators, live for the whole loop. They+ must be ``HD`` columns apart: the head loop reaches head ``nh``'s as+ ``acc_o[0]`` shifted by ``nh * HD``.bars: ``(qkbar, pvbar)`` MMA completion barriers, re-initialised per tile.m_sm: ``[lane, nh]`` fixed softmax offsets, from the caller's ``||q||`` pass.l_sm: ``[lane, wg, nh]`` softmax denominators, accumulated per warpgroup.@@ -372,6 +382,14 @@ def attention_tile(``None`` is NATTEN's inward shift, ``[lt0, lt0 + kt)`` etc., and traces to thebyte-identical pre-keyframe code. The two agree only in the interior, so thechoice belongs to the caller's operator, not to this function.+ rowtab: optional ``(2, M)`` Int32 SMEM table. Given, K/V tiles go global -> SMEM by+ cooperative ``cp.async`` (:func:`~ltx_kernels.vae.fna_smem._cp_kv_rows`) instead of+ through ``kvfrag``: each lane publishes its key's source row in the table, a step+ ahead, and the copy of step ``s+2``'s K (``s+1``'s V) is issued as soon as its+ buffer frees. ``None`` traces the register path.+ copy_bars: with ``rowtab``, three SMEM mbarriers -- ``K full`` per buffer and+ ``P/V full`` -- that replace the two CTA barriers per step: the threads that+ wrote an operand fence and arrive, and only the MMA-issuing warp waits.A keyframe tail is this same call with ``PT=1`` and ``kt=1``: ``panel_key_row`` thenreduces to ``flatten_3d(plane, ...)`` and the mask's ``rdt`` test to ``0 <= 0 < 1``, soa spatial-only window over one keyframe plane needs no separate code. Because the@@ -385,9 +403,10 @@ def attention_tile(for *its own* Q@K, issued a whole step earlier, and before returning it issuesP@V(s) and Q@K(s+1) and stages step s+1's K/V row into the other SMEM buffer. Soa step's source read, its P@V and the next step's Q@K all run underneath the- current softmax, and only the softmax is on the critical path. Both buffer- indices are ``nh % 2``, compile-time because ``NHG`` is even, which keeps every- SMEM slice statically aligned and every barrier phase constant.+ current softmax, and only the softmax is on the critical path. The heads are walked+ a pair at a time by a device loop, and both buffer indices are the head's place in+ its pair -- compile-time, because ``NHG`` is even -- which keeps every SMEM slice+ statically aligned.K and V are staged by warpgroups 0 and 1 respectively -- two arms of the same``wg`` test, which costs nothing (the arms already diverge at the store, K beingK-major and V MN-major) and lets the NA caller pass its own two tensors. A@@ -395,7 +414,7 @@ def attention_tile(NVVM backend, not the frontend."""frQ, frK, frP, frV = frags- sK, sV, sP = smem+ sK, sV, sP_own = smemqkbar, pvbar = barsk_src, v_src = kv_srck_base, v_base = kv_base@@ -417,18 +436,41 @@ def attention_tile(# Re-initialised per query tile: each barrier then sees exactly ``2 * NKV`` waits# in the loop (plus one after it for barrier 0, which retires the pipeline's# trailing Q@K), so the phase a given ``nh`` waits on never depends on ``kv``.+ # Every warp must have finished polling the previous tile's barriers before warp 0+ # resets them: a warp still waiting on a re-initialised barrier never sees the phase+ # it expects, and the persistent grid spins forever.+ cute.arch.sync_threads()if warp_i == 0:with cute.arch.elect_one():cute.arch.mbarrier_init(qkbar.iterator, 1)cute.arch.mbarrier_init(qkbar.iterator + 1, 1)cute.arch.mbarrier_init(pvbar.iterator, 1)+ if cutlass.const_expr(copy_bars is not None):+ cute.arch.mbarrier_init(copy_bars.iterator, THREADS // 2)+ cute.arch.mbarrier_init(copy_bars.iterator + 1, THREADS // 2)+ cute.arch.mbarrier_init(copy_bars.iterator + 2, THREADS)cute.arch.mbarrier_init_fence()cute.arch.sync_threads()key_row = panel_key_row(0, lane, panel, N_KEYS, PANEL_HW, PW)+ if cutlass.const_expr(rowtab is not None):+ # Prologue: K of steps 0 and 1 and V of step 0 in flight, step 0's landed.+ rowtab[0, lane] = key_row+ rowtab[1, lane] = panel_key_row(1, lane, panel, N_KEYS, PANEL_HW, PW)+ cute.arch.sync_threads()+ if wg == 0:+ _cp_kv_rows(k_src, k_base, rowtab, 0, KVRM, sK, 0, lane, False)+ cute.arch.cp_async_commit_group()+ _cp_kv_rows(k_src, k_base + kv_hstride, rowtab, 0, KVRM, sK, 1, lane, False)+ cute.arch.cp_async_commit_group()+ cute.arch.cp_async_wait_group(1)+ elif wg == 1:+ _cp_kv_rows(v_src, v_base, rowtab, 0, KVRM, sV, 0, lane, True)+ cute.arch.cp_async_commit_group()+ cute.arch.cp_async_wait_group(0)# Prologue: fill buffer 0 with step (0, 0), leave step (0, 1) in registers, and# put Q@K(0, 0) in flight.- if wg == 0:+ elif wg == 0:_load_head_row(k_src, k_base + key_row * KVRM, kvfrag)_store_k_row(kvfrag, sK, lane, 0)_load_head_row(k_src, k_base + kv_hstride + key_row * KVRM, kvfrag)@@ -494,95 +536,159 @@ def attention_tile(# Whatever filled the K/V source wrote every panel token exactly once, so this# is a straight gather -- no per-query projection, norm or RoPE.next_key_row = panel_key_row(kv + 1, lane, panel, N_KEYS, PANEL_HW, PW)- for nh in cutlass.range_constexpr(NHG):- # Buffer this step owns, and the one the next step will own. ``kv`` is a- # runtime value -- the KV loop is a real device loop -- so the buffer- # index has to come from ``nh`` alone, which is what keeps every SMEM- # slice statically aligned. ``NHG`` is even, so ``nh % 2`` is the flat- # step's parity.- bf = nh % 2- nf = 1 - bf- # Wait phases. ``pvbar`` is waited once per step, so its phase is the step- # parity. ``qkbar[bf]`` is waited every second step, so its phase advances- # by ``NHG / 2`` per KV tile: a constant when that is even, one runtime- # AND when it is not.- qk_ph = (cutlass.Int32(kv) * (NHG // 2) + nh // 2) % 2- # Stage the *next* step's K, start the fetch two steps out, then issue the- # next step's Q@K -- all before touching this step's scores. That ordering- # is the point: Q@K(s+1) enters the MMA queue one full softmax before step- # s+1 waits on it, and the global load has a full step to land. Issuing it- # after the softmax measures within noise of no pipeline at all.- # sK[nf] currently holds K(s-1), whose Q@K was waited on last step, so it- # is free; sV[nf] is not, which is why V is staged further down, behind- # its own P@V barrier.- if wg == 0:- _store_k_row(kvfrag, sK, lane, nf)- if cutlass.const_expr(nh + 2 < NHG):- _load_head_row(k_src, k_base + (nh + 2) * kv_hstride + key_row * KVRM, kvfrag)+ # Defined on both paths: the staged regions below capture it even where it is unused.+ tab = kv % 2+ if cutlass.const_expr(rowtab is not None):+ # Read first by step (kv, NHG-2)'s copy, which several barriers follow; the+ # slot's previous rows (tile kv-1) were last read before this tile began.+ rowtab[1 - tab, lane] = next_key_row+ if cutlass.const_expr(copy_bars is not None):+ # The per-step barriers are gone, so the table needs its own.+ cute.arch.sync_threads()+ # Heads go a pair at a time through a device loop, so the step body is traced once+ # per buffer parity ``j`` rather than once per head. The runtime head reaches only+ # source rows, ``m_sm``/``l_sm`` indices, the Q operand's stage and TMEM addresses;+ # every SMEM slice index below is ``j`` or a constant, which keeps them statically+ # aligned.+ n_pairs = NHG // 2+ for hp in cutlass.range(n_pairs):+ # The look-ahead steps wrap into the next KV tile only from the last pair: their+ # head drops by ``NHG`` and their key rows are the next tile's.+ last = cutlass.Int32(1) if hp == n_pairs - 1 else cutlass.Int32(0)+ ld_row = key_row + (next_key_row - key_row) * last+ ld_tab = tab + (1 - 2 * tab) * last+ for j in cutlass.range_constexpr(2):+ nh = hp * 2 + j+ # Buffer this step owns, and the one the next step will own. ``kv`` and ``hp``+ # are runtime values, so the buffer index has to come from ``j`` alone; with+ # ``NHG`` even, ``j`` is the flat step's parity.+ bf = j+ nf = 1 - bf+ # Wait phases. ``pvbar`` is waited once per step, so its phase is the step+ # parity. ``qkbar[bf]`` is waited every second step, so its phase advances+ # by ``NHG / 2`` per KV tile and by one per pair.+ qk_ph = (cutlass.Int32(kv) * n_pairs + hp) % 2+ # Head of the step one out (its Q operand and, on the cp.async path, its V and+ # the table slot that V's rows sit in) and of the step two out (its K, and its+ # V on the register path).+ if cutlass.const_expr(j == 0):+ next_head = nh + 1+ next_tab = tabelse:- _load_head_row(k_src, k_base + (nh + 2 - NHG) * kv_hstride + next_key_row * KVRM, kvfrag)- _publish_smem()- _mma_issue("n128", frQ, frK, acc_qk[nf], 1, (nh + 1) % NHG, nf, False, qkbar.iterator + nf)- # Now wait for *this* step's Q@K, issued one step ago.- cute.arch.mbarrier_wait(qkbar.iterator + bf, qk_ph)- # P@V(s-1) has drained, which frees both sP and the V buffer this step is- # about to stage into. The wait sits here rather than after the- # exponentials, where it would have more slack: moving it down measures- # worse, because the V store has to follow it and the global load that- # follows the store then loses a whole exp loop of latency hiding.- cute.arch.mbarrier_wait(pvbar.iterator, bf)- if wg == 1:- _store_v_row(kvfrag, sV, lane, 2 * nf)- if cutlass.const_expr(nh + 2 < NHG):- _load_head_row(v_src, v_base + (nh + 2) * kv_hstride + key_row * KVRM, kvfrag)+ next_head = nh + 1 - NHG * last+ next_tab = ld_tab+ ld_head = nh + 2 - NHG * last+ # Stage the *next* step's K, start the fetch two steps out, then issue the+ # next step's Q@K -- all before touching this step's scores. That ordering+ # is the point: Q@K(s+1) enters the MMA queue one full softmax before step+ # s+1 waits on it, and the global load has a full step to land. Issuing it+ # after the softmax measures within noise of no pipeline at all.+ # sK[nf] currently holds K(s-1), whose Q@K was waited on last step, so it+ # is free; sV[nf] is not, which is why V is staged further down, behind+ # its own P@V barrier.+ if cutlass.const_expr(rowtab is not None):+ # K(s+1) was issued a step ago, as soon as sK[nf] freed.+ if wg == 0:+ cute.arch.cp_async_wait_group(0)+ if cutlass.const_expr(copy_bars is not None):+ cute.arch.fence_proxy("async.shared", space="cta")+ cute.arch.mbarrier_arrive(copy_bars.iterator + nf)+ elif wg == 0:+ _store_k_row(kvfrag, sK, lane, nf)+ _load_head_row(k_src, k_base + ld_head * kv_hstride + ld_row * KVRM, kvfrag)+ if cutlass.const_expr(copy_bars is not None):+ if warp_i == 0:+ cute.arch.mbarrier_wait(copy_bars.iterator + nf, qk_ph)else:- _load_head_row(v_src, v_base + (nh + 2 - NHG) * kv_hstride + next_key_row * KVRM, kvfrag)- # Softmax over this warpgroup's ``ACC_COLS`` score columns against a *fixed*- # row offset (the caller put ``||q|| * max||k||`` in ``m_sm``). Nothing- # here crosses the warpgroup boundary: no max, no max reduction, no- # rescale of the O accumulator, no denominator reduction. That deletes two- # CTA-wide barriers and a TMEM read-modify-write from every one of the- # ``NKV * NHG`` steps.- m_new = m_sm[lane, nh]- # One TMEM read. The chunk index may be a runtime value here because a- # TMEM address is a register either way; only SMEM slices need constexpr.- _tmem_load(acc_qk[bf], lane, areg, wg)- cute.arch.fence_view_async_tmem_load()- # sP still has to be written for a fully masked group (P@V reads all ``M``- # columns), but zeroing is far cheaper than 32 exps.- # The group is 32 keys because **only a warp-uniform skip pays**: a warp- # shares ``qt``, so a group whose panel rows all sit outside the query's dt- # window is skipped by the whole warp. Finer groups skip more often on- # paper, but ``lw0`` varies across a warp's lanes, so both arms issue and- # only the extra tests are real -- measured 9% slower per nibble.- sm4 = [ACC(0.0) for _ in range(4)]- for hb in cutlass.range_constexpr(NBW):- bw = mask[hb]- if bw != 0:- for e in cutlass.range_constexpr(32):- # Masking the *score* rather than the exponential keeps the ex2- # and the running sum unconditional, so the sum splits across- # four accumulators too. ex2 of -1e30 underflows to +0, which- # is the point.- v = areg[hb * 32 + e] * LOG2E - m_new- if (bw >> e) & 1 == 0:- v = ACC(-1.0e30)- p = cute.math.exp2(v, fastmath=True)- areg[hb * 32 + e] = p- sm4[e % 4] = sm4[e % 4] + p+ _publish_smem()+ _mma_issue("n128", frQ, frK, acc_qk[nf], 1, next_head, nf, False, qkbar.iterator + nf)+ # Now wait for *this* step's Q@K, issued one step ago.+ cute.arch.mbarrier_wait(qkbar.iterator + bf, qk_ph)+ if cutlass.const_expr(rowtab is not None): # noqa: SIM102 -- a staged ``if`` cannot share a constexpr test+ # Q@K(s) is done with sK[bf]: start K(s+2) into it.+ if wg == 0:+ _cp_kv_rows(k_src, k_base + ld_head * kv_hstride, rowtab, ld_tab, KVRM, sK, bf, lane, False)+ cute.arch.cp_async_commit_group()+ if cutlass.const_expr(rowtab is None):+ # P@V(s-1) has drained, which frees both sP and the V buffer this step is+ # about to stage into. The wait sits here rather than after the+ # exponentials, where it would have more slack: moving it down measures+ # worse, because the V store has to follow it and the global load that+ # follows the store then loses a whole exp loop of latency hiding.+ cute.arch.mbarrier_wait(pvbar.iterator, bf)+ if wg == 1:+ _store_v_row(kvfrag, sV, lane, 2 * nf)+ _load_head_row(v_src, v_base + ld_head * kv_hstride + ld_row * KVRM, kvfrag)+ # Softmax over this warpgroup's ``ACC_COLS`` score columns against a *fixed*+ # row offset (the caller put ``||q|| * max||k||`` in ``m_sm``). Nothing+ # here crosses the warpgroup boundary: no max, no max reduction, no+ # rescale of the O accumulator, no denominator reduction. That deletes two+ # CTA-wide barriers and a TMEM read-modify-write from every one of the+ # ``NKV * NHG`` steps.+ m_new = m_sm[lane, nh]+ # One TMEM read. The chunk index may be a runtime value here because a+ # TMEM address is a register either way; only SMEM slices need constexpr.+ _tmem_load(acc_qk[bf], lane, areg, wg)+ cute.arch.fence_view_async_tmem_load()+ # sP still has to be written for a fully masked group (P@V reads all ``M``+ # columns), but zeroing is far cheaper than 32 exps.+ # The group is 32 keys because **only a warp-uniform skip pays**: a warp+ # shares ``qt``, so a group whose panel rows all sit outside the query's dt+ # window is skipped by the whole warp. Finer groups skip more often on+ # paper, but ``lw0`` varies across a warp's lanes, so both arms issue and+ # only the extra tests are real -- measured 9% slower per nibble.+ sm4 = [ACC(0.0) for _ in range(4)]+ for hb in cutlass.range_constexpr(NBW):+ bw = mask[hb]+ if bw != 0:+ for e in cutlass.range_constexpr(32):+ # Masking the *score* rather than the exponential keeps the ex2+ # and the running sum unconditional, so the sum splits across+ # four accumulators too. ex2 of -1e30 underflows to +0, which+ # is the point.+ v = areg[hb * 32 + e] * LOG2E - m_new+ if (bw >> e) & 1 == 0:+ v = ACC(-1.0e30)+ p = cute.math.exp2(v, fastmath=True)+ areg[hb * 32 + e] = p+ sm4[e % 4] = sm4[e % 4] + p+ else:+ for e in cutlass.range_constexpr(32):+ areg[hb * 32 + e] = ACC(0.0)+ l_sm[lane, wg, nh] = l_sm[lane, wg, nh] + ((sm4[0] + sm4[1]) + (sm4[2] + sm4[3]))+ if cutlass.const_expr(rowtab is not None):+ # P@V(s-1) still reads sP and V(s-1)'s buffer; its remaining time overlapped+ # the exponentials above. Then V(s+1) can start into the freed buffer.+ cute.arch.mbarrier_wait(pvbar.iterator, bf)+ if wg == 1:+ _cp_kv_rows(+ v_src, v_base + next_head * kv_hstride, rowtab, next_tab, KVRM, sV, 2 * nf, lane, True+ )+ cute.arch.cp_async_commit_group()+ # Stage 0 of the shifted view is this warpgroup's own ``ACC_COLS`` chunk of sP.+ _fill_a_chunk(sP_own, lane, areg, 0, ACC_COLS)+ if cutlass.const_expr(rowtab is not None): # noqa: SIM102 -- a staged ``if`` cannot share a constexpr test+ # V(s) was issued a step ago; V(s+1), issued above, may still be in flight.+ if wg == 1:+ cute.arch.cp_async_wait_group(1)+ if cutlass.const_expr(copy_bars is not None):+ # Every thread wrote its P row (and wg1 its V copies): fence, count in, and+ # let only the issuing warp wait.+ cute.arch.fence_proxy("async.shared", space="cta")+ cute.arch.mbarrier_arrive(copy_bars.iterator + 2)+ if warp_i == 0:+ cute.arch.mbarrier_wait(copy_bars.iterator + 2, bf)else:- for e in cutlass.range_constexpr(32):- areg[hb * 32 + e] = ACC(0.0)- l_sm[lane, wg, nh] = l_sm[lane, wg, nh] + ((sm4[0] + sm4[1]) + (sm4[2] + sm4[3]))- for g in cutlass.range_constexpr(NUM_WARPGROUPS):- if wg == g:- _fill_a_chunk(sP, lane, areg, g * ACC_COLS, ACC_COLS)- # Publishes sP for the P@V below and the V buffer for the next step's.- _publish_smem()- _mma_issue("pv", frP, frV, acc_o[nh], 2, 0, 2 * bf, True, pvbar.iterator)+ # Publishes sP for the P@V below and the V buffer for the next step's.+ _publish_smem()+ _mma_issue("pv", frP, frV, _tmem_shift(acc_o[0], nh * HD), 2, 0, 2 * bf, True, pvbar.iterator)key_row = next_key_rowkv += 1+ if cutlass.const_expr(rowtab is not None):+ # The look-ahead copies past the last step land in buffers nothing reads; they still+ # have to finish before the caller reuses the pool.+ cute.arch.cp_async_wait_group(0)# Retire the pipeline. P@V is the last MMA of the last step, so its barrier proves# every accumulator is final -- and, being issued after it, that the trailing# Q@K's commit has landed too, which is what makes re-initialising the barriers@@ -591,13 +697,15 @@ def attention_tile(@cute.jit-def attn_epilogue(acc_o, l_sm, oreg, sO, lane, wg, NHG: cutlass.Constexpr):- """Divide the attention accumulators by the denominator, into ``sO``.- ``sO`` is an A-operand-shaped SMEM buffer whose channel ``nh * HD + d`` is head- ``nh``'s output: the fused kernel hands it ``sA`` and feeds the out-projection- straight from it, the NA kernel hands it a buffer it then scatters to global.+def attn_epilogue(acc_o, l_sm, oreg, o_buf, o_layout, lane, wg, NHG: cutlass.Constexpr):+ """Divide the attention accumulators by the denominator, into the operand ``sO``.+ ``sO`` is the A-operand view ``o_layout`` over the raw storage ``o_buf``, and its+ channel ``nh * HD + d`` is head ``nh``'s output: the fused kernel hands it ``sA`` and+ feeds the out-projection straight from it, the NA kernel hands it a buffer it then+ scatters to global. ``HD`` is one K-stage, so the device head loop reaches head+ ``nh`` through a view shifted ``nh`` stages and every slice coordinate stays static."""- for nh in cutlass.range_constexpr(NHG):+ for nh in cutlass.range(NHG):# Each warpgroup summed its own slice of every score row.lden = l_sm[lane, 0, nh]for g in cutlass.range_constexpr(1, NUM_WARPGROUPS):@@ -605,12 +713,15 @@ def attn_epilogue(acc_o, l_sm, oreg, sO, lane, wg, NHG: cutlass.Constexpr):linv = ACC(0.0)if lden > ACC(0.0):linv = cute.arch.rcp_approx(lden)+ _tmem_load(_tmem_shift(acc_o[0], nh * HD), lane, oreg, wg, HEAD_COLS)+ for d in cutlass.range_constexpr(HEAD_COLS):+ oreg[d] = oreg[d] * linv+ s_head = _alias(o_buf, o_layout, nh * (M * TILE_K))+ # A ``HEAD_COLS`` chunk is half a stage, not a whole one, so the warpgroup split+ # stays a constexpr arm here.for g in cutlass.range_constexpr(NUM_WARPGROUPS):if wg == g:- _tmem_load(acc_o[nh], lane, oreg, g, HEAD_COLS)- for d in cutlass.range_constexpr(HEAD_COLS):- oreg[d] = oreg[d] * linv- _fill_a_chunk(sO, lane, oreg, nh * HD + g * HEAD_COLS, HEAD_COLS)+ _fill_a_chunk(s_head, lane, oreg, g * HEAD_COLS, HEAD_COLS)@cute.jit@@ -619,9 +730,7 @@ def attn_zero_accumulators(acc_o, l_sm, oreg, lane, wg, NHG: cutlass.Constexpr):for d in cutlass.range_constexpr(HEAD_COLS):oreg[d] = ACC(0.0)for nh in cutlass.range_constexpr(NHG):- for g in cutlass.range_constexpr(NUM_WARPGROUPS):- if wg == g:- _tmem_store(acc_o[nh], lane, oreg, g, HEAD_COLS)+ _tmem_store(acc_o[nh], lane, oreg, wg, HEAD_COLS)l_sm[lane, wg, nh] = ACC(0.0)cute.arch.fence_view_async_tmem_store()cute.arch.sync_threads()diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/fna_gemm.py b/packages/ltx-kernels/src/ltx_kernels/vae/fna_gemm.pyindex bc1bdc8..9e2210c 100644--- a/packages/ltx-kernels/src/ltx_kernels/vae/fna_gemm.py+++ b/packages/ltx-kernels/src/ltx_kernels/vae/fna_gemm.py@@ -5,6 +5,8 @@ Extracted from :mod:`ltx_kernels.vae.fna_attn_core`. Constants:# NOTE: no `from __future__ import annotations` -- cute.struct needs real types.+from typing import Any, NamedTuple+import cutlassimport cutlass.cute as cuteimport cutlass.pipeline as pipeline@@ -113,6 +115,15 @@ def _tmem_store(acc, lane, regs, i=0, w=ACC_COLS):cute.copy(st, regs, dst[None, None, i])+def _tmem_shift(acc, cols):+ """``acc`` moved ``cols`` TMEM columns along, with its layout.+ How a runtime index picks one of an evenly spaced run of accumulators -- a head's+ ``acc_o``, an N-tile's ``acc_main`` -- without unrolling the loop that indexes it: a+ TMEM address is a register, so ``cols`` may be a runtime value.+ """+ return cute.make_tensor(acc.iterator + cols, acc.layout)++def _reg_shape(tiled_mma, acc, lane, n_cols, w=ACC_COLS):"""Shape every register array must use: the TMEM copy's ``partition_D``.An identity tensor stands in for the destination -- it carries the layout without@@ -168,43 +179,54 @@ def _mma_issue(tcgen05.commit(mbar)+class GemmJob(NamedTuple):+ """One ``M=128`` GEMM of a :func:`gemm_tma` call: B from ``(tma_atom, m_b)`` against ``fr_a``.+ ``fr_a2`` continues the reduction in a second A operand from K-tile ``a_split`` on, for a+ reduction wider than one operand can hold. ``kt0`` / ``n_kt`` pick the K-tiles of B.+ ``n_idx``, ``kt0`` and ``accumulate`` may be device values (a device loop's index and the+ expressions built from it); the other counts are trace-time ints.+ """++ tma_atom: Any+ m_b: Any+ fr_a: Any+ acc: Any+ n_idx: Any+ kt0: Any+ n_kt: int+ a_st: int+ accumulate: Any+ fr_a2: Any = None+ a_split: int = 0++@cute.jitdef gemm_tma(kind: cutlass.Constexpr,storage,- tma_atom,- mB,sB,- frA,- acc,- n_idx,- kt0: cutlass.Constexpr,- n_kt: cutlass.Constexpr,- a_st: cutlass.Constexpr,- accumulate: cutlass.Constexpr,stage_bytes: cutlass.Constexpr,- frA2=None,- a_split: cutlass.Constexpr = 0,+ jobs: cutlass.Constexpr,+ num_stages: cutlass.Constexpr = 2,):- """One ``M=128`` GEMM with the B operand streamed in by TMA.+ """Independent ``M=128`` GEMMs with their B operands streamed in by TMA.A is already in SMEM (the kernel computes it); only B is fetched. The- ``PipelineTmaUmma`` cycles ``sB``'s two stages, so one call covers every K-tile of- a C-wide (or Hidd-wide) reduction with the next tile's copy in flight during the- current tile's MMA.- ``frA2`` continues the reduction in a second A operand from K-tile ``a_split`` on, for a- reduction wider than one operand can hold. It is not a convenience: standing up the two- pipelines and waiting out the accumulator costs more than the K-tiles themselves at- these shapes, and two *calls* would pay that fixed cost twice. ``kt`` is compile-time,- so the choice folds away.+ ``PipelineTmaUmma`` cycles ``sB``'s ``num_stages`` stages over every job's K-tiles as+ one sequence, keeping ``num_stages - 1`` copies -- the next job's first ones included --+ in flight during the current tile's MMA, and the CTA waits for the accumulators once.+ Standing up the two pipelines and that final wait cost more than the K-tiles+ themselves at these shapes, which is why independent GEMMs share a call.+ ``jobs`` is a trace-time tuple of :class:`GemmJob`."""tiled_mma = cute.make_tiled_mma(_mma_op(kind))tiler = _mma_tiler(kind)warp_idx = cute.arch.make_warp_uniform(cute.arch.warp_idx())if warp_idx == 0:- cpasync.prefetch_descriptor(tma_atom)+ for job in jobs:+ cpasync.prefetch_descriptor(job.tma_atom)ab_prod, ab_cons = pipeline.PipelineTmaUmma.create(- num_stages=2,+ num_stages=num_stages,producer_group=pipeline.CooperativeGroup(pipeline.Agent.Thread),consumer_group=pipeline.CooperativeGroup(pipeline.Agent.Thread),tx_count=stage_bytes,@@ -217,39 +239,57 @@ def gemm_tma(barrier_storage=storage.acc_mbar_ptr.data_ptr(),).make_participants()- gB = cute.local_tile(mB, tiler, (0, n_idx, None), proj=(None, 1, 1))- tCgB = tiled_mma.get_slice(0).partition_B(gB)- tBsB, tBgB = cpasync.tma_partition(- tma_atom,- 0,- cute.make_layout(1),- cute.group_modes(sB, 0, 3),- cute.group_modes(tCgB, 0, 3),- )+ parts = []+ for job in jobs:+ gB = cute.local_tile(job.m_b, tiler, (0, job.n_idx, None), proj=(None, 1, 1))+ tCgB = tiled_mma.get_slice(0).partition_B(gB)+ parts.append(+ cpasync.tma_partition(+ job.tma_atom,+ 0,+ cute.make_layout(1),+ cute.group_modes(sB, 0, 3),+ cute.group_modes(tCgB, 0, 3),+ )+ )frB = tiled_mma.make_fragment_B(sB)- n_kblk = cute.size(frA, mode=[2])+ steps = [(j, kt) for j, job in enumerate(jobs) for kt in range(job.n_kt)]- tiled_mma.set(tcgen05.Field.ACCUMULATE, accumulate)+ tiled_mma.set(tcgen05.Field.ACCUMULATE, jobs[0].accumulate)if warp_idx == 0:acc_empty = acc_prod.acquire_and_advance()- # One-deep prefetch: K-tile k+1's copy is issued before the wait on k, so- # sB's two stages actually overlap TMA with MMA.- e = ab_prod.acquire_and_advance()- cute.copy(tma_atom, tBgB[(None, kt0)], tBsB[(None, e.index)], tma_bar_ptr=e.barrier)- for kt in cutlass.range_constexpr(n_kt):- if kt + 1 < n_kt:+ # ``num_stages - 1`` copies ahead: step i+look's copy is issued before the wait on+ # step i, so sB's stages actually overlap TMA with MMA.+ look = num_stages - 1+ for i in cutlass.range_constexpr(min(look, len(steps))):+ pj, pkt = steps[i]+ pre = ab_prod.acquire_and_advance()+ cute.copy(+ jobs[pj].tma_atom,+ parts[pj][1][(None, jobs[pj].kt0 + pkt)],+ parts[pj][0][(None, pre.index)],+ tma_bar_ptr=pre.barrier,+ )+ for i in cutlass.range_constexpr(len(steps)):+ if cutlass.const_expr(i + look < len(steps)):+ nj, nkt = steps[i + look]nxt = ab_prod.acquire_and_advance()cute.copy(- tma_atom,- tBgB[(None, kt0 + kt + 1)],- tBsB[(None, nxt.index)],+ jobs[nj].tma_atom,+ parts[nj][1][(None, jobs[nj].kt0 + nkt)],+ parts[nj][0][(None, nxt.index)],tma_bar_ptr=nxt.barrier,)+ j, kt = steps[i]+ job = jobs[j]+ if cutlass.const_expr(j > 0 and kt == 0):+ tiled_mma.set(tcgen05.Field.ACCUMULATE, job.accumulate)f = ab_cons.wait_and_advance()- src = frA if cutlass.const_expr(a_split == 0 or kt < a_split) else frA2- a_kt = a_st + kt if cutlass.const_expr(a_split == 0 or kt < a_split) else kt - a_split- for kb in cutlass.range_constexpr(n_kblk):- cute.gemm(tiled_mma, acc, src[None, None, kb, a_kt], frB[None, None, kb, f.index], acc)+ first_operand = job.a_split == 0 or kt < job.a_split+ src = job.fr_a if cutlass.const_expr(first_operand) else job.fr_a2+ a_kt = job.a_st + kt if cutlass.const_expr(first_operand) else kt - job.a_split+ for kb in cutlass.range_constexpr(cute.size(job.fr_a, mode=[2])):+ cute.gemm(tiled_mma, job.acc, src[None, None, kb, a_kt], frB[None, None, kb, f.index], job.acc)tiled_mma.set(tcgen05.Field.ACCUMULATE, True)f.release()acc_empty.commit()diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/fna_smem.py b/packages/ltx-kernels/src/ltx_kernels/vae/fna_smem.pyindex 0ccbc2f..0285e38 100644--- a/packages/ltx-kernels/src/ltx_kernels/vae/fna_smem.py+++ b/packages/ltx-kernels/src/ltx_kernels/vae/fna_smem.py@@ -7,6 +7,7 @@ TMA/UMMA helpers can stay focused. Constants: :mod:`ltx_kernels.vae.fna_types`.import cutlassimport cutlass.cute as cute+from cutlass.cute.nvgpu import cpasyncfrom ltx_kernels.vae.fna_types import ACC, HD, IO, VEC, VEC_ROWS, M@@ -27,8 +28,11 @@ def _fill_a_chunk(sA, row, vals, c0, n):"""Write ``vals`` into channels ``[c0, c0+n)`` of A-operand row ``row``.``c0`` and ``n`` must be compile-time and multiples of ``_KRUN``: a dynamiccoordinate loses the slice's static alignment and ``autovec_copy`` falls back to- scalar stores. That is why callers spell the warpgroup split as a constexpr loop- over ``g`` guarded by ``wg == g`` rather than as an arithmetic offset.+ scalar stores. A runtime offset that is a whole K-stage -- a warpgroup's chunk, a+ head, a loop-carried chunk index -- goes into the view instead (:func:`_alias` with+ a runtime ``off``), so ``c0`` stays static. A sub-stage split, such as a+ ``HEAD_COLS`` half of a head, still needs a constexpr loop over ``g`` guarded by+ ``wg == g``."""frag = cute.make_rmem_tensor(cute.make_layout(_KRUN), IO)for k in range(n // _KRUN):@@ -172,6 +176,39 @@ def _store_v_row(frag, sV, key, st_off):cute.autovec_copy(frag, sV[((None, key % 64 % 16), 0, key % 64 // 16, st_off + key // 64)])+# Cooperative cp.async staging of a KV tile: four lanes per key row, one ``_KRUN`` (32 B) each,+# so a warp moves eight whole 128 B rows per instruction pair and no lane holds a row in+# registers. A run goes as its two 16 B halves -- cp.async's widest -- split with+# ``zipped_divide``, the form that stays correct on the swizzled operand.+_CP_LANES_PER_ROW = HD // _KRUN+_CP_ROWS_PER_PASS = M // _CP_LANES_PER_ROW+_CP_HALF = _KRUN // 2+++def _cp_kv_rows(src, base, rowtab, tab, kvrm, dst, st, lane, v_major: bool):+ """``cp.async`` one head's ``M`` key rows of a KV tile into a K or V operand stage.+ ``rowtab[tab, key]`` is the source row of key ``key`` (see ``panel_key_row``), so a lane+ copies rows other lanes own. ``v_major`` picks the MN-major V layout over K-major K.+ The copies join the thread's open cp.async group; the caller commits and waits.+ """+ atom = cute.make_copy_atom(cpasync.CopyG2SOp(), IO, num_bits_per_copy=_CP_HALF * 16)+ run = lane % _CP_LANES_PER_ROW+ for p in range(M // _CP_ROWS_PER_PASS):+ key = p * _CP_ROWS_PER_PASS + lane // _CP_LANES_PER_ROW+ row = base + rowtab[tab, key] * kvrm+ if v_major:+ krow = dst[((None, key % 64 % 16), 0, key % 64 // 16, st + key // 64)]+ drun = cute.zipped_divide(krow, cute.make_layout(_KRUN))[None, run]+ else:+ drun = dst[((key, None), 0, run, st)]+ halves = cute.zipped_divide(drun, cute.make_layout(_CP_HALF))+ for h in range(_KRUN // _CP_HALF):+ src_half = cute.make_tensor(+ _row_vec_ptr(src, row, run * _KRUN + h * _CP_HALF, HD), cute.make_layout(_CP_HALF)+ )+ cute.copy(atom, src_half, halves[None, h])++def _store_head_row(dst, row, vals, n=HD, c0: int = 0):"""Write ``n`` accumulator values to ``dst[row, c0:c0+n]``, vectorised.``dst`` rows are always ``HD`` wide; ``c0``/``n`` let a warpgroup that owns only@@ -187,14 +224,18 @@ def _store_head_row(dst, row, vals, n=HD, c0: int = 0):)-def _alias(buf, layout, off: int = 0):+def _alias(buf, layout, off=0):"""A UMMA-operand view over ``buf``'s storage, ``off`` elements in.tcgen05 wants the swizzle on the pointer rather than folded into the layout, sothe composed layout is split apart here the way ``SmemAllocator.allocate_tensor``would have done it. Re-asserting the alignment matters: adding an offset drops apointer's static alignment and a swizzled operand needs its 128 bytes back.+ ``buf`` must be raw (unswizzled) storage. ``off`` may be a runtime value, and must+ then be a whole number of K-stages (``M * TILE_K`` elements): the swizzle only+ permutes address bits below a stage, so the shifted view swizzles exactly like the+ unshifted one and its stage ``s`` is the unshifted view's stage ``s + off / (M * TILE_K)``."""- ptr = buf.iterator if off == 0 else (buf.iterator + off).align(128)+ ptr = buf.iterator if isinstance(off, int) and off == 0 else (buf.iterator + off).align(128)return cute.make_tensor(cute.recast_ptr(ptr, layout.inner, dtype=IO), layout.outer)diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/fna_types.py b/packages/ltx-kernels/src/ltx_kernels/vae/fna_types.pyindex 53b398e..72f2b75 100644--- a/packages/ltx-kernels/src/ltx_kernels/vae/fna_types.py+++ b/packages/ltx-kernels/src/ltx_kernels/vae/fna_types.py@@ -28,6 +28,9 @@ TILE_K = TILER_N128[2]assert TILE_N == M, "attention_tile indexes an N=TILE_N key tile by lane (0..M-1)"EPS = 1e-6HD = 64 # head_dim; fixed by the DiffVAE attention+# A head is one K-stage of an A operand, so a runtime head index reaches its channels by+# shifting the operand view whole stages (``fna_smem._alias``) with static slice coordinates.+assert HD == TILE_K, "per-head operand views assume one head per K-stage"# Warpgroups per CTA -- 128 threads, four warps, the tcgen05 scheduling unit. Both# kernels hold all 512 TMEM columns, so they are pinned to one CTA per SM and the only@@ -44,10 +47,15 @@ HEAD_COLS = HD // NUM_WARPGROUPSHEADS_PER_TILE = TILE_N // HD # heads in one N=128 projection tile# Warpgroups sharing a head. Two warpgroups gives one, so the Q/K RMSNorms and RoPE# stay register-local; four gives two, and they then need a partial reduction over the-# pair (``head_sum2``) and a channel-ranged RoPE. Both are written generically, so the-# two-warpgroup build is the ``WARPGROUPS_PER_HEAD == 1`` special case of the same code.+# pair (``head_sum2``) and a channel-ranged RoPE. ``head_sum2`` and the K/V pass are+# written generically, so the two-warpgroup build is the ``WARPGROUPS_PER_HEAD == 1``+# special case of the same code.WARPGROUPS_PER_HEAD = NUM_WARPGROUPS // HEADS_PER_TILEassert HEADS_PER_TILE * WARPGROUPS_PER_HEAD == NUM_WARPGROUPS, "NUM_WARPGROUPS must be a multiple of TILE_N // HD"+# The other row passes -- the query pass, the residual and AdaLN passes, the P and MLP+# hidden fills -- are one code copy for all warpgroups: each reaches its ``ACC_COLS``+# chunk through an operand view shifted by whole K-stages, so a chunk must be one stage.+assert ACC_COLS == TILE_K, "warpgroup-agnostic row passes need one K-stage per warpgroup chunk"# One cooperative-copy step moves a whole ``_KRUN`` run per thread, so both ends of# the copy vectorise: 32B from global (2x LDG.E.128) and one swizzled-slice store.VEC = 16 # bf16 elements per thread per cooperative-copy stepdiff --git a/packages/ltx-kernels/src/ltx_kernels/vae/na_attn_dsl.py b/packages/ltx-kernels/src/ltx_kernels/vae/na_attn_dsl.pyindex 6533fd2..fa05a3c 100644--- a/packages/ltx-kernels/src/ltx_kernels/vae/na_attn_dsl.py+++ b/packages/ltx-kernels/src/ltx_kernels/vae/na_attn_dsl.py@@ -26,6 +26,7 @@ import cutlass.utils.blackwell_helpers as sm100_utilsimport torchfrom cutlass.utils import SmemAllocator+from ltx_kernels.devices import upload_from_hostfrom ltx_kernels.vae.aot import KERNEL_CACHE, fake_dynamic_matrix, fake_dynamic_vectorfrom ltx_kernels.vae.availability import UNSUPPORTED_MESSAGE, gpu_supports_dsl_kernelsfrom ltx_kernels.vae.fna_attn_core import (@@ -179,6 +180,8 @@ def _na_kernel(pool = smem.allocate_tensor(IO, cute.make_layout((pool_elems,)), byte_alignment=128)sO = _alias(pool, o_layout)sP = _alias(pool, p_layout)+ # This warpgroup's chunk of ``sP`` at stage 0, for ``attention_tile``'s P fill.+ sP_own = _alias(pool, p_layout, wg * (M * TILE_K))sK = _alias(pool, k_layout, 2 * M * TILE_K)sV = _alias(pool, v_layout, 4 * M * TILE_K)sQ = smem.allocate_tensor(IO, q_layout.outer, byte_alignment=128, swizzle=q_layout.inner)@@ -323,7 +326,7 @@ def _na_kernel(if deep:attention_tile((frQ, frK, frP, frV),- (sK, sV, sP),+ (sK, sV, sP_own),acc_qk,acc_o,(qkbar, pvbar),@@ -390,7 +393,7 @@ def _na_kernel(v_src = select_tensor(video_kv != 0, v, v_kf)attention_tile((frQ, frK, frP, frV),- (sK, sV, sP),+ (sK, sV, sP_own),acc_qk,acc_o,(qkbar, pvbar),@@ -422,7 +425,7 @@ def _na_kernel(pass_i += 1cute.arch.sync_threads()- attn_epilogue(acc_o, l_sm, oreg, sO, lane, wg, HG)+ attn_epilogue(acc_o, l_sm, oreg, pool, o_layout, lane, wg, HG)cute.arch.sync_threads()_scatter_a(sO, y_dst, rowix_w, copy_slot, HG * HD, c_off)cute.arch.sync_threads()@@ -743,9 +746,9 @@ def _keyframe_operands(enabled=True,k=_dyn_act(rows(k_keyframes)),v=_dyn_act(rows(v_keyframes)),- slots=_dyn(slots_cpu.reshape(-1).to(device=device)),- planes=_dyn(planes.reshape(-1).to(device=device)),- counts=_dyn(counts.to(device=device)),+ slots=_dyn(upload_from_host(slots_cpu.reshape(-1).contiguous(), device)),+ planes=_dyn(upload_from_host(planes.reshape(-1).contiguous(), device)),+ counts=_dyn(upload_from_host(counts, device)),)@@ -844,8 +847,8 @@ def run_na_attention_bound(y_kf_buf = torch.empty((n_kf * H * W + M, NH * hd), device=q_.device, dtype=torch.bfloat16)y_kf_dyn = _dyn_act(y_kf_buf)q_kf_ = _dyn_act(q_kf_rows)- kfv = _dyn(video_frames.reshape(-1).to(device=q_.device))- kfv_counts = _dyn((video_frames >= 0).sum(dim=-1).to(dtype=torch.int32, device=q_.device))+ kfv = _dyn(upload_from_host(video_frames.reshape(-1).contiguous(), q_.device))+ kfv_counts = _dyn(upload_from_host((video_frames >= 0).sum(dim=-1).to(dtype=torch.int32), q_.device))n_kfq_tiles = n_kf * grid_h * grid_welse:_, _, q_kf_, y_kf_dyn = _inert_operands(hd, NH * hd, q_.device)diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/rope_dsl.py b/packages/ltx-kernels/src/ltx_kernels/vae/rope_dsl.pynew file mode 100644index 0000000..f9ce454--- /dev/null+++ b/packages/ltx-kernels/src/ltx_kernels/vae/rope_dsl.py@@ -0,0 +1,195 @@+"""Absolute 3D RoPE over a whole ``(B, T, H, W, NH, HD)`` volume in one CuTe DSL launch.+The rotation is :func:`ltx_kernels.vae.fna_attn_core.apply_rope`, the same code the fused+DiffVAE block runs on its Q/K rows; here thread ``i`` of a block owns head row ``i``, rows+are staged through SMEM so global traffic stays coalesced, and the cos/sin come from+global tables. No tensor-core instructions are+involved, so it builds for any CUDA architecture the DSL targets.+The caller builds the tables, so they can hold exactly the angles a reference+implementation uses -- integer or fractional positions alike.+"""++# NOTE: no `from __future__ import annotations` -- cute.jit needs real annotations.++import cutlass+import cutlass.cute as cute+import torch+from cutlass.cute.runtime import from_dlpack+from cutlass.utils import SmemAllocator++from ltx_kernels.vae.aot import KERNEL_CACHE, fake_dynamic_matrix+from ltx_kernels.vae.fna_attn_core import apply_rope+from ltx_kernels.vae.fna_smem import _row_vec_ptr+from ltx_kernels.vae.fna_types import ACC, HD, IO++_THREADS = 128+# Global traffic moves in 16B slices, consecutive lanes on consecutive slices, so a warp's+# loads and stores cover whole 128B lines. Rows are staged through SMEM, padded by one+# slice so the per-thread row reads of an 8-lane phase land on distinct banks.+_VEC = 8+_LANES_PER_ROW = HD // _VEC+_ROW_STRIDE = HD + _VEC+++def _rope_kernel(+ x: cute.Tensor, # (rows, HD) bf16, rows ordered (B, T, H, W, NH)+ y: cute.Tensor, # (rows, HD) bf16+ cos_t: cute.Tensor, # (T, d_t / 2) f32+ sin_t: cute.Tensor,+ cos_h: cute.Tensor, # (H, d_h / 2) f32+ sin_h: cute.Tensor,+ cos_w: cute.Tensor, # (W, d_w / 2) f32+ sin_w: cute.Tensor,+ n_rows: cutlass.Int32,+ T: cutlass.Int32,+ H: cutlass.Int32,+ W: cutlass.Int32,+ NH: cutlass.Int32,+ d_t: cutlass.Constexpr,+ d_h: cutlass.Constexpr,+ d_w: cutlass.Constexpr,+) -> None:+ tidx, _, _ = cute.arch.thread_idx()+ bidx, _, _ = cute.arch.block_idx()+ row0 = bidx * _THREADS+ smem = SmemAllocator()+ tile = smem.allocate_tensor(IO, cute.make_layout((_THREADS * _ROW_STRIDE,)), byte_alignment=16)+ frag = cute.make_rmem_tensor(cute.make_layout(_VEC), IO)++ # Coalesced fill: consecutive lanes take consecutive 16B slices of consecutive rows.+ for k in range(_LANES_PER_ROW):+ r = (k * _THREADS + tidx) // _LANES_PER_ROW+ c = (k * _THREADS + tidx) % _LANES_PER_ROW * _VEC+ if row0 + r < n_rows:+ cute.autovec_copy(cute.make_tensor(_row_vec_ptr(x, row0 + r, c, HD), cute.make_layout(_VEC)), frag)+ cute.autovec_copy(frag, _smem_slice(tile, r, c))+ cute.arch.sync_threads()++ # Thread ``i`` rotates row ``i`` in registers.+ row = row0 + tidx+ if row < n_rows:+ token = row // NH+ w = token % W+ h = (token // W) % H+ t = (token // (W * H)) % T+ reg = cute.make_rmem_tensor(cute.make_layout(HD), ACC)+ for k in range(_LANES_PER_ROW):+ cute.autovec_copy(_smem_slice(tile, tidx, k * _VEC), frag)+ for e in range(_VEC):+ reg[k * _VEC + e] = ACC(frag[e])+ apply_rope(reg, d_t, d_h, d_w, (cos_t, sin_t, cos_h, sin_h, cos_w, sin_w), t, h, w)+ for k in range(_LANES_PER_ROW):+ for e in range(_VEC):+ frag[e] = IO(reg[k * _VEC + e])+ cute.autovec_copy(frag, _smem_slice(tile, tidx, k * _VEC))+ cute.arch.sync_threads()++ for k in range(_LANES_PER_ROW):+ r = (k * _THREADS + tidx) // _LANES_PER_ROW+ c = (k * _THREADS + tidx) % _LANES_PER_ROW * _VEC+ if row0 + r < n_rows:+ cute.autovec_copy(_smem_slice(tile, r, c), frag)+ cute.autovec_copy(frag, cute.make_tensor(_row_vec_ptr(y, row0 + r, c, HD), cute.make_layout(_VEC)))+++def _smem_slice(tile, row, col):+ """The 16B slice at ``(row, col)`` of the padded SMEM row tile."""+ return cute.make_tensor((tile.iterator + (row * _ROW_STRIDE + col)).align(16), cute.make_layout(_VEC))+++def _launch(+ x: cute.Tensor,+ y: cute.Tensor,+ cos_t: cute.Tensor,+ sin_t: cute.Tensor,+ cos_h: cute.Tensor,+ sin_h: cute.Tensor,+ cos_w: cute.Tensor,+ sin_w: cute.Tensor,+ n_rows: cutlass.Int32,+ T: cutlass.Int32,+ H: cutlass.Int32,+ W: cutlass.Int32,+ NH: cutlass.Int32,+ d_t: cutlass.Constexpr,+ d_h: cutlass.Constexpr,+ d_w: cutlass.Constexpr,+) -> None:+ _rope_kernel(x, y, cos_t, sin_t, cos_h, sin_h, cos_w, sin_w, n_rows, T, H, W, NH, d_t, d_h, d_w).launch(+ grid=[(n_rows + _THREADS - 1) // _THREADS, 1, 1], block=[_THREADS, 1, 1]+ )+++def _compile_kernel(*, rope_dims: "tuple[int, int, int]", options: str | None = None):+ """Compile one ``(d_t, d_h, d_w)`` split from storage-free descriptors."""+ rows = fake_dynamic_matrix(cutlass.BFloat16, assumed_align=16)+ tables = [fake_dynamic_matrix(cutlass.Float32, assumed_align=4) for _ in range(6)]+ return cute.compile(+ _launch,+ rows,+ fake_dynamic_matrix(cutlass.BFloat16, assumed_align=16),+ *tables,+ 1,+ 1,+ 1,+ 1,+ 1,+ *rope_dims,+ **({"options": options} if options else {}),+ )+++def abs_rope_supported(x: torch.Tensor, rope_dims: "tuple[int, int, int]") -> bool:+ """Whether :func:`run_abs_rope` takes ``x``: a contiguous CUDA bf16 ``(B, T, H, W, NH, 64)`` volume."""+ d_t, d_h, d_w = rope_dims+ return (+ x.is_cuda+ and x.dtype == torch.bfloat16+ and x.dim() == 6+ and x.shape[-1] == HD+ and x.is_contiguous()+ and x.numel() // HD < 2**31+ and d_t + d_h + d_w == HD+ and d_t % 2 == 0+ and d_h % 2 == 0+ )+++def run_abs_rope(+ x: torch.Tensor,+ rope_dims: "tuple[int, int, int]",+ tables: "tuple[torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor]",+ *,+ out: torch.Tensor | None = None,+) -> torch.Tensor:+ """Rotate every head row of ``x`` by its (t, h, w) angles, into ``out`` (a new tensor by default).+ ``tables`` is ``(cos_t, sin_t, cos_h, sin_h, cos_w, sin_w)``, fp32 ``(span, dim / 2)``+ tables over ``T``, ``H`` and ``W`` of ``x``. Check :func:`abs_rope_supported` first.+ ``out`` may be ``x`` itself: each CTA reads all of its rows into shared memory before it+ writes any of them back. Otherwise it must not overlap ``x``. It must be on ``x``'s device.+ """+ _, t, h, w, nh, _ = x.shape+ rows = x.view(-1, HD)+ if out is None:+ y = torch.empty_like(rows)+ else:+ if out.shape != x.shape or out.dtype != x.dtype or out.device != x.device or not out.is_contiguous():+ raise ValueError(+ f"out must be a contiguous {tuple(x.shape)} {x.dtype} tensor on {x.device}, "+ f"got {tuple(out.shape)} {out.dtype} on {out.device}"+ )+ y = out.view(-1, HD)+ with torch.cuda.device(x.device):+ compiled = KERNEL_CACHE.get(_compile_kernel, x.device, rope_dims=tuple(rope_dims))+ compiled(+ from_dlpack(rows, assumed_align=16).mark_layout_dynamic(),+ from_dlpack(y, assumed_align=16).mark_layout_dynamic(),+ *(from_dlpack(table.contiguous(), assumed_align=4).mark_layout_dynamic() for table in tables),+ rows.shape[0],+ t,+ h,+ w,+ nh,+ )+ return y.view(x.shape)diff --git a/packages/ltx-kernels/src/ltx_kernels/vae/workspace.py b/packages/ltx-kernels/src/ltx_kernels/vae/workspace.pynew file mode 100644index 0000000..aa2c2df--- /dev/null+++ b/packages/ltx-kernels/src/ltx_kernels/vae/workspace.py@@ -0,0 +1,164 @@+"""Reusable device buffers for one tiled DiffVAE decode.+Per-tile allocation of the large stage-5 buffers leaves the caching allocator 25-45 GiB above what is ever live;+:class:`DecodeWorkspace` keeps them for the whole decode as activation volumes leased per tile, uninitialised like+``torch.empty``; without a workspace every helper is a plain one.+"""++from __future__ import annotations++import contextvars+import math+from collections.abc import Iterator+from contextlib import contextmanager+from dataclasses import dataclass++import torch++__all__ = [+ "DecodeWorkspace",+ "active_decode_workspace",+ "decode_workspace",+ "workspace_phase_empty",+ "workspace_phase_reset",+ "workspace_volume",+]++# Phase tensors start on a multiple of this many elements (keeps every carved buffer aligned).+_PHASE_ALIGN_ELEMS = 128++_ACTIVE: contextvars.ContextVar[DecodeWorkspace | None] = contextvars.ContextVar(+ "ltx_kernels_decode_workspace", default=None+)+++@dataclass+class _VolumeSlot:+ buffer: torch.Tensor # flat, one dtype / device+ leased: bool = False+ phase_offset: int = 0 # elements handed out by phase_take since the last phase_reset++ @property+ def capacity(self) -> int:+ return self.buffer.numel()+++class DecodeWorkspace:+ """Per-tile leased volumes and the phase arena carved from the idle ones; see the module docstring."""++ def __init__(self, large_volume_numel: int = 0) -> None:+ """``large_volume_numel``: capacity of every large slot, the decode's biggest volume, so later tiles reuse it.+ A request counts as large from half that size; smaller ones (keyframe planes) get exactly what they ask for.+ """+ self._volumes: list[_VolumeSlot] = []+ self._large_numel = max(0, int(large_volume_numel))++ # -- leased volumes ------------------------------------------------------------------+ def _is_large(self, numel: int) -> bool:+ return bool(self._large_numel) and numel * 2 >= self._large_numel++ def volume(self, numel: int, *, dtype: torch.dtype, device: torch.device) -> torch.Tensor:+ """Lease the smallest free buffer that holds ``numel`` elements, or allocate a new one.+ Large requests use large-capacity slots; small ones never lease a large slot, so they cannot displace one.+ """+ large = self._is_large(numel)+ best: _VolumeSlot | None = None+ for slot in self._volumes:+ buf = slot.buffer+ if slot.leased or buf.dtype != dtype or buf.device != device or slot.capacity < numel:+ continue+ if slot.phase_offset:+ continue+ if self._large_numel and (slot.capacity >= self._large_numel) != large:+ continue+ if best is None or slot.capacity < best.capacity:+ best = slot+ if best is None:+ capacity = max(numel, 1)+ if large:+ capacity = max(capacity, self._large_numel)+ best = _VolumeSlot(torch.empty(capacity, dtype=dtype, device=device))+ self._volumes.append(best)+ best.leased = True+ return best.buffer[:numel]++ def release_tile(self) -> None:+ """End of a tile: every leased volume is free for the next one (nothing is deallocated)."""+ for slot in self._volumes:+ slot.leased = False+ self.phase_reset()++ # -- phase arena: short-lived tensors carved out of the idle volume slots --------------+ def phase_take(+ self,+ numel: int,+ *,+ dtype: torch.dtype,+ device: torch.device,+ not_aliasing: tuple[torch.Tensor, ...] = (),+ ) -> torch.Tensor | None:+ """Bump-allocate ``numel`` elements from a free (unleased) volume slot, or ``None`` if none has room.+ For a phase whose tensors all die before the volumes are next leased; :meth:`phase_reset` frees them all.+ ``not_aliasing``: slots backing these tensors are skipped (a custom op's output may not alias its inputs).+ """+ avoid = {t.untyped_storage().data_ptr() for t in not_aliasing}+ for slot in self._volumes:+ buf = slot.buffer+ if slot.leased or buf.dtype != dtype or buf.device != device:+ continue+ if avoid and buf.untyped_storage().data_ptr() in avoid:+ continue+ offset = slot.phase_offset+ if offset + numel > slot.capacity:+ continue+ slot.phase_offset = offset + (-(-numel // _PHASE_ALIGN_ELEMS)) * _PHASE_ALIGN_ELEMS+ return buf[offset : offset + numel]+ return None++ def phase_reset(self) -> None:+ for slot in self._volumes:+ slot.phase_offset = 0+++def active_decode_workspace() -> DecodeWorkspace | None:+ return _ACTIVE.get()+++@contextmanager+def decode_workspace(workspace: DecodeWorkspace | None) -> Iterator[DecodeWorkspace | None]:+ """Make ``workspace`` the one :func:`workspace_volume` / :func:`workspace_phase_empty` draw from."""+ token = _ACTIVE.set(workspace)+ try:+ yield workspace+ finally:+ _ACTIVE.reset(token)+++def workspace_phase_empty(+ shape: tuple[int, ...],+ *,+ dtype: torch.dtype,+ device: torch.device,+ not_aliasing: tuple[torch.Tensor, ...] = (),+) -> torch.Tensor:+ """``torch.empty(shape)`` carved from the active workspace's idle volumes (see+ :meth:`DecodeWorkspace.phase_take`); a plain ``empty`` without a workspace or when nothing fits."""+ ws = _ACTIVE.get()+ if ws is not None:+ flat = ws.phase_take(math.prod(shape), dtype=dtype, device=device, not_aliasing=not_aliasing)+ if flat is not None:+ return flat.view(shape)+ return torch.empty(shape, dtype=dtype, device=device)+++def workspace_phase_reset() -> None:+ ws = _ACTIVE.get()+ if ws is not None:+ ws.phase_reset()+++def workspace_volume(shape: tuple[int, ...], *, dtype: torch.dtype, device: torch.device) -> torch.Tensor:+ """``torch.empty(shape)`` leased from the active workspace until its tile ends (plain ``empty`` without one)."""+ ws = _ACTIVE.get()+ if ws is None:+ return torch.empty(shape, dtype=dtype, device=device)+ return ws.volume(math.prod(shape), dtype=dtype, device=device).view(shape)diff --git a/packages/ltx-pipelines/CLAUDE.md b/packages/ltx-pipelines/CLAUDE.mdindex 18f70d5..1fcc225 100644--- a/packages/ltx-pipelines/CLAUDE.md+++ b/packages/ltx-pipelines/CLAUDE.md@@ -22,16 +22,18 @@ Pipelines stay at `ltx_pipelines/<name>.py`. Distilled, TI2Vid two-stage, A2VidCLI: on `python -m ltx_pipelines.distilled`, `python -m ltx_pipelines.ti2vid_two_stages` (and their MGPU runners), `python -m ltx_pipelines.a2vid_two_stage`, or `python -m ltx_pipelines.dubit`, pass `--chunk-pixel-frames` (default 97) and/or `--chunk-carry-frames` (default 25) via `add_chunk_layout_args` (`utils/args.py`); `--chunk-blend-frames` controls decoded video seam crossfade length (default: full carry, `0` for a hard cut). When any layout flag is set, `chunk_config_from_args` builds a `ChunkConfig` and the CLI/MGPU driver streams `stream_chunks` through `split_decoded_chunks` into `encode_video`; otherwise it encodes `PipelineOutput` from `__call__`. With no chunk flags, `stream_chunks` / `uniform_chunk_layouts` plans one chunk covering the snapped clip.+TI2Vid two-stage freezes stage-1 audio during stage 2 by default. Pass `refine_audio=True` (CLI: `--refine-audio`)+to denoise both modalities in stage 2, including when replacing the former keyframe-interpolation pipeline.+## Pipeline selection| Pipeline | File | Stages | Model | Sampler | Use case ||----------|------|--------|-------|---------|----------|| `TI2VidOneStagePipeline` | `ti2vid_one_stage.py` | 1 | Full | Euler | Simple text/image-to-video || `T2AOneStagePipeline` | `t2a_one_stage.py` | 1 | Full | Euler | Text-to-audio (audio-only output, no video branch) |-| `TI2VidTwoStagesPipeline` | `ti2vid_two_stages.py` | 2 | Full + distilled LoRA | Euler | Guided two-stage (CFG/STG) |+| `TI2VidTwoStagesPipeline` | `ti2vid_two_stages.py` | 2 | Full + distilled LoRA | Euler | Guided two-stage (CFG/STG), including keyframe interpolation || `TI2VidTwoStagesHQPipeline` | `ti2vid_two_stages_hq.py` | 2 | Full + distilled LoRA (both stages) | Res2s | Guided two-stage, res_2s, fewer steps || `A2VidPipelineTwoStage` | `a2vid_two_stage.py` | 2 | Full + distilled LoRA | Euler | Audio-conditioned video |-| `KeyframeInterpolationPipeline` | `keyframe_interpolation.py` | 2 | Full + distilled LoRA | Euler | Keyframe interpolation || `DFRPipeline` | `dfr_pipeline.py` | 2 (+ optional tiled temporal) | Distilled checkpoint + detailing IC-LoRA stage 2 | Euler ancestral (2.5+) | Production quality (keyframes + detailing + optional temporal x2) || `DistilledPipeline` | `distilled.py` | 2 | Distilled only | Euler ancestral (2.5+) | Fastest inference || `ICLoraPipeline` | `ic_lora.py` | 2 | Distilled only | Euler | Video-to-video with IC-LoRA control |@@ -57,23 +59,23 @@ CLI: on `python -m ltx_pipelines.distilled`, `python -m ltx_pipelines.ti2vid_two## LoRA conventions- No default LoRAs. `loras` param defaults to empty list/tuple. `DEFAULT_LORA_STRENGTH = 1.0`.-- Two-stage non-distilled pipelines require `distilled_lora` (applied to stage 2 only in TI2Vid/A2Vid/Keyframe).+- Two-stage non-distilled pipelines require `distilled_lora` (applied to stage 2 only in TI2Vid/A2Vid).- HQ applies distilled LoRA to **both** stages with separate `distilled_lora_strength_stage_1` / `_stage_2` params.- DFR loads a distilled checkpoint (no distilled LoRA). Stage 2 adds a detailing IC-LoRA via `with_loras`.## Shared building blocks (`utils/blocks.py`)-- `DiffusionStage` -- owns transformer lifecycle; builds model on call, frees on exit via `gpu_model()` context manager (moves params to meta device to release GPU/CPU memory). Accepts an optional `loop` override. `__init__` takes a pre-built transformer builder; pipelines construct it via the `DiffusionStage.from_checkpoint(checkpoint_path, ..., loras=...)` classmethod, which builds the standard (and, when offloading, streaming) builders. `with_builder` / `with_loras` return a new stage with a swapped builder / LoRA set without re-specifying config. ``supports_generated_keyframes`` reads ``use_keyframes_abs_pos_embedding`` from the checkpoint config (no weights loaded). ``__call__`` re-checks slot conditionings via ``_assert_supports_conditionings`` as a backstop for callers that build items directly.+- `DiffusionStage` -- owns transformer lifecycle; builds model on call, frees on exit via `gpu_model()` context manager (moves params to meta device to release GPU/CPU memory), except that a builder whose weights the registry keeps GPU-resident (`keeps_gpu_resident_weights`) under `AllocatorTrimStrategy.DEFER` leaves its build assigned: every build reassigns those resident weights, so a dispose would free nothing. Accepts an optional `loop` override. `__init__` takes a pre-built transformer builder; pipelines construct it via the `DiffusionStage.from_checkpoint(checkpoint_path, ..., loras=...)` classmethod, which builds the standard (and, when offloading, streaming) builders. `with_builder` / `with_loras` return a new stage with a swapped builder / LoRA set without re-specifying config. ``supports_generated_keyframes`` reads ``use_keyframes_abs_pos_embedding`` from the checkpoint config (no weights loaded). ``__call__`` re-checks slot conditionings via ``_assert_supports_conditionings`` as a backstop for callers that build items directly.`__call__` takes `modalities: VideoAudio[ModalitySpec]` and is a latent entry point: each `ModalitySpec` brings the latent to denoise, and that latent is the stage's only source of shape -- pixel dimensions never reach it, and it never sizes a latent itself. A pipeline's *first* stage seeds the latent with `create_initial_video_latent` / `create_initial_audio_latent` / `create_initial_av_latents` (`utils/helpers.py`); every later stage passes the previous stage's output, upscaled or not. So latents are allocated once per run, at the pipeline head. Sizing audio off a playback rate that differs from the video RoPE time base is done by building the audio latent at that rate, not by a stage argument. Returns `VideoAudio`, which unpacks as `(video, audio)`.- `PromptEncoder` -- Gemma text encoder + embeddings processor (video 4096-dim, audio 2048-dim).-- `ImageConditioner` / `AudioConditioner` -- temporary encoder scope; builds encoder, passes to callable, frees. `ImageConditioner` is additionally the single owner of the image-conditioning CRF: `resolve_crf(images)` (called near the top of every pipeline's `__call__`) fills in the H.264 CRF of any `ImageConditioningInput` (`utils.types`) that left it unset, reading it from the checkpoint's `model_version` (`detect_params`, lazily and cached). So omitting `crf` means "use what matches this model" (33 through LTX-2.3, 18 from 2.4), while an explicit `crf` -- including `0` for no re-compression -- is always honoured.+- `ImageConditioner` / `AudioConditioner` -- temporary encoder scope; builds encoder, passes to callable, frees. `ImageConditioner` is additionally the single owner of the image-conditioning CRF: `resolve_crf(images)` fills in the H.264 CRF of any `ImageConditioningInput` (`utils.types`) that left it unset, reading it from the checkpoint's `model_version` (`detect_params`, lazily and cached). So omitting `crf` means "use what matches this model" (33 through LTX-2.3, 18 from 2.4), while an explicit `crf` -- including `0` for no re-compression -- is always honoured. `load_conditionings(images, color_space=None)` (called near the top of every pipeline's `__call__`) resolves the CRF that way and decodes each distinct still once for the run (SDR re-compressed at its CRF; EXR with the run's `color_space`). It returns `LoadedImageConditioning`s (`utils.types`): the decoded still plus the input's `binding` and `strength`, with `frame_idx` / `assert_in_clip` / `map_frame_index` / `to_item`. Everything after it -- chunk and DFR rebasing, stage/tile/window plumbing, `image_conditionings()` -- takes only `LoadedImageConditioning`s; rebasing keeps the still, so encodes only resize per resolution. `ImageConditioningInput` is the user-facing request (path, strength, binding, CRF).- `VideoUpsampler` -- 2x spatial upsampling via encoder + upsampler.- `VideoDecoder` / `AudioDecoder` -- latent-to-pixel decoding (iterator for video, `Audio` for audio). `VideoDecoder` takes a single decoder `checkpoint_path` and `diffvae_optimization` (`DiffVAEMode`, default `CHUNKED_EAGER`); pipelines pass `model_paths.video_vae()`, the same file `ImageConditioner` and `VideoUpsampler` build their encoder from -- a video VAE checkpoint carries encoder and decoder together. Decoder kind (conv vs diffusion) is chosen from checkpoint metadata (`is_diffusion_video_vae`); distilled DiffVAE uses fixed 2 Euler steps. CLI: `--video-vae-path` is the video VAE slot in both monolith (optional override; defaults via `ModelPaths.from_monolith`) and split modes; `--diffvae-optimization` (see `docs/optimization.md#diffusion-vae-decoder` for mode meanings and relative compile/runtime/VRAM factors). Multi-GPU pipelines (`*_mgpu.py`) accept the same flags and wrap the loaded decoder in `DistributedVideoDecoder`.- `DurationPredictor` -- predicts a frame count from `PromptEncoder`'s connector token outputs via `DurationHead` (`ltx_core.duration_head`), snapped to the VAE's temporal grid. Unlike other blocks, it holds the built model directly (`__init__` takes a `DurationHead`, not a builder) since the checkpoint is only a few MB -- no build-on-call/free-on-exit needed. Pipelines construct it via `DurationPredictor.from_checkpoint(checkpoint_path, dtype, device)`, which returns `None` instead of a predictor if the checkpoint has no `duration_head.*` weights (checkpoints predating LTX-2.5 / gemma4). Called with `(video_encoding, audio_encoding, frame_rate=...)` -- either may be `None` but not both. Wired into `TI2VidOneStagePipeline`, `TI2VidTwoStagesPipeline`, `TI2VidTwoStagesHQPipeline`, `DistilledPipeline`, `DFRPipeline`, and `T2AOneStagePipeline` (audio-only: `video_encoding=None`), plus their MGPU runners: when `num_frames` is omitted (`None`), `require_num_frames_source` (called at the top of `__call__`, before any work) raises immediately if no `DurationPredictor` is available, otherwise it's auto-predicted from the caption. Auto-duration resolves inside `__call__`; read the final count from ``PipelineOutput.num_frames`` (or the local ``num_frames`` variable before return).### Memory management-- **Model lifecycle**: All blocks build their model on call and free it on exit. `gpu_model()` moves params to `"meta"` device on exit, immediately releasing storage. No model persists between calls.+- **Model lifecycle**: All blocks build their model on call and free it on exit. `gpu_model()` moves params to `"meta"` device on exit, immediately releasing storage. The one exception is a `DiffusionStage` transformer with registry-resident weights under `DEFER`, which stays assigned (see `DiffusionStage` above); otherwise no model persists between calls.- **Block streaming**: When offloading is enabled, `DiffusionStage` wraps the transformer in `BlockStreamingWrapper`. Blocks live on pinned CPU memory; only 2 blocks are buffered on GPU at a time (one for compute, one for async H2D copy on a separate CUDA stream).- **Batch splitting**: `BatchSplitAdapter` wraps the transformer and splits inputs exceeding `max_batch_size` into sequential chunks. If guidance needs B=4 but `max_batch_size=1`, it runs 4 sequential B=1 passes. Higher `max_batch_size` reduces layer-streaming PCIe transfers at the cost of peak memory.@@ -81,7 +83,7 @@ CLI: on `python -m ltx_pipelines.distilled`, `python -m ltx_pipelines.ti2vid_two- `SimpleDenoiser` -- single forward pass (B=1), no guidance. Used by distilled pipelines and all stage 2.- `GuidedDenoiser` -- CFG/STG with static `MultiModalGuider` instances (HQ, A2Vid, Retake non-distilled).-- `FactoryGuidedDenoiser` -- per-step guider creation via factory (OneStageTI2Vid, TwoStagesTI2Vid, Keyframe).+- `FactoryGuidedDenoiser` -- per-step guider creation via factory (OneStageTI2Vid, TwoStagesTI2Vid).All denoisers return `VideoAudio` of `DenoisedLatentResult` (defined in `utils/types.py`). It still unpacks as `(video_result, audio_result)`; either side may be `None` for absent modalities. `DenoisedLatentResult.denoised` is the final blended tensor. Guided denoisers additionally populate per-pass fields (`.cond`, `.uncond`, `.ptb`, `.mod`) on each result; `SimpleDenoiser` leaves these `None`.@@ -97,9 +99,9 @@ Every video pipeline ``__call__`` returns ``PipelineOutput``: ``video``, ``audio- **HQ**: Res2s second-order sampler for **both** stages, latent-dependent sigma schedule, distilled LoRA on both stages with separate strengths.- **A2Vid**: Audio frozen in both stages (`frozen=True, noise_scale=0.0`). That zeros `denoise_mask` and forces audio `Modality.sigma=0` (prompt AdaLN / a2v gate). Returns original audio (not VAE-decoded); no `AudioDecoder`. Chunked mode seeds each chunk from the encoded source latent (`generate_uniform_chunks(audio_latent=)`) and muxes the original waveform after decode (`replace_chunks_audio`). CLI `--num-frames` and `--audio-max-duration` are mutually exclusive: with `--audio-max-duration`, `num_frames` is omitted (`None`) and derived from the decoded clip (capped by remaining audio after `--audio-start-time`); with `--num-frames` or neither, that frame count is used and audio is clipped to match. Returns ``PipelineOutput`` like the other two-stage pipelines.-- **IC-LoRA**: `VideoConditionByReferenceLatent`, `reference_downscale_factor` / `reference_temporal_scale_factor` from LoRA metadata, `skip_stage_2`, attention mask downsampling. Default stage 2 is LoRA-free (`combined_image_conditionings` only); `apply_ic_lora` / `--stage-2-ic-lora` keeps the reference on a stage. CLI is two stages (`--tile` plus `--tile-height` / `--tile-width` for transformer windows). Longer ladders use `ICLoraPipeline(stages=...)`. Temporal chunking: `stream_chunks()` / `chunk_config` compose the shared chunk operators with per-stage LoRA and `diffusion_stage_with_tiling`; reference video uses `SequentialVideoFrameSource` and chunk-local mask/temporal scaling via `reference_video_conditionings_for_chunk`. CLI `--chunk-pixel-frames` / `--chunk-carry-frames` / `--chunk-blend-frames` stream encode like Distilled/Dub-It.+- **IC-LoRA**: `VideoConditionByReferenceLatent`, `reference_downscale_factor` / `reference_temporal_scale_factor` from LoRA metadata, `skip_stage_2`, attention mask downsampling. Default stage 2 is LoRA-free (`image_conditionings` only); `apply_ic_lora` / `--stage-2-ic-lora` keeps the reference on a stage. CLI is two stages (`--tile` plus `--tile-height` / `--tile-width` for transformer windows). Longer ladders use `ICLoraPipeline(stages=...)`. Temporal chunking: `stream_chunks()` / `chunk_config` compose the shared chunk operators with per-stage LoRA and `diffusion_stage_with_tiling`; reference video uses `SequentialVideoFrameSource` and chunk-local mask/temporal scaling via `reference_video_conditionings_for_chunk`. CLI `--chunk-pixel-frames` / `--chunk-carry-frames` / `--chunk-blend-frames` stream encode like Distilled/Dub-It.- **Dub-It**: Standalone pipeline; IC reference **video** helpers and audio-reference patchify (negative RoPE) in `iclora_utils.py`. Appends frozen audio-reference tokens via `AudioConditionByReferenceLatent` (ltx-core), matching video token order (`[target | ref]`) while keeping reference RoPE positions negative (training-compatible). Single IC-LoRA on both stages; full IC-LoRA video conditioning at stage 1 and 2; stage-2 audio is frozen with S1 latent as initial state and uses S1-derived ref. Final audio decoded from stage 1 latent. Chunked mode walks the IC-LoRA reference video once through `SequentialVideoFrameSource` (overlap lookback) and encodes audio-reference tokens in the chunk factories (stage 2 swaps the audio factory to self-reference via `replace_audio_conditionings`). The Dub-It CLI does not expose `--conditioning-attention-mask`; use `ic_lora.py` if you need spatial IC attention masking.-- **Keyframe**: Uses `image_conditionings_by_adding_guiding_latent` in both stages (all frames as keyframe guidance, no replacement) -- unlike TI2Vid which uses `combined_image_conditionings` (frame_idx=0 replaces, others guide).+- **TI2Vid two-stage (image conditioning)**: Preserves each caller-supplied binding through chunking and DFR rebasing. Use `ImageConditioningInput.replace()` / `.guide()` / `.at()` (or `--image` with optional trailing `replace` / `guide`) explicitly; the pipeline does not rewrite bindings. Keyframe interpolation uses `.guide()` at endpoint frames plus optional `refine_audio=True` (see `ti2vid_two_stages.py`).- **Retake**: `TemporalRegionMask` for selective time-window regeneration. `regenerate_video`/`regenerate_audio` flags. Conditional distilled/full behavior.- **Distilled**: Single `self.stage` reused for both stages (not `stage_1`/`stage_2`). The sampler is resolved in `__init__` from the checkpoint generation -- `should_use_ancestral_sampler(path)` (`detect_model_version(...) >= (2, 5)`) sets `self.use_ancestral_sampler`, so an LTX-2.5+ checkpoint samples every stage with `euler_ancestral_denoising_loop(..., eta=1.0, s_noise=1.0)` (ComfyUI's `sample_euler_ancestral_RF` defaults), and anything older samples with plain Euler. There is no per-call override; assign the attribute before calling to pin a sampler (e.g. to A/B both on one checkpoint). `distilled_mgpu.py` detects per rank from the same checkpoint, so ranks stay symmetric. Each pass has its own noise-seed offset (`ANCESTRAL_NOISE_SEED_OFFSET`, `ANCESTRAL_STAGE_2_NOISE_SEED_OFFSET`), so no two of them inject the same noise and stage 1's first draw is not bit-identical to the initial `GaussianNoiser` noise. Within a pass the seed advances by one on each denoise call, so sequential chunks do not reuse ancestral noise across a seam; a single `DiffusionStage` call is index 0, so one-shot output matches a fixed-seed partial.- **DFR** (Diffusion Fidelity Rendering, `dfr_pipeline.py` orchestrator + `dfr_stages.py` +@@ -203,8 +205,8 @@ Every video pipeline ``__call__`` returns ``PipelineOutput``: ``video``, ``audio## Image conditioning helpers (`utils/helpers.py`)-- `combined_image_conditionings()` -- images with `frame_idx==0` replace latent (`VideoConditionByLatentIndex`), others guide (`VideoConditionByKeyframeIndex`).-- `image_conditionings_by_adding_guiding_latent()` -- all images become keyframe guidance regardless of `frame_idx`.+- `image_conditionings()` -- encodes `LoadedImageConditioning`s using each `binding` (`ImageReplace` / `ImageGuide`). Both bindings carry pixel-frame indices; `ImageReplace` accepts causal latent starts (`0` or `1 + 8 * k`) and converts to a latent index only when lowered to `VideoConditionByLatentIndex`.+- `ImageConditioningInput.at()` -- combined i2v default: frame 0 replaces latent, later frames guide.- `evenly_spaced_keyframe_positions()` -- evenly spaced **interior** positions (both endpoints excluded).- `generated_keyframe_conditionings()` / `resolve_generated_keyframes()` -- turn the pipeline-level ``generated_keyframes`` arg into ``VideoGeneratedKeyframeSlots`` or pixel indices.- `decode_keyframes_from_slots()` -- build ``DecodeKeyframes`` from carried slot latents, dropping planes past a trimmed canvas.diff --git a/packages/ltx-pipelines/README.md b/packages/ltx-pipelines/README.mdindex a088df2..12bbb5b 100644--- a/packages/ltx-pipelines/README.md+++ b/packages/ltx-pipelines/README.md@@ -43,6 +43,7 @@ See [Installation & Usage](docs/installation.md) for full setup, CLI modules, an| [Available Pipelines](docs/pipelines.md) | Full reference for all 12 pipelines, including [Running DFR](docs/pipelines.md#running-dfr) || [Conditioning Types](docs/conditioning.md) | Image and video conditioning methods || [HDR Support](docs/hdr.md) | Native EXR/`--hdr` colour spaces, I/O, and examples |+| [VideoSeal marking](docs/marking.md) | Opt-in post-hoc VideoSeal / AudioSeal (`--marker-root-path`) || [Multimodal Guidance](docs/multimodal-guidance.md) | CFG / STG / modality guidance parameters and tuning || [Optimization Tips](docs/optimization.md) | FP8 quantization, `torch.compile`, gradient estimation || [Multi-GPU Inference](docs/multigpu/README.md) | Run a single generation across GPUs for latency (SP, TDP, distributed VAE, distributed Gemma) |diff --git a/packages/ltx-pipelines/docs/conditioning.md b/packages/ltx-pipelines/docs/conditioning.mdindex 5cb68b2..fab7fb0 100644--- a/packages/ltx-pipelines/docs/conditioning.md+++ b/packages/ltx-pipelines/docs/conditioning.md@@ -4,15 +4,27 @@ Pipelines use different conditioning methods from [`ltx-core`](../../ltx-core/)## Image Conditioning-All pipelines support image conditioning, but with different methods:--- **Replacing Latents** ([`image_conditionings_by_replacing_latent`](../src/ltx_pipelines/utils/helpers.py)):- - Replaces the latent at a specific frame with the encoded image- - Strong control over specific frames--- **Guiding Latents** ([`image_conditionings_by_adding_guiding_latent`](../src/ltx_pipelines/utils/helpers.py)):- - Adds the image as a guiding signal rather than replacing- - Better for smooth interpolation between keyframes+Stills are passed as [`ImageConditioningInput`](../src/ltx_pipelines/utils/types.py) values with an+[`ImageReplace`](../src/ltx_pipelines/utils/types.py) or [`ImageGuide`](../src/ltx_pipelines/utils/types.py)+binding. [`image_conditionings()`](../src/ltx_pipelines/utils/helpers.py) encodes each still and+emits the matching core conditioning item (`VideoConditionByLatentIndex` or+`VideoConditionByKeyframeIndex`).++`ImageConditioningInput` is a pixel-space API: every `frame_idx` is a zero-based index on the+decoded output video's frame timeline, never an index into the VAE latent. This applies equally to+`ImageConditioningInput.at()`, `.replace()`, `.guide()`, `ImageReplace`, `ImageGuide`, and the+`--image` (optional trailing `replace` / `guide` binding keyword) CLI flags. Chunking and temporal rebasing also+map these values in pixel space.++`ImageGuide` accepts any frame in the clip. `ImageReplace` must point to the start of a causal VAE+latent: pixel frame `0`, or `1 + 8 * k` (`1`, `9`, `17`, ...). The pipeline converts that+pixel-frame index to a latent index only when it creates the core conditioning item:++```python+opening = ImageConditioningInput.replace("opening.png", frame_idx=0, strength=1.0)+later_replacement = ImageConditioningInput.replace("later.png", frame_idx=9, strength=1.0)+guide = ImageConditioningInput.guide("guide.png", frame_idx=80, strength=0.8)+```## Video Conditioningdiff --git a/packages/ltx-pipelines/docs/installation.md b/packages/ltx-pipelines/docs/installation.mdindex 44e1005..cde19ef 100644--- a/packages/ltx-pipelines/docs/installation.md+++ b/packages/ltx-pipelines/docs/installation.md@@ -77,7 +77,6 @@ what pipeline constructors take.- `ltx_pipelines.ti2vid_one_stage` - Single-stage text/image-to-video. ([docs](pipelines.md#3-ti2vidonestagepipeline), [source](../src/ltx_pipelines/ti2vid_one_stage.py))- `ltx_pipelines.t2a_one_stage` - Single-stage text-to-audio (audio-only output). ([docs](pipelines.md#11-t2aonestagepipeline), [source](../src/ltx_pipelines/t2a_one_stage.py))- `ltx_pipelines.ic_lora` - Video-to-video with IC-LoRA. ([docs](pipelines.md#5-iclorapipeline), [source](../src/ltx_pipelines/ic_lora.py))-- `ltx_pipelines.keyframe_interpolation` - Keyframe interpolation. ([docs](pipelines.md#6-keyframeinterpolationpipeline), [source](../src/ltx_pipelines/keyframe_interpolation.py))- `ltx_pipelines.a2vid_two_stage` - Audio-to-video generation conditioned on an input audio. ([docs](pipelines.md#7-a2vidpipelinetwostage), [source](../src/ltx_pipelines/a2vid_two_stage.py))- `ltx_pipelines.retake` - Regenerate a time region of an existing video. ([docs](pipelines.md#8-retakepipeline), [source](../src/ltx_pipelines/retake.py))- `ltx_pipelines.hdr_ic_lora` - Video-to-video SDR→HDR: writes a BT.2020/HLG master plus an EXR sequence (`--exr-colorspace`, default ACEScg). ([docs](pipelines.md#9-hdriclorapipeline), [source](../src/ltx_pipelines/hdr_ic_lora.py))@@ -99,6 +98,7 @@ These flags are shared across the pipeline CLIs (they come from a common base pa- `--lora <path> [strength]` - apply a LoRA (repeatable; default strength 1.0).- `--enhance-prompt` - rewrite the prompt with the built-in enhancer before generation.- `--hdr {SRGB_LINEAR,ACESCG,ACESCCT}` - on pipelines that accept image / video conditioning (not Dub-It, HDR IC-LoRA, or T2A): declare the source colour space for EXR stills / EXR-frame folders. Required whenever any EXR input is passed; omit for SDR. With `--hdr` set, encode writes half EXR frames plus a BT.2020/HLG master. See [HDR Support](hdr.md).+- `--marker-root-path <dir>` - optional local `path/to/marking` pack for post-hoc VideoSeal (SDR pixels) and AudioSeal (stereo PCM). Requires `uv sync --extra videoseal --extra audioseal`. HDR encode skips video stamping and still stamps audio. See [VideoSeal marking](marking.md).- `--video-vae-path <path>` - video VAE `.safetensors` (encoder + decoder). Split: the `vae/` component. Monolith: optional override of the VAE bundled in `--checkpoint-path`/`--distilled-checkpoint-path`. When it is a diffusion VAE, decoding builds a `DiffusionVideoDecoder`, which for best performance requires the `natten` extra. See [Optimization Tips](optimization.md#diffusion-vae-decoder).- `--diffvae-optimization {chunked_eager,chunked_compile,combined_compile,blackwell_dsl}` - DiffVAE decode preset (ignored for convolutional VAEs). `blackwell_dsl` is the fast path on datacenter Blackwell.- `--chunk-pixel-frames` / `--chunk-carry-frames` - on `distilled`, `ti2vid_two_stages` (and their MGPU runners), `a2vid_two_stage`, and `dubit`: pass either flag to generate in windows instead of one full-clip chunk. Defaults are 97 pixel frames and 25-pixel carry when the other is set. Pixel frame counts must be on the causal grid (`8k+1`). Off-grid `--num-frames` is floored to the causal grid.diff --git a/packages/ltx-pipelines/docs/marking.md b/packages/ltx-pipelines/docs/marking.mdnew file mode 100644index 0000000..cc422c4--- /dev/null+++ b/packages/ltx-pipelines/docs/marking.md@@ -0,0 +1,95 @@+# VideoSeal marking++Opt-in post-hoc VideoSeal on SDR H.264 encode and AudioSeal on stereo PCM. These are **not** cryptographic signatures; `detected` is each marker's own call (VideoSeal: bit-test `log10(p) < -6`; AudioSeal: presence score). `bit_accuracy` / `p_value` are the binomial test of decoded bits vs the baked payload.++## Install++```bash+uv sync --extra videoseal --extra audioseal+# or+pip install 'ltx-pipelines[videoseal,audioseal]'+```++PyPI `videoseal` 1.0.1 predates the PixelSeal card, so the extra installs videoseal from git. AudioSeal is PyPI `audioseal` 0.2+.++The library does not download weights. Pass a local `path/to/marking` pack directory on `--marker-root-path`. Expected files (all required):++- `pixelseal.pth`+- `pixelseal.yaml`+- `videoseal_attenuation.yaml`+- `audioseal_generator.pth`+- `audioseal_detector.pth`++## CLI++Video-generation and editing CLIs, HDR IC-LoRA, and text-to-audio (`t2a_one_stage`) accept `--marker-root-path DIR`. Omit the flag for unchanged pixels / PCM (`markers_from_args` returns `(None, None)`). HDR encode skips VideoSeal (logs that video marking is unavailable with HDR) and still stamps AudioSeal PCM before AAC/HLG mux. A2Vid two-stage stamps video only (source audio is muxed unmarked). Text-to-audio uses the same flag and only the audio marker (the pack still loads both).++```bash+python -m ltx_pipelines.distilled \+ ... \+ --marker-root-path path/to/marking \+ --output-path out.mp4++python -m ltx_pipelines.t2a_one_stage \+ ... \+ --marker-root-path path/to/marking \+ --output-path out.wav+```++Detect from Python with a loaded marker (there is no inspect CLI):++```python+import torch+from ltx_pipelines.marking import create_markers, detect_audio_mark, detect_video_mark++video_marker, audio_marker = create_markers("path/to/marking", torch.device("cuda"))+print(detect_video_mark(video_marker, "out.mp4"))+print(detect_audio_mark(audio_marker, "out.mp4"))+```++## Library++`encode_video(..., video_marker=None, audio_marker=None)` leaves pixels and PCM unchanged. Pass resident markers; encode does not load weights. HDR callers omit `video_marker`. A2Vid callers omit `audio_marker`. Each yielded `Audio` is stamped independently (encode does not stitch). `encode_audio(..., audio_marker=None)` is the WAV path.++From a pack (`create` loads, `eval()`s, drops grad, and moves to `device`):++```python+import torch+from ltx_pipelines.marking import create_markers+from ltx_pipelines.utils.media_io import encode_video++device = torch.device("cuda")+video_marker, audio_marker = create_markers("path/to/marking", device)+with torch.inference_mode():+ encode_video(..., video_marker=video_marker, audio_marker=audio_marker)+```++Without a pack, construct the seal classes from file paths:++```python+import torch+from ltx_pipelines.marking.audio_seal import AudioSealMarker+from ltx_pipelines.marking.video_seal import VideoSealMarker++device = torch.device("cuda")+video_marker = VideoSealMarker.create(+ "path/to/marking/pixelseal.pth",+ card="path/to/marking/pixelseal.yaml",+ attenuation_config="path/to/marking/videoseal_attenuation.yaml",+ device=device,+)+audio_marker = AudioSealMarker.create(+ "path/to/marking/audioseal_generator.pth",+ "path/to/marking/audioseal_detector.pth",+ device=device,+)++with torch.inference_mode():+ encode_video(..., video_marker=video_marker, audio_marker=audio_marker)+```++`VideoMarker.embed` stamps one `[F, H, W, C]` RGB tensor in `[0, 1]`. `VideoMarker.detect` takes an iterator of `[H, W, C]` frames in the same domain. `AudioMarker.embed` / `AudioMarker.detect` take `Audio`. File I/O is `detect_video_mark` / `detect_audio_mark` in `ltx_pipelines.marking`.++Pack load uses the local card YAML (not packaged `videoseal/cards/`). Stereo is stamped as a batch of two mono channels (no downmix). Native 24/48 kHz; AudioSeal 0.2 does not resample.++Public API in `ltx_pipelines.marking` is the `VideoMarker` / `AudioMarker` protocols, `create_markers`, `MarkDetectResult`, and detect helpers. `VideoSealMarker.create` / `AudioSealMarker.create` live in `ltx_pipelines.marking.video_seal` and `ltx_pipelines.marking.audio_seal` (not re-exported, so importing `ltx_pipelines.marking` does not require extras) and return those protocols. Filename constants for the pack (`PIXELSEAL_CHECKPOINT_NAME`, …) live on the root helper.diff --git a/packages/ltx-pipelines/docs/multigpu/controller.md b/packages/ltx-pipelines/docs/multigpu/controller.mdindex b78dfd4..966dfe4 100644--- a/packages/ltx-pipelines/docs/multigpu/controller.md+++ b/packages/ltx-pipelines/docs/multigpu/controller.md@@ -17,12 +17,17 @@ MGPUController(num_gpus: int | None = None, # GPUs 0..num_gpus-1 (default: all visible)devices: Sequence[int] | None = None, # place on specific physical GPUs, e.g. [2, 3]logs_specs: LogsSpecs | None = None,+ cpu_threads_per_rank: int | None = 8, # intra-op CPU threads per rank (None: torch's default))````num_gpus` and `devices` are mutually exclusive. `devices=[2, 3]` puts rank r on`cuda:devices[r]`, so two controllers can share one machine on disjoint GPU sets.+`cpu_threads_per_rank` sets every rank's intra-op CPU threads with `torch.set_num_threads`;+`None` leaves torch's default. Ranks run only small host tensor ops, which a pool as wide as+the machine stalls (tens to hundreds of ms per op on a 176-CPU host).+Lifecycle:| Method | What it does |diff --git a/packages/ltx-pipelines/docs/multigpu/gemma.md b/packages/ltx-pipelines/docs/multigpu/gemma.mdindex ac022bb..bca65aa 100644--- a/packages/ltx-pipelines/docs/multigpu/gemma.md+++ b/packages/ltx-pipelines/docs/multigpu/gemma.md@@ -14,6 +14,18 @@ lightweight `AccelerateGemmaWrapper` stub that receives the encoded embeddings oNCCL. The source rank fuses all prompts into one Gemma call, then broadcasts eachoutput.+Gemma's shards share GPUs with the other ranks, so the non-source ranks must not+post their NCCL receives while Gemma is still running: a receive kernel spins on+its GPU, and the two processes then time-slice it. Each builder therefore owns a+`BroadcastCoordinator` (`ltx_core.multigpu.broadcast`) on the default store and+hands it to every wrapper it builds. The source rank synchronizes the devices+holding Gemma's outputs and signals; the other ranks poll the store every 1 ms on+the host and only then receive. `enhance_*` results are gated the same way.+A waiting rank raises `TimeoutError` after 10 minutes (NCCL's default collective+timeout), e.g. when the source rank failed before broadcasting. Construct the+builders on every rank in `setup()`, in the same order and before any job runs:+each builder's store key comes from its construction order.+The first `build()` loads Gemma via `load_gemma_with_device_map` (directory`from_pretrained(device_map="auto")`, or single-file TE with the same balancedplacement) and caches the full state dict (including non-persistent buffers) indiff --git a/packages/ltx-pipelines/docs/optimization.md b/packages/ltx-pipelines/docs/optimization.mdindex c11891c..a857840 100644--- a/packages/ltx-pipelines/docs/optimization.md+++ b/packages/ltx-pipelines/docs/optimization.md@@ -217,12 +217,12 @@ That `natten` extra pins `natten==0.21.7+torch2130cu132` with `torch==2.13.0` (c| stage-5 VRAM coefficient | 5 | 7 | 11 | 2.5 || withheld reserve | 1 GiB | 2 GiB | 2 GiB | 2 GiB || `joint_na3d`, with keyframes | Triton > pure-torch | Triton > pure-torch | Triton > pure-torch | `na_attention_joint_dsl` |-| stage-5 VRAM coefficient | 5 | **5** | **15** | 2.5 |+| stage-5 VRAM coefficient | 5 | **5** | **15** | **4.0** || withheld reserve | 1 GiB (2 on MATH SDPA) | 1 GiB (2 on MATH SDPA) | 1 GiB (2 on MATH SDPA) | 2 GiB |-A keyframe decode is **not** compiled -- the compiled entry points are the keyframe-free ones -- so its diff blocks run eager whatever the mode asked for, and the budget follows that: `chunked_compile` drops to the `chunked_eager` coefficient, and `combined_compile` needs **15** rather than 11 because it holds several live full-volume intermediates where the compiled path holds one fused buffer. On a host whose joint attention falls back to torch's MATH SDPA kernel (no CUDA, e.g. MPS) the reserve grows by a GiB, since that kernel materializes its score block.+A keyframe decode is **not** compiled -- the compiled entry points are the keyframe-free ones -- so its diff blocks run eager whatever the mode asked for, and the budget follows that: `chunked_compile` drops to the `chunked_eager` coefficient, and `combined_compile` needs **15** rather than 11 because it holds several live full-volume intermediates where the compiled path holds one fused buffer. `blackwell_dsl` serves keyframes with its fused joint kernel, but the keyframe decode's eager det blocks and keyframe planes raise its measured working set to **4.0**. On a host whose joint attention falls back to torch's MATH SDPA kernel (no CUDA, e.g. MPS) the reserve grows by a GiB, since that kernel materializes its score block.-Pipeline `AUTO_TILING` resolves via `tiling_config_for_vae`: for a **Conv VAE** that is aspect-coupled long-side `768/64` spatial tiles plus temporal `80/24` (`TileSizeConfig.from_long_side`). For **DiffVAE**, recommended decode tiling enumerates legal per-axis tile **sizes** on the VAE grid (via the same `split_by_size` path decode uses) and emits a `TileSizeConfig` with independent `frames` / `height` / `width` `DimensionSizeConfig` values. Tile sizes are floored at ``2xOverlap`` so blend masks stay complementary. Stage-5 coefficients and reserves come from the grid above; `tiling_config_for_vae(..., keyframes=True)` selects the keyframe column, and a `chunked_compile` remapped to the Triton/eager fallback uses the `chunked_eager` row.+Pipeline `AUTO_TILING` resolves via `tiling_config_for_vae`: for a **Conv VAE** that is aspect-coupled long-side `768/64` spatial tiles plus temporal `80/24` (`TileSizeConfig.from_long_side`). For **DiffVAE**, recommended decode tiling enumerates legal per-axis tile **sizes** on the VAE grid (via the same `split_by_size` path decode uses) and emits a `TileSizeConfig` with independent `frames` / `height` / `width` `DimensionSizeConfig` values. Tile sizes are floored at ``2xOverlap`` so blend masks stay complementary. The resolved tile is logged at INFO. Stage-5 coefficients and reserves come from the grid above; `tiling_config_for_vae(..., keyframes=True)` selects the keyframe column, and a `chunked_compile` remapped to the Triton/eager fallback uses the `chunked_eager` row.**DFR** always keyframe-decodes (`decode_video(keyframes=)`), so `AUTO_TILING` uses the **keyframe** VRAM column. At 4K with temporal rounds, decode is usually the memory cliff, not the denoise; there is no decode-tile CLI flag. `--spatial-upscalings 2` keeps stage-2 denoise at half resolution. Each temporal tile reloads the transformer. See [Running DFR](pipelines.md#running-dfr).diff --git a/packages/ltx-pipelines/docs/pipeline-selection.md b/packages/ltx-pipelines/docs/pipeline-selection.mdindex 9ccd3b1..8455785 100644--- a/packages/ltx-pipelines/docs/pipeline-selection.md+++ b/packages/ltx-pipelines/docs/pipeline-selection.md@@ -19,7 +19,7 @@ Do you need to condition on existing images/videos?├─ YES → Do you have reference videos for video-to-video?│ ├─ YES → Use ICLoraPipeline│ └─ NO → Do you have multiple keyframe images to interpolate?-│ ├─ YES → Use KeyframeInterpolationPipeline+│ ├─ YES → Use TI2VidTwoStagesPipeline with `--image … guide` bindings│ └─ NO → Image-to-video (one still)│ ├─ Production quality / higher fps → DFRPipeline│ └─ Guided CFG/STG two-stage → TI2VidTwoStagesPipeline@@ -43,11 +43,10 @@ Do you need to condition on existing images/videos?| -------- | ------ | --- | ---------- | ------------- | -------- || [**DistilledPipeline**](pipelines.md#4-distilledpipeline) | 2 | ❌ | ✅ | Image | Fastest inference (starting point) || [**DFRPipeline**](pipelines.md#12-dfrpipeline) | 2 (+ optional temporal rounds) | ❌ | ✅ spatial + optional temporal 2x | Image + [generated keyframes](conditioning.md#generated-keyframe-slots) | **Production quality**; higher effective frame rate |-| [**TI2VidTwoStagesPipeline**](pipelines.md#1-ti2vidtwostagespipeline) | 2 | ✅ | ✅ | Image | Guided two-stage with CFG/STG |+| [**TI2VidTwoStagesPipeline**](pipelines.md#1-ti2vidtwostagespipeline) | 2 | ✅ | ✅ | Image + keyframes | Guided two-stage with CFG/STG; animation and interpolation || [**TI2VidTwoStagesHQPipeline**](pipelines.md#2-ti2vidtwostageshqpipeline) | 2 | ✅ | ✅ | Image | Same guided flow, res_2s sampler || [**TI2VidOneStagePipeline**](pipelines.md#3-ti2vidonestagepipeline) | 1 | ✅ | ❌ | Image | Educational, prototyping || [**ICLoraPipeline**](pipelines.md#5-iclorapipeline) | 2 | ✅ | ✅ | Image + Video | Video-to-video transformations |-| [**KeyframeInterpolationPipeline**](pipelines.md#6-keyframeinterpolationpipeline) | 2 | ✅ | ✅ | Keyframes | Animation, interpolation || [**A2VidPipelineTwoStage**](pipelines.md#7-a2vidpipelinetwostage) | 2 | ✅ | ✅ | Audio + Image | Audio-driven video generation || [**RetakePipeline**](pipelines.md#8-retakepipeline) | 1 | ✅ | ❌ | Source Video | Regenerating a time region of a video || [**HDRICLoraPipeline**](pipelines.md#9-hdriclorapipeline) | 1 | ❌ | ❌ | Video | SDR-to-HDR with an HDR IC-LoRA (HLG master + EXR via `--exr-colorspace`); see also native [`--hdr`](hdr.md) |diff --git a/packages/ltx-pipelines/docs/pipelines.md b/packages/ltx-pipelines/docs/pipelines.mdindex 5ca9213..9b39bd1 100644--- a/packages/ltx-pipelines/docs/pipelines.md+++ b/packages/ltx-pipelines/docs/pipelines.md@@ -6,15 +6,22 @@ Full reference for each pipeline. See the [Pipeline Selection Guide](pipeline-se## 1. TI2VidTwoStagesPipeline-**Best for:** Guided two-stage text/image-to-video with CFG/STG and 2x upsampling.+**Best for:** Guided two-stage text/image-to-video and keyframe interpolation with CFG/STG and 2x upsampling.**Source**: [`src/ltx_pipelines/ti2vid_two_stages.py`](../src/ltx_pipelines/ti2vid_two_stages.py)-Two-stage generation: Stage 1 generates low-resolution video with [multimodal guidance](multimodal-guidance.md), Stage 2 upsamples to 2x resolution with distilled LoRA refinement. Supports image conditioning. Slower than DistilledPipeline; for production quality use [`DFRPipeline`](#12-dfrpipeline).+Two-stage generation: Stage 1 generates low-resolution video with [multimodal guidance](multimodal-guidance.md),+Stage 2 upsamples to 2x resolution with distilled LoRA refinement. Supports image conditioning, including+[keyframe interpolation](#6-keyframe-interpolation) through explicit keyframe-guide bindings. Slower than+DistilledPipeline; for production quality use [`DFRPipeline`](#12-dfrpipeline).-**Extra CLI arguments:** `--chunk-pixel-frames` (default 97) and/or `--chunk-carry-frames` (default 25) enable long-video windowing; `--chunk-blend-frames` controls the decoded seam crossfade (default: full carry, `0` for a hard cut). Also on `ltx_pipelines.ti2vid_two_stages_mgpu`.+**Extra CLI arguments:** `--chunk-pixel-frames` (default 97) and/or `--chunk-carry-frames` (default 25) enable+long-video windowing; `--chunk-blend-frames` controls the decoded seam crossfade (default: full carry, `0` for a hard cut).+`--refine-audio` denoises audio again in stage 2 instead of preserving stage-1 audio. These flags are also available on+`ltx_pipelines.ti2vid_two_stages_mgpu`.-**Use when:** You want CFG/STG-guided two-stage text/image-to-video, not the DFR detailing path.+**Use when:** You want CFG/STG-guided two-stage text/image-to-video or keyframe interpolation, not the DFR detailing+path.---@@ -115,13 +122,19 @@ python -m ltx_pipelines.ic_lora \----## 6. KeyframeInterpolationPipeline+## 6. Keyframe interpolation**Best for:** Generating videos by interpolating between keyframe images.-**Source**: [`src/ltx_pipelines/keyframe_interpolation.py`](../src/ltx_pipelines/keyframe_interpolation.py)+Use [`TI2VidTwoStagesPipeline`](#1-ti2vidtwostagespipeline) with explicit keyframe-guide image bindings. In Python,+construct each image with `ImageConditioningInput.guide(path, frame_idx, strength)`, where `frame_idx` is a+zero-based pixel-frame index on the decoded output timeline; on the CLI, pass each image as+`--image PATH FRAME_IDX STRENGTH guide [CRF]` and add `--refine-audio`. The CLI index uses the same pixel-frame+unit.-Two-stage generation with keyframe interpolation. Uses guiding latents (additive conditioning) instead of replacing latents for smoother transitions. [Multimodal guidance](multimodal-guidance.md) in stage 1, upsampling in stage 2.+The two-stage pipeline uses guiding latents (additive conditioning) for these inputs, with+[multimodal guidance](multimodal-guidance.md) in stage 1 and upsampling in stage 2. Pass `refine_audio=True` to+also refine audio in stage 2, preserving the former keyframe-interpolation behavior.**Use when:** You have keyframe images and want to interpolate between them, creating smooth transitions, or animation/motion interpolation tasks.diff --git a/packages/ltx-pipelines/pyproject.toml b/packages/ltx-pipelines/pyproject.tomlindex 767bd72..618a46d 100644--- a/packages/ltx-pipelines/pyproject.toml+++ b/packages/ltx-pipelines/pyproject.toml@@ -1,11 +1,20 @@[project]name = "ltx-pipelines"-version = "1.4.2"+version = "1.5.0"description = "Pipelines implementation for Lightricks' LTX-2 model"readme = "README.md"requires-python = ">=3.10"dependencies = ["ltx-core", "av", "tqdm", "pillow", "openimageio", "cloudpickle>=3.1"]+[project.optional-dependencies]+# Videoseal 1.0.1 on PyPI predates PixelSeal; pin a commit that includes pixelseal.yaml.+videoseal = [+ "videoseal @ git+https://github.com/facebookresearch/videoseal.git@870ca7fb33578b90f14c602016b6c2788096226e",+]+audioseal = [+ "audioseal>=0.2,<1",+]+[build-system]requires = ["uv_build>=0.9.8,<0.10.0"]build-backend = "uv_build"diff --git a/packages/ltx-pipelines/src/ltx_pipelines/__init__.py b/packages/ltx-pipelines/src/ltx_pipelines/__init__.pyindex 4a6af55..9ecb4f1 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/__init__.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/__init__.py@@ -7,7 +7,6 @@ This package provides ready-to-use pipelines for video generation:- DistilledPipeline: Fast distilled two-stage generation- DubItPipeline: Dub-It with IC-LoRA and audio conditioning- ICLoraPipeline: Image/video conditioning with distilled LoRA-- KeyframeInterpolationPipeline: Keyframe-based video interpolation- DFRPipeline: keyframe slots → spatial detailing → optional tiled temporal x2 (Diffusion Fidelity Rendering)- RetakePipeline: Regenerate a time region (retake) of an existing videoFor more detailed components and utilities, import from specific submodules@@ -27,7 +26,6 @@ if TYPE_CHECKING:from ltx_pipelines.distilled import DistilledPipelinefrom ltx_pipelines.dubit import DubItPipelinefrom ltx_pipelines.ic_lora import ICLoraPipeline- from ltx_pipelines.keyframe_interpolation import KeyframeInterpolationPipelinefrom ltx_pipelines.retake import RetakePipelinefrom ltx_pipelines.t2a_one_stage import T2AOneStagePipelinefrom ltx_pipelines.ti2vid_one_stage import TI2VidOneStagePipeline@@ -42,7 +40,6 @@ _EXPORTS = {"DistilledPipeline": "ltx_pipelines.distilled","DubItPipeline": "ltx_pipelines.dubit","ICLoraPipeline": "ltx_pipelines.ic_lora",- "KeyframeInterpolationPipeline": "ltx_pipelines.keyframe_interpolation","RetakePipeline": "ltx_pipelines.retake","T2AOneStagePipeline": "ltx_pipelines.t2a_one_stage","TI2VidOneStagePipeline": "ltx_pipelines.ti2vid_one_stage",@@ -55,7 +52,6 @@ __all__ = ["DistilledPipeline","DubItPipeline","ICLoraPipeline",- "KeyframeInterpolationPipeline","PipelineOutput","RetakePipeline","T2AOneStagePipeline",diff --git a/packages/ltx-pipelines/src/ltx_pipelines/a2vid_two_stage.py b/packages/ltx-pipelines/src/ltx_pipelines/a2vid_two_stage.pyindex 744adb7..3f1b9cf 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/a2vid_two_stage.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/a2vid_two_stage.py@@ -42,6 +42,7 @@ from ltx_pipelines.utils.args import (add_keyframe_decode_arg,chunk_config_from_args,default_2_stage_arg_parser,+ markers_from_args,resolve_cli_params,)from ltx_pipelines.utils.blocks import (@@ -75,7 +76,13 @@ from ltx_pipelines.utils.media_io import (vae_dtype_for_hdr,)from ltx_pipelines.utils.model_paths import ModelPaths-from ltx_pipelines.utils.types import ImageConditioningInput, OffloadMode, PipelineOutput, VideoAudio+from ltx_pipelines.utils.types import (+ ImageConditioningInput,+ LoadedImageConditioning,+ OffloadMode,+ PipelineOutput,+ VideoAudio,+)logger = logging.getLogger(__name__)@@ -90,7 +97,7 @@ class _A2VidRunContext:noiser: GaussianNoiserdtype: torch.dtypevae_dtype: torch.dtype- images: list[ImageConditioningInput]+ images: list[LoadedImageConditioning]v_context_p: torch.Tensora_context_p: torch.Tensorv_context_n: torch.Tensor@@ -207,6 +214,7 @@ class A2VidPipelineTwoStage:num_frames: int | None,frame_rate: float,images: list[ImageConditioningInput],+ color_space: EXRColorSpace | None,audio_path: str,audio_start_time: float,audio_max_duration: float | None,@@ -219,7 +227,8 @@ class A2VidPipelineTwoStage:chunk_config: ChunkConfig | None = None,) -> _A2VidRunContext:"""Validate inputs, decode/encode audio, and resolve frame count / tiling before chunk planning."""- images = self.image_conditioner.resolve_crf(images)+ enhance_prompt_image = images[0].path if images else None+ images = self.image_conditioner.load_conditionings(images, color_space)assert_resolution(height=height, width=width, is_two_stage=True)assert_generated_keyframes_request(decode_with_keyframes, generated_keyframes, self.stage_1)@@ -250,7 +259,7 @@ class A2VidPipelineTwoStage:[prompt, negative_prompt],enhance_first_prompt=enhance_prompt,enhance_static_cache=enhance_static_cache,- enhance_prompt_image=images[0][0] if len(images) > 0 else None,+ enhance_prompt_image=enhance_prompt_image,)v_context_p, a_context_p = ctx_p.video_encoding, ctx_p.audio_encodingv_context_n, _ = ctx_n.video_encoding, ctx_n.audio_encoding@@ -300,9 +309,8 @@ class A2VidPipelineTwoStage:def _video_conditionings(self,- images: list[ImageConditioningInput],+ images: list[LoadedImageConditioning],num_frames: int,- color_space: EXRColorSpace | None,) -> MakeVideoConditionings:assert_image_frames_in_clip(images, num_frames)@@ -314,7 +322,6 @@ class A2VidPipelineTwoStage:chunk,images=images,video_encoder=enc,- color_space=color_space,))@@ -415,6 +422,7 @@ class A2VidPipelineTwoStage:num_frames=num_frames,frame_rate=frame_rate,images=images,+ color_space=color_space,audio_path=audio_path,audio_start_time=audio_start_time,audio_max_duration=audio_max_duration,@@ -459,7 +467,7 @@ class A2VidPipelineTwoStage:config=ctx.chunk_config,video_scale_factors=self.stage_1.video_scale_factors,audio_latent=ctx.encoded_audio_latent,- make_video_conditionings=self._video_conditionings(ctx.images, ctx.num_frames, color_space),+ make_video_conditionings=self._video_conditionings(ctx.images, ctx.num_frames),generated_keyframes=generated_keyframes,)base = denoise_chunks(@@ -590,6 +598,8 @@ def main() -> None:params = resolve_cli_params()parser = add_chunk_layout_args(add_keyframe_decode_arg(add_generated_keyframes_arg(build_a2vid_arg_parser(params))))args = parser.parse_args()+ hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)+ video_marker, _ = markers_from_args(args, device=get_device(), color_space=hdr)num_frames, audio_max_duration = resolve_a2vid_duration_or_exit(parser, args, default_num_frames=params.num_frames)pipeline = A2VidPipelineTwoStage(model_paths=args.model_paths,@@ -602,7 +612,6 @@ def main() -> None:prompt_enhancer_gemma_root=args.prompt_enhancer_gemma_root,diffvae_optimization=args.diffvae_optimization,)- hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)chunk_config = chunk_config_from_args(args)video_guider_params = MultiModalGuiderParams(@@ -656,6 +665,7 @@ def main() -> None:video_chunks_number=video_chunks_number,color_space=hdr,audio_sampling_rate=audio_sampling_rate,+ video_marker=video_marker,)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/alpha_gen.py b/packages/ltx-pipelines/src/ltx_pipelines/alpha_gen.pyindex dc3e0c5..6fe05fe 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/alpha_gen.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/alpha_gen.py@@ -37,10 +37,10 @@ from ltx_pipelines.utils.blocks import (from ltx_pipelines.utils.denoisers import FactoryGuidedDenoiserfrom ltx_pipelines.utils.helpers import (assert_resolution,- combined_image_conditionings,create_initial_video_latent,ensure_tiling_config,get_device,+ image_conditionings,tiling_scale_factors_for_vae,)from ltx_pipelines.utils.media_io import EXRColorSpace, encode_video, resolve_hdr_color_space, vae_dtype_for_hdr@@ -49,6 +49,7 @@ from ltx_pipelines.utils.types import (DEFAULT_AUTO_DURATION,AutoDuration,ImageConditioningInput,+ LoadedImageConditioning,ModalitySpec,OffloadMode,PipelineOutput,@@ -128,7 +129,7 @@ class AlphaGenPipeline:if not video_conditioning:raise ValueError("Alpha-gen requires at least one reference video")require_num_frames_source(num_frames, self.duration_predictor)- images = self.image_conditioner.resolve_crf(images)+ images = self.image_conditioner.load_conditionings(images, color_space)assert_resolution(height=height, width=width, is_two_stage=False)generator = torch.Generator(device=self.device).manual_seed(seed)@@ -189,21 +190,20 @@ class AlphaGenPipeline:def _conditionings(self,video_encoder: VideoEncoder,- images: list[ImageConditioningInput],+ images: list[LoadedImageConditioning],video_conditioning: list[tuple[str, float]],height: int,width: int,num_frames: int,color_space: EXRColorSpace | None,) -> list[ConditioningItem]:- conditionings = combined_image_conditionings(+ conditionings = image_conditionings(images=images,height=height,width=width,video_encoder=video_encoder,dtype=self.dtype,device=self.device,- color_space=color_space,)append_ic_lora_reference_video_conditionings(conditionings,diff --git a/packages/ltx-pipelines/src/ltx_pipelines/chunks/conditionings.py b/packages/ltx-pipelines/src/ltx_pipelines/chunks/conditionings.pyindex 883be7d..6dbc661 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/chunks/conditionings.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/chunks/conditionings.py@@ -26,48 +26,46 @@ from ltx_pipelines.iclora_utils import (append_ic_lora_reference_video_conditionings,patchify_audio_reference_latent,)-from ltx_pipelines.utils.helpers import combined_image_conditionings+from ltx_pipelines.utils.helpers import image_conditioningsfrom ltx_pipelines.utils.media_io import is_exr_dirfrom ltx_pipelines.utils.media_io.color_config import EXRColorSpace-from ltx_pipelines.utils.types import ImageConditioningInput+from ltx_pipelines.utils.types import LoadedImageConditioning-def assert_image_frames_in_clip(images: Sequence[ImageConditioningInput], num_frames: int) -> None:- """Raise if any image ``frame_idx`` lies outside the planned stitched clip."""+def assert_image_frames_in_clip(images: Sequence[LoadedImageConditioning], num_frames: int) -> None:+ """Raise if any image binding index lies outside the planned stitched clip."""for image in images:- if image.frame_idx < 0 or image.frame_idx >= num_frames:- raise ValueError(f"frame_idx={image.frame_idx} is outside the planned stitched range [0, {num_frames})")+ image.assert_in_clip(num_frames)-def images_for_chunk(chunk: Chunk, images: Sequence[ImageConditioningInput]) -> list[ImageConditioningInput]:+def images_for_chunk(chunk: Chunk, images: Sequence[LoadedImageConditioning]) -> list[LoadedImageConditioning]:"""Remap clip-level image inputs onto ``chunk``'s stitched kept window."""- return [- image._replace(frame_idx=local)- for image in images- if (local := chunk.layout.local_frame_index(image.frame_idx)) is not None- ]+ local_images: list[LoadedImageConditioning] = []+ for image in images:+ local_image = image.map_frame_index(chunk.layout.local_frame_index)+ if local_image is not None:+ local_images.append(local_image)+ return local_imagesdef image_conditionings_for_chunk(chunk: Chunk,*,- images: Sequence[ImageConditioningInput],+ images: Sequence[LoadedImageConditioning],video_encoder: VideoEncoder,- color_space: EXRColorSpace | None = None,) -> list[ConditioningItem]:"""Encode image conditionings that land in ``chunk``, at its current latent spatial size."""local = images_for_chunk(chunk, images)if not local or chunk.video is None:return []height, width = _pixel_hw(chunk)- return combined_image_conditionings(- images=local,+ return image_conditionings(+ images=list(local),height=height,width=width,video_encoder=video_encoder,dtype=chunk.video.dtype,device=chunk.video.device,- color_space=color_space,)@@ -155,10 +153,7 @@ def audio_reference_conditionings_for_chunk(strength: float = 1.0,negative_positions: bool = True,) -> list[ConditioningItem]:- """Build audio-reference tokens for ``chunk``, sliced to its layout window.- ``latent`` is the full-clip VAE latent. ``latent is None`` means use the chunk's own- audio latent (stage-2 self-reference).- """+ """Build audio-reference tokens for ``chunk``, sliced to its layout window."""source = chunk.audio if latent is None else audio_latent_for_layout(latent, chunk.layout, fps)if source is None:raise ValueError("audio reference with latent=None requires chunk.audio")diff --git a/packages/ltx-pipelines/src/ltx_pipelines/chunks/operators.py b/packages/ltx-pipelines/src/ltx_pipelines/chunks/operators.pyindex cd3cdcc..cd43294 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/chunks/operators.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/chunks/operators.py@@ -270,6 +270,8 @@ def _decode_chunk_video(decoded_batches = list(video_decoder(chunk.video, tiling_config, generator, **decode_kwargs))if not decoded_batches:return None+ if len(decoded_batches) == 1:+ return decoded_batches[0]return torch.cat(decoded_batches, dim=0)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/dfr_helpers/lanczos.py b/packages/ltx-pipelines/src/ltx_pipelines/dfr_helpers/lanczos.pynew file mode 100644index 0000000..abcb983--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/dfr_helpers/lanczos.py@@ -0,0 +1,95 @@+"""Pillow's 8-bit Lanczos-x2 resample as torch ops, so it runs on the frames' own device.+Pillow resizes in two separable passes of fixed-point Lanczos-3 weights (22 fractional bits),+rounding and clamping to a byte after each pass. The same integer arithmetic in float64 is exact+(every product and partial sum stays below 2**53), so the bytes match Pillow's on any device.+"""++from __future__ import annotations++import math+from functools import lru_cache++import torch++from ltx_core.devices import upload_from_host++FIXED_POINT_BITS = 22+LANCZOS_SUPPORT = 3.0+++def rgb_to_uint8_fhwc(frames: torch.Tensor) -> torch.Tensor:+ """``(F, H, W, C)`` RGB in ``[0, 1]`` to bytes, rounding the way the Pillow path does on the host."""+ return (frames.detach().float().clamp(0, 1) * 255.0).round().to(torch.uint8)+++def _lanczos3(x: float) -> float:+ if not -LANCZOS_SUPPORT <= x < LANCZOS_SUPPORT:+ return 0.0++ def sinc(v: float) -> float:+ if v == 0.0:+ return 1.0+ v *= math.pi+ return math.sin(v) / v++ return sinc(x) * sinc(x / LANCZOS_SUPPORT)+++@lru_cache(maxsize=8)+def _lanczos_weights(in_size: int, out_size: int) -> torch.Tensor:+ """Pillow's fixed-point Lanczos-3 taps as a ``(2, out_size, taps)`` float64 tensor.+ ``[0]`` is each tap's source index and ``[1]`` its weight, normalised per output row and+ rounded to ``FIXED_POINT_BITS``. A row with fewer taps than the widest pads with weight 0+ on index 0. Indices are small integers, exact in float64, so one tensor carries both.+ """+ scale = in_size / out_size+ filterscale = max(scale, 1.0)+ support = LANCZOS_SUPPORT * filterscale+ ss = 1.0 / filterscale+ rows: list[tuple[int, list[float]]] = []+ for xx in range(out_size):+ center = (xx + 0.5) * scale+ xmin = max(int(center - support + 0.5), 0)+ xmax = min(int(center + support + 0.5), in_size) - xmin+ k = [_lanczos3((x + xmin - center + 0.5) * ss) for x in range(xmax)]+ total = sum(k)+ fixed = []+ for w in k:+ weight = w / total if total != 0.0 else w+ v = weight * (1 << FIXED_POINT_BITS)+ fixed.append(float(int(-0.5 + v) if weight < 0 else int(0.5 + v)))+ rows.append((xmin, fixed))+ taps = torch.zeros(2, out_size, max(len(fixed) for _, fixed in rows), dtype=torch.float64)+ for xx, (xmin, fixed) in enumerate(rows):+ taps[0, xx, : len(fixed)] = torch.arange(xmin, xmin + len(fixed), dtype=torch.float64)+ taps[1, xx, : len(fixed)] = torch.tensor(fixed, dtype=torch.float64)+ return taps+++def _fixed_point_pass(v: torch.Tensor, k: torch.Tensor) -> torch.Tensor:+ """Resample the last axis of ``v`` with taps ``k``: fixed-point accumulate, rounding shift, byte clamp.+ One gather and multiply-add per tap; every product and partial sum is an integer below+ 2**53, so the float64 sum is exact in any order.+ """+ index = k[0].long()+ acc = torch.full((*v.shape[:-1], k.shape[1]), float(1 << (FIXED_POINT_BITS - 1)), dtype=v.dtype, device=v.device)+ for tap in range(k.shape[2]):+ acc.addcmul_(v[..., index[:, tap]], k[1, :, tap])+ return torch.floor(acc / float(1 << FIXED_POINT_BITS)).clamp_(0.0, 255.0)+++@lru_cache(maxsize=8)+def _device_lanczos_weights(in_size: int, out_size: int, device: torch.device) -> torch.Tensor:+ """:func:`_lanczos_weights` on ``device``, uploaded once without waiting for the device."""+ return upload_from_host(_lanczos_weights(in_size, out_size), device)+++def lanczos_x2_u8_hwc(frame: torch.Tensor) -> torch.Tensor:+ """``(H, W, C)`` uint8 to ``(2H, 2W, C)`` uint8, byte-identical to ``Image.resize((2W, 2H), LANCZOS)``."""+ height, width, _ = frame.shape+ kx = _device_lanczos_weights(width, 2 * width, frame.device)+ ky = _device_lanczos_weights(height, 2 * height, frame.device)+ x = frame.to(torch.float64).permute(2, 0, 1)+ x = _fixed_point_pass(x, kx)+ x = _fixed_point_pass(x.transpose(1, 2), ky).transpose(1, 2)+ return x.permute(1, 2, 0).to(torch.uint8)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/dfr_helpers/ops.py b/packages/ltx-pipelines/src/ltx_pipelines/dfr_helpers/ops.pyindex cdc9fca..4cbdfc5 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/dfr_helpers/ops.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/dfr_helpers/ops.py@@ -5,6 +5,7 @@ No ``DiffusionStage`` / Gemma / ``ModelPaths``. Imported by ``dfr_stages`` and `from __future__ import annotationsimport logging+import mathfrom collections.abc import Iterable, Sequencefrom dataclasses import dataclass, replace@@ -15,7 +16,8 @@ from PIL import Imagefrom ltx_core.conditioning import ConditioningItem, VideoConditionByKeyframeIndex, VideoConditionByLatentIndexfrom ltx_core.tiling import DimensionTilingConfig, TileCountConfigfrom ltx_core.types import Audio, AudioLatentShape, VideoPixelShape-from ltx_pipelines.utils.types import ImageConditioningInput+from ltx_pipelines.dfr_helpers.lanczos import lanczos_x2_u8_hwc, rgb_to_uint8_fhwc+from ltx_pipelines.utils.types import LoadedImageConditioninglogger = logging.getLogger(__name__)@@ -36,32 +38,43 @@ def conditioning_fps(playback_fps: float) -> float:return MAX_CONDITIONING_FPS if playback_fps > SNAP_CONDITIONING_FPS_ABOVE else playback_fps-# HDR stage-2 spatial overlap. Temporal windows are sequential and pinned; spatial tiles inside-# a time window blend after every Euler step; time windows are not blended. The first detailing-# step is 2x2; remaining steps retile to 4x4. A two-value schedule stays 2x2.-EPILOGUE_SPATIAL_OVERLAP = 10-EPILOGUE_SPATIAL_COARSE_TILES = 2-EPILOGUE_SPATIAL_TILES = 4+# Spatial epilogue tiling. Temporal windows are sequential and pinned and are not blended; spatial+# tiles inside a time window blend after every Euler step. Every step uses one grid of tiles at most+# EPILOGUE_TILE_CELLS latent cells per side, overlapping by EPILOGUE_SPATIAL_OVERLAP cells: the blend+# only has to hide two tiles' per-step disagreement on the shared cells, which 2 cells (64 px) keep+# within ~0.5 dB of the rest of the frame.+EPILOGUE_SPATIAL_OVERLAP = 2+EPILOGUE_TILE_CELLS = 32EPILOGUE_KEYFRAME_STRENGTH = 1.0DETAILING_LORA_STRENGTH = 0.5+def epilogue_tile_count(cells: int) -> int:+ """Tiles along a latent axis of ``cells`` cells: the fewest whose overlapping tiles fit ``EPILOGUE_TILE_CELLS``."""+ stride = EPILOGUE_TILE_CELLS - EPILOGUE_SPATIAL_OVERLAP+ return max(1, math.ceil((cells - EPILOGUE_SPATIAL_OVERLAP) / stride))++def lanczos_x2_fhwc(frames: torch.Tensor) -> torch.Tensor:- """Stretch each RGB frame 2x with Lanczos. ``frames`` is ``(F, H, W, C)`` in ``[0, 1]``."""+ """Stretch each RGB frame 2x with Lanczos. ``frames`` is ``(F, H, W, C)`` in ``[0, 1]``.+ CUDA frames are resampled on their device by :mod:`ltx_pipelines.dfr_helpers.lanczos`, CPU+ frames by Pillow; both quantize to bytes first and return the same pixels.+ """if frames.ndim != 4:raise ValueError(f"Expected (F, H, W, C) RGB, got shape {tuple(frames.shape)}")if frames.shape[0] < 1:raise ValueError("Need at least one frame to Lanczos-upsample")+ if frames.shape[-1] not in (1, 3):+ raise ValueError(f"Lanczos x2 expects 1 or 3 channels, got {frames.shape[-1]}")+ if frames.is_cuda:+ resized = torch.stack([lanczos_x2_u8_hwc(frame) for frame in rgb_to_uint8_fhwc(frames)]).float()+ # A tensor divisor keeps this an IEEE division, as on the host (a Python scalar becomes a reciprocal multiply).+ return resized / torch.full((), 255.0, device=resized.device)out: list[torch.Tensor] = []for frame in frames:height, width, channels = frame.shape- array = (frame.detach().float().clamp(0, 1).cpu().numpy() * 255.0).round().astype(np.uint8)- if channels == 1:- image = Image.fromarray(array[..., 0], mode="L")- elif channels == 3:- image = Image.fromarray(array, mode="RGB")- else:- raise ValueError(f"Lanczos x2 expects 1 or 3 channels, got {channels}")+ array = rgb_to_uint8_fhwc(frame).numpy()+ image = Image.fromarray(array[..., 0], mode="L") if channels == 1 else Image.fromarray(array, mode="RGB")image = image.resize((width * 2, height * 2), resample=Image.Resampling.LANCZOS)resized = torch.from_numpy(np.asarray(image, dtype=np.float32) / 255.0)if resized.ndim == 2:@@ -279,22 +292,28 @@ def lead_in_carryover(def rebase_image_conditionings(- images: Sequence[ImageConditioningInput],+ images: Sequence[LoadedImageConditioning],*,pixel_scale: int,pixel_start: int = 0,pixel_end: int | None = None,-) -> list[ImageConditioningInput]:+) -> list[LoadedImageConditioning]:"""Map user ``frame_idx`` values from the requested canvas onto a temporally-upsampled grid.After ``r`` temporal rounds the same moment sits at ``frame_idx * 2**r``. When ``pixel_end`` isset, only images inside ``[pixel_start, pixel_end]`` are kept and indices become tile-local."""- rebased: list[ImageConditioningInput] = []- for image in images:- scaled = image.frame_idx * pixel_scale++ def map_frame(frame_idx: int) -> int | None:+ scaled = frame_idx * pixel_scaleif pixel_end is not None and not (pixel_start <= scaled <= pixel_end):- continue- rebased.append(ImageConditioningInput(image.path, scaled - pixel_start, image.strength, image.crf))+ return None+ return scaled - pixel_start++ rebased: list[LoadedImageConditioning] = []+ for image in images:+ rebased_image = image.map_frame_index(map_frame)+ if rebased_image is not None:+ rebased.append(rebased_image)return rebaseddiff --git a/packages/ltx-pipelines/src/ltx_pipelines/dfr_mgpu.py b/packages/ltx-pipelines/src/ltx_pipelines/dfr_mgpu.pyindex 785041d..963b0e9 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/dfr_mgpu.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/dfr_mgpu.py@@ -13,8 +13,8 @@ from __future__ import annotationsimport loggingfrom collections.abc import Callable, Iterator, Sequencefrom contextlib import contextmanager, nullcontext-from functools import wrapsfrom multiprocessing import SimpleQueue+from types import SimpleNamespacefrom typing import Anyimport torch@@ -24,21 +24,23 @@ from ltx_core.allocator_trim_strategy import AllocatorTrimStrategyfrom ltx_core.loader import LoraPathStrengthAndSDOpsfrom ltx_core.loader.registry import ModelRegistryfrom ltx_core.model.transformer.compiling import CompilationConfig-from ltx_core.model.video_vae import AUTO_TILING, AutoTiling, TilingConfig, get_video_chunks_number+from ltx_core.model.video_vae import AUTO_TILING, get_video_chunks_numberfrom ltx_core.model.video_vae.transformer import DiffVAEModefrom ltx_core.multigpu.transformer.attention import AttentionManager-from ltx_core.multigpu.vae.distributed_decoder import dist_rank_tile_pixel_shapefrom ltx_core.quantization import QuantizationPolicyfrom ltx_core.quantization.fp8_cast import build_policy as _build_fp8_cast_policyfrom ltx_core.tiling import DimensionTilingConfig, TileCountConfig, balanced_tile_split-from ltx_core.types import VIDEO_SCALE_FACTORS, SpatioTemporalScaleFactors, VideoLatentShape, VideoPixelShape+from ltx_core.types import VIDEO_SCALE_FACTORS, SpatioTemporalScaleFactorsfrom ltx_pipelines.dfr_pipeline import DFRPipeline, add_dfr_cli_argsfrom ltx_pipelines.multigpu.controller import MGPUController+from ltx_pipelines.multigpu.decode_tiling import dist_auto_tilingfrom ltx_pipelines.multigpu.gemma_builders import AccelerateGemmaBuilderfrom ltx_pipelines.multigpu.runner import MGPURunnerfrom ltx_pipelines.multigpu.sp_builder import SequenceParallelBuilderfrom ltx_pipelines.multigpu.vae_builders import DistributedDecoderBuilderfrom ltx_pipelines.multigpu.weight_tracker import TransformerWeightTracker+from ltx_pipelines.utils.args import markers_from_args+from ltx_pipelines.utils.helpers import get_devicefrom ltx_pipelines.utils.media_io import encode_videofrom ltx_pipelines.utils.model_paths import ModelPathsfrom ltx_pipelines.utils.types import DEFAULT_AUTO_DURATION, AutoDuration@@ -62,62 +64,20 @@ _DEFAULT_VAE_TEMPORAL_BATCHES = 16def _dfr_dist_tile_auto_tiling(vae_tiling: TileCountConfig,*,- rank: int,world_size: int,+ group: dist.ProcessGroup | None,scale: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,) -> Iterator[None]:- """Rewrite ``DFRPipeline`` AUTO recommend to this rank's Dist-tile pixel shape.+ """Plan ``DFRPipeline``'s AUTO decode tiling for the group's Dist tiles.Patches ``ltx_pipelines.dfr_pipeline.ensure_tiling_config`` (the name- ``DFRPipeline.__call__`` looks up) for the duration of the ``with`` block.- Dist stays a pass-through; the pipeline still returns the resolved- ``TilingConfig``. Explicit configs and ``None`` are unchanged.+ ``DFRPipeline.__call__`` looks up) with :func:`dist_auto_tiling` for the duration of the+ ``with`` block, so every rank resolves the same ``TilingConfig``. Explicit configs and+ ``None`` are unchanged."""import ltx_pipelines.dfr_pipeline as dfr_mod # noqa: PLC0415orig = dfr_mod.ensure_tiling_config-- @wraps(orig)- def ensure_tiling_config_dist_tile(- tiling_config: TilingConfig | AutoTiling | None,- *,- video_shape: VideoPixelShape,- scale_factors: SpatioTemporalScaleFactors,- vae_checkpoint_path: str,- diffvae_optimization: DiffVAEMode = DiffVAEMode.CHUNKED_EAGER,- device: torch.device | None = None,- free_bytes: int | None = None,- keyframes: bool = False,- ) -> TilingConfig | None:- if isinstance(tiling_config, AutoTiling):- latent_shape = VideoLatentShape.from_pixel_shape(video_shape, scale_factors=scale)- tile_shape = dist_rank_tile_pixel_shape(- vae_tiling,- latent_shape,- scale,- rank,- world_size,- fps=video_shape.fps,- batch=video_shape.batch,- )- logger.info(- "rank=%s AUTO_TILING Dist tile video_shape=%s (full canvas %s)",- rank,- tile_shape,- video_shape,- )- video_shape = tile_shape- return orig(- tiling_config,- video_shape=video_shape,- scale_factors=scale_factors,- vae_checkpoint_path=vae_checkpoint_path,- diffvae_optimization=diffvae_optimization,- device=device,- free_bytes=free_bytes,- keyframes=keyframes,- )-- dfr_mod.ensure_tiling_config = ensure_tiling_config_dist_tile+ dfr_mod.ensure_tiling_config = dist_auto_tiling(orig, vae_tiling, world_size=world_size, group=group, scale=scale)try:yieldfinally:@@ -244,15 +204,20 @@ class DFRRunner(MGPURunner):enhance_static_cache: bool = False,temporal_upscalings: int = 0,spatial_upscalings: int = 1,+ marker_root_path: str | None = None,) -> Iterator[str | None]:# The pipeline raises ValueError on invalid input (symmetric across ranks); the controller# catches that and turns it into a recoverable RunnerError. Anything else is fatal.+ video_marker, audio_marker = markers_from_args(+ SimpleNamespace(marker_root_path=marker_root_path),+ device=get_device(),+ )vae_tiling = self._vae_tilingauto_tile = (_dfr_dist_tile_auto_tiling(vae_tiling,- rank=dist.get_rank(self.groups.vae_group),world_size=dist.get_world_size(self.groups.vae_group),+ group=self.groups.vae_group,)if vae_tiling is not Noneelse nullcontext()@@ -282,12 +247,19 @@ class DFRRunner(MGPURunner):audio=result.audio,output_path=output_path,video_chunks_number=get_video_chunks_number(result.num_frames, result.tiling_config),+ video_marker=video_marker,+ audio_marker=audio_marker,+ # The distributed decoder's batches are complete on every stream when handed out.+ chunks_complete=True,)yield output_pathif __name__ == "__main__":- from ltx_pipelines.utils.args import default_2_stage_distilled_arg_parser, resolve_cli_params+ from ltx_pipelines.utils.args import (+ default_2_stage_distilled_arg_parser,+ resolve_cli_params,+ )logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")params = resolve_cli_params(distilled=True)@@ -321,6 +293,7 @@ if __name__ == "__main__":enhance_static_cache=args.enhance_static_cache,temporal_upscalings=args.temporal_upscalings,spatial_upscalings=args.spatial_upscalings,+ marker_root_path=args.marker_root_path,):pass # drive the job to completion; the runner writes the file as a side effectfinally:diff --git a/packages/ltx-pipelines/src/ltx_pipelines/dfr_pipeline.py b/packages/ltx-pipelines/src/ltx_pipelines/dfr_pipeline.pyindex 146b3c8..60c022a 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/dfr_pipeline.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/dfr_pipeline.py@@ -22,7 +22,8 @@ from ltx_core.model.video_vae import (get_video_chunks_number,)from ltx_core.model.video_vae.transformer import DiffVAEMode-from ltx_core.quantization import QuantizationPolicy+from ltx_core.quantization import QuantizationPolicy, fp8_cast_fuse_rule+from ltx_core.quantization.fp8_cast import supports_fp8_gemm, with_rowwise_fp8_gemmfrom ltx_core.types import VIDEO_SCALE_FACTORS, VideoPixelShapefrom ltx_pipelines.dfr_helpers.layout import resolve_canvasfrom ltx_pipelines.dfr_helpers.ops import (@@ -47,6 +48,7 @@ from ltx_pipelines.iclora_utils import read_lora_reference_downscale_factorfrom ltx_pipelines.utils.args import (LoraAction,default_2_stage_distilled_arg_parser,+ markers_from_args,resolve_cli_params,resolve_existing_path,)@@ -241,6 +243,19 @@ class DFRPipeline:fps=fps,)+ def _epilogue_stage(self) -> DiffusionStage:+ """The detailing stage the spatial epilogue denoises with.+ On an fp8-cast policy and a device with fp8 GEMMs, its long video projections multiply+ e4m3 activations (:func:`~ltx_core.quantization.fp8_cast.with_rowwise_fp8_gemm`): the+ epilogue is the last pass, where the precision costs least. Derived per call so it+ follows whatever builder the stage carries (e.g. a multi-GPU runner's).+ """+ stage = self.stage_detailing+ policy = stage.quantization+ if policy is None or policy.fuse_rule is not fp8_cast_fuse_rule or not supports_fp8_gemm(self.device):+ return stage+ return stage.with_quantization(with_rowwise_fp8_gemm(policy))+def __call__( # noqa: PLR0913, PLR0915self,prompt: str,@@ -267,7 +282,8 @@ class DFRPipeline:raise ValueError(f"spatial_upscalings must be 1 or 2, got {spatial_upscalings}")require_num_frames_source(num_frames, self.duration_predictor)- images = self.image_conditioner.resolve_crf(images)+ enhance_prompt_image = images[0].path if images else None+ images = self.image_conditioner.load_conditionings(images)assert_resolution(height=height,width=width,@@ -284,7 +300,7 @@ class DFRPipeline:[prompt],enhance_first_prompt=enhance_prompt,enhance_static_cache=enhance_static_cache,- enhance_prompt_image=images[0][0] if len(images) > 0 else None,+ enhance_prompt_image=enhance_prompt_image,)video_context, audio_context = ctx_p.video_encoding, ctx_p.audio_encoding@@ -355,47 +371,49 @@ class DFRPipeline:)last_window_seams: list[int] = []- for round_idx in range(1, temporal_upscalings + 1):- assert self.temporal_upsampler is not None- if not keyframes:- raise RuntimeError(f"Temporal round {round_idx}: missing carry-forward keyframes")- video_shape = video_shape._replace(frames=2 * (video_shape.frames - 1) + 1, fps=2 * video_shape.fps)- video_state, keyframes, last_window_seams = run_one_temporal_round(- round_idx=round_idx,- stage=self.stage,- temporal_upsampler=self.temporal_upsampler,- image_conditioner=self.image_conditioner,- video_state=video_state,- keyframes=keyframes,- video_shape=video_shape,- images=images,- video_context=video_context,- audio_context=audio_context,- audio_latent=audio_latent,- source_duration=source_duration,- seed=seed,- noiser=noiser,- sigmas=DISTILLED_SIGMAS[4:].to(dtype=torch.float32, device=self.device),- device=self.device,- dtype=dtype,- temporal_scale=temporal_scale,- )- self._emit_stage(- f"t{round_idx}",- latent=video_state.latent,- keyframes=keyframes,- height=video_shape.height,- width=video_shape.width,- num_frames=video_shape.frames,- fps=video_shape.fps,- )+ # One transformer for every temporal tile of every round instead of a build per tile.+ with self.stage.resident():+ for round_idx in range(1, temporal_upscalings + 1):+ assert self.temporal_upsampler is not None+ if not keyframes:+ raise RuntimeError(f"Temporal round {round_idx}: missing carry-forward keyframes")+ video_shape = video_shape._replace(frames=2 * (video_shape.frames - 1) + 1, fps=2 * video_shape.fps)+ video_state, keyframes, last_window_seams = run_one_temporal_round(+ round_idx=round_idx,+ stage=self.stage,+ temporal_upsampler=self.temporal_upsampler,+ image_conditioner=self.image_conditioner,+ video_state=video_state,+ keyframes=keyframes,+ video_shape=video_shape,+ images=images,+ video_context=video_context,+ audio_context=audio_context,+ audio_latent=audio_latent,+ source_duration=source_duration,+ seed=seed,+ noiser=noiser,+ sigmas=DISTILLED_SIGMAS[4:].to(dtype=torch.float32),+ device=self.device,+ dtype=dtype,+ temporal_scale=temporal_scale,+ )+ self._emit_stage(+ f"t{round_idx}",+ latent=video_state.latent,+ keyframes=keyframes,+ height=video_shape.height,+ width=video_shape.width,+ num_frames=video_shape.frames,+ fps=video_shape.fps,+ )num_frames = video_shape.framescurrent_fps = video_shape.fps# --- Spatial epilogue (spatial_upscalings == 2): s2 is h/2; this pass is final H x W ---if spatial_upscalings == 2:epi = run_spatial_epilogue(- stage=self.stage_detailing,+ stage=self._epilogue_stage(),upsampler=self.upsampler,image_conditioner=self.image_conditioner,video_decoder=self.video_decoder,@@ -498,6 +516,7 @@ def main() -> None:params = resolve_cli_params(distilled=True)parser = add_dfr_cli_args(default_2_stage_distilled_arg_parser(params=params, supports_auto_duration=True))args = parser.parse_args()+ video_marker, audio_marker = markers_from_args(args, device=get_device())pipeline = DFRPipeline(model_paths=args.model_paths,@@ -532,6 +551,8 @@ def main() -> None:audio=result.audio,output_path=args.output_path,video_chunks_number=get_video_chunks_number(result.num_frames, result.tiling_config),+ video_marker=video_marker,+ audio_marker=audio_marker,)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/dfr_stages.py b/packages/ltx-pipelines/src/ltx_pipelines/dfr_stages.pyindex f6aa71e..6dc32bf 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/dfr_stages.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/dfr_stages.py@@ -21,6 +21,7 @@ import torchfrom ltx_core.components.noisers import GaussianNoiser, Noiserfrom ltx_core.conditioning import ConditioningItem, VideoConditionByReferenceLatent, VideoGeneratedKeyframeSlots+from ltx_core.devices import upload_from_hostfrom ltx_core.model.video_vae import VideoEncoderfrom ltx_core.tiling import DimensionTilingConfig, TileCountConfigfrom ltx_core.tools import LatentTools, VideoLatentTools@@ -29,14 +30,13 @@ from ltx_pipelines.dfr_helpers.layout import TemporalTile, TemporalTilePlanfrom ltx_pipelines.dfr_helpers.ops import (ANCHOR_KEYFRAME_STRENGTH,EPILOGUE_KEYFRAME_STRENGTH,- EPILOGUE_SPATIAL_COARSE_TILES,EPILOGUE_SPATIAL_OVERLAP,- EPILOGUE_SPATIAL_TILES,TEMPORAL_ANCESTRAL_ETA,TilePrefix,audio_latent_for_tile,clamp_tile_to_latent,conditioning_fps,+ epilogue_tile_count,keyframe_conditionings_from_latents,lanczos_x2_fhwc,lead_in_carryover,@@ -47,14 +47,14 @@ from ltx_pipelines.dfr_helpers.ops import (from ltx_pipelines.utils.blocks import DiffusionStage, ImageConditioner, VideoDecoder, VideoUpsamplerfrom ltx_pipelines.utils.denoisers import SimpleDenoiserfrom ltx_pipelines.utils.helpers import (- combined_image_conditionings,create_initial_audio_latent,create_initial_video_latent,+ image_conditionings,)from ltx_pipelines.utils.media_io import to_vae_rangefrom ltx_pipelines.utils.samplers import euler_ancestral_denoising_loopfrom ltx_pipelines.utils.tiled_diffusion import TiledDiffusionModel-from ltx_pipelines.utils.types import ImageConditioningInput, ModalitySpec, VideoAudio+from ltx_pipelines.utils.types import LoadedImageConditioning, ModalitySpec, VideoAudiologger = logging.getLogger(__name__)@@ -152,7 +152,7 @@ def _stack_keyframes(keyframes: KeyframeCarry) -> tuple[list[int], torch.Tensor]def _image_conditionings(image_conditioner: ImageConditioner,- images: Sequence[ImageConditioningInput],+ images: Sequence[LoadedImageConditioning],*,height: int,width: int,@@ -160,7 +160,7 @@ def _image_conditionings(device: torch.device,) -> list[ConditioningItem]:return image_conditioner(- lambda enc: combined_image_conditionings(+ lambda enc: image_conditionings(images=list(images),height=height,width=width,@@ -174,7 +174,7 @@ def _image_conditionings(def _prepare_stage1_specs(*,image_conditioner: ImageConditioner,- images: list[ImageConditioningInput],+ images: list[LoadedImageConditioning],video_shape: VideoPixelShape,positions: Sequence[int],video_context: torch.Tensor,@@ -229,7 +229,7 @@ def _prepare_stage2_specs( # noqa: PLR0913*,upsampler: VideoUpsampler,image_conditioner: ImageConditioner,- images: list[ImageConditioningInput],+ images: list[LoadedImageConditioning],video_shape: VideoPixelShape,keyframes: KeyframeCarry,half_res_reference: torch.Tensor,@@ -287,7 +287,7 @@ def denoise_stage1( # noqa: PLR0913*,stage: DiffusionStage,image_conditioner: ImageConditioner,- images: list[ImageConditioningInput],+ images: list[LoadedImageConditioning],video_shape: VideoPixelShape,positions: Sequence[int],video_context: torch.Tensor,@@ -306,7 +306,8 @@ def denoise_stage1( # noqa: PLR0913``1.0``, which would noise the latent before freezing it).Transformer fps snaps above 30; audio duration stays at playback ``video_shape.fps``."""- sigmas = sigmas.to(dtype=torch.float32, device=device)+ # Kept on the host: the loop uploads the schedule, and host reads of it stay free.+ sigmas = sigmas.to(dtype=torch.float32)prepared = _prepare_stage1_specs(image_conditioner=image_conditioner,images=images,@@ -344,7 +345,7 @@ def denoise_stage2( # noqa: PLR0913stage: DiffusionStage,image_conditioner: ImageConditioner,upsampler: VideoUpsampler,- images: list[ImageConditioningInput],+ images: list[LoadedImageConditioning],video_shape: VideoPixelShape,keyframes: KeyframeCarry,half_res_reference: torch.Tensor,@@ -362,7 +363,8 @@ def denoise_stage2( # noqa: PLR0913Upsamples ``half_res_reference`` and ``keyframes`` as its first step, matching temporalrounds and the epilogue. Audio is re-noised for cross-modal attention only."""- sigmas = sigmas.to(dtype=torch.float32, device=device)+ # Kept on the host: the loop uploads the schedule, and host reads of it stay free.+ sigmas = sigmas.to(dtype=torch.float32)prepared = _prepare_stage2_specs(upsampler=upsampler,image_conditioner=image_conditioner,@@ -425,7 +427,7 @@ def _temporal_tile_video_input(def _temporal_tile_conditionings( # noqa: PLR0913*,image_conditioner: ImageConditioner,- images: list[ImageConditioningInput],+ images: list[LoadedImageConditioning],canvas: VideoPixelShape,tile_video: TemporalTileVideo,tile: TemporalTile,@@ -499,7 +501,7 @@ def _temporal_tile_conditionings( # noqa: PLR0913def _prepare_temporal_tile_specs( # noqa: PLR0913*,image_conditioner: ImageConditioner,- images: list[ImageConditioningInput],+ images: list[LoadedImageConditioning],canvas: VideoPixelShape,video_latent: torch.Tensor,tile_index: int,@@ -606,7 +608,7 @@ def _denoise_temporal_tile( # noqa: PLR0913*,stage: DiffusionStage,image_conditioner: ImageConditioner,- images: list[ImageConditioningInput],+ images: list[LoadedImageConditioning],canvas: VideoPixelShape,video_latent: torch.Tensor,tile_index: int,@@ -683,7 +685,7 @@ def run_one_temporal_round( # noqa: PLR0913video_state: LatentState,keyframes: KeyframeCarry,video_shape: VideoPixelShape,- images: list[ImageConditioningInput],+ images: list[LoadedImageConditioning],video_context: torch.Tensor,audio_context: torch.Tensor,audio_latent: torch.Tensor,@@ -751,51 +753,47 @@ def run_one_temporal_round( # noqa: PLR0913return next_video_state, dict(sorted(plane_at.items())), seam_positions-def _decode_carry_keyframes(+def _rebuild_carry_keyframes(video_decoder: VideoDecoder,+ image_conditioner: ImageConditioner,keyframes: KeyframeCarry,*,seed: int,device: torch.device,-) -> dict[int, torch.Tensor]:- """Decode each carry keyframe as its own 1-frame RGB clip.- The VAE is causal, so a stacked encode/decode would bleed neighbouring planes. Each- plane is a standalone one-pixel-frame latent, matching ``DecodeKeyframes``. Dist- ``decode_single_frames`` splits the planes across the ranks and gathers them back, so- every rank gets every plane instead of the empty worker yield from ``decode_video``.- Generators are per plane, which keeps the pixels independent of the world size.+ dtype: torch.dtype,+) -> KeyframeCarry:+ """Decode, Lanczos-x2 and re-encode each carry plane on the rank that owns it.+ Each plane is a standalone one-pixel-frame clip with its own generator, so its pixels do+ not depend on the world size. A Dist decoder splits the planes round-robin over the group;+ each rank resamples and encodes its own planes and only the small encoded latents cross the+ group. The Lanczos runs on the decode device right after each plane; every encode is queued+ before any read-back. Returns every plane on every rank."""for position, plane in keyframes.items():if plane.ndim != 5 or plane.shape[2] != 1:raise ValueError(f"Expected one keyframe plane at {position}, got {tuple(plane.shape)}")- n_planes = len(keyframes)- logger.info("Epilogue: decoding %d carry keyframes separately", n_planes)- planes = list(keyframes.values())- generators = [torch.Generator(device=device).manual_seed(seed + 4000 + index) for index in range(n_planes)]- rgb_planes: list[torch.Tensor] = []- for index, rgb in enumerate(video_decoder.decode_single_frames(planes, generators)):+ positions = list(keyframes)+ logger.info("Epilogue: decoding %d carry keyframes separately", len(positions))+ generators = [torch.Generator(device=device).manual_seed(seed + 4000 + index) for index in range(len(positions))]+ owned = video_decoder.decode_owned_single_frames(list(keyframes.values()), generators)++ resampled: dict[int, torch.Tensor] = {}+ for index, rgb in zip(owned.indices, owned.planes, strict=True):if rgb.shape[0] < 1:raise RuntimeError(f"Decoder returned no pixels for carry keyframe {index}")- rgb_planes.append(rgb.detach().cpu())- if len(rgb_planes) != n_planes:- raise RuntimeError(f"Expected {n_planes} decoded carry keyframes, got {len(rgb_planes)}")- return dict(zip(keyframes, rgb_planes, strict=True))---def _encode_keyframe_pixels(- enc: VideoEncoder,- pixel_keyframes: dict[int, torch.Tensor],- *,- dtype: torch.dtype,- device: torch.device,-) -> KeyframeCarry:- """Encode each Lanczos RGB clip as a one-frame latent plane."""+ resampled[index] = lanczos_x2_fhwc(rgb.detach())- def encode(rgb: torch.Tensor) -> torch.Tensor:- sample = to_vae_range(rgb.to(device=device, dtype=torch.float32))- return enc(sample.permute(3, 0, 1, 2).unsqueeze(0).contiguous().to(dtype=dtype))+ def encode(enc: VideoEncoder) -> dict[int, torch.Tensor]:+ on_device = {}+ for index, rgb in resampled.items():+ sample = to_vae_range(rgb.to(device=device, dtype=torch.float32))+ on_device[index] = enc(sample.permute(3, 0, 1, 2).unsqueeze(0).contiguous().to(dtype=dtype))+ return {index: plane.cpu() for index, plane in on_device.items()}- return {position: encode(rgb) for position, rgb in pixel_keyframes.items()}+ encoded = owned.gather(image_conditioner(encode))+ if len(encoded) != len(positions):+ raise RuntimeError(f"Expected {len(positions)} encoded carry keyframes, got {len(encoded)}")+ return {position: upload_from_host(encoded[index], device) for index, position in enumerate(positions)}def _rebuild_epilogue_keyframes(@@ -804,24 +802,21 @@ def _rebuild_epilogue_keyframes(image_conditioner: ImageConditioner,keyframes: KeyframeCarry,video_state: LatentState,- images: Sequence[ImageConditioningInput],+ images: Sequence[LoadedImageConditioning],temporal_upscalings: int,seed: int,device: torch.device,dtype: torch.dtype,) -> tuple[KeyframeCarry, torch.Tensor | None]:- """Decode carry planes, Lanczos-x2, and encode. Opening plane if no frame-0 image.- Decoder is freed before the encoder is built: decode completes, then- ``image_conditioner`` loads the encoder for the RGB planes only.- """+ """Rebuild the carry planes at the epilogue resolution, plus the opening plane if no image conditions frame 0."""opening_position = -1to_decode = dict(keyframes)epi_images = rebase_image_conditionings(images, pixel_scale=2**temporal_upscalings)if not any(image.frame_idx == 0 for image in epi_images):to_decode[opening_position] = video_state.latent[:1, :, :1]- pixel_keyframes = _decode_carry_keyframes(video_decoder, to_decode, seed=seed, device=device)- pixel_keyframes = {position: lanczos_x2_fhwc(rgb) for position, rgb in pixel_keyframes.items()}- encoded = image_conditioner(lambda enc: _encode_keyframe_pixels(enc, pixel_keyframes, dtype=dtype, device=device))+ encoded = _rebuild_carry_keyframes(+ video_decoder, image_conditioner, to_decode, seed=seed, device=device, dtype=dtype+ )opening = encoded.pop(opening_position, None)return encoded, opening@@ -1004,7 +999,7 @@ def _execute_tiled_spatial_denoise()video_state, _ = tiled_stage(denoiser=SimpleDenoiser(video_context, audio_context),- sigmas=sigmas.to(dtype=torch.float32, device=device),+ sigmas=sigmas.to(dtype=torch.float32),noiser=GaussianNoiser(generator=torch.Generator(device=device).manual_seed(seed + 2000)),modalities=VideoAudio(video=prepared.video, audio=prepared.audio),loop=loop,@@ -1046,92 +1041,89 @@ def _run_epilogue_windows( # noqa: PLR0913temporal_scale: int,loop: Callable[..., tuple[LatentState | None, LatentState | None]] | None = None,) -> torch.Tensor:- """Detail each temporal window with one coarse-grid step, then any remaining steps on the fine grid."""+ """Detail each temporal window with one step, then any remaining steps, on one spatial tile grid."""stitched: list[torch.Tensor] = []previous: tuple[int, torch.Tensor] | None = Nonecoarse_sigmas, fine_sigmas = _epilogue_sigma_phases(sigmas)- coarse_tiling = TileCountConfig(+ tiling = TileCountConfig(frames=DimensionTilingConfig(1, overlap=0),- height=DimensionTilingConfig(EPILOGUE_SPATIAL_COARSE_TILES, EPILOGUE_SPATIAL_OVERLAP),- width=DimensionTilingConfig(EPILOGUE_SPATIAL_COARSE_TILES, EPILOGUE_SPATIAL_OVERLAP),+ height=DimensionTilingConfig(epilogue_tile_count(int(latent.shape[3])), EPILOGUE_SPATIAL_OVERLAP),+ width=DimensionTilingConfig(epilogue_tile_count(int(latent.shape[4])), EPILOGUE_SPATIAL_OVERLAP),)scale_factors = stage.video_scale_factors- for index, window in enumerate(windows):- prepared_window = _prepare_epilogue_window_specs(- index=index,- num_windows=len(windows),- window=window,- prefix=prefixes[index],- latent=latent,- guide=guide,- plane_at=keyframes,- window_images=window_images[index],- keyframes=keyframes,- opening_plane=opening_plane,- previous=previous,- downscale_factor=downscale_factor,- canvas=canvas,- audio_latent=audio_latent,- source_duration=source_duration,- video_context=video_context,- audio_context=audio_context,- noise_scale=float(coarse_sigmas[0].item()),- temporal_scale=temporal_scale,- scale_factors=scale_factors,- device=device,- dtype=dtype,- )- window_latent = _execute_tiled_spatial_denoise(- stage,- prepared=prepared_window.denoise,- latent=prepared_window.window_latent,- tiling=coarse_tiling,- sigmas=coarse_sigmas,- video_context=video_context,- audio_context=audio_context,- seed=seed + 100 * index,- device=device,- loop=loop,- )- if fine_sigmas is not None:- fine_tiling = TileCountConfig(- frames=DimensionTilingConfig(1, overlap=0),- height=DimensionTilingConfig(EPILOGUE_SPATIAL_TILES, EPILOGUE_SPATIAL_OVERLAP),- width=DimensionTilingConfig(EPILOGUE_SPATIAL_TILES, EPILOGUE_SPATIAL_OVERLAP),- )- audio_latent_in = prepared_window.denoise.audio.latent- if audio_latent_in is None:- raise RuntimeError("Epilogue window prepared no audio latent")- fine_prepared = _prepare_tiled_spatial_denoise_specs(- latent=window_latent,- conditionings=list(prepared_window.denoise.video.conditionings),+ # One transformer for every window and tile instead of a build per tile.+ with stage.resident():+ for index, window in enumerate(windows):+ prepared_window = _prepare_epilogue_window_specs(+ index=index,+ num_windows=len(windows),+ window=window,+ prefix=prefixes[index],+ latent=latent,+ guide=guide,+ plane_at=keyframes,+ window_images=window_images[index],+ keyframes=keyframes,+ opening_plane=opening_plane,+ previous=previous,+ downscale_factor=downscale_factor,+ canvas=canvas,+ audio_latent=audio_latent,+ source_duration=source_duration,video_context=video_context,audio_context=audio_context,- audio_latent=audio_latent_in,- fps=prepared_window.denoise.canvas.fps,+ noise_scale=float(coarse_sigmas[0].item()),+ temporal_scale=temporal_scale,scale_factors=scale_factors,device=device,dtype=dtype,- noise_scale=0.0,)window_latent = _execute_tiled_spatial_denoise(stage,- prepared=fine_prepared,- latent=window_latent,- tiling=fine_tiling,- sigmas=fine_sigmas,+ prepared=prepared_window.denoise,+ latent=prepared_window.window_latent,+ tiling=tiling,+ sigmas=coarse_sigmas,video_context=video_context,audio_context=audio_context,seed=seed + 100 * index,device=device,loop=loop,)- prefix = prepared_window.prefix- previous = (- prefix.video_start_cell - 1 if prefix else prepared_window.interval_start,- window_latent.detach(),- )- stitched.append(window_latent[:, :, prepared_window.pinned_cells :])+ if fine_sigmas is not None:+ audio_latent_in = prepared_window.denoise.audio.latent+ if audio_latent_in is None:+ raise RuntimeError("Epilogue window prepared no audio latent")+ fine_prepared = _prepare_tiled_spatial_denoise_specs(+ latent=window_latent,+ conditionings=list(prepared_window.denoise.video.conditionings),+ video_context=video_context,+ audio_context=audio_context,+ audio_latent=audio_latent_in,+ fps=prepared_window.denoise.canvas.fps,+ scale_factors=scale_factors,+ device=device,+ dtype=dtype,+ noise_scale=0.0,+ )+ window_latent = _execute_tiled_spatial_denoise(+ stage,+ prepared=fine_prepared,+ latent=window_latent,+ tiling=tiling,+ sigmas=fine_sigmas,+ video_context=video_context,+ audio_context=audio_context,+ seed=seed + 100 * index,+ device=device,+ loop=loop,+ )+ prefix = prepared_window.prefix+ previous = (+ prefix.video_start_cell - 1 if prefix else prepared_window.interval_start,+ window_latent.detach(),+ )+ stitched.append(window_latent[:, :, prepared_window.pinned_cells :])return torch.cat(stitched, dim=2)@@ -1165,7 +1157,7 @@ def _encode_epilogue_window_images(*,windows: TemporalTilePlan,prefixes: Sequence[TilePrefix | None],- images: Sequence[ImageConditioningInput],+ images: Sequence[LoadedImageConditioning],temporal_upscalings: int,video_shape: VideoPixelShape,dtype: torch.dtype,@@ -1188,7 +1180,7 @@ def _encode_epilogue_window_images(if image.frame_idx + origin >= resume]per_window.append(- combined_image_conditionings(+ image_conditionings(images=inside,height=video_shape.height,width=video_shape.width,@@ -1210,7 +1202,7 @@ def run_spatial_epilogue( # noqa: PLR0913video_decoder: VideoDecoder,video_state: LatentState,keyframes: KeyframeCarry,- images: Sequence[ImageConditioningInput],+ images: Sequence[LoadedImageConditioning],video_shape: VideoPixelShape,temporal_upscalings: int,last_window_seams: Sequence[int],diff --git a/packages/ltx-pipelines/src/ltx_pipelines/distilled.py b/packages/ltx-pipelines/src/ltx_pipelines/distilled.pyindex b80b0ee..0db6c5a 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/distilled.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/distilled.py@@ -39,6 +39,7 @@ from ltx_pipelines.utils.args import (add_keyframe_decode_arg,chunk_config_from_args,default_2_stage_distilled_arg_parser,+ markers_from_args,resolve_cli_params,)from ltx_pipelines.utils.blocks import (@@ -69,6 +70,7 @@ from ltx_pipelines.utils.types import (AutoDuration,Denoiser,ImageConditioningInput,+ LoadedImageConditioning,OffloadMode,PipelineOutput,VideoAudio,@@ -145,8 +147,8 @@ class _DistilledRunContext:noiser: GaussianNoiserdtype: torch.dtypevae_dtype: torch.dtype- #: ``images`` with every unset CRF filled in, so callers do not re-resolve it.- images: list[ImageConditioningInput]+ #: ``images`` loaded once for the run (CRF resolved, stills decoded), so callers do not reload them.+ images: list[LoadedImageConditioning]video_context: torch.Tensoraudio_context: torch.Tensornum_frames: int@@ -248,6 +250,7 @@ class DistilledPipeline:width: int,frame_rate: float,images: list[ImageConditioningInput],+ color_space: EXRColorSpace | None,num_frames: int | AutoDuration,vae_dtype: torch.dtype | None,tiling_config: TilingConfig | AutoTiling | None,@@ -259,7 +262,8 @@ class DistilledPipeline:) -> _DistilledRunContext:"""Validate inputs, encode prompts, and resolve frame count / tiling before chunk planning."""require_num_frames_source(num_frames, self.duration_predictor)- images = self.image_conditioner.resolve_crf(images)+ enhance_prompt_image = images[0].path if images else None+ images = self.image_conditioner.load_conditionings(images, color_space)assert_resolution(height=height, width=width, is_two_stage=True)assert_generated_keyframes_request(decode_with_keyframes, generated_keyframes, self.stage)@@ -273,7 +277,7 @@ class DistilledPipeline:[prompt],enhance_first_prompt=enhance_prompt,enhance_static_cache=enhance_static_cache,- enhance_prompt_image=images[0][0] if len(images) > 0 else None,+ enhance_prompt_image=enhance_prompt_image,)video_context, audio_context = ctx_p.video_encoding, ctx_p.audio_encoding@@ -334,9 +338,8 @@ class DistilledPipeline:def _video_conditionings(self,- images: list[ImageConditioningInput],+ images: list[LoadedImageConditioning],num_frames: int,- color_space: EXRColorSpace | None,) -> MakeVideoConditionings:assert_image_frames_in_clip(images, num_frames)@@ -348,7 +351,6 @@ class DistilledPipeline:chunk,images=images,video_encoder=enc,- color_space=color_space,))@@ -438,6 +440,7 @@ class DistilledPipeline:width=width,frame_rate=frame_rate,images=images,+ color_space=color_space,num_frames=num_frames,vae_dtype=vae_dtype,tiling_config=tiling_config,@@ -465,7 +468,7 @@ class DistilledPipeline:dtype=ctx.dtype,config=ctx.chunk_config,video_scale_factors=self.stage.video_scale_factors,- make_video_conditionings=self._video_conditionings(ctx.images, ctx.num_frames, color_space),+ make_video_conditionings=self._video_conditionings(ctx.images, ctx.num_frames),generated_keyframes=generated_keyframes,)@@ -515,6 +518,8 @@ def main() -> None:))args = parser.parse_args()+ hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)+ video_marker, audio_marker = markers_from_args(args, device=get_device(), color_space=hdr)pipeline = DistilledPipeline(model_paths=args.model_paths,spatial_upsampler_path=args.spatial_upsampler_path,@@ -525,7 +530,6 @@ def main() -> None:prompt_enhancer_gemma_root=args.prompt_enhancer_gemma_root,diffvae_optimization=args.diffvae_optimization,)- hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)chunk_config = chunk_config_from_args(args)chunks, num_frames, tiling_config = pipeline.stream_chunks(@@ -564,6 +568,8 @@ def main() -> None:video_chunks_number=video_chunks_number,color_space=hdr,audio_sampling_rate=audio_sampling_rate,+ video_marker=video_marker,+ audio_marker=audio_marker,)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/distilled_mgpu.py b/packages/ltx-pipelines/src/ltx_pipelines/distilled_mgpu.pyindex fad2d63..42381ce 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/distilled_mgpu.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/distilled_mgpu.py@@ -13,6 +13,7 @@ from __future__ import annotationsimport loggingfrom collections.abc import Callable, Iterator, Sequencefrom multiprocessing import SimpleQueue+from types import SimpleNamespacefrom typing import Anyimport torch@@ -34,6 +35,8 @@ from ltx_pipelines.multigpu.runner import MGPURunnerfrom ltx_pipelines.multigpu.sp_builder import SequenceParallelBuilderfrom ltx_pipelines.multigpu.vae_builders import DistributedDecoderBuilderfrom ltx_pipelines.multigpu.weight_tracker import TransformerWeightTracker+from ltx_pipelines.utils.args import markers_from_args+from ltx_pipelines.utils.helpers import get_devicefrom ltx_pipelines.utils.media_io import EXRColorSpace, encode_video, resolve_hdr_color_space, vae_dtype_for_hdrfrom ltx_pipelines.utils.model_paths import ModelPathsfrom ltx_pipelines.utils.types import DEFAULT_AUTO_DURATION, AutoDuration@@ -135,6 +138,9 @@ class DistilledRunner(MGPURunner):vae_tiling=vae_tiling,driver_rank=_DRIVER_RANK,registry=registry,+ # The driver concatenates the decoded clip on one device anyway, so assembling it+ # in more temporal batches only adds a copy of the canvas.+ num_temporal_batches=1,)self._pipeline = pipeline@@ -157,44 +163,55 @@ class DistilledRunner(MGPURunner):generated_keyframes: int | Sequence[int] = 0,decode_with_keyframes: bool = False,chunk_config: ChunkConfig | None = None,+ marker_root_path: str | None = None,) -> Iterator[str | None]:# The pipeline raises ValueError on invalid input (symmetric across ranks); the controller# catches that and turns it into a recoverable RunnerError. Anything else is fatal.hdr = resolve_hdr_color_space(images=images or [], hdr=hdr)- vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)- chunks, _, _ = self._pipeline.stream_chunks(- prompt=prompt,- seed=seed,- height=height,- width=width,- frame_rate=frame_rate,- images=images or [],- num_frames=num_frames,- vae_dtype=vae_dtype,- tiling_config=None,- enhance_prompt=enhance_prompt,- enhance_static_cache=enhance_static_cache,- color_space=hdr,- generated_keyframes=generated_keyframes,- decode_with_keyframes=decode_with_keyframes,- chunk_config=chunk_config,- )- if dist.get_rank() != _DRIVER_RANK:- for _ in chunks:- pass- yield None- return- video, audio, audio_sampling_rate = split_decoded_chunks(chunks)- encode_video(- video=video,- fps=frame_rate,- audio=audio,- output_path=output_path,- video_chunks_number=1,+ video_marker, audio_marker = markers_from_args(+ SimpleNamespace(marker_root_path=marker_root_path),+ device=get_device(),color_space=hdr,- audio_sampling_rate=audio_sampling_rate,)- yield output_path+ vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)+ # One transformer for both passes of every chunk instead of a build per call. Under DEFER the SP+ # builder's builds stay assigned anyway, so holding one for the whole job costs no memory.+ with self._pipeline.stage.resident():+ chunks, _, _ = self._pipeline.stream_chunks(+ prompt=prompt,+ seed=seed,+ height=height,+ width=width,+ frame_rate=frame_rate,+ images=images or [],+ num_frames=num_frames,+ vae_dtype=vae_dtype,+ tiling_config=None,+ enhance_prompt=enhance_prompt,+ enhance_static_cache=enhance_static_cache,+ color_space=hdr,+ generated_keyframes=generated_keyframes,+ decode_with_keyframes=decode_with_keyframes,+ chunk_config=chunk_config,+ )+ is_driver = dist.get_rank() == _DRIVER_RANK+ if not is_driver:+ for _ in chunks:+ pass+ else:+ video, audio, audio_sampling_rate = split_decoded_chunks(chunks)+ encode_video(+ video=video,+ fps=frame_rate,+ audio=audio,+ output_path=output_path,+ video_chunks_number=1,+ color_space=hdr,+ audio_sampling_rate=audio_sampling_rate,+ video_marker=video_marker,+ audio_marker=audio_marker,+ )+ yield output_path if is_driver else Noneif __name__ == "__main__":@@ -244,6 +261,7 @@ if __name__ == "__main__":generated_keyframes=args.num_generated_keyframes,decode_with_keyframes=args.decode_with_keyframes,chunk_config=chunk_config_from_args(args),+ marker_root_path=args.marker_root_path,):pass # drive the job to completion; the runner writes the file as a side effectfinally:diff --git a/packages/ltx-pipelines/src/ltx_pipelines/dubit.py b/packages/ltx-pipelines/src/ltx_pipelines/dubit.pyindex 0f00d5e..5e3021d 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/dubit.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/dubit.py@@ -57,6 +57,7 @@ from ltx_pipelines.utils.args import (add_keyframe_decode_arg,chunk_config_from_args,dubit_arg_parser,+ markers_from_args,resolve_cli_params,)from ltx_pipelines.utils.blocks import (@@ -79,7 +80,13 @@ from ltx_pipelines.utils.helpers import ()from ltx_pipelines.utils.media_io import EXRColorSpace, decode_audio_from_file, encode_video, get_videostream_metadatafrom ltx_pipelines.utils.model_paths import ModelPaths-from ltx_pipelines.utils.types import ImageConditioningInput, OffloadMode, PipelineOutput, VideoAudio+from ltx_pipelines.utils.types import (+ ImageConditioningInput,+ LoadedImageConditioning,+ OffloadMode,+ PipelineOutput,+ VideoAudio,+)__all__ = ["DubItPipeline", "main", "patchify_dubit_audio_reference_latent"]@@ -94,7 +101,7 @@ class _DubItRunContext:noiser: GaussianNoiserdtype: torch.dtypevae_dtype: torch.dtype- images: list[ImageConditioningInput]+ images: list[LoadedImageConditioning]video_context: torch.Tensoraudio_context: torch.Tensornum_frames: int@@ -204,6 +211,7 @@ class DubItPipeline:height: int,width: int,images: list[ImageConditioningInput],+ color_space: EXRColorSpace | None,reference_video_path: str,reference_strength: float,enhance_prompt: bool,@@ -215,7 +223,8 @@ class DubItPipeline:chunk_config: ChunkConfig | None = None,) -> _DubItRunContext:"""Validate inputs, encode prompts/reference, and resolve frame count / tiling before chunk planning."""- images = self.image_conditioner.resolve_crf(images)+ enhance_prompt_image = images[0].path if images else None+ images = self.image_conditioner.load_conditionings(images, color_space)assert_resolution(height=height, width=width, is_two_stage=True)assert_generated_keyframes_request(decode_with_keyframes, generated_keyframes, self.stage)@@ -234,7 +243,7 @@ class DubItPipeline:[prompt],enhance_first_prompt=enhance_prompt,enhance_static_cache=enhance_static_cache,- enhance_prompt_image=images[0][0] if len(images) > 0 else None,+ enhance_prompt_image=enhance_prompt_image,)video_context, audio_context = ctx_p.video_encoding, ctx_p.audio_encoding@@ -276,7 +285,7 @@ class DubItPipeline:def _video_conditionings(self,- images: list[ImageConditioningInput],+ images: list[LoadedImageConditioning],num_frames: int,reference_video_path: str,reference_strength: float,@@ -291,7 +300,7 @@ class DubItPipeline:def encode(enc: VideoEncoder) -> list[ConditioningItem]:items = [- *image_conditionings_for_chunk(chunk, images=images, video_encoder=enc, color_space=color_space),+ *image_conditionings_for_chunk(chunk, images=images, video_encoder=enc),*reference_video_conditionings_for_chunk(chunk,video_conditioning=[(reference_video_path, reference_strength)],@@ -386,6 +395,7 @@ class DubItPipeline:height=height,width=width,images=images,+ color_space=color_space,reference_video_path=reference_video_path,reference_strength=reference_strength,enhance_prompt=enhance_prompt,@@ -476,6 +486,7 @@ def main() -> None:add_keyframe_decode_arg(add_generated_keyframes_arg(dubit_arg_parser(params=params))))args = parser.parse_args()+ video_marker, audio_marker = markers_from_args(args, device=get_device())if not args.lora or len(args.lora) != 1:raise ValueError("Dub-It requires exactly one --lora (the Dub-It IC-LoRA).")@@ -526,6 +537,8 @@ def main() -> None:video_chunks_number=video_chunks_number,color_space=None,audio_sampling_rate=audio_sampling_rate,+ video_marker=video_marker,+ audio_marker=audio_marker,)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/hdr_ic_lora.py b/packages/ltx-pipelines/src/ltx_pipelines/hdr_ic_lora.pyindex 764af40..540d9bf 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/hdr_ic_lora.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/hdr_ic_lora.py@@ -25,7 +25,10 @@ from ltx_core.model.video_vae.transformer import DiffVAEModefrom ltx_core.quantization import QuantizationPolicyfrom ltx_core.types import LatentState, SpatioTemporalScaleFactors, VideoPixelShapefrom ltx_pipelines.dfr_helpers.layout import resolve_canvas-from ltx_pipelines.utils.args import hdr_ic_lora_arg_parser+from ltx_pipelines.utils.args import (+ hdr_ic_lora_arg_parser,+ markers_from_args,+)from ltx_pipelines.utils.blocks import DiffusionStage, ImageConditioner, VideoDecoderfrom ltx_pipelines.utils.constants import DISTILLED_SIGMA_VALUESfrom ltx_pipelines.utils.denoisers import SimpleDenoiser@@ -527,6 +530,7 @@ class HDRICLoraPipeline:def main() -> None:logging.basicConfig(level=logging.INFO)args = hdr_ic_lora_arg_parser().parse_args()+ video_marker, audio_marker = markers_from_args(args, device=get_device(), color_space=args.exr_colorspace)logger.info("Loading HDRICLoraPipeline...")pipeline = HDRICLoraPipeline(@@ -554,6 +558,8 @@ def main() -> None:output_path=str(output_path),video_chunks_number=1,color_space=args.exr_colorspace,+ video_marker=video_marker,+ audio_marker=audio_marker,)logger.info("Done → %s (+ EXR)", args.output_path)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/ic_lora.py b/packages/ltx-pipelines/src/ltx_pipelines/ic_lora.pyindex 46baa6a..3008a51 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/ic_lora.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/ic_lora.py@@ -47,6 +47,7 @@ from ltx_pipelines.utils.args import (add_keyframe_decode_arg,chunk_config_from_args,default_2_stage_distilled_arg_parser,+ markers_from_args,resolve_cli_params,)from ltx_pipelines.utils.blocks import (@@ -89,6 +90,7 @@ from ltx_pipelines.utils.tiled_diffusion import ()from ltx_pipelines.utils.types import (ImageConditioningInput,+ LoadedImageConditioning,OffloadMode,PipelineOutput,VideoAudio,@@ -265,7 +267,7 @@ class _ICLoraRunContext:noiser: GaussianNoiserdtype: torch.dtypevae_dtype: torch.dtype- images: list[ImageConditioningInput]+ images: list[LoadedImageConditioning]video_context: torch.Tensoraudio_context: torch.Tensornum_frames: int@@ -442,7 +444,8 @@ class ICLoraPipeline:decode_with_keyframes: bool = False,chunk_config: ChunkConfig | None = None,) -> _ICLoraRunContext:- images = self.image_conditioner.resolve_crf(images)+ enhance_prompt_image = images[0].path if images else None+ images = self.image_conditioner.load_conditionings(images, color_space)assert_generated_keyframes_request(decode_with_keyframes,generated_keyframes,@@ -478,7 +481,7 @@ class ICLoraPipeline:[prompt],enhance_first_prompt=enhance_prompt,enhance_static_cache=enhance_static_cache,- enhance_prompt_image=images[0][0] if len(images) > 0 else None,+ enhance_prompt_image=enhance_prompt_image,)video_context, audio_context = ctx_p.video_encoding, ctx_p.audio_encoding@@ -540,7 +543,6 @@ class ICLoraPipeline:chunk,images=ctx.images,video_encoder=enc,- color_space=ctx.color_space,),*reference_video_conditionings_for_chunk(chunk,@@ -887,6 +889,12 @@ def main() -> None:),)args = parser.parse_args()+ hdr = resolve_hdr_color_space(+ images=args.images,+ video_paths=[path for path, _ in args.video_conditioning],+ hdr=args.hdr,+ )+ video_marker, audio_marker = markers_from_args(args, device=get_device(), color_space=hdr)# Load mask video if provided via --conditioning-attention-maskconditioning_attention_mask = None@@ -921,11 +929,6 @@ def main() -> None:skip_stage_2=args.skip_stage_2,),)- hdr = resolve_hdr_color_space(- images=args.images,- video_paths=[path for path, _ in args.video_conditioning],- hdr=args.hdr,- )vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)chunk_config = chunk_config_from_args(args)chunks, num_frames, tiling_config = pipeline.stream_chunks(@@ -968,6 +971,8 @@ def main() -> None:video_chunks_number=video_chunks_number,color_space=hdr,audio_sampling_rate=audio_sampling_rate,+ video_marker=video_marker,+ audio_marker=audio_marker,)except _IC_LORA_OOM as exc:reraise_ic_lora_oom(exc, tile=args.tile)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/keyframe_interpolation.py b/packages/ltx-pipelines/src/ltx_pipelines/keyframe_interpolation.pydeleted file mode 100644index ca96a77..0000000--- a/packages/ltx-pipelines/src/ltx_pipelines/keyframe_interpolation.py+++ /dev/null@@ -1,375 +0,0 @@-import logging--import torch--from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy-from ltx_core.components.guiders import (- MultiModalGuiderFactory,- MultiModalGuiderParams,- create_multimodal_guider_factory,-)-from ltx_core.components.noisers import GaussianNoiser-from ltx_core.components.schedulers import LTX2Scheduler-from ltx_core.loader import LoraPathStrengthAndSDOps-from ltx_core.loader.registry import Registry-from ltx_core.model.transformer.compiling import CompilationConfig-from ltx_core.model.video_vae import AUTO_TILING, AutoTiling, TilingConfig, get_video_chunks_number-from ltx_core.model.video_vae.transformer import DiffVAEMode-from ltx_core.quantization import QuantizationPolicy-from ltx_core.types import VideoPixelShape-from ltx_pipelines.utils.args import (- default_2_stage_arg_parser,- resolve_cli_params,-)-from ltx_pipelines.utils.blocks import (- AudioDecoder,- DiffusionStage,- ImageConditioner,- PromptEncoder,- VideoDecoder,- VideoUpsampler,-)-from ltx_pipelines.utils.constants import (- STAGE_2_DISTILLED_SIGMAS,-)-from ltx_pipelines.utils.denoisers import FactoryGuidedDenoiser, SimpleDenoiser-from ltx_pipelines.utils.helpers import (- assert_resolution,- create_initial_av_latents,- ensure_tiling_config,- get_device,- image_conditionings_by_adding_guiding_latent,- tiling_scale_factors_for_vae,-)-from ltx_pipelines.utils.media_io import (- EXRColorSpace,- encode_video,- resolve_hdr_color_space,- vae_dtype_for_hdr,-)-from ltx_pipelines.utils.model_paths import ModelPaths-from ltx_pipelines.utils.types import (- ImageConditioningInput,- ModalitySpec,- OffloadMode,- PipelineOutput,- VideoAudio,-)---class KeyframeInterpolationPipeline:- """- Keyframe-based Two-stage video interpolation pipeline.- Interpolates between keyframes to generate a video with smoother transitions.- Stage 1 generates video at half of the target resolution, then Stage 2 upsamples- by 2x and refines with additional denoising steps for higher quality output.- Stage 1 uses full model while Stage 2 uses distilled LORA for efficiency,- as the upsampled video already has good quality and just needs refinement.- """-- def __init__( # noqa: PLR0913- self,- model_paths: ModelPaths,- distilled_lora: list[LoraPathStrengthAndSDOps],- spatial_upsampler_path: str,- loras: list[LoraPathStrengthAndSDOps],- device: torch.device | None = None,- quantization: QuantizationPolicy | None = None,- registry: Registry | None = None,- compilation_config: CompilationConfig | None = None,- offload_mode: OffloadMode = OffloadMode.NONE,- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,- prompt_enhancer_gemma_root: str | None = None,- diffvae_optimization: DiffVAEMode = DiffVAEMode.CHUNKED_EAGER,- ):- self.device = device or get_device()- self.dtype = torch.bfloat16- self._scheduler = LTX2Scheduler()-- self.prompt_encoder = PromptEncoder(- model_paths,- self.dtype,- self.device,- registry=registry,- offload_mode=offload_mode,- alloc_trim_strategy=alloc_trim_strategy,- prompt_enhancer_gemma_root=prompt_enhancer_gemma_root,- )- self.image_conditioner = ImageConditioner(- model_paths.video_vae(),- self.dtype,- self.device,- registry=registry,- alloc_trim_strategy=alloc_trim_strategy,- )- self.stage_1 = DiffusionStage.from_checkpoint(- model_paths.transformer(),- self.dtype,- self.device,- loras=tuple(loras),- quantization=quantization,- registry=registry,- compilation_config=compilation_config,- offload_mode=offload_mode,- alloc_trim_strategy=alloc_trim_strategy,- )- stage_2_loras = (*tuple(loras), *tuple(distilled_lora))- self.stage_2 = DiffusionStage.from_checkpoint(- model_paths.transformer(),- self.dtype,- self.device,- loras=stage_2_loras,- quantization=quantization,- registry=registry,- compilation_config=compilation_config,- offload_mode=offload_mode,- alloc_trim_strategy=alloc_trim_strategy,- )- self.upsampler = VideoUpsampler(- model_paths.video_vae(),- spatial_upsampler_path,- self.dtype,- self.device,- registry=registry,- alloc_trim_strategy=alloc_trim_strategy,- )- self.video_decoder = VideoDecoder(- model_paths.video_vae(),- self.dtype,- self.device,- registry=registry,- alloc_trim_strategy=alloc_trim_strategy,- diffvae_optimization=diffvae_optimization,- )- self.audio_decoder = AudioDecoder(- model_paths.audio_vae(),- self.dtype,- self.device,- registry=registry,- alloc_trim_strategy=alloc_trim_strategy,- )-- def __call__( # noqa: PLR0913- self,- prompt: str,- negative_prompt: str,- seed: int,- height: int,- width: int,- num_frames: int,- frame_rate: float,- num_inference_steps: int,- video_guider_params: MultiModalGuiderParams | MultiModalGuiderFactory,- audio_guider_params: MultiModalGuiderParams | MultiModalGuiderFactory,- images: list[ImageConditioningInput],- vae_dtype: torch.dtype | None = None,- tiling_config: TilingConfig | AutoTiling | None = AUTO_TILING,- enhance_prompt: bool = False,- enhance_static_cache: bool = False,- max_batch_size: int = 1,- stage_1_sigmas: torch.Tensor | None = None,- stage_2_sigmas: torch.Tensor = STAGE_2_DISTILLED_SIGMAS,- color_space: EXRColorSpace | None = None,- ) -> PipelineOutput:- images = self.image_conditioner.resolve_crf(images)- assert_resolution(height=height, width=width, is_two_stage=True)-- generator = torch.Generator(device=self.device).manual_seed(seed)- noiser = GaussianNoiser(generator=generator)- dtype = torch.bfloat16- if vae_dtype is None:- vae_dtype = dtype-- ctx_p, ctx_n = self.prompt_encoder(- [prompt, negative_prompt],- enhance_first_prompt=enhance_prompt,- enhance_static_cache=enhance_static_cache,- enhance_prompt_image=images[0][0] if len(images) > 0 else None,- )- v_context_p, a_context_p = ctx_p.video_encoding, ctx_p.audio_encoding- v_context_n, a_context_n = ctx_n.video_encoding, ctx_n.audio_encoding-- scale_factors = tiling_scale_factors_for_vae(self.video_decoder.checkpoint_path)- tiling_config = ensure_tiling_config(- tiling_config,- scale_factors=scale_factors,- vae_checkpoint_path=self.video_decoder.checkpoint_path,- video_shape=VideoPixelShape(batch=1, frames=num_frames, height=height, width=width, fps=frame_rate),- diffvae_optimization=self.video_decoder.diffvae_optimization,- device=self.device,- )-- # Stage 1: Initial low resolution video generation.- sigmas = (- stage_1_sigmas if stage_1_sigmas is not None else self._scheduler.execute(steps=num_inference_steps)- ).to(dtype=torch.float32, device=self.device)-- stage_1_output_shape = VideoPixelShape(- batch=1,- frames=num_frames,- width=width // 2,- height=height // 2,- fps=frame_rate,- )- stage_1_conditionings = self.image_conditioner(- lambda enc: image_conditionings_by_adding_guiding_latent(- images=images,- height=stage_1_output_shape.height,- width=stage_1_output_shape.width,- video_encoder=enc,- dtype=dtype,- device=self.device,- color_space=color_space,- )- )-- video_guider_factory = create_multimodal_guider_factory(- params=video_guider_params,- negative_context=v_context_n,- )- audio_guider_factory = create_multimodal_guider_factory(- params=audio_guider_params,- negative_context=a_context_n,- )-- video_latent, audio_latent = create_initial_av_latents(- width=stage_1_output_shape.width,- height=stage_1_output_shape.height,- frames=num_frames,- fps=frame_rate,- device=self.device,- dtype=self.dtype,- video_scale_factors=self.stage_1.video_scale_factors,- )- video_state, audio_state = self.stage_1(- denoiser=FactoryGuidedDenoiser(- v_context=v_context_p,- a_context=a_context_p,- video_guider_factory=video_guider_factory,- audio_guider_factory=audio_guider_factory,- ),- sigmas=sigmas,- noiser=noiser,- modalities=VideoAudio(- video=ModalitySpec(- latent=video_latent,- conditioning_fps=frame_rate,- context=v_context_p,- conditionings=stage_1_conditionings,- ),- audio=ModalitySpec(- latent=audio_latent,- conditioning_fps=frame_rate,- context=a_context_p,- ),- ),- max_batch_size=max_batch_size,- )-- # Stage 2: Upsample and refine the video at higher resolution with distilled LORA.- upscaled_video_latent = self.upsampler(video_state.latent[:1])-- stage_2_sigmas = stage_2_sigmas.to(dtype=torch.float32, device=self.device)- stage_2_output_shape = VideoPixelShape(batch=1, frames=num_frames, width=width, height=height, fps=frame_rate)- stage_2_conditionings = self.image_conditioner(- lambda enc: image_conditionings_by_adding_guiding_latent(- images=images,- height=stage_2_output_shape.height,- width=stage_2_output_shape.width,- video_encoder=enc,- dtype=dtype,- device=self.device,- color_space=color_space,- )- )-- video_state, audio_state = self.stage_2(- denoiser=SimpleDenoiser(v_context_p, a_context_p),- sigmas=stage_2_sigmas,- noiser=noiser,- modalities=VideoAudio(- video=ModalitySpec(- latent=upscaled_video_latent,- conditioning_fps=frame_rate,- context=v_context_p,- conditionings=stage_2_conditionings,- noise_scale=stage_2_sigmas[0].item(),- ),- audio=ModalitySpec(- latent=audio_state.latent,- conditioning_fps=frame_rate,- context=a_context_p,- noise_scale=stage_2_sigmas[0].item(),- ),- ),- )-- decoded_video = self.video_decoder(video_state.latent, tiling_config, generator, dtype=vae_dtype)- decoded_audio = self.audio_decoder(audio_state.latent)- return PipelineOutput(decoded_video, decoded_audio, num_frames, tiling_config)--[email protected]_mode()-def main() -> None:- logging.basicConfig(level=logging.INFO)- params = resolve_cli_params()- parser = default_2_stage_arg_parser(params=params)- args = parser.parse_args()- pipeline = KeyframeInterpolationPipeline(- model_paths=args.model_paths,- distilled_lora=args.distilled_lora,- spatial_upsampler_path=args.spatial_upsampler_path,- loras=tuple(args.lora) if args.lora else (),- quantization=args.quantization,- compilation_config=args.compile,- offload_mode=args.offload_mode,- prompt_enhancer_gemma_root=args.prompt_enhancer_gemma_root,- diffvae_optimization=args.diffvae_optimization,- )- hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)- vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)- result = pipeline(- prompt=args.prompt,- negative_prompt=args.negative_prompt,- seed=args.seed,- height=args.height,- width=args.width,- num_frames=args.num_frames,- frame_rate=args.frame_rate,- num_inference_steps=args.num_inference_steps,- video_guider_params=MultiModalGuiderParams(- cfg_scale=args.video_cfg_guidance_scale,- stg_scale=args.video_stg_guidance_scale,- rescale_scale=args.video_rescale_scale,- modality_scale=args.a2v_guidance_scale,- skip_step=args.video_skip_step,- stg_blocks=args.video_stg_blocks,- ),- audio_guider_params=MultiModalGuiderParams(- cfg_scale=args.audio_cfg_guidance_scale,- stg_scale=args.audio_stg_guidance_scale,- rescale_scale=args.audio_rescale_scale,- modality_scale=args.v2a_guidance_scale,- skip_step=args.audio_skip_step,- stg_blocks=args.audio_stg_blocks,- ),- images=args.images,- vae_dtype=vae_dtype,- color_space=hdr,- enhance_prompt=args.enhance_prompt,- enhance_static_cache=args.enhance_static_cache,- tiling_config=AUTO_TILING,- max_batch_size=args.max_batch_size,- )- encode_video(- video=result.video,- fps=args.frame_rate,- audio=result.audio,- output_path=args.output_path,- video_chunks_number=get_video_chunks_number(result.num_frames, result.tiling_config),- color_space=hdr,- )---if __name__ == "__main__":- main()diff --git a/packages/ltx-pipelines/src/ltx_pipelines/marking/__init__.py b/packages/ltx-pipelines/src/ltx_pipelines/marking/__init__.pynew file mode 100644index 0000000..3df6254--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/marking/__init__.py@@ -0,0 +1,76 @@+"""Post-hoc video and audio marking (Videoseal / AudioSeal opt-in).+Importing this package does not require the ``videoseal`` or ``audioseal`` extras.+A pack directory is loaded with :func:`create_markers`. Seal classes live in+``ltx_pipelines.marking.video_seal`` and ``ltx_pipelines.marking.audio_seal``.+"""++from __future__ import annotations++from pathlib import Path++import torch++from ltx_pipelines.marking.mark_detection import detect_audio_mark, detect_video_mark+from ltx_pipelines.marking.types import AudioMarker, MarkDetectResult, VideoMarker++PIXELSEAL_CHECKPOINT_NAME = "pixelseal.pth"+PIXELSEAL_CARD_NAME = "pixelseal.yaml"+PIXELSEAL_ATTENUATION_NAME = "videoseal_attenuation.yaml"+AUDIOSEAL_GENERATOR_NAME = "audioseal_generator.pth"+AUDIOSEAL_DETECTOR_NAME = "audioseal_detector.pth"++_PACK_FILES = (+ PIXELSEAL_CHECKPOINT_NAME,+ PIXELSEAL_CARD_NAME,+ PIXELSEAL_ATTENUATION_NAME,+ AUDIOSEAL_GENERATOR_NAME,+ AUDIOSEAL_DETECTOR_NAME,+)+++def _require_marking_pack(root: Path | str) -> Path:+ pack = Path(root)+ missing = [name for name in _PACK_FILES if not (pack / name).is_file()]+ if missing:+ raise FileNotFoundError(f"marking pack {pack} requires {', '.join(missing)}")+ return pack+++def create_markers(root: Path | str, device: torch.device) -> tuple[VideoMarker, AudioMarker]:+ """Load VideoSeal and AudioSeal markers from a marking pack directory.+ All pack files must be present. Both extras are imported (fail fast).+ """+ pack = _require_marking_pack(root)+ try:+ from ltx_pipelines.marking.audio_seal import AudioSealMarker # noqa: PLC0415+ from ltx_pipelines.marking.video_seal import VideoSealMarker # noqa: PLC0415+ except ModuleNotFoundError as exc:+ raise ImportError("videoseal and audioseal packages are required") from exc+ return (+ VideoSealMarker.create(+ pack / PIXELSEAL_CHECKPOINT_NAME,+ card=pack / PIXELSEAL_CARD_NAME,+ attenuation_config=pack / PIXELSEAL_ATTENUATION_NAME,+ device=device,+ ),+ AudioSealMarker.create(+ pack / AUDIOSEAL_GENERATOR_NAME,+ pack / AUDIOSEAL_DETECTOR_NAME,+ device=device,+ ),+ )+++__all__ = [+ "AUDIOSEAL_DETECTOR_NAME",+ "AUDIOSEAL_GENERATOR_NAME",+ "PIXELSEAL_ATTENUATION_NAME",+ "PIXELSEAL_CARD_NAME",+ "PIXELSEAL_CHECKPOINT_NAME",+ "AudioMarker",+ "MarkDetectResult",+ "VideoMarker",+ "create_markers",+ "detect_audio_mark",+ "detect_video_mark",+]diff --git a/packages/ltx-pipelines/src/ltx_pipelines/marking/audio_seal.py b/packages/ltx-pipelines/src/ltx_pipelines/marking/audio_seal.pynew file mode 100644index 0000000..b964847--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/marking/audio_seal.py@@ -0,0 +1,126 @@+"""AudioSeal marker wrapping Meta's generator and detector."""++from __future__ import annotations++import logging+from pathlib import Path++import torch+from audioseal import AudioSeal+from einops import rearrange+from torch import nn++from ltx_core.types import Audio+from ltx_pipelines.marking.score import score_bits+from ltx_pipelines.marking.types import AudioMarker, MarkDetectResult++logger = logging.getLogger(__name__)++# Baked payload seed shared by embed and detect. Not a signature and not a model version.+AUDIOSEAL_PAYLOAD_SEED = 0x4C5458 # ASCII "LTX"+AUDIOSEAL_NBITS = 16+++class AudioSealMarker(AudioMarker):+ """Adapter around loaded AudioSeal generator and detector modules."""++ def __init__(self, generator: nn.Module, detector: nn.Module, *, nbits: int) -> None:+ self._generator = generator+ self._detector = detector+ self._nbits = nbits+ self._payload = baked_audioseal_payload(nbits)++ @classmethod+ def create(+ cls,+ generator: Path | str,+ detector: Path | str,+ *,+ device: torch.device,+ nbits: int = AUDIOSEAL_NBITS,+ ) -> AudioSealMarker:+ """Load local generator and detector; ``eval``, drop grad, move to ``device``."""+ logger.info("Loading AudioSeal from %s and %s", generator, detector)+ gen, det = load_audioseal(generator, detector, nbits=nbits)+ for module in (gen, det):+ module.eval()+ module.requires_grad_(False)+ module.to(device)+ return cls(gen, det, nbits=nbits)++ @property+ def name(self) -> str:+ return "audioseal"++ def embed(self, audio: Audio) -> Audio:+ """Stamp one stereo ``Audio`` as a batch of two mono channels (no downmix)."""+ device = next(self._generator.parameters()).device+ wav, pattern = waveform_to_channel_batch(audio.waveform)+ wav = wav.to(device=device, dtype=torch.float32)+ message = self._payload.to(device=device, dtype=torch.long).unsqueeze(0).expand(wav.shape[0], -1)+ stamped = self._generator(wav, message=message)+ return Audio(+ waveform=rearrange(stamped, f"b 1 t -> {pattern}").to(+ device=audio.waveform.device, dtype=audio.waveform.dtype+ ),+ sampling_rate=audio.sampling_rate,+ )++ def detect(self, audio: Audio) -> MarkDetectResult:+ """Run the detector; ``detected`` is the clip-level presence score (mean >= 0.5)."""+ device = next(self._detector.parameters()).device+ wav, _ = waveform_to_channel_batch(audio.waveform)+ wav = wav.to(device=device, dtype=torch.float32)+ detection, message = self._detector(wav)+ detect_score = float((detection[:, 1, :] > 0.5).to(dtype=torch.float32).mean().item())+ decoded_bits = (message.mean(dim=0) > 0.5).to(dtype=torch.float32)+ bit_accuracy, p_value = score_bits(decoded_bits, self._payload)+ return MarkDetectResult(+ detected=detect_score >= 0.5,+ bit_accuracy=bit_accuracy,+ p_value=p_value,+ payload=self._payload.detach().cpu(),+ )+++def baked_audioseal_payload(nbits: int = AUDIOSEAL_NBITS) -> torch.Tensor:+ """Return the baked payload as a ``[nbits]`` integer ``{0, 1}`` tensor."""+ generator = torch.Generator(device="cpu")+ generator.manual_seed(AUDIOSEAL_PAYLOAD_SEED)+ return torch.randint(0, 2, (nbits,), generator=generator)+++def waveform_to_channel_batch(waveform: torch.Tensor) -> tuple[torch.Tensor, str]:+ """Convert PCM to AudioSeal ``(b, 1, t)`` (stereo as batch-2, no downmix).+ The returned string is the einops source pattern (``1 b t``, ``t b``, or ``b t``).+ """+ if waveform.ndim == 3:+ if waveform.shape[0] != 1:+ raise ValueError(f"expected batched waveform (1, C, T); got {tuple(waveform.shape)}")+ pattern = "1 b t"+ elif waveform.ndim != 2:+ raise ValueError(f"audio.waveform must be (C, T), (T, C), or (1, C, T); got {tuple(waveform.shape)}")+ elif waveform.shape[1] == 2:+ pattern = "t b"+ elif waveform.shape[0] == 2:+ pattern = "b t"+ else:+ raise ValueError(f"audio.waveform must be stereo (2, N) or (N, 2); got {tuple(waveform.shape)}")+ return rearrange(waveform, f"{pattern} -> b 1 t"), pattern+++def load_audioseal(+ generator: Path | str,+ detector: Path | str,+ *,+ nbits: int = AUDIOSEAL_NBITS,+) -> tuple[nn.Module, nn.Module]:+ """Load generator and detector from local file paths (no Hub download)."""+ generator_path = Path(generator)+ detector_path = Path(detector)+ missing = [str(path) for path in (generator_path, detector_path) if not path.is_file()]+ if missing:+ raise FileNotFoundError(f"AudioSeal checkpoints missing: {', '.join(missing)}")+ generator = AudioSeal.load_generator(str(generator_path), nbits=nbits)+ detector = AudioSeal.load_detector(str(detector_path), nbits=nbits)+ return generator, detectordiff --git a/packages/ltx-pipelines/src/ltx_pipelines/marking/mark_detection.py b/packages/ltx-pipelines/src/ltx_pipelines/marking/mark_detection.pynew file mode 100644index 0000000..6651113--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/marking/mark_detection.py@@ -0,0 +1,32 @@+"""File-level mark detect: decode stamp-domain media, then call the marker."""++from __future__ import annotations++from collections.abc import Iterator+from pathlib import Path++import torch+from torch._prims_common import DeviceLikeType++from ltx_core.devices import get_preferred_device+from ltx_pipelines.marking.types import AudioMarker, MarkDetectResult, VideoMarker+from ltx_pipelines.utils.media_io.decode import decode_audio_from_file, decode_video_by_frame+++def detect_video_mark(marker: VideoMarker, path: str | Path) -> MarkDetectResult:+ """Detects a marked file at ``path`` if it was marked by marker."""+ return marker.detect(_sdr_detect_frames(path, get_preferred_device()))+++def detect_audio_mark(marker: AudioMarker, path: str | Path) -> MarkDetectResult | None:+ """Detects a marked file at ``path`` if it was marked by marker."""+ decoded = decode_audio_from_file(str(path), get_preferred_device())+ if decoded is None:+ return None+ return marker.detect(decoded)+++def _sdr_detect_frames(path: str | Path, device: DeviceLikeType) -> Iterator[torch.Tensor]:+ """Yield SDR RGB frames ``[H, W, C]`` in ``[0, 1]``."""+ for frame in decode_video_by_frame(str(path), device):+ yield frame.squeeze(0).to(dtype=torch.float32).div_(255.0)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/marking/score.py b/packages/ltx-pipelines/src/ltx_pipelines/marking/score.pynew file mode 100644index 0000000..d48df92--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/marking/score.py@@ -0,0 +1,30 @@+"""Binomial bit match of decoded bits against a baked payload.+Does not decide ``detected`` — each marker applies its own call.+"""++from __future__ import annotations++import torch+from scipy import stats+++def score_bits(decoded: torch.Tensor, payload: torch.Tensor) -> tuple[float, float]:+ """Binomial bit match of ``decoded`` against ``payload``.+ Args:+ decoded: ``{0, 1}`` bits (any shape). Flattened before the test.+ payload: ``{0, 1}`` bits (any shape). Flattened length must match ``decoded``.+ Returns:+ ``(bit_accuracy, p_value)``. ``p_value`` is a one-tailed binomial test+ (``alternative="greater"``) against a chance match of ``p=0.5``.+ Raises:+ ValueError: If the flattened lengths differ.+ """+ decoded = decoded.reshape(-1).to(dtype=torch.float32)+ expected = payload.reshape(-1).to(device=decoded.device, dtype=torch.float32)+ if decoded.numel() != expected.numel():+ raise ValueError(f"detector bits {decoded.numel()} != payload bits {expected.numel()}")+ matches = int((decoded == expected).sum().item())+ nbits = int(expected.numel())+ bit_accuracy = matches / nbits+ p_value = float(stats.binomtest(matches, nbits, 0.5, alternative="greater").pvalue)+ return bit_accuracy, p_valuediff --git a/packages/ltx-pipelines/src/ltx_pipelines/marking/types.py b/packages/ltx-pipelines/src/ltx_pipelines/marking/types.pynew file mode 100644index 0000000..18ee44a--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/marking/types.py@@ -0,0 +1,59 @@+"""Public video and audio marker protocols and detect result."""++from __future__ import annotations++from collections.abc import Iterator+from typing import NamedTuple, Protocol++import torch++from ltx_core.types import Audio+++class MarkDetectResult(NamedTuple):+ """Clip-level detector output.+ ``bit_accuracy`` and ``p_value`` are the binomial test of decoded bits vs the baked+ payload. ``detected`` is this marker's own call (not that test).+ """++ detected: bool+ bit_accuracy: float+ p_value: float+ payload: torch.Tensor | None+++class VideoMarker(Protocol):+ """Embeds a watermark into RGB frames and detects it in a frame stream."""++ @property+ def name(self) -> str:+ """Short identifier."""+ ...++ def embed(self, frames: torch.Tensor) -> torch.Tensor:+ """Stamp one ``[F, H, W, C]`` RGB tensor in ``[0, 1]``; return the same layout."""+ ...++ def detect(self, frames: Iterator[torch.Tensor]) -> MarkDetectResult:+ """Detect the watermark in a stream of ``[H, W, C]`` RGB frames in ``[0, 1]``."""+ ...+++class AudioMarker(Protocol):+ """Embeds a watermark into stereo PCM and detects it in an ``Audio`` track."""++ @property+ def name(self) -> str:+ """Short identifier."""+ ...++ def embed(self, audio: Audio) -> Audio:+ """Stamp one yielded stereo ``Audio`` track."""+ ...++ def detect(self, audio: Audio) -> MarkDetectResult:+ """Detect the audio watermark in one stereo ``Audio`` track."""+ ...+++__all__ = ["AudioMarker", "MarkDetectResult", "VideoMarker"]diff --git a/packages/ltx-pipelines/src/ltx_pipelines/marking/video_seal.py b/packages/ltx-pipelines/src/ltx_pipelines/marking/video_seal.pynew file mode 100644index 0000000..6f001a9--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/marking/video_seal.py@@ -0,0 +1,141 @@+"""Videoseal video marker wrapping Meta's ``Videoseal`` module."""++from __future__ import annotations++import logging+import math+from collections.abc import Iterator+from itertools import islice+from pathlib import Path++import torch+from einops import rearrange+from omegaconf import OmegaConf+from videoseal.utils.cfg import SubModelConfig, VideosealConfig, setup_model++from ltx_pipelines.marking.score import score_bits+from ltx_pipelines.marking.types import MarkDetectResult, VideoMarker++logger = logging.getLogger(__name__)++# Baked payload seed shared by embed and detect. Not a signature and not a model version.+VIDEOSEAL_PAYLOAD_SEED = 0x4C5458 # ASCII "LTX"++# Max sampled frames held on GPU for one ``detect`` call. Logits are accumulated.+_DETECT_FRAME_BATCH = 8+++class VideoSealMarker(VideoMarker):+ """Adapter around a loaded Meta ``Videoseal`` module (PixelSeal, ChunkySeal, …)."""++ def __init__(self, model: torch.nn.Module, *, name: str, nbits: int) -> None:+ self._model = model+ self._name = name+ self._payload = baked_videoseal_payload(nbits)++ @classmethod+ def create(+ cls,+ checkpoint: Path | str,+ *,+ card: Path | str,+ attenuation_config: Path | str,+ device: torch.device,+ ) -> VideoSealMarker:+ """Load local checkpoint, card, and attenuation; ``eval``, drop grad, move to ``device``."""+ card_path = Path(card)+ logger.info("Loading Videoseal card %s from %s", card_path, checkpoint)+ model, nbits = load_videoseal(+ checkpoint,+ card=card_path,+ attenuation_config=attenuation_config,+ )+ model.eval()+ model.requires_grad_(False)+ model.to(device)+ return cls(model, name=card_path.stem, nbits=nbits)++ @property+ def name(self) -> str:+ return self._name++ def embed(self, frames: torch.Tensor) -> torch.Tensor:+ """Stamp one ``[F, H, W, C]`` RGB tensor; Videoseal resizes internally to 256 and back."""+ device = next(self._model.parameters()).device+ nchw = rearrange(frames.to(device=device, dtype=torch.float32), "f h w c -> f c h w")+ payload = self._payload.to(device=device, dtype=torch.float32)+ stamped = self._model.embed(nchw, msgs=payload, is_video=True, lowres_attenuation=True)["imgs_w"]+ return rearrange(stamped, "f c h w -> f h w c").clamp(0.0, 1.0).to(device=frames.device)++ def detect(self, frames: Iterator[torch.Tensor]) -> MarkDetectResult:+ """Sample at ``step_size``, average logits; ``detected`` is ``log10(p_value) < -6``."""+ device = next(self._model.parameters()).device+ step_size = int(getattr(self._model, "step_size", 8))+ logits: list[torch.Tensor] = []+ sampled = _nchw_sampled_frames(frames, step_size=step_size, device=device)+ for video in _nchw_batches(sampled, _DETECT_FRAME_BATCH):+ bit_logits = self._model.detect(video, is_video=True)["preds"][:, 1:]+ while bit_logits.ndim > 2:+ bit_logits = bit_logits.mean(dim=-1)+ logits.append(bit_logits)+ if not logits:+ raise ValueError("no frames to detect")+ mean_logits = torch.cat(logits, dim=0).mean(dim=0)+ decoded = (mean_logits > 0).to(dtype=torch.float32)+ bit_accuracy, p_value = score_bits(decoded, self._payload)+ detected = p_value == 0.0 or math.log10(p_value) < -6+ return MarkDetectResult(+ detected=detected,+ bit_accuracy=bit_accuracy,+ p_value=p_value,+ payload=self._payload.detach().cpu(),+ )+++def _nchw_sampled_frames(+ frames: Iterator[torch.Tensor], *, step_size: int, device: torch.device+) -> Iterator[torch.Tensor]:+ for index, frame in enumerate(frames):+ if index % step_size != 0:+ continue+ rgb = frame.to(device=device, dtype=torch.float32)+ if rgb.ndim != 3 or rgb.shape[-1] != 3:+ raise ValueError(f"expected frame [H, W, C]; got {tuple(rgb.shape)}")+ yield rearrange(rgb, "h w c -> c h w")+++def _nchw_batches(frames: Iterator[torch.Tensor], batch_size: int) -> Iterator[torch.Tensor]:+ while batch := list(islice(frames, batch_size)):+ yield torch.stack(batch, dim=0)+++def baked_videoseal_payload(nbits: int) -> torch.Tensor:+ """Return the baked payload as a ``[1, nbits]`` float ``{0, 1}`` tensor."""+ generator = torch.Generator(device="cpu")+ generator.manual_seed(VIDEOSEAL_PAYLOAD_SEED)+ return torch.randint(0, 2, (1, nbits), generator=generator).float()+++def load_videoseal(+ checkpoint: Path | str,+ *,+ card: Path | str,+ attenuation_config: Path | str,+) -> tuple[torch.nn.Module, int]:+ """Build a Videoseal module from local checkpoint, card YAML, and attenuation YAML paths.+ Videoseal's ``setup_model_from_checkpoint`` looks up repo-root ``configs/*.yaml``+ that the installed package does not ship. Pass the JND attenuation YAML from the pack+ (``videoseal_attenuation.yaml``).+ """+ card_path = Path(card)+ if not card_path.is_file():+ raise FileNotFoundError(f"Videoseal model card not found: {card_path}")+ raw = OmegaConf.load(card_path)+ raw.args.attenuation_config = str(Path(attenuation_config))+ nbits = int(raw.args.nbits)+ config = VideosealConfig(+ args=raw.args,+ embedder=SubModelConfig(model=str(raw.embedder.model), params=raw.embedder.params),+ extractor=SubModelConfig(model=str(raw.extractor.model), params=raw.extractor.params),+ )+ return setup_model(config, Path(checkpoint)), nbitsdiff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/controller.py b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/controller.pyindex d90122d..b340e24 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/controller.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/controller.py@@ -161,12 +161,18 @@ class MGPUController:num_gpus: int | None = None,devices: Sequence[int] | None = None,logs_specs: LogsSpecs | None = None,+ cpu_threads_per_rank: int | None = 8,) -> None:"""`num_gpus` uses GPUs 0..num_gpus-1 (default: all). `devices` places the fleet on specificphysical GPUs (e.g. `[2, 3]` -> rank r on cuda:devices[r]), so several controllers can share abox on disjoint GPU sets; the two are mutually exclusive. Indices are as the controller seesthem; every GPU stays visible to each worker, which simply binds to its assigned one.+ `cpu_threads_per_rank` sets each rank's intra-op CPU threads (`torch.set_num_threads`); None+ keeps torch's default. Ranks only run small host tensor ops, which a pool as wide as the machine+ stalls."""+ if cpu_threads_per_rank is not None and cpu_threads_per_rank < 1:+ raise ValueError(f"cpu_threads_per_rank must be >= 1: {cpu_threads_per_rank}.")if devices is not None:if num_gpus is not None:raise ValueError("Pass either num_gpus or devices, not both.")@@ -176,6 +182,7 @@ class MGPUController:self._devices = list(devices) if devices is not None else Noneself._num_gpus = num_gpusself._logs_specs = logs_specs+ self._cpu_threads_per_rank = cpu_threads_per_rankself._spawn_ctx = torch_mp.get_context("spawn")self._fleet: _RunnersFleet | None = None@@ -232,6 +239,7 @@ class MGPUController:num_gpus=num_gpus,logs_specs=self._logs_specs or DefaultLogsSpecs(tee=Std.ALL),device_ids=device_ids,+ cpu_threads=self._cpu_threads_per_rank,)try:self._await_ready(num_gpus, timeout.total_seconds())diff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/decode_tiling.py b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/decode_tiling.pynew file mode 100644index 0000000..1ec8c74--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/decode_tiling.py@@ -0,0 +1,143 @@+"""Distributed-decode tile planning shared by the multi-GPU runners.+A Dist decode splits the latent over the ranks of a VAE group, and each rank decodes its share+with a :class:`TilingConfig` of its own. :func:`dist_auto_tiling` resolves ``AUTO_TILING`` to one+plan for the whole group: every rank plans the group's largest Dist tile from the group's+smallest budgets, so the plans, and whether the fallback budget is needed, agree on every rank,+cold or warm, without a further collective.+"""++from __future__ import annotations++from dataclasses import dataclass+from functools import wraps+from typing import Protocol++import torch+import torch.distributed as dist++from ltx_core.devices import cuda_allocatable_bytes, resolve_device+from ltx_core.model.video_vae import AutoTiling, TilingConfig+from ltx_core.model.video_vae.transformer import DiffVAEMode+from ltx_core.multigpu.vae.distributed_decoder import dist_rank_tile_pixel_shape, dist_rank_tiles+from ltx_core.tiling import TileCountConfig+from ltx_core.types import VIDEO_SCALE_FACTORS, SpatioTemporalScaleFactors, VideoLatentShape, VideoPixelShape++# Share of device memory the decode plan may fill. The rest serves what runs beside the decode:+# streamed chunks waiting for their temporal batch, the batch being blended, and the encoder.+DECODE_LIVE_SHARE = 0.858+# How ensure_tiling_config's ValueError starts when no tile fits the budget.+_NO_TILE_FITS = "Cannot fit a DiffVAE decode tile"+++class EnsureTilingConfig(Protocol):+ """The signature of :func:`ltx_pipelines.utils.helpers.ensure_tiling_config`."""++ def __call__(+ self,+ tiling_config: TilingConfig | AutoTiling | None,+ *,+ video_shape: VideoPixelShape,+ scale_factors: SpatioTemporalScaleFactors,+ vae_checkpoint_path: str,+ diffvae_optimization: DiffVAEMode = DiffVAEMode.CHUNKED_EAGER,+ device: torch.device | None = None,+ free_bytes: int | None = None,+ keyframes: bool = False,+ ) -> TilingConfig | None: ...+++@dataclass(frozen=True)+class GroupDecodeBudget:+ """The smallest decode budgets over a VAE group, in bytes.+ Attributes:+ planned: ``DECODE_LIVE_SHARE`` of device memory minus allocated bytes. Allocated bytes do+ not depend on what the caching allocator holds, so this is the same cold or warm.+ fallback: Everything a new allocation can still get (:func:`cuda_allocatable_bytes`), for+ a clip the planned budget fits no tile for.+ """++ planned: int+ fallback: int+++def group_decode_budget(device: torch.device | None, group: dist.ProcessGroup | None) -> GroupDecodeBudget:+ """Each rank's complete budgets, MIN-reduced over ``group``; one collective and one host sync."""+ resolved = resolve_device(device)+ total = torch.cuda.get_device_properties(resolved).total_memory+ planned = max(0, int(DECODE_LIVE_SHARE * total) - int(torch.cuda.memory_allocated(resolved)))+ budgets = torch.tensor([planned, cuda_allocatable_bytes(resolved)], dtype=torch.int64, device=resolved)+ dist.all_reduce(budgets, op=dist.ReduceOp.MIN, group=group)+ planned, fallback = budgets.tolist()+ return GroupDecodeBudget(planned=planned, fallback=fallback)+++def dist_max_tile_pixel_shape(+ vae_tiling: TileCountConfig,+ video_shape: VideoPixelShape,+ world_size: int,+ scale: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,+) -> VideoPixelShape:+ """:func:`dist_rank_tile_pixel_shape` of the rank with the largest Dist tile, the same on every rank."""+ latent_shape = VideoLatentShape.from_pixel_shape(video_shape, scale_factors=scale)+ shapes = [+ dist_rank_tile_pixel_shape(+ vae_tiling, latent_shape, scale, rank, world_size, fps=video_shape.fps, batch=video_shape.batch+ )+ for rank in range(world_size)+ if dist_rank_tiles(vae_tiling, latent_shape, scale, rank, world_size)+ ]+ return max(shapes, key=lambda shape: shape.frames * shape.height * shape.width)+++def dist_auto_tiling(+ ensure: EnsureTilingConfig,+ vae_tiling: TileCountConfig,+ *,+ world_size: int,+ group: dist.ProcessGroup | None,+ scale: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,+) -> EnsureTilingConfig:+ """Wrap ``ensure`` so ``AUTO_TILING`` without ``free_bytes`` plans one Dist tile for the group.+ The wrapper plans :func:`dist_max_tile_pixel_shape` from :func:`group_decode_budget`'s+ planned budget, and from its fallback budget if no tile fits that. Every rank joins the one+ collective and plans the same shape from the same numbers, so every rank resolves the same+ plan or raises the same error. Explicit configs, ``None`` and calls with ``free_bytes`` pass+ through unchanged.+ """++ @wraps(ensure)+ def ensure_dist_tile(+ tiling_config: TilingConfig | AutoTiling | None,+ *,+ video_shape: VideoPixelShape,+ scale_factors: SpatioTemporalScaleFactors,+ vae_checkpoint_path: str,+ diffvae_optimization: DiffVAEMode = DiffVAEMode.CHUNKED_EAGER,+ device: torch.device | None = None,+ free_bytes: int | None = None,+ keyframes: bool = False,+ ) -> TilingConfig | None:+ def plan(shape: VideoPixelShape, budget: int | None) -> TilingConfig | None:+ return ensure(+ tiling_config,+ video_shape=shape,+ scale_factors=scale_factors,+ vae_checkpoint_path=vae_checkpoint_path,+ diffvae_optimization=diffvae_optimization,+ device=device,+ free_bytes=budget,+ keyframes=keyframes,+ )++ if not isinstance(tiling_config, AutoTiling) or free_bytes is not None:+ return plan(video_shape, free_bytes)+ tile_shape = dist_max_tile_pixel_shape(vae_tiling, video_shape, world_size, scale)+ budget = group_decode_budget(device, group)+ try:+ return plan(tile_shape, budget.planned)+ except ValueError as exc:+ if not str(exc).startswith(_NO_TILE_FITS):+ raise+ return plan(tile_shape, budget.fallback)++ return ensure_dist_tilediff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/delegating_builder.py b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/delegating_builder.pyindex 0be95cb..6061171 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/delegating_builder.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/delegating_builder.py@@ -72,6 +72,10 @@ class DelegatingBuilder(Generic[InnerModelT]):def keeps_gpu_resident_weights(self) -> bool:return self._inner.keeps_gpu_resident_weights+ @property+ def builds_per_shape(self) -> bool:+ return self._inner.builds_per_shape+def with_registry(self, registry: Registry) -> Self:clone = copy.copy(self)clone._inner = self._inner.with_registry(registry)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/fleet.py b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/fleet.pyindex 420a191..fe1e45d 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/fleet.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/fleet.py@@ -9,6 +9,7 @@ lives in ``runner.py``.from __future__ import annotationsimport contextlib+import gcimport loggingimport osimport socket@@ -26,7 +27,7 @@ from torch.distributed.elastic.multiprocessing import start_processesfrom torch.distributed.elastic.multiprocessing.api import LogsSpecsfrom torch.distributed.elastic.multiprocessing.errors import ProcessFailure, record-from ltx_pipelines.multigpu._broadcast import BroadcastCoordinator+from ltx_core.multigpu.broadcast import BroadcastCoordinatorfrom ltx_pipelines.multigpu.nccl_groups import create_local_nccl_groupsfrom ltx_pipelines.multigpu.runner import MGPURunner, RunnerError@@ -160,6 +161,18 @@ class _RankLink:return shell+def _freeze_long_lived_heap() -> None:+ """After a rank's first job, move everything that survived it out of the cyclic GC's reach.+ The first job builds the models, compiled graphs and caches the later jobs reuse; full+ collections otherwise rescan those objects every time (3.3 s of pauses in a cold 4K DFR job).+ Collected first, so only survivors are frozen; anything a later job leaves in a cycle is+ still collected.+ """+ if gc.get_freeze_count() == 0:+ gc.collect()+ gc.freeze()++def _run_job(runner: MGPURunner, kwargs: dict[str, Any], job_id: int, rank: int, result: Queue[_JobResult]) -> None:"""Run one job on this rank and put each item on the result queue: every yield is a `_JobResult`(tagged by rank), forwarded straight to the caller; the rank's end -- a clean StopIteration or a@@ -194,6 +207,7 @@ def _relay_loop(runner: MGPURunner, link: _RankLink, channels: _Channels) -> Nonif job is None: # shutdown sentinel (already broadcast to the others)return_run_job(runner, job.kwargs, job.job_id, _RELAY_RANK, channels.results)+ _freeze_long_lived_heap()def _worker_loop(runner: MGPURunner, link: _RankLink, channels: _Channels, rank: int) -> None:@@ -203,6 +217,7 @@ def _worker_loop(runner: MGPURunner, link: _RankLink, channels: _Channels, rank:if job is None: # shutdown sentinelreturn_run_job(runner, job.kwargs, job.job_id, rank, channels.results)+ _freeze_long_lived_heap()def _shutdown_distributed() -> None:@@ -223,9 +238,15 @@ def _worker_entrypoint(init_timeout: timedelta,channels: _Channels,device_ids: list[int] | None = None,+ cpu_threads: int | None = 8,) -> None:"""Per-worker entry. @record turns a crash into a readable failure on the controller."""local_rank = int(os.environ["LOCAL_RANK"])+ # Torch's CPU pool defaults to one thread per core. A rank runs only small host tensor ops (conditioning+ # masks, tile plans), which a pool that wide stalls -- tens to hundreds of ms per op on a 176-CPU host --+ # while the GPU waits for the work they gate. None keeps torch's default.+ if cpu_threads is not None:+ torch.set_num_threads(cpu_threads)# All GPUs stay visible; we bind this rank to its physical GPU by index. (Setting# CUDA_VISIBLE_DEVICES here is unreliable -- the spawn bootstrap can touch CUDA, freezing# the device list, before a late env override would apply.)@@ -320,6 +341,7 @@ class _RunnersFleet:num_gpus: int,logs_specs: LogsSpecs,device_ids: list[int] | None = None,+ cpu_threads: int | None = 8,) -> _RunnersFleet:port = _find_free_port()envs = {@@ -334,7 +356,7 @@ class _RunnersFleet:}# device_ids maps rank -> physical GPU (None = identity); the worker binds to it. The runner# class ships by value so a __main__/test-module runner survives the spawn (see _RunnerShipper).- packed = (_RunnerShipper(runner_cls), setup_kwargs, init_timeout, channels, device_ids)+ packed = (_RunnerShipper(runner_cls), setup_kwargs, init_timeout, channels, device_ids, cpu_threads)args = dict.fromkeys(range(num_gpus), packed)logger.info("Spawning %d MGPU workers...", num_gpus)return cls(diff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/gemma_builders.py b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/gemma_builders.pyindex 7822dcb..d02152f 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/gemma_builders.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/gemma_builders.py@@ -15,6 +15,7 @@ dispatch hooks. The outer wrapper is not cached.from __future__ import annotationsimport copy+import itertoolsimport loggingfrom pathlib import Pathfrom typing import TYPE_CHECKING, Any@@ -24,8 +25,10 @@ import torch.distributed as distfrom accelerate import dispatch_modelfrom transformers import AutoModelForImageTextToText+from ltx_core.loader.helpers import assign_state_dictfrom ltx_core.loader.primitives import BuilderProtocol, StateDictfrom ltx_core.loader.registry import Registry, module_registry_key+from ltx_core.multigpu.broadcast import BroadcastCoordinatorfrom ltx_core.multigpu.gemma.accelerate_wrapper import AccelerateGemmaWrapperfrom ltx_core.multigpu.gemma.loader import load_gemma_with_device_mapfrom ltx_core.text_encoders.gemma import gemma_model_config@@ -36,6 +39,9 @@ if TYPE_CHECKING:logger = logging.getLogger(__name__)+# Numbers this process's builders in construction order, which every rank shares, to give each its own store key.+_builder_ids = itertools.count()+def _assign_non_persistent(model: torch.nn.Module, non_persistent_sd: dict[str, torch.Tensor]) -> None:for name, tensor in non_persistent_sd.items():@@ -69,6 +75,9 @@ class AccelerateGemmaBuilder(BuilderProtocol):Conforms to the builder interface expected by ``PromptEncoder``:``build(device, dtype) -> model``. Non-source ranks get a lightweight:class:`AccelerateGemmaWrapper` that receives embeddings via broadcast.+ Every rank must construct its builders in the same order (in ``setup``, before any job): each+ gets its own :class:`BroadcastCoordinator`, keyed by construction order, which it hands to every+ wrapper it builds."""def __init__(@@ -88,6 +97,10 @@ class AccelerateGemmaBuilder(BuilderProtocol):self._src_rank = src_rankself._is_src = dist.get_rank() == src_rankself._dtype = dtype+ store = dist.distributed_c10d.PrefixStore(+ f"ltx_gemma_broadcast/{next(_builder_ids)}/", dist.distributed_c10d._get_default_store()+ )+ self._coordinator = BroadcastCoordinator(store=store, is_driver=self._is_src)# Cached on the src rank after first build (non-tensor objects).self._config: object | None = Noneself._hf_device_map: dict[str, int | str] | None = None@@ -133,6 +146,7 @@ class AccelerateGemmaBuilder(BuilderProtocol):src_rank=dist.get_group_rank(self._broadcast_group, self._src_rank),dtype=dtype,device=device,+ coordinator=self._coordinator,)def _build_encoder(self, dtype: torch.dtype) -> LTXGemmaTextEncoder:@@ -147,7 +161,7 @@ class AccelerateGemmaBuilder(BuilderProtocol):"registry must outlive the builder.")encoder = cached_model # type: ignore[assignment]- encoder.model.load_state_dict(cached_sd.sd, strict=False, assign=True)+ assign_state_dict(encoder.model, cached_sd.sd)encoder.dtype = dtypereturn encoderdiff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/tdp_builder.py b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/tdp_builder.pyindex 71d586c..7792c1d 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/tdp_builder.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/tdp_builder.py@@ -50,6 +50,11 @@ class TiledDataParallelBuilder(DelegatingBuilder[InnerModelT], Generic[InnerModeself._tiling = tilingself._normalize_positions = normalize_positions+ @property+ def builds_per_shape(self) -> bool:+ # Each build wraps the model for one call's video tools, so it cannot serve another shape.+ return True+@propertydef keeps_gpu_resident_weights(self) -> bool:# Weight tracker keeps registry tensors GPU-resident and rebinds them across builds.diff --git a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/weight_tracker.py b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/weight_tracker.pyindex 7d5d314..d7db6f9 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/multigpu/weight_tracker.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/multigpu/weight_tracker.py@@ -4,11 +4,16 @@ Shared infrastructure used by both TDP and SP builders.from __future__ import annotations+import functools+from collections.abc import Iterable++import safetensorsimport torchimport torch.distributed as distfrom ltx_core.loader.fuse_loras import fuse_lora_weightsfrom ltx_core.loader.primitives import LoraPathStrengthAndSDOps, LoraStateDictWithStrength, StateDict+from ltx_core.loader.sd_ops import SDOpsfrom ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builderfrom ltx_core.model.model_protocol import ModelTypefrom ltx_core.multigpu.sharded_sd import ShardedSD@@ -20,8 +25,10 @@ def _apply_loras_inplace(loras: tuple[LoraPathStrengthAndSDOps, ...],builder: Builder, # type: ignore[type-arg]lora_keys: frozenset[str],-) -> None:- """Reset *target* to clean weights from *source*, then fuse all LoRAs in one pass."""+) -> list[str]:+ """Reset *target* to clean weights from *source*, then fuse all LoRAs in one pass.+ Returns the keys the fusion wrote.+ """for key, clean_weight in source.items():target[key].copy_(clean_weight)@@ -41,8 +48,32 @@ def _apply_loras_inplace(target_sd = StateDict(sd=target, device=next(iter(target.values())).device, size=0, dtype={next(iter(target.values())).dtype})+ written: list[str] = []for key, fused in fuse_lora_weights(target_sd, lora_sds, fuse_rule=builder.fuse_rule):target[key].copy_(fused)+ written.append(key)+ return written+++def _lora_module_prefixes(path: str, sd_ops: SDOps) -> frozenset[str]:+ """The model modules a LoRA file targets, from its header's ``lora_A`` keys mapped through ``sd_ops``."""+ suffix = ".lora_A.weight"+ with safetensors.safe_open(path, framework="pt") as f:+ names = [sd_ops.apply_to_key(key) for key in f.keys()] # noqa: SIM118 -- safe_open is not a mapping+ return frozenset(name[: -len(suffix)] for name in names if name is not None and name.endswith(suffix))+++def lora_footprint(keys: Iterable[str], loras: Iterable[LoraPathStrengthAndSDOps]) -> frozenset[str]:+ """The keys of ``keys`` a LoRA set can write: every tensor of a module one of its non-zero LoRAs targets.+ That covers the fused ``.weight`` and any companion a fuse rule rewrites with it (``.weight_scale``, ...).+ Read from the LoRA files' headers, so every rank derives the same set without a collective.+ """+ prefixes: set[str] = set()+ for lora in loras:+ if lora.strength != 0:+ prefixes |= _lora_module_prefixes(lora.path, lora.sd_ops)+ return frozenset(key for key in keys if key.rpartition(".")[0] in prefixes)class TransformerWeightTracker:@@ -57,6 +88,9 @@ class TransformerWeightTracker:After in-place LoRA fusion on the owning rank, this broadcasts thefused results to all other ranks so every rank sees the same weights.Both are created together and are always either both ``None`` or both set.+ A swap touches only the LoRA footprints (:func:`lora_footprint`): the keys the outgoing set+ wrote (``dirty``) are restored and the incoming set's are re-fused, and only their union is+ broadcast. Weights neither set targets stay as they are, clean.With ``no_lora_swap``, the LoRA set is assumed fixed (none, or one set):the backup clone is skipped and any swap or reset raises."""@@ -71,6 +105,7 @@ class TransformerWeightTracker:self.stored_sd: ShardedSD | None = Noneself.broadcast_sd: ShardedSD | None = Noneself.loras: list[tuple[str, float, str]] = []+ self.dirty: frozenset[str] = frozenset() # keys holding LoRA-fused values, the same on every rank@propertydef staging(self) -> torch.Tensor:@@ -86,7 +121,7 @@ class TransformerWeightTracker:return sorted(lora_list) == sorted(self.loras)def reset_loras(self, target_sd: dict[str, torch.Tensor]) -> None:- """Restore *target_sd* to original (pre-LoRA) weights.+ """Restore *target_sd* to original (pre-LoRA) weights: the keys the applied LoRAs wrote.No-op if no LoRAs are currently applied. This is a cooperativeoperation — all ranks must call it simultaneously."""@@ -96,8 +131,8 @@ class TransformerWeightTracker:raise RuntimeError("no_lora_swap tracker has no backup to reset from")if self.stored_sd is None:raise RuntimeError("stored_sd must be initialised before reset_loras")- self.loras = []- self.stored_sd.broadcast_shards_into(target_sd, self.staging)+ dirty, self.loras, self.dirty = self.dirty, [], frozenset()+ self.stored_sd.subset(dirty).broadcast_shards_into(target_sd, self.staging)def _local_lora_keys(self) -> frozenset[str]:"""Derive LoRA key names from the locally owned model keys."""@@ -119,9 +154,10 @@ class TransformerWeightTracker:builder: Builder, # type: ignore[type-arg]) -> None:"""Fuse *loras* into *target_sd* in-place (trailing ``_`` denotes in-place).- Skips work when the requested LoRAs already match. Restores stored- weights before applying new LoRAs. This is a cooperative operation —- all ranks must call it simultaneously.+ Skips work when the requested LoRAs already match. Restores the stored+ weights of the outgoing set's footprint, fuses the new set into its own,+ and broadcasts only the union. This is a cooperative operation — all+ ranks must call it simultaneously."""new_loras = [(lora.path, lora.strength, lora.sd_ops.name) for lora in loras]@@ -138,13 +174,19 @@ class TransformerWeightTracker:if self.stored_sd is None or self.broadcast_sd is None:raise RuntimeError("ShardedSDs must be initialised before apply_loras_ (call build first)")- source = self.stored_sd.local_shard- target = {k: v for k, v in target_sd.items() if k in source}- lora_keys = self._local_lora_keys()- _apply_loras_inplace(source, target, loras, builder, lora_keys)-- self.broadcast_sd.broadcast_shards_into(target_sd, self.staging)+ footprint = lora_footprint(self.stored_sd.keys, loras)+ touched = footprint | self.dirty+ source = {k: v for k, v in self.stored_sd.local_shard.items() if k in touched}+ target = {k: target_sd[k] for k in source}+ written = _apply_loras_inplace(source, target, loras, builder, self._local_lora_keys()) if target else []+ stray = [key for key in written if key not in footprint]+ if stray:+ # Other ranks would never receive these: refuse rather than leave the ranks' weights apart.+ raise RuntimeError(f"LoRA fusion wrote keys outside the LoRA set's footprint: {stray[:5]}")++ self.broadcast_sd.subset(touched).broadcast_shards_into(target_sd, self.staging)self.loras = new_loras+ self.dirty = footprintdef build(self,diff --git a/packages/ltx-pipelines/src/ltx_pipelines/retake.py b/packages/ltx-pipelines/src/ltx_pipelines/retake.pyindex cdf2ad4..6f9d3eb 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/retake.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/retake.py@@ -19,7 +19,10 @@ from ltx_core.types import (SpatioTemporalScaleFactors,VideoPixelShape,)-from ltx_pipelines.utils.args import video_editing_arg_parser+from ltx_pipelines.utils.args import (+ markers_from_args,+ video_editing_arg_parser,+)from ltx_pipelines.utils.blocks import (AudioConditioner,AudioDecoder,@@ -340,6 +343,8 @@ def main() -> None:parser = video_editing_arg_parser(distilled=True)parser.description = "Retake: regenerate a time region of a video with LTX-2."args = parser.parse_args()+ hdr = resolve_hdr_color_space(video_paths=[args.video_path], hdr=args.hdr)+ video_marker, audio_marker = markers_from_args(args, device=get_device(), color_space=hdr)if args.start_time >= args.end_time:raise ValueError("start_time must be less than end_time")@@ -367,7 +372,6 @@ def main() -> None:diffvae_optimization=args.diffvae_optimization,)params = detect_params(args.model_paths.transformer())- hdr = resolve_hdr_color_space(video_paths=[args.video_path], hdr=args.hdr)vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)result = pipeline(video_path=args.video_path,@@ -393,6 +397,8 @@ def main() -> None:output_path=args.output_path,video_chunks_number=get_video_chunks_number(result.num_frames, result.tiling_config),color_space=hdr,+ video_marker=video_marker,+ audio_marker=audio_marker,)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/t2a_one_stage.py b/packages/ltx-pipelines/src/ltx_pipelines/t2a_one_stage.pyindex 4c1848b..fafdb70 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/t2a_one_stage.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/t2a_one_stage.py@@ -19,6 +19,7 @@ from ltx_core.types import Audio, AudioLatentShapefrom ltx_pipelines.utils import get_devicefrom ltx_pipelines.utils.args import (default_1_stage_t2a_arg_parser,+ markers_from_args,resolve_cli_params,)from ltx_pipelines.utils.blocks import (@@ -179,6 +180,7 @@ def main() -> None:params = resolve_cli_params()parser = default_1_stage_t2a_arg_parser(params=params)args = parser.parse_args()+ _, audio_marker = markers_from_args(args, device=get_device())pipeline = T2AOneStagePipeline(model_paths=args.model_paths,loras=tuple(args.lora) if args.lora else (),@@ -209,7 +211,7 @@ def main() -> None:max_batch_size=args.max_batch_size,)- encode_audio(audio=audio, output_path=args.output_path)+ encode_audio(audio=audio, output_path=args.output_path, audio_marker=audio_marker)if __name__ == "__main__":diff --git a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_one_stage.py b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_one_stage.pyindex a9dd0cc..56cb271 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_one_stage.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_one_stage.py@@ -22,6 +22,7 @@ from ltx_pipelines.utils.args import (add_generated_keyframes_arg,add_keyframe_decode_arg,default_1_stage_arg_parser,+ markers_from_args,resolve_cli_params,)from ltx_pipelines.utils.blocks import (@@ -38,12 +39,12 @@ from ltx_pipelines.utils.denoisers import FactoryGuidedDenoiserfrom ltx_pipelines.utils.helpers import (assert_generated_keyframes_request,assert_resolution,- combined_image_conditionings,create_initial_av_latents,decode_keyframes_from_slots,ensure_tiling_config,generated_keyframe_conditionings,get_device,+ image_conditionings,resolve_generated_keyframes,tiling_scale_factors_for_vae,)@@ -164,7 +165,8 @@ class TI2VidOneStagePipeline:decode_with_keyframes: bool = False,) -> PipelineOutput:require_num_frames_source(num_frames, self.duration_predictor)- images = self.image_conditioner.resolve_crf(images)+ enhance_prompt_image = images[0].path if images else None+ images = self.image_conditioner.load_conditionings(images, color_space)assert_resolution(height=height, width=width, is_two_stage=False)assert_generated_keyframes_request(decode_with_keyframes, generated_keyframes, self.stage)@@ -178,7 +180,7 @@ class TI2VidOneStagePipeline:[prompt, negative_prompt],enhance_first_prompt=enhance_prompt,enhance_static_cache=enhance_static_cache,- enhance_prompt_image=images[0][0] if len(images) > 0 else None,+ enhance_prompt_image=enhance_prompt_image,)v_context_p, a_context_p = ctx_p.video_encoding, ctx_p.audio_encodingv_context_n, a_context_n = ctx_n.video_encoding, ctx_n.audio_encoding@@ -203,14 +205,13 @@ class TI2VidOneStagePipeline:)stage_1_conditionings = self.image_conditioner(- lambda enc: combined_image_conditionings(+ lambda enc: image_conditionings(images=images,height=height,width=width,video_encoder=enc,dtype=dtype,device=self.device,- color_space=color_space,))stage_1_conditionings.extend(generated_keyframe_conditionings(generated_keyframes, num_frames))@@ -292,6 +293,8 @@ def main() -> None:add_generated_keyframes_arg(default_1_stage_arg_parser(params=params, supports_auto_duration=True)))args = parser.parse_args()+ hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)+ video_marker, audio_marker = markers_from_args(args, device=get_device(), color_space=hdr)pipeline = TI2VidOneStagePipeline(model_paths=args.model_paths,loras=tuple(args.lora) if args.lora else (),@@ -301,7 +304,6 @@ def main() -> None:prompt_enhancer_gemma_root=args.prompt_enhancer_gemma_root,diffvae_optimization=args.diffvae_optimization,)- hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)result = pipeline(prompt=args.prompt,@@ -346,6 +348,8 @@ def main() -> None:output_path=args.output_path,video_chunks_number=get_video_chunks_number(result.num_frames, result.tiling_config),color_space=hdr,+ video_marker=video_marker,+ audio_marker=audio_marker,)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.pyindex 8238658..611e847 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages.py@@ -43,6 +43,7 @@ from ltx_pipelines.utils.args import (add_keyframe_decode_arg,chunk_config_from_args,default_2_stage_arg_parser,+ markers_from_args,resolve_cli_params,)from ltx_pipelines.utils.blocks import (@@ -74,6 +75,7 @@ from ltx_pipelines.utils.types import (DEFAULT_AUTO_DURATION,AutoDuration,ImageConditioningInput,+ LoadedImageConditioning,OffloadMode,PipelineOutput,VideoAudio,@@ -90,8 +92,8 @@ class _TI2VidRunContext:noiser: GaussianNoiserdtype: torch.dtypevae_dtype: torch.dtype- #: ``images`` with every unset CRF filled in, so callers do not re-resolve it.- images: list[ImageConditioningInput]+ #: ``images`` loaded once for the run (CRF resolved, stills decoded), so callers do not reload them.+ images: list[LoadedImageConditioning]v_context_p: torch.Tensora_context_p: torch.Tensorv_context_n: torch.Tensor@@ -211,6 +213,7 @@ class TI2VidTwoStagesPipeline:width: int,frame_rate: float,images: list[ImageConditioningInput],+ color_space: EXRColorSpace | None,num_frames: int | AutoDuration,vae_dtype: torch.dtype | None,tiling_config: TilingConfig | AutoTiling | None,@@ -222,7 +225,8 @@ class TI2VidTwoStagesPipeline:) -> _TI2VidRunContext:"""Validate inputs, encode prompts, and resolve frame count / tiling before chunk planning."""require_num_frames_source(num_frames, self.duration_predictor)- images = self.image_conditioner.resolve_crf(images)+ enhance_prompt_image = images[0].path if images else None+ images = self.image_conditioner.load_conditionings(images, color_space)assert_resolution(height=height, width=width, is_two_stage=True)assert_generated_keyframes_request(decode_with_keyframes, generated_keyframes, self.stage_1)@@ -235,7 +239,7 @@ class TI2VidTwoStagesPipeline:[prompt, negative_prompt],enhance_first_prompt=enhance_prompt,enhance_static_cache=enhance_static_cache,- enhance_prompt_image=images[0][0] if len(images) > 0 else None,+ enhance_prompt_image=enhance_prompt_image,)v_context_p, a_context_p = ctx_p.video_encoding, ctx_p.audio_encodingv_context_n, a_context_n = ctx_n.video_encoding, ctx_n.audio_encoding@@ -289,9 +293,8 @@ class TI2VidTwoStagesPipeline:def _video_conditionings(self,- images: list[ImageConditioningInput],+ images: list[LoadedImageConditioning],num_frames: int,- color_space: EXRColorSpace | None,) -> MakeVideoConditionings:assert_image_frames_in_clip(images, num_frames)@@ -303,7 +306,6 @@ class TI2VidTwoStagesPipeline:chunk,images=images,video_encoder=enc,- color_space=color_space,))@@ -333,9 +335,11 @@ class TI2VidTwoStagesPipeline:generated_keyframes: int | Sequence[int] = 0,decode_with_keyframes: bool = False,chunk_config: ChunkConfig | None = None,+ refine_audio: bool = False,) -> PipelineOutput:"""Generate a video.``chunk_config=None`` plans a single chunk covering the snapped clip.+ Set ``refine_audio=True`` to denoise audio in stage 2; by default stage-1 audio is preserved."""chunks, num_frames, tiling_config = self.stream_chunks(prompt=prompt,@@ -360,6 +364,7 @@ class TI2VidTwoStagesPipeline:generated_keyframes=generated_keyframes,decode_with_keyframes=decode_with_keyframes,chunk_config=chunk_config,+ refine_audio=refine_audio,)return pipeline_output_from_chunks(chunks, num_frames=num_frames, tiling_config=tiling_config)@@ -387,9 +392,11 @@ class TI2VidTwoStagesPipeline:generated_keyframes: int | Sequence[int] = 0,decode_with_keyframes: bool = False,chunk_config: ChunkConfig | None = None,+ refine_audio: bool = False,) -> tuple[Iterator[DecodedChunk], int, TilingConfig]:"""Generate a video as a stream of decoded chunks.``chunk_config=None`` plans a single chunk covering the snapped clip.+ Set ``refine_audio=True`` to denoise audio in stage 2; by default stage-1 audio is preserved."""ctx = self._prepare_run(prompt=prompt,@@ -399,6 +406,7 @@ class TI2VidTwoStagesPipeline:width=width,frame_rate=frame_rate,images=images,+ color_space=color_space,num_frames=num_frames,vae_dtype=vae_dtype,tiling_config=tiling_config,@@ -440,7 +448,7 @@ class TI2VidTwoStagesPipeline:dtype=ctx.dtype,config=ctx.chunk_config,video_scale_factors=self.stage_1.video_scale_factors,- make_video_conditionings=self._video_conditionings(ctx.images, ctx.num_frames, color_space),+ make_video_conditionings=self._video_conditionings(ctx.images, ctx.num_frames),generated_keyframes=generated_keyframes,)base = denoise_chunks(@@ -459,7 +467,7 @@ class TI2VidTwoStagesPipeline:sigmas=stage_2_sigmas,noiser=ctx.noiser,fps=frame_rate,- freeze_audio=True,+ freeze_audio=not refine_audio,)return (decode_chunks(@@ -486,7 +494,14 @@ def main() -> None:add_generated_keyframes_arg(default_2_stage_arg_parser(params=params, supports_auto_duration=True))))+ parser.add_argument(+ "--refine-audio",+ action="store_true",+ help="Denoise audio in stage 2 instead of preserving the stage-1 audio.",+ )args = parser.parse_args()+ hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)+ video_marker, audio_marker = markers_from_args(args, device=get_device(), color_space=hdr)pipeline = TI2VidTwoStagesPipeline(model_paths=args.model_paths,distilled_lora=args.distilled_lora,@@ -498,7 +513,6 @@ def main() -> None:prompt_enhancer_gemma_root=args.prompt_enhancer_gemma_root,diffvae_optimization=args.diffvae_optimization,)- hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)chunk_config = chunk_config_from_args(args)video_guider_params = MultiModalGuiderParams(@@ -538,6 +552,7 @@ def main() -> None:generated_keyframes=args.num_generated_keyframes,decode_with_keyframes=args.decode_with_keyframes,chunk_config=chunk_config,+ refine_audio=args.refine_audio,)if chunk_config is not None:video, audio, audio_sampling_rate = split_decoded_chunks(chunks)@@ -558,6 +573,8 @@ def main() -> None:video_chunks_number=video_chunks_number,color_space=hdr,audio_sampling_rate=audio_sampling_rate,+ video_marker=video_marker,+ audio_marker=audio_marker,)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq.py b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq.pyindex 854ea94..d0a890b 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq.py@@ -19,6 +19,7 @@ from ltx_pipelines.utils.args import (add_generated_keyframes_arg,add_keyframe_decode_arg,hq_2_stage_arg_parser,+ markers_from_args,)from ltx_pipelines.utils.blocks import (AudioDecoder,@@ -39,12 +40,12 @@ from ltx_pipelines.utils.denoisers import GuidedDenoiser, SimpleDenoiserfrom ltx_pipelines.utils.helpers import (assert_generated_keyframes_request,assert_resolution,- combined_image_conditionings,create_initial_av_latents,decode_keyframes_from_slots,ensure_tiling_config,generated_keyframe_conditionings,get_device,+ image_conditionings,resolve_generated_keyframes,tiling_scale_factors_for_vae,)@@ -207,7 +208,8 @@ class TI2VidTwoStagesHQPipeline:decode_with_keyframes: bool = False,) -> PipelineOutput:require_num_frames_source(num_frames, self.duration_predictor)- images = self.image_conditioner.resolve_crf(images)+ enhance_prompt_image = images[0].path if images else None+ images = self.image_conditioner.load_conditionings(images, color_space)assert_resolution(height=height, width=width, is_two_stage=True)assert_generated_keyframes_request(decode_with_keyframes, generated_keyframes, self.stage_1)@@ -221,7 +223,7 @@ class TI2VidTwoStagesHQPipeline:[prompt, negative_prompt],enhance_first_prompt=enhance_prompt,enhance_static_cache=enhance_static_cache,- enhance_prompt_image=images[0][0] if len(images) > 0 else None,+ enhance_prompt_image=enhance_prompt_image,)v_context_p, a_context_p = ctx_p.video_encoding, ctx_p.audio_encodingv_context_n, a_context_n = ctx_n.video_encoding, ctx_n.audio_encoding@@ -256,14 +258,13 @@ class TI2VidTwoStagesHQPipeline:fps=frame_rate,)stage_1_conditionings = self.image_conditioner(- lambda enc: combined_image_conditionings(+ lambda enc: image_conditionings(images=images,height=stage_1_output_shape.height,width=stage_1_output_shape.width,video_encoder=enc,dtype=dtype,device=self.device,- color_space=color_space,))stage_1_conditionings.extend(generated_keyframe_conditionings(generated_keyframes, num_frames))@@ -325,14 +326,13 @@ class TI2VidTwoStagesHQPipeline:stage_2_sigmas = stage_2_sigmas.to(dtype=torch.float32, device=self.device)stage_2_output_shape = VideoPixelShape(batch=1, frames=num_frames, width=width, height=height, fps=frame_rate)stage_2_conditionings = self.image_conditioner(- lambda enc: combined_image_conditionings(+ lambda enc: image_conditionings(images=images,height=stage_2_output_shape.height,width=stage_2_output_shape.width,video_encoder=enc,dtype=dtype,device=self.device,- color_space=color_space,))if decode_with_keyframes:@@ -397,6 +397,8 @@ def main() -> None:add_generated_keyframes_arg(hq_2_stage_arg_parser(params=LTX_2_3_HQ_PARAMS, supports_auto_duration=True)))args = parser.parse_args()+ hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)+ video_marker, audio_marker = markers_from_args(args, device=get_device(), color_space=hdr)pipeline = TI2VidTwoStagesHQPipeline(model_paths=args.model_paths,distilled_lora=args.distilled_lora,@@ -410,7 +412,6 @@ def main() -> None:prompt_enhancer_gemma_root=args.prompt_enhancer_gemma_root,diffvae_optimization=args.diffvae_optimization,)- hdr = resolve_hdr_color_space(images=args.images, hdr=args.hdr)vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)result = pipeline(prompt=args.prompt,@@ -455,6 +456,8 @@ def main() -> None:output_path=args.output_path,video_chunks_number=get_video_chunks_number(result.num_frames, result.tiling_config),color_space=hdr,+ video_marker=video_marker,+ audio_marker=audio_marker,)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq_mgpu.py b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq_mgpu.pyindex 273fb35..96e742d 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq_mgpu.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_hq_mgpu.py@@ -15,6 +15,7 @@ from __future__ import annotationsimport loggingfrom collections.abc import Iterator, Sequencefrom multiprocessing import SimpleQueue+from types import SimpleNamespaceimport torchimport torch.distributed as dist@@ -37,7 +38,9 @@ from ltx_pipelines.multigpu.tdp_builder import TiledDataParallelBuilderfrom ltx_pipelines.multigpu.vae_builders import DistributedDecoderBuilderfrom ltx_pipelines.multigpu.weight_tracker import TransformerWeightTrackerfrom ltx_pipelines.ti2vid_two_stages_hq import TI2VidTwoStagesHQPipeline+from ltx_pipelines.utils.args import markers_from_argsfrom ltx_pipelines.utils.constants import TDP_DISTILLED_SIGMAS+from ltx_pipelines.utils.helpers import get_devicefrom ltx_pipelines.utils.media_io import EXRColorSpace, encode_video, resolve_hdr_color_space, vae_dtype_for_hdrfrom ltx_pipelines.utils.model_paths import ModelPathsfrom ltx_pipelines.utils.types import DEFAULT_AUTO_DURATION, AutoDuration@@ -181,10 +184,16 @@ class TI2VidTwoStagesHQRunner(MGPURunner):hdr: EXRColorSpace | None = None,generated_keyframes: int | Sequence[int] = 0,decode_with_keyframes: bool = False,+ marker_root_path: str | None = None,) -> Iterator[str | None]:# The pipeline raises ValueError on invalid input (symmetric across ranks); the controller# catches that and turns it into a recoverable RunnerError. Anything else is fatal.hdr = resolve_hdr_color_space(images=images or [], hdr=hdr)+ video_marker, audio_marker = markers_from_args(+ SimpleNamespace(marker_root_path=marker_root_path),+ device=get_device(),+ color_space=hdr,+ )vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)result = self._pipeline(prompt=prompt,@@ -217,12 +226,18 @@ class TI2VidTwoStagesHQRunner(MGPURunner):output_path=output_path,video_chunks_number=get_video_chunks_number(result.num_frames, result.tiling_config),color_space=hdr,+ video_marker=video_marker,+ audio_marker=audio_marker,)yield output_pathif __name__ == "__main__":- from ltx_pipelines.utils.args import add_generated_keyframes_arg, add_keyframe_decode_arg, hq_2_stage_arg_parser+ from ltx_pipelines.utils.args import (+ add_generated_keyframes_arg,+ add_keyframe_decode_arg,+ hq_2_stage_arg_parser,+ )from ltx_pipelines.utils.constants import LTX_2_3_HQ_PARAMSlogging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")@@ -277,6 +292,7 @@ if __name__ == "__main__":hdr=args.hdr,generated_keyframes=args.num_generated_keyframes,decode_with_keyframes=args.decode_with_keyframes,+ marker_root_path=args.marker_root_path,):pass # drive the job to completion; the runner writes the file as a side effectfinally:diff --git a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_mgpu.py b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_mgpu.pyindex d1f8196..344b777 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_mgpu.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/ti2vid_two_stages_mgpu.py@@ -12,6 +12,7 @@ from __future__ import annotationsimport loggingfrom collections.abc import Callable, Iterator, Sequencefrom multiprocessing import SimpleQueue+from types import SimpleNamespaceimport torchimport torch.distributed as dist@@ -35,7 +36,9 @@ from ltx_pipelines.multigpu.tdp_builder import TiledDataParallelBuilderfrom ltx_pipelines.multigpu.vae_builders import DistributedDecoderBuilderfrom ltx_pipelines.multigpu.weight_tracker import TransformerWeightTrackerfrom ltx_pipelines.ti2vid_two_stages import TI2VidTwoStagesPipeline+from ltx_pipelines.utils.args import markers_from_argsfrom ltx_pipelines.utils.constants import TDP_DISTILLED_SIGMAS+from ltx_pipelines.utils.helpers import get_devicefrom ltx_pipelines.utils.media_io import EXRColorSpace, encode_video, resolve_hdr_color_space, vae_dtype_for_hdrfrom ltx_pipelines.utils.model_paths import ModelPathsfrom ltx_pipelines.utils.types import DEFAULT_AUTO_DURATION, AutoDuration@@ -181,10 +184,17 @@ class TI2VidTwoStagesRunner(MGPURunner):generated_keyframes: int | Sequence[int] = 0,decode_with_keyframes: bool = False,chunk_config: ChunkConfig | None = None,+ refine_audio: bool = False,+ marker_root_path: str | None = None,) -> Iterator[str | None]:# The pipeline raises ValueError on invalid input (symmetric across ranks); the controller# catches that and turns it into a recoverable RunnerError. Anything else is fatal.hdr = resolve_hdr_color_space(images=images or [], hdr=hdr)+ video_marker, audio_marker = markers_from_args(+ SimpleNamespace(marker_root_path=marker_root_path),+ device=get_device(),+ color_space=hdr,+ )vae_dtype = vae_dtype_for_hdr(hdr, torch.bfloat16)chunks, _, _ = self._pipeline.stream_chunks(prompt=prompt,@@ -207,6 +217,7 @@ class TI2VidTwoStagesRunner(MGPURunner):generated_keyframes=generated_keyframes,decode_with_keyframes=decode_with_keyframes,chunk_config=chunk_config,+ refine_audio=refine_audio,)if dist.get_rank() != _DRIVER_RANK:for _ in chunks:@@ -222,6 +233,8 @@ class TI2VidTwoStagesRunner(MGPURunner):video_chunks_number=1,color_space=hdr,audio_sampling_rate=audio_sampling_rate,+ video_marker=video_marker,+ audio_marker=audio_marker,)yield output_path@@ -243,6 +256,11 @@ if __name__ == "__main__":add_generated_keyframes_arg(default_2_stage_arg_parser(params=params, supports_auto_duration=True))))+ parser.add_argument(+ "--refine-audio",+ action="store_true",+ help="Denoise audio in stage 2 instead of preserving the stage-1 audio.",+ )args = parser.parse_args()vae_queue = torch.multiprocessing.get_context("spawn").SimpleQueue()@@ -290,6 +308,8 @@ if __name__ == "__main__":generated_keyframes=args.num_generated_keyframes,decode_with_keyframes=args.decode_with_keyframes,chunk_config=chunk_config_from_args(args),+ refine_audio=args.refine_audio,+ marker_root_path=args.marker_root_path,):pass # drive the job to completion; the runner writes the file as a side effectfinally:diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/__init__.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/__init__.pyindex 6d0cc5e..bead6ed 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/__init__.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/__init__.py@@ -13,12 +13,11 @@ from ltx_pipelines.utils.helpers import (assert_resolution,assert_stage_supports_generated_keyframes,cleanup_memory,- combined_image_conditionings,decode_keyframes_from_slots,evenly_spaced_keyframe_positions,generated_keyframe_conditionings,get_device,- image_conditionings_by_adding_guiding_latent,+ image_conditionings,resolve_generated_keyframes,)from ltx_pipelines.utils.samplers import (@@ -58,7 +57,6 @@ __all__ = ["assert_resolution","assert_stage_supports_generated_keyframes","cleanup_memory",- "combined_image_conditionings","decode_keyframes_from_slots","euler_ancestral_denoising_loop","euler_cfg_pp_denoising_loop",@@ -67,7 +65,7 @@ __all__ = ["generated_keyframe_conditionings","get_device","gradient_estimating_euler_denoising_loop",- "image_conditionings_by_adding_guiding_latent",+ "image_conditionings","res2s_audio_video_denoising_loop","resolve_generated_keyframes",]diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/args.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/args.pyindex e369035..3ca30bf 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/args.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/args.py@@ -7,17 +7,21 @@ from collections.abc import Sequencefrom pathlib import Pathfrom typing import Any+import torch+from ltx_core.loader import LTXV_LORA_COMFY_RENAMING_MAP, LoraPathStrengthAndSDOpsfrom ltx_core.model.transformer.compiling import CompilationConfigfrom ltx_core.model.video_vae.transformer import DiffVAEModefrom ltx_core.quantization import QuantizationPolicyfrom ltx_pipelines.chunks import ChunkConfig+from ltx_pipelines.marking import AudioMarker, VideoMarker, create_markersfrom ltx_pipelines.utils.constants import (DEFAULT_LORA_STRENGTH,DEFAULT_NEGATIVE_PROMPT,LTX_2_3_HQ_PARAMS,LTX_2_3_PARAMS,LTX_2_4_PARAMS,+ VIDEO_SCALE_FACTORS,PipelineParams,detect_params,)@@ -78,6 +82,44 @@ def add_hdr_args(parser: argparse.ArgumentParser) -> argparse.ArgumentParser:return parser+def add_marker_root_args(parser: argparse.ArgumentParser) -> argparse.ArgumentParser:+ """Add ``--marker-root-path``. Requires the ``videoseal`` and ``audioseal`` extras."""+ parser.add_argument(+ "--marker-root-path",+ dest="marker_root_path",+ type=resolve_existing_dir,+ default=None,+ help=(+ "Path to a local path/to/marking pack directory (VideoSeal checkpoint/card/"+ "attenuation and AudioSeal generator/detector). Stamps SDR pixels and stereo PCM. "+ "HDR encode skips video and still stamps audio. "+ "Requires the videoseal and audioseal extras. "+ "Default: no marking."+ ),+ )+ return parser+++def markers_from_args(+ args: object,+ device: torch.device,+ color_space: EXRColorSpace | None = None,+) -> tuple[VideoMarker | None, AudioMarker | None]:+ """Load resident markers for this CLI, or ``(None, None)`` when the flag is omitted.+ When ``--marker-root-path`` is set, both extras and pack files are checked+ immediately. HDR (``color_space`` set) drops the video marker, stamps audio,+ and logs that video marking is unavailable with HDR.+ """+ path = getattr(args, "marker_root_path", None)+ if path is None:+ return None, None+ video_marker, audio_marker = create_markers(path, device)+ if color_space is not None:+ logger.warning("video marking is unavailable with HDR")+ return None, audio_marker+ return video_marker, audio_marker++class VideoConditioningAction(argparse.Action):def __call__(self,@@ -135,10 +177,21 @@ class AutoDurationAction(argparse.Action):setattr(namespace, self.dest, AutoDuration(min_seconds=min_seconds, max_seconds=max_seconds))+def _append_image_conditioning(namespace: argparse.Namespace, dest: str, item: ImageConditioningInput) -> None:+ current = getattr(namespace, dest) or []+ current.append(item)+ setattr(namespace, dest, current)+++_IMAGE_BINDING_KEYWORDS = frozenset({"auto", "replace", "guide"})++class ImageAction(argparse.Action):- """Parse ``--image PATH FRAME_IDX STRENGTH [CRF]``.- An omitted CRF stays ``None``, which the pipeline resolves from the checkpoint it runs- (see ``ImageConditioner.resolve_crf``) -- the CLI does not need to know the value.+ """Parse ``--image PATH FRAME_IDX STRENGTH [BINDING] [CRF]``.+ ``BINDING`` is ``auto`` (default when omitted), ``replace``, or ``guide``. With ``auto``, frame 0+ replaces the latent and later frames guide as keyframes. When only one trailing token is given,+ a keyword selects ``BINDING`` and an integer selects ``CRF``. An omitted CRF stays ``None`` for+ the pipeline to resolve from its checkpoint."""def __call__(@@ -148,19 +201,43 @@ class ImageAction(argparse.Action):values: list[str],option_string: str | None = None,) -> None:- if len(values) not in (3, 4):- msg = f"{option_string} requires 3 or 4 arguments (PATH FRAME_IDX STRENGTH [CRF]), got {len(values)}"+ if len(values) not in (3, 4, 5):+ msg = (+ f"{option_string} requires 3 to 5 arguments "+ f"(PATH FRAME_IDX STRENGTH [BINDING] [CRF]), got {len(values)}"+ )raise argparse.ArgumentError(self, msg)- conditioning = ImageConditioningInput(- path=resolve_existing_path(values[0]),- frame_idx=int(values[1]),- strength=float(values[2]),- crf=int(values[3]) if len(values) > 3 else None,- )- current = getattr(namespace, self.dest) or []- current.append(conditioning)- setattr(namespace, self.dest, current)+ path = resolve_existing_path(values[0])+ frame_idx = int(values[1])+ strength = float(values[2])+ binding_mode = "auto"+ crf: int | None = None++ if len(values) == 4:+ if values[3] in _IMAGE_BINDING_KEYWORDS:+ binding_mode = values[3]+ else:+ crf = int(values[3])+ elif len(values) == 5:+ if values[3] not in _IMAGE_BINDING_KEYWORDS:+ msg = f"{option_string} BINDING must be auto, replace, or guide, got {values[3]!r}"+ raise argparse.ArgumentError(self, msg)+ binding_mode = values[3]+ crf = int(values[4])++ match binding_mode:+ case "auto":+ image = ImageConditioningInput.at(path, frame_idx, strength, crf)+ case "replace":+ image = ImageConditioningInput.replace(path, frame_idx, strength, crf)+ case "guide":+ image = ImageConditioningInput.guide(path, frame_idx, strength, crf)+ case _:+ msg = f"{option_string} BINDING must be auto, replace, or guide, got {binding_mode!r}"+ raise argparse.ArgumentError(self, msg)++ _append_image_conditioning(namespace, self.dest, image)class LoraAction(argparse.Action):@@ -319,6 +396,14 @@ def resolve_existing_path(path: str) -> str:return resolved+def resolve_existing_dir(path: str) -> str:+ """Resolve *path* and verify it is an existing directory."""+ resolved = resolve_existing_path(path)+ if not Path(resolved).is_dir():+ raise argparse.ArgumentError(None, f"Not a directory: {resolved}")+ return resolved++QUANTIZATION_POLICIES = tuple(k.value for k in QuantizationKind)@@ -922,17 +1007,21 @@ def new_video_gen_arg_parser(metavar="ARG",default=[],help=(- "Image conditioning input: PATH FRAME_IDX STRENGTH [CRF]. "+ "Image conditioning: PATH FRAME_IDX STRENGTH [BINDING] [CRF]. ""PATH is a still file — PNG/JPEG (SDR) or scene-linear .exr (HDR). "- "FRAME_IDX is the target frame index, STRENGTH is the conditioning strength "- "(all three required). "- "CRF is the optional H.264 compression quality for SDR stills (0=lossless); when "- "omitted, the value matching the checkpoint's model version is used. "- "Can be specified multiple times. Example: --image path/to/image1.jpg 0 0.8 "- "--image path/to/plate.exr 0 1.0 --hdr SRGB_LINEAR"+ "BINDING is auto (default when omitted), replace, or guide. With auto, FRAME_IDX 0 "+ "replaces the latent and later indices guide as keyframes. replace pins a still on a "+ f"causal latent start (0 or 1 + {VIDEO_SCALE_FACTORS.time} * k pixel frames). guide adds "+ "a keyframe at any pixel frame. When only one trailing token is given, a binding keyword "+ "selects BINDING and an integer selects CRF. STRENGTH is the conditioning strength. CRF "+ "optionally sets H.264 compression quality for SDR stills (0=lossless); when omitted, "+ "the checkpoint's default is used. Examples: --image path/to/start.jpg 0 1.0 or "+ "--image path/to/end.jpg 160 1.0 guide or --image path/to/middle.jpg 65 1.0 replace or "+ "--image path/to/start.jpg 0 1.0 7 or --image path/to/plate.exr 0 1.0 --hdr SRGB_LINEAR."),)add_hdr_args(parser)+ add_marker_root_args(parser)return parser@@ -1069,6 +1158,7 @@ def video_editing_arg_parser(parser.add_argument("--start-time", type=float, required=True, help="Start time of the region to regenerate (s).")parser.add_argument("--end-time", type=float, required=True, help="End time of the region to regenerate (s).")add_hdr_args(parser)+ add_marker_root_args(parser)return parser@@ -1183,6 +1273,7 @@ def hdr_ic_lora_arg_parser(params: PipelineParams = LTX_2_4_PARAMS) -> argparse."ignored with --no-keyframes). Default: 0.95."),)+ add_marker_root_args(parser)return parser@@ -1233,6 +1324,7 @@ def dubit_arg_parser(default=1.0,help="Strength for IC-LoRA video reference conditioning (default: 1.0).",)+ add_marker_root_args(parser)return parser@@ -1426,6 +1518,7 @@ def default_1_stage_t2a_arg_parser(params: PipelineParams = LTX_2_3_PARAMS) -> adefault=audio_guider.skip_step,help=f"Audio skip step (default: {audio_guider.skip_step}).",)+ add_marker_root_args(parser)return parserdiff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks.pydeleted file mode 100644index e279c41..0000000--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks.py+++ /dev/null@@ -1,1300 +0,0 @@-"""Pipeline blocks — each block owns its model lifecycle.-Blocks build a model on each ``__call__``, use it, then free GPU memory.-This eliminates manual ``del model; cleanup_memory()`` in pipelines: each-block is self-contained, so no central model-coordinator object is needed.-"""--from __future__ import annotations--import copy-import logging-from collections.abc import Iterator, Sequence-from contextlib import AbstractContextManager, contextmanager-from dataclasses import replace-from typing import Callable, TypeVar--import torch--from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy-from ltx_core.batch_split import BatchSplitAdapter-from ltx_core.block_streaming import DISK_CPU_SLOTS, StreamingModelBuilder-from ltx_core.components.noisers import Noiser-from ltx_core.components.patchifiers import AudioPatchifier, VideoLatentPatchifier-from ltx_core.conditioning import ConditioningItem, VideoGeneratedKeyframeSlots-from ltx_core.duration_head import (- DURATION_HEAD_KEY_OPS,- DurationHead,- DurationHeadConfigurator,-)-from ltx_core.loader import SDOps-from ltx_core.loader.attention_ops import set_attention_module_op-from ltx_core.loader.fuse_loras import bf16_fuse_rule-from ltx_core.loader.module_ops import ModuleOps-from ltx_core.loader.primitives import BuilderProtocol, LoraPathStrengthAndSDOps, ModelBuilderProtocol-from ltx_core.loader.registry import ModelRegistry, Registry-from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder-from ltx_core.model.audio_vae import (- AUDIO_VAE_DECODER_COMFY_KEYS_FILTER,- AUDIO_VAE_ENCODER_COMFY_KEYS_FILTER,- VOCODER_COMFY_KEYS_FILTER,- AudioDecoderConfigurator,- AudioEncoderConfigurator,- VocoderConfigurator,-)-from ltx_core.model.audio_vae import (- decode_audio as vae_decode_audio,-)-from ltx_core.model.model_protocol import LTXModelProtocol, ModelConfigurator-from ltx_core.model.transformer import (- LTXV_MODEL_COMFY_RENAMING_MAP,- LTXModelConfigurator,- X0Model,-)-from ltx_core.model.transformer.attention import (- AttentionCallable,- AttentionFunction,-)-from ltx_core.model.transformer.compiling import (- CompilationConfig,- build_compile_transformer_op,- modify_sd_ops_for_compilation,-)-from ltx_core.model.upsampler import LatentUpsamplerConfigurator, upsample_video-from ltx_core.model.video_vae import (- CHANNELS_LAST_3D_WEIGHTS,- MEMORY_EFFICIENT_DECODE,- VAE_DECODER_COMFY_KEYS_FILTER,- VAE_ENCODER_COMFY_KEYS_FILTER,- TilingConfig,- VideoDecoderConfigurator,- VideoEncoder,- VideoEncoderConfigurator,- is_diffusion_video_vae,- video_decoder_sd_ops_for_checkpoint,-)-from ltx_core.model.video_vae.keyframes import DecodeKeyframes-from ltx_core.model.video_vae.transformer import (- DiffVAEMode,- NAttentionKind,- build_diffvae_mode_op,- na_dsl_available,-)-from ltx_core.quantization import QuantizationPolicy, fp8_cast_fuse_rule-from ltx_core.text_encoders.gemma import (- EMBEDDINGS_PROCESSOR_KEY_OPS,- EmbeddingsProcessorConfigurator,- GemmaTextEncoderConfigurator,- gemma_model_type,- get_gemma_ops,- resolve_gemma_weight_paths,-)-from ltx_core.text_encoders.gemma.embeddings_processor import EmbeddingsProcessor, EmbeddingsProcessorOutput-from ltx_core.tools import AudioLatentTools, LatentTools, VideoLatentTools-from ltx_core.types import (- VIDEO_SCALE_FACTORS,- Audio,- AudioLatentShape,- LatentState,- SpatioTemporalScaleFactors,- VideoLatentShape,-)-from ltx_pipelines.utils.constants import detect_params-from ltx_pipelines.utils.gpu_model import gpu_model-from ltx_pipelines.utils.helpers import (- assert_stage_supports_generated_keyframes,- cleanup_memory,- create_noised_state,- generate_enhanced_prompt,- seconds_to_clamped_num_frames,-)-from ltx_pipelines.utils.model_paths import ModelPaths-from ltx_pipelines.utils.samplers import euler_denoising_loop-from ltx_pipelines.utils.types import (- AutoDuration,- Denoiser,- ImageConditioningInput,- ModalitySpec,- OffloadMode,- VideoAudio,-)--_ENCODE_MODEL_TYPES = frozenset({"gemma3", "gemma4", "gemma4_unified"})--logger = logging.getLogger(__name__)--ModelWrapper = Callable[[torch.nn.Module, "LatentTools | None"], torch.nn.Module]-"""Wraps a built transformer for one :meth:`DiffusionStage.__call__` (see-:meth:`DiffusionStage.with_model_wrapper`). Must keep the-``(video, audio, perturbations) -> (video, audio)`` forward signature."""--T = TypeVar("T")-_M = TypeVar("_M", bound=torch.nn.Module)--DenoisingLoop = Callable[..., VideoAudio[LatentState]]---# ----------------------------------------------------------------------------# Internal helpers-# ------------------------------------------------------------------------------def _chain_quantization(- sd_ops: SDOps,- module_ops: tuple[ModuleOps, ...],- quantization: QuantizationPolicy,-) -> tuple[SDOps, tuple[ModuleOps, ...]]:- chained_sd_ops = sd_ops- if quantization.sd_ops is not None:- chained_sd_ops = SDOps(- name=f"sd_ops_chain_{sd_ops.name}+{quantization.sd_ops.name}",- mapping=(*sd_ops.mapping, *quantization.sd_ops.mapping),- )- return chained_sd_ops, (*module_ops, *quantization.module_ops)---def _apply_compile_ops(- sd_ops: SDOps,- module_ops: tuple[ModuleOps, ...],- loras: tuple[LoraPathStrengthAndSDOps, ...],- number_of_layers: int,- compilation_config: CompilationConfig,-) -> tuple[SDOps, tuple[ModuleOps, ...], tuple[LoraPathStrengthAndSDOps, ...]]:- """Rewrite sd_ops/module_ops/LoRAs for compiled blocks (params land under ``_orig_mod``)."""- sd_ops = modify_sd_ops_for_compilation(sd_ops, number_of_layers)- compile_op = build_compile_transformer_op(compilation_config)- module_ops = (*module_ops, compile_op)- loras = tuple(- LoraPathStrengthAndSDOps(- lora.path,- lora.strength,- modify_sd_ops_for_compilation(lora.sd_ops, number_of_layers),- )- for lora in loras- )- return sd_ops, module_ops, loras---def _ensure_cudagraph_compatible_builder(- builder: ModelBuilderProtocol[LTXModelProtocol], compilation_config: CompilationConfig | None-) -> None:- """CUDA-graph compile paths need GPU-resident weights across dispose/rebuild.- Covers ``capture=True`` and inductor cudagraph modes (``reduce-overhead`` /- ``max-autotune``). Checked lazily at prepare/build time so MGPU runners can- wrap the stage's builder with SP/TDP (+ weight tracker) after construction.- """- if compilation_config is None:- return- cudagraph_modes = frozenset({"reduce-overhead", "max-autotune"})- needs_resident = compilation_config.capture or compilation_config.mode in cudagraph_modes- if not needs_resident or builder.keeps_gpu_resident_weights:- return- raise ValueError(- "CompilationConfig capture / mode=reduce-overhead|max-autotune requires a builder "- "with keeps_gpu_resident_weights=True (SequenceParallelBuilder / "- "TiledDataParallelBuilder with a weight tracker, or StreamingModelBuilder). "- "Plain SingleGPUModelBuilder reloads weights onto fresh GPU storages each stage, "- "which invalidates CUDA graphs (or repeatedly recaptures until Dynamo gives up)."- )---@contextmanager-def _streaming_model(- builder: StreamingModelBuilder,- target_device: torch.device,- dtype: torch.dtype,- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,-) -> Iterator:- """Build a streaming wrapper, yield it, then tear down and free memory.- The builder's own ``cpu_slots_count`` selects RAM vs disk streaming.- ``teardown()`` always runs -- it releases non-memory resources (forward- hooks, the disk I/O worker thread, open file handles) that GC would not- reclaim promptly. ``dispose`` always metas parameter storage; ``TRIM``- also runs ``cleanup_memory()``, while ``DEFER`` leaves the CUDA cache warm.- """- wrapped = builder.build(device=target_device, dtype=dtype)- try:- yield wrapped- finally:- wrapped.teardown()- wrapped.dispose()- if alloc_trim_strategy == AllocatorTrimStrategy.TRIM:- cleanup_memory()---def _build_state(- spec: ModalitySpec,- tools: LatentTools,- noiser: Noiser,- dtype: torch.dtype,- device: torch.device,-) -> LatentState:- """Create a noised latent state from a modality spec and tools."""- state = create_noised_state(- tools=tools,- conditionings=spec.conditionings,- noiser=noiser,- dtype=dtype,- device=device,- noise_scale=spec.noise_scale,- initial_latent=spec.latent,- )- if spec.frozen:- state = replace(- state,- denoise_mask=torch.zeros_like(state.denoise_mask),- frozen=True,- )- return state---def _cleanup_iter(- it: Iterator[torch.Tensor],- model: torch.nn.Module,- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,-) -> Iterator[torch.Tensor]:- """Wrap an iterator to release *model* memory (per ``alloc_trim_strategy``) once exhausted or abandoned."""- with gpu_model(model, alloc_trim_strategy=alloc_trim_strategy):- yield from it---# ----------------------------------------------------------------------------# DiffusionStage-# ------------------------------------------------------------------------------class DiffusionStage:- """Owns transformer lifecycle. Builds on each call, frees on exit.- Replaces the manual build-transformer / ``del transformer`` pattern that- every pipeline previously repeated.- """-- def __init__(- self,- transformer_builder: ModelBuilderProtocol[LTXModelProtocol],- dtype: torch.dtype,- device: torch.device,- *,- quantization: QuantizationPolicy | None = None,- compilation_config: CompilationConfig | None = None,- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,- scale_factors: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,- ) -> None:- """Construct a stage from a single pre-built transformer ``builder``.- Holds only that builder plus build-time configuration (dtype, device,- quantization, compilation). Turning a checkpoint path + LoRA set into a- builder -- and choosing a :class:`StreamingModelBuilder` when offloading --- lives in :meth:`from_checkpoint`, which is how pipelines normally create a- stage. A :class:`StreamingModelBuilder` selects the block-streaming build- path; any other builder uses the standard (all-on-GPU) path.- ``quantization`` and ``compilation_config`` are applied lazily by- :meth:`_prepared_builder` on both the standard and streaming paths.- ``scale_factors`` are the video VAE's spatiotemporal downscaling factors,- defaulting to the 32x32x8 layout.- """- self._transformer_builder = transformer_builder- self._dtype = dtype- self._device = device- self._quantization = quantization- self._compilation_config = compilation_config- self._alloc_trim_strategy = alloc_trim_strategy- self.video_scale_factors = scale_factors- self._model_wrapper: ModelWrapper | None = None-- @classmethod- def from_checkpoint( # noqa: PLR0913- cls,- checkpoint_path: str,- dtype: torch.dtype,- device: torch.device,- loras: tuple[LoraPathStrengthAndSDOps, ...] = (),- quantization: QuantizationPolicy | None = None,- registry: Registry | None = None,- compilation_config: CompilationConfig | None = None,- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,- offload_mode: OffloadMode = OffloadMode.NONE,- model_configurator: type[ModelConfigurator] = LTXModelConfigurator,- model_sd_ops: SDOps = LTXV_MODEL_COMFY_RENAMING_MAP,- scale_factors: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,- ) -> "DiffusionStage":- """Build a stage from a checkpoint path and LoRA set.- Constructs a single transformer builder from ``checkpoint_path`` +- ``loras`` and delegates to ``__init__``. When ``offload_mode != OffloadMode.NONE``- that builder is a :class:`StreamingModelBuilder` (fuse rule +- ``cpu_slots_count`` for the mode); otherwise it is the standard single-GPU- builder. Quantization and compilation stay on the stage and are applied- lazily at build time. This is the high-level entry point used by pipelines;- ``__init__`` itself takes an already-built builder.- ``model_configurator`` / ``model_sd_ops`` let callers (e.g. the audio-only- T2A pipeline) override the model class configurator and the state-dict key- mapping.- """- transformer_builder: ModelBuilderProtocol[LTXModelProtocol]- if offload_mode == OffloadMode.NONE:- transformer_builder = Builder(- model_path=checkpoint_path,- model_class_configurator=model_configurator,- model_sd_ops=model_sd_ops,- loras=tuple(loras),- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),- )- else:- transformer_builder = cls._build_streaming_builder(- checkpoint_path=checkpoint_path,- configurator=model_configurator,- model_sd_ops=model_sd_ops,- loras=tuple(loras),- quantization=quantization,- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),- offload_mode=offload_mode,- )-- return cls(- transformer_builder,- dtype,- device,- quantization=quantization,- compilation_config=compilation_config,- alloc_trim_strategy=alloc_trim_strategy,- scale_factors=scale_factors,- )-- @staticmethod- def _build_streaming_builder(- *,- checkpoint_path: str,- configurator: type[ModelConfigurator],- model_sd_ops: SDOps,- loras: tuple[LoraPathStrengthAndSDOps, ...],- quantization: QuantizationPolicy | None,- registry: Registry,- offload_mode: OffloadMode,- ) -> StreamingModelBuilder:- """Construct the streaming transformer builder for an offloading stage.- Returns a :class:`StreamingModelBuilder` with the checkpoint path, LoRAs,- fuse rule (bf16 or fp8_cast), and ``cpu_slots_count`` for ``offload_mode``.- Compilation / quantization sd_ops and module_ops are not applied here --- the stage applies them lazily via :meth:`_prepared_builder`.- """- # WeightsProvider currently only supports plain bf16 + fp8_cast LoRA fusion- # (no companion-key emission). Quantization policies that emit- # companion keys (e.g. ``.weight_scale``) cannot be streamed yet.- if quantization is not None and quantization.fuse_rule is not fp8_cast_fuse_rule:- raise ValueError(- "Block streaming is not supported with this quantization policy "- "(only bf16 and fp8_cast are currently supported)."- )- return StreamingModelBuilder(- model_class_configurator=configurator,- model_path=checkpoint_path,- model_sd_ops=model_sd_ops,- loras=loras,- registry=registry,- fuse_rule=quantization.fuse_rule if quantization is not None else bf16_fuse_rule,- blocks_attr="transformer_blocks",- blocks_prefix="transformer_blocks",- cpu_slots_count=DISK_CPU_SLOTS if offload_mode == OffloadMode.DISK else None,- )-- @property- def supports_generated_keyframes(self) -> bool:- """Whether this stage's checkpoint was trained with the keyframe absolute-position embedding.- Read from the checkpoint's transformer config, so it is answered before any weights are- built. Generated keyframe slots on a checkpoint without it would be denoised as unmarked- tokens while still costing a full latent frame of tokens each, so pipelines refuse rather- than silently degrade.- """- transformer_config = self._transformer_builder.model_config().get("transformer", {})- return bool(transformer_config.get("use_keyframes_abs_pos_embedding", False))-- def _assert_supports_conditionings(self, conditionings: Sequence[ConditioningItem] | None) -> None:- if conditionings is None or self.supports_generated_keyframes:- return- if any(isinstance(item, VideoGeneratedKeyframeSlots) for item in conditionings):- assert_stage_supports_generated_keyframes(self)-- @property- def dtype(self) -> torch.dtype:- """Dtype this stage builds its transformer and sizes its latents in."""- return self._dtype-- @property- def device(self) -> torch.device:- """Device this stage builds its transformer and sizes its latents on."""- return self._device-- def with_attention(self, attention: AttentionFunction | AttentionCallable | None) -> "DiffusionStage":- """Return a new ``DiffusionStage`` that pins the transformer build to ``attention``.- Functional: never mutates ``self``. The returned stage shares all other- configuration with the original; only the underlying builders' ``module_ops``- gain a ``set_attention_module_op(attention)`` entry so subsequent transformer- builds use that kernel. ``attention=None`` is a no-op (returns ``self``).- """- if attention is None:- return self- op = set_attention_module_op(attention)- new = copy.copy(self)- new._transformer_builder = self._transformer_builder.with_module_ops(- (*self._transformer_builder.module_ops, op),- )- return new-- def with_builder(self, builder: ModelBuilderProtocol[LTXModelProtocol]) -> "DiffusionStage":- """Return a new ``DiffusionStage`` that builds its transformer from ``builder``.- Functional: never mutates ``self``; shares all other configuration (dtype, device,- quantization, compilation). Affects the standard (non-offload) build path.- """- new = copy.copy(self)- new._transformer_builder = builder- return new-- def with_model_wrapper(self, wrapper: "ModelWrapper | None") -> "DiffusionStage":- """Return a new stage that passes each built transformer through ``wrapper``.- Functional: never mutates ``self``. The wrapper is applied inside the transformer- context -- after the model is built, before the denoising loop -- on both the standard- and block-streaming paths, and it receives the call's video tools. Wrapping *here*- rather than around the builder is what lets a wrapper compose with offloading: a- wrapping builder would hide a :class:`StreamingModelBuilder` from :attr:`_is_streaming`,- so the stage would take the plain ``gpu_model`` path and never tear the stream down.- ``wrapper=None`` clears any previously set wrapper.- """- new = copy.copy(self)- new._model_wrapper = wrapper- return new-- def with_loras(self, loras: tuple[LoraPathStrengthAndSDOps, ...]) -> "DiffusionStage":- """Return a new ``DiffusionStage`` built with exactly ``loras`` (replacing the current set)."""- return self.with_builder(self._transformer_builder.with_loras(loras))-- def _prepared_builder(self) -> ModelBuilderProtocol[LTXModelProtocol]:- """Return the configured builder with the stage's build-time ops applied.- Compilation and quantization live on the stage (not on the builder) and are- applied here, lazily, for both the standard and streaming paths. This keeps- the builder holding only raw sd_ops/module_ops/LoRAs, so ``with_loras`` /- ``with_builder`` swap them consistently regardless of the build path. The- returned copy preserves the builder's concrete type (e.g. a- ``StreamingModelBuilder`` stays one).- """- builder = self._transformer_builder- _ensure_cudagraph_compatible_builder(builder, self._compilation_config)- sd_ops = builder.model_sd_ops- module_ops = builder.module_ops- loras = builder.loras- if self._quantization is not None:- sd_ops, module_ops = _chain_quantization(sd_ops, module_ops, self._quantization)- builder = builder.with_fuse_rule(self._quantization.fuse_rule)- if self._compilation_config is not None:- number_of_layers = builder.model_config()["transformer"]["num_layers"]- sd_ops, module_ops, loras = _apply_compile_ops(- sd_ops, module_ops, loras, number_of_layers, self._compilation_config- )- return builder.with_module_ops(module_ops).with_sd_ops(sd_ops).with_loras(loras)-- def _build_transformer(self, *, device: torch.device | None = None, **kwargs: object) -> X0Model:- target = device or self._device- return X0Model(self._prepared_builder().build(device=target, **kwargs)).to(target).eval()-- @property- def _is_streaming(self) -> bool:- """Whether the configured builder uses the block-streaming build path."""- return isinstance(self._transformer_builder, StreamingModelBuilder)-- @contextmanager- def _streaming_transformer_ctx(self) -> Iterator[X0Model]:- builder = self._prepared_builder()- assert isinstance(builder, StreamingModelBuilder)- with _streaming_model(builder, self._device, self._dtype, self._alloc_trim_strategy) as streaming_wrapper:- yield X0Model(streaming_wrapper).eval()-- def _transformer_ctx(self, **kwargs: object) -> AbstractContextManager:- if self._is_streaming:- return self._streaming_transformer_ctx()- return gpu_model(self._build_transformer(**kwargs), alloc_trim_strategy=self._alloc_trim_strategy)-- def __call__(- self,- denoiser: Denoiser,- sigmas: torch.Tensor,- noiser: Noiser,- modalities: VideoAudio[ModalitySpec],- loop: DenoisingLoop | None = None,- max_batch_size: int = 1,- ) -> VideoAudio[LatentState]:- """Build transformer -> run denoising loop -> free transformer.- Returns a :class:`VideoAudio` of unpatchified :class:`LatentState` with cleared- conditionings for present modalities.- """- video, audio = modalities- if video is None and audio is None:- raise ValueError("At least one of `modalities.video` or `modalities.audio` must be provided")- if video is not None:- self._assert_supports_conditionings(video.conditionings)-- if loop is None:- loop = euler_denoising_loop-- video_latent = video.latent if video is not None else None- audio_latent = audio.latent if audio is not None else None- conditioning_fps = video.conditioning_fps if video is not None else None-- # Build video_tools up front so it can be forwarded to the transformer- # context (required by TiledDataParallelBuilder in multi-GPU mode).- video_tools = (- VideoLatentTools(- VideoLatentPatchifier(patch_size=1),- VideoLatentShape.from_torch_shape(video_latent.shape),- conditioning_fps,- scale_factors=self.video_scale_factors,- )- if video_latent is not None- else None- )-- mode = "streaming" if self._is_streaming else "standard"- logger.info("Building transformer (%s) from %s", mode, self._transformer_builder.checkpoint)- with self._transformer_ctx(video_tools=video_tools) as built:- transformer = built if self._model_wrapper is None else self._model_wrapper(built, video_tools)- logger.info(- "Running denoising loop (%d steps, video latent %s, audio latent %s @ %.1f fps)",- len(sigmas) - 1,- tuple(video_latent.shape) if video_latent is not None else None,- tuple(audio_latent.shape) if audio_latent is not None else None,- conditioning_fps,- )- video_state: LatentState | None = None- if video is not None and video_tools is not None:- video_state = _build_state(video, video_tools, noiser, self._dtype, self._device)-- audio_tools = (- AudioLatentTools(- AudioPatchifier(patch_size=1),- AudioLatentShape.from_torch_shape(audio_latent.shape),- )- if audio_latent is not None- else None- )- audio_state: LatentState | None = None- if audio is not None and audio_tools is not None:- audio_state = _build_state(audio, audio_tools, noiser, self._dtype, self._device)-- wrapped = BatchSplitAdapter(transformer, max_batch_size=max_batch_size) # type: ignore[arg-type]- video_state, audio_state = loop(- sigmas=sigmas,- video_state=video_state,- audio_state=audio_state,- transformer=wrapped,- denoiser=denoiser,- )-- if video_state is not None and video_tools is not None:- video_state = video_tools.clear_conditioning(video_state)- video_state = video_tools.unpatchify(video_state)- if audio_state is not None and audio_tools is not None:- audio_state = audio_tools.clear_conditioning(audio_state)- audio_state = audio_tools.unpatchify(audio_state)-- return VideoAudio(video_state, audio_state)---class RecordingDiffusionStage:- """Wraps a :class:`DiffusionStage` and records the states each call produced.- Diagnostics wrapper for a stage's intermediate ``LatentState``s. Use this to inspect- per-stage states (including half-res stage-1 slots) without a lifecycle callback. Pipelines- hold their stage in a plain attribute, so a caller substitutes this in::- pipeline.stage = RecordingDiffusionStage(pipeline.stage)- result = pipeline(...)- keyframes = pipeline.stage.generated_keyframes- Delegates every other attribute to the wrapped stage, so a substituted stage stays usable- wherever the original was. Recording holds references to the returned states, which keeps- their latents alive -- fine for diagnostics on one run, not something to leave enabled in a- long-lived serving process.- """-- def __init__(self, stage: DiffusionStage) -> None:- self._stage = stage- self.video_states: list[LatentState] = []- self.audio_states: list[LatentState | None] = []-- def __call__(self, *args: object, **kwargs: object) -> VideoAudio[LatentState]:- video_state, audio_state = self._stage(*args, **kwargs) # type: ignore[arg-type]- if video_state is not None:- self.video_states.append(video_state)- self.audio_states.append(audio_state)- return VideoAudio(video_state, audio_state)-- @property- def generated_keyframes(self) -> torch.Tensor | None:- """Generated keyframe latents from the first recorded call, if it produced any."""- for state in self.video_states:- if state.generated_keyframes is not None:- return state.generated_keyframes- return None-- def __getattr__(self, name: str) -> object:- return getattr(self._stage, name)---# ----------------------------------------------------------------------------# PromptEncoder-# ------------------------------------------------------------------------------class PromptEncoder:- """Owns text encoder + embeddings processor lifecycle.- Loads Gemma, optionally enhances then encodes prompts, frees Gemma, then loads the- embeddings processor. ``_enhancer_text_encoder_builder`` is always set: when enhance- and encode share a checkpoint it aliases ``_text_encoder_builder`` (same object /- residency); a distinct builder means sequential enhance-then-encode contexts.- """-- def __init__(- self,- model_paths: ModelPaths,- dtype: torch.dtype,- device: torch.device,- registry: Registry | None = None,- offload_mode: OffloadMode = OffloadMode.NONE,- text_encoder_builder: BuilderProtocol | None = None,- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,- prompt_enhancer_gemma_root: str | None = None,- enhancer_text_encoder_builder: BuilderProtocol | None = None,- ) -> None:- text_encoder_path = model_paths.text_encoder()- embeddings_paths = model_paths.embeddings_weight_paths- if not embeddings_paths:- raise ValueError(- "PromptEncoder requires ModelPaths.embeddings_weight_paths "- "(transformer checkpoint and, in split mode, the text-encoder file)."- )- self._text_encoder_path = text_encoder_path- self._embeddings_paths = embeddings_paths- self._dtype = dtype- self._device = device- self._offload_mode = offload_mode- self._alloc_trim_strategy = alloc_trim_strategy- self._prompt_enhancer_gemma_root = prompt_enhancer_gemma_root-- registry = registry or ModelRegistry(cache_models=True, cache_weights=False)- self._registry = registry-- if text_encoder_builder is not None:- encode_type = text_encoder_builder.model_config().get("model_type") or gemma_model_type(text_encoder_path)- else:- encode_type = gemma_model_type(text_encoder_path)- if encode_type not in _ENCODE_MODEL_TYPES:- raise ValueError(- f"Encode root model_type={encode_type!r} is not supported for encoding; "- f"expected one of {sorted(_ENCODE_MODEL_TYPES)}."- )- self._encode_model_type = encode_type-- if text_encoder_builder is not None:- if offload_mode != OffloadMode.NONE:- raise ValueError(- "text_encoder_builder cannot be used with offload_mode != OffloadMode.NONE "- "because no streaming text encoder builder is available."- )- self._text_encoder_builder = text_encoder_builder- self._streaming_text_encoder_builder = None- else:- gemma_sd_ops, gemma_module_ops = get_gemma_ops(text_encoder_path)- weight_paths = resolve_gemma_weight_paths(text_encoder_path)- self._text_encoder_builder = Builder(- model_path=weight_paths,- model_class_configurator=GemmaTextEncoderConfigurator.with_gemma_model_path(text_encoder_path),- model_sd_ops=gemma_sd_ops,- module_ops=gemma_module_ops,- registry=registry,- )- self._streaming_text_encoder_builder = StreamingModelBuilder(- model_path=weight_paths,- model_class_configurator=GemmaTextEncoderConfigurator.with_gemma_model_path(text_encoder_path),- model_sd_ops=gemma_sd_ops,- module_ops=gemma_module_ops,- registry=registry,- blocks_attr="model.model.language_model.layers",- blocks_prefix="model.model.language_model.layers",- cpu_slots_count=DISK_CPU_SLOTS if offload_mode == OffloadMode.DISK else None,- )-- # Always set: shared enhance aliases the encode builder (same object / residency).- if enhancer_text_encoder_builder is not None:- self._enhancer_text_encoder_builder = enhancer_text_encoder_builder- elif prompt_enhancer_gemma_root is not None and prompt_enhancer_gemma_root != text_encoder_path:- enhancer_sd_ops, enhancer_module_ops = get_gemma_ops(prompt_enhancer_gemma_root)- enhancer_paths = resolve_gemma_weight_paths(prompt_enhancer_gemma_root)- self._enhancer_text_encoder_builder = Builder(- model_path=enhancer_paths,- model_class_configurator=GemmaTextEncoderConfigurator.with_gemma_model_path(prompt_enhancer_gemma_root),- model_sd_ops=enhancer_sd_ops,- module_ops=enhancer_module_ops,- registry=registry,- )- else:- self._enhancer_text_encoder_builder = self._text_encoder_builder-- self._embeddings_processor_builder = Builder(- model_path=embeddings_paths,- model_class_configurator=EmbeddingsProcessorConfigurator.with_gemma_model_path(text_encoder_path),- model_sd_ops=EMBEDDINGS_PROCESSOR_KEY_OPS,- registry=registry,- )-- def _build_text_encoder(self) -> torch.nn.Module:- """Build the Gemma text encoder (non-streaming path)."""- return self._text_encoder_builder.build(device=self._device, dtype=self._dtype).eval()-- def _build_enhancer_text_encoder(self) -> torch.nn.Module:- return self._enhancer_text_encoder_builder.build(device=self._device, dtype=self._dtype).eval()-- def _build_embeddings_processor(self) -> EmbeddingsProcessor:- """Build the embeddings processor on the target device."""- return self._embeddings_processor_builder.build(device=self._device, dtype=self._dtype).eval()-- def _text_encoder_ctx(self) -> AbstractContextManager:- if self._offload_mode != OffloadMode.NONE:- return _streaming_model(- self._streaming_text_encoder_builder, self._device, self._dtype, self._alloc_trim_strategy- )- return gpu_model(self._build_text_encoder(), alloc_trim_strategy=self._alloc_trim_strategy)-- def __call__(- self,- prompts: list[str],- *,- enhance_first_prompt: bool = False,- enhance_prompt_image: str | None = None,- enhance_prompt_seed: int = 42,- enhance_static_cache: bool = False,- ) -> list[EmbeddingsProcessorOutput]:- """Encode *prompts* through Gemma -> embeddings processor, freeing each model after use."""- prompts = list(prompts)- enhance_kwargs = {- "image_path": enhance_prompt_image,- "seed": enhance_prompt_seed,- "static_cache": enhance_static_cache,- }- separate_enhancer = self._enhancer_text_encoder_builder is not self._text_encoder_builder-- if enhance_first_prompt and separate_enhancer:- logger.info(- "Enhancing with separate Gemma%s",- f" from {self._prompt_enhancer_gemma_root}" if self._prompt_enhancer_gemma_root else "",- )- with gpu_model(- self._build_enhancer_text_encoder(),- alloc_trim_strategy=self._alloc_trim_strategy,- ) as enhancer:- prompts[0] = generate_enhanced_prompt(enhancer, prompts[0], **enhance_kwargs)- elif enhance_first_prompt and self._encode_model_type != "gemma3":- raise ValueError(- f"Prompt enhancement with encode root model_type={self._encode_model_type!r} "- "requires --prompt-enhancer-gemma-root pointing at a generative instruct checkpoint "- "(e.g. gemma3 or gemma4 E2B-it)."- )-- logger.info("Building text encoder from %s", self._text_encoder_path)- with self._text_encoder_ctx() as text_encoder:- if enhance_first_prompt and not separate_enhancer:- prompts[0] = generate_enhanced_prompt(text_encoder, prompts[0], **enhance_kwargs)- raw_outputs = text_encoder.encode(prompts)- logger.info("Text encoder done, building embeddings processor from %s", self._embeddings_paths)-- with gpu_model(- self._build_embeddings_processor(), alloc_trim_strategy=self._alloc_trim_strategy- ) as embeddings_processor:- result = [embeddings_processor.process_hidden_states(hs, mask) for hs, mask in raw_outputs]- logger.info("Prompt encoding complete")- return result---# ----------------------------------------------------------------------------# DurationPredictor-# ------------------------------------------------------------------------------class DurationPredictor:- """Predicts shot duration (in frames) from ``PromptEncoder`` output.- Unlike most blocks, the model is held directly rather than rebuilt on every call:- DurationHead is a few MB, so there's no memory pressure motivating the- build-on-call / free-on-exit pattern used for the large transformer/VAE blocks.- Construct via :meth:`from_checkpoint`.- """-- def __init__(self, head: DurationHead) -> None:- """Construct from an already-built, already-loaded head."""- self._head = head-- @classmethod- def from_checkpoint(- cls,- checkpoint_path: str | None,- dtype: torch.dtype,- device: torch.device,- model_configurator: type[ModelConfigurator[DurationHead]] = DurationHeadConfigurator,- model_sd_ops: SDOps = DURATION_HEAD_KEY_OPS,- ) -> "DurationPredictor | None":- """Build a predictor from a checkpoint path, or ``None`` if unavailable.- Returns ``None`` when ``checkpoint_path`` is ``None`` (split pack without a- duration-head file), or when the file has no DurationHead weights (pre-2.5 /- gemma3 monoliths). The underlying loader uses ``strict=False``, so a missing- head does not raise -- it silently leaves parameters on the meta device;- checked here so callers get ``None`` instead of a predictor that would crash- later.- No registry: the head is a few MB, so there's no benefit to caching its weights or- shell across pipeline instances the way the large transformer/VAE builders do.- """- if checkpoint_path is None:- return None- builder = Builder(- model_path=checkpoint_path,- model_class_configurator=model_configurator,- model_sd_ops=model_sd_ops,- )- head = builder.build(device=device, dtype=dtype).eval()- if any(param.is_meta for param in head.parameters()):- logger.info(- "No DurationHead weights found in %s; auto-duration prediction unavailable.",- checkpoint_path,- )- return None- return cls(head)-- def __call__(- self,- video_encoding: torch.Tensor | None,- audio_encoding: torch.Tensor | None,- *,- frame_rate: float,- min_seconds: float = 1.0,- max_seconds: float = 20.0,- ) -> int:- """Predict a frame count from caption connector tokens, snapped to the VAE's grid.- ``min_seconds``/``max_seconds`` clamp the prediction so a misbehaving prediction can't- request a degenerate or OOM-sized generation; the defaults are 1s and 20s. The result is- a frame count snapped to the VAE's ``8k + 1`` causal temporal grid. Callers doing offline- analysis of raw predictions (rather than feeding a real generation) can pass a much wider- range to see the unclamped duration.- """- if video_encoding is None and audio_encoding is None:- raise ValueError("DurationPredictor requires at least one of video_encoding / audio_encoding")- seconds_pred = self._head(video_encoding, audio_encoding)- if seconds_pred.shape != (1,):- raise ValueError(- f"DurationPredictor only supports a single-item batch, got prediction shape {tuple(seconds_pred.shape)}"- )- seconds = seconds_pred.item()- min_frames = round(min_seconds * frame_rate)- max_frames = round(max_seconds * frame_rate)- num_frames = seconds_to_clamped_num_frames(- seconds, frame_rate=frame_rate, min_frames=min_frames, max_frames=max_frames- )- if seconds > max_seconds or seconds < min_seconds:- logger.warning(- "DurationHead prediction clamped: raw %.2fs outside [%.2fs, %.2fs], using %.2fs (%d frames) @ %.2f fps",- seconds,- min_seconds,- max_seconds,- num_frames / frame_rate,- num_frames,- frame_rate,- )- else:- logger.info("DurationHead predicted %.2fs (%d frames @ %.2f fps)", seconds, num_frames, frame_rate)- return num_frames---def require_num_frames_source(num_frames: int | AutoDuration, duration_predictor: DurationPredictor | None) -> None:- """Guard against an unsatisfiable auto-duration request.- Call at the very top of a pipeline's ``__call__`` -- before prompt encoding or any other- work -- so a checkpoint without DurationHead weights (anything predating 2.5) fails fast- with a clear message instead of after paying for work whose result would be discarded.- """- if isinstance(num_frames, AutoDuration) and duration_predictor is None:- raise ValueError(- "num_frames was AutoDuration but this checkpoint has no DurationHead weights to "- "auto-predict duration from (DurationHead ships from LTX 2.5 / gemma4 onward). "- "Pass num_frames explicitly."- )---def resolve_num_frames(- num_frames: int | AutoDuration,- duration_predictor: DurationPredictor | None,- *,- video_encoding: torch.Tensor | None,- audio_encoding: torch.Tensor | None,- frame_rate: float,-) -> int:- """Resolve ``num_frames`` to a concrete frame count, predicting it if ``AutoDuration``.- Call after prompt encoding (once ``video_encoding``/``audio_encoding`` exist) and after- ``require_num_frames_source`` has already validated a predictor is available when needed.- """- if not isinstance(num_frames, AutoDuration):- return num_frames- return duration_predictor(- video_encoding,- audio_encoding,- frame_rate=frame_rate,- min_seconds=num_frames.min_seconds,- max_seconds=num_frames.max_seconds,- )---# ----------------------------------------------------------------------------# ImageConditioner-# ------------------------------------------------------------------------------class ImageConditioner:- """Owns video encoder lifecycle, and the CRF image conditionings are re-compressed at.- Builds the encoder, passes it to the user-supplied callable, then frees it.- Also the single place that knows what H.264 CRF an image conditioning should be- re-compressed at: that value is a property of the model generation, and this block- already holds the checkpoint it belongs to. Pipelines call ``resolve_crf`` so callers- may omit the CRF and still condition the way the model was trained.- """-- def __init__(- self,- checkpoint_path: str,- dtype: torch.dtype,- device: torch.device,- registry: Registry | None = None,- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,- ) -> None:- self._checkpoint_path = checkpoint_path- self._default_image_crf: int | None = None- self._dtype = dtype- self._device = device- self._encoder_builder = Builder(- model_path=checkpoint_path,- model_class_configurator=VideoEncoderConfigurator,- model_sd_ops=VAE_ENCODER_COMFY_KEYS_FILTER,- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),- )- self._alloc_trim_strategy = alloc_trim_strategy-- @property- def default_image_crf(self) -> int:- """The H.264 CRF image conditionings should be re-compressed at for this checkpoint.- Read from the checkpoint's ``model_version`` metadata (see ``detect_params``) rather- than hardcoded, because the value the model was trained with changed between- generations. Resolved on first use and cached, so a run without image conditionings- never reads the header.- """- if self._default_image_crf is None:- self._default_image_crf = detect_params(self._checkpoint_path).default_image_crf- return self._default_image_crf-- def resolve_crf(self, images: Sequence[ImageConditioningInput]) -> list[ImageConditioningInput]:- """Fill in ``default_image_crf`` for every conditioning that left its CRF unset (``crf=None``).- Call near the top of a pipeline's ``__call__``: omitting the CRF then means "use what- matches this model", while an explicitly passed CRF (including ``0`` for no- re-compression) is always honoured.- """- return [image if image.crf is not None else image._replace(crf=self.default_image_crf) for image in images]-- def _build_encoder(self) -> VideoEncoder:- return self._encoder_builder.build(device=self._device, dtype=self._dtype).eval()-- def __call__(self, fn: Callable[[VideoEncoder], T]) -> T:- """Build video encoder → call *fn(encoder)* → free encoder."""- with gpu_model(self._build_encoder(), alloc_trim_strategy=self._alloc_trim_strategy) as encoder:- return fn(encoder)---# ----------------------------------------------------------------------------# VideoUpsampler-# ------------------------------------------------------------------------------class VideoUpsampler:- """Owns video encoder + spatial upsampler lifecycle."""-- def __init__(- self,- checkpoint_path: str,- upsampler_path: str,- dtype: torch.dtype,- device: torch.device,- registry: Registry | None = None,- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,- ) -> None:- self._upsampler_path = upsampler_path- self._dtype = dtype- self._device = device- self._encoder_builder = Builder(- model_path=checkpoint_path,- model_class_configurator=VideoEncoderConfigurator,- model_sd_ops=VAE_ENCODER_COMFY_KEYS_FILTER,- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),- )- self._upsampler_builder = Builder(- model_path=upsampler_path,- model_class_configurator=LatentUpsamplerConfigurator,- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),- )- self._alloc_trim_strategy = alloc_trim_strategy-- def __call__(self, latent: torch.Tensor) -> torch.Tensor:- """Upsample *latent* using video encoder + spatial upsampler, then free both."""- logger.info("Building video encoder + spatial upsampler from %s", self._upsampler_path)- with (- gpu_model(- self._encoder_builder.build(device=self._device, dtype=self._dtype).eval(),- alloc_trim_strategy=self._alloc_trim_strategy,- ) as encoder,- gpu_model(- self._upsampler_builder.build(device=self._device, dtype=self._dtype).eval(),- alloc_trim_strategy=self._alloc_trim_strategy,- ) as upsampler,- ):- return upsample_video(latent=latent, video_encoder=encoder, upsampler=upsampler)---# ----------------------------------------------------------------------------# VideoDecoder-# ------------------------------------------------------------------------------class VideoDecoder:- """Owns video decoder lifecycle.- Returns an iterator that cleans up the decoder after all chunks are consumed.- """-- def __init__(- self,- checkpoint_path: str,- dtype: torch.dtype,- device: torch.device,- registry: Registry | None = None,- memory_efficient: bool = True,- decoder_builder: BuilderProtocol | None = None,- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,- diffvae_optimization: DiffVAEMode = DiffVAEMode.CHUNKED_EAGER,- ) -> None:- self._checkpoint_path = checkpoint_path- self._dtype = dtype- self._device = device- self._diffvae_optimization = diffvae_optimization- diffusion_vae = is_diffusion_video_vae(self._checkpoint_path)- self._diffusion_vae = diffusion_vae- cfg = diffvae_optimization.resolve() if diffusion_vae else None- use_dsl = cfg is not None and cfg.attention is NAttentionKind.BLACKWELL_DSL- if decoder_builder is not None:- self._decoder_builder = decoder_builder- else:- if diffusion_vae and use_dsl and not na_dsl_available():- raise RuntimeError(- "diffvae_optimization=blackwell_dsl needs a datacenter Blackwell GPU and "- "nvidia-cutlass-dsl (uv sync --group kernels)"- )- sd_ops = (- video_decoder_sd_ops_for_checkpoint(- self._checkpoint_path,- diffusion_vae=True,- na_dsl_kernel=use_dsl,- )- if diffusion_vae- else VAE_DECODER_COMFY_KEYS_FILTER- )- module_ops: tuple[ModuleOps, ...] = ()- # channels-last + memory-efficient decode apply to ConvVideoDecoder only- if memory_efficient and not diffusion_vae:- sd_ops = SDOps(- name=f"sd_ops_chain_{sd_ops.name}+{CHANNELS_LAST_3D_WEIGHTS.name}",- mapping=(*sd_ops.mapping, *CHANNELS_LAST_3D_WEIGHTS.mapping),- )- module_ops = (MEMORY_EFFICIENT_DECODE,)- self._decoder_builder = Builder(- model_path=self._checkpoint_path,- model_class_configurator=VideoDecoderConfigurator,- model_sd_ops=sd_ops,- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),- module_ops=module_ops,- )- self._alloc_trim_strategy = alloc_trim_strategy-- @property- def checkpoint_path(self) -> str:- return self._checkpoint_path-- @property- def diffvae_optimization(self) -> DiffVAEMode:- return self._diffvae_optimization-- def with_builder(self, builder: BuilderProtocol) -> "VideoDecoder":- """Return a new ``VideoDecoder`` that builds from ``builder``.- Functional: never mutates ``self``; shares checkpoint, dtype, device, and- trim strategy. Mirrors :meth:`DiffusionStage.with_builder`.- """- new = copy.copy(self)- new._decoder_builder = builder- return new-- def _prepared_builder(self) -> BuilderProtocol:- """Return the builder with DiffVAE mode ops applied at build time.- DiffVAE ``module_ops`` live on the decoder (not on the caller-held builder),- matching :meth:`DiffusionStage._prepared_builder`. That keeps- ``with_builder`` / Dist ``with_tiling`` swaps on the raw builder, while- every build still installs the configured DiffVAE policy.- """- builder = self._decoder_builder- if not self._diffusion_vae:- return builder- op = build_diffvae_mode_op(- self._diffvae_optimization,- CompilationConfig() if self._diffvae_optimization.resolve().compile_blocks else None,- )- return builder.with_module_ops((*builder.module_ops, op))-- def __call__(- self,- latent: torch.Tensor,- tiling_config: TilingConfig | None = None,- generator: torch.Generator | None = None,- *,- dtype: torch.dtype | None = None,- keyframes: DecodeKeyframes | None = None,- device_fn: Callable[[int], str | torch.device] | None = None,- ) -> Iterator[torch.Tensor]:- """Decode *latent* to pixel-space video chunks. Decoder freed after exhaustion.- ``dtype`` overrides the constructor dtype for this call only (e.g. float32 for- HDR raw-in/raw-out). Omitted → same dtype as on construction (SDR default).- ``keyframes`` anchors the decode on already-encoded planes. Every decoder in this repo- takes it, so this never branches on which VAE was built: a diffusion VAE uses the planes,- a conv VAE logs that it is dropping them, and a distributed decoder splits them across- its ranks. The keyword is only passed when there is something to pass, so a decoder- written against the older signature still serves plain decodes.- """- build_dtype = self._dtype if dtype is None else dtype- latent = latent.to(dtype=build_dtype)- logger.info("Building video decoder from %s", self._checkpoint_path)- decoder = self._prepared_builder().build(device=self._device, dtype=build_dtype).eval()- extra = {} if keyframes is None else {"keyframes": keyframes}- if device_fn is not None:- extra["device_fn"] = device_fn- chunks = decoder.decode_video(latent, tiling_config, generator, **extra)- return _cleanup_iter(- chunks,- decoder,- alloc_trim_strategy=self._alloc_trim_strategy,- )-- def decode_single_frames(- self,- latents: Sequence[torch.Tensor],- generator: torch.Generator | Sequence[torch.Generator | None] | None = None,- *,- dtype: torch.dtype | None = None,- ) -> Iterator[torch.Tensor]:- """Decode each latent as its own one-frame clip on one VAE build, then free it.- A causal VAE cannot decode stacked independent planes without bleeding neighbours.- A Dist build splits the list across its ranks and gathers the pixels back, so every- rank returns the whole list; it does not go through the volume-splitting- ``decode_video`` path, whose workers yield nothing.- """- if not latents:- return iter(())- build_dtype = self._dtype if dtype is None else dtype- moved = [latent.to(dtype=build_dtype) for latent in latents]- logger.info("Building video decoder from %s", self._checkpoint_path)- decoder = self._prepared_builder().build(device=self._device, dtype=build_dtype).eval()- chunks = decoder.decode_single_frames(moved, generator)- return _cleanup_iter(- chunks,- decoder,- alloc_trim_strategy=self._alloc_trim_strategy,- )---# ----------------------------------------------------------------------------# AudioDecoder-# ------------------------------------------------------------------------------class AudioDecoder:- """Owns audio decoder + vocoder lifecycle."""-- def __init__(- self,- checkpoint_path: str,- dtype: torch.dtype,- device: torch.device,- registry: Registry | None = None,- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,- ) -> None:- self._checkpoint_path = checkpoint_path- self._dtype = dtype- self._device = device- self._decoder_builder = Builder(- model_path=checkpoint_path,- model_class_configurator=AudioDecoderConfigurator,- model_sd_ops=AUDIO_VAE_DECODER_COMFY_KEYS_FILTER,- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),- )- self._vocoder_builder = Builder(- model_path=checkpoint_path,- model_class_configurator=VocoderConfigurator,- model_sd_ops=VOCODER_COMFY_KEYS_FILTER,- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),- )- self._alloc_trim_strategy = alloc_trim_strategy-- def __call__(self, latent: torch.Tensor) -> Audio:- """Decode audio *latent* through VAE decoder + vocoder, then free both."""- logger.info("Building audio decoder + vocoder from %s", self._checkpoint_path)- # The vocoder always runs in fp32 (bf16 accumulation degrades spectral- # metrics). On CUDA/CPU it is stored in bf16 and autocast upcasts per-op to- # save memory; MPS has no fp32 autocast, so store it in fp32 directly and- # avoid the per-call cast. Negligible footprint for this small model.- vocoder_dtype = torch.float32 if self._device.type == "mps" else self._dtype- with (- gpu_model(- self._decoder_builder.build(device=self._device, dtype=self._dtype).eval(),- alloc_trim_strategy=self._alloc_trim_strategy,- ) as decoder,- gpu_model(- self._vocoder_builder.build(device=self._device, dtype=vocoder_dtype).eval(),- alloc_trim_strategy=self._alloc_trim_strategy,- ) as vocoder,- ):- return vae_decode_audio(latent, decoder, vocoder)---# ----------------------------------------------------------------------------# AudioEncoder-# ------------------------------------------------------------------------------class AudioConditioner:- """Owns audio encoder lifecycle.- Builds the encoder, passes it to the user-supplied callable, then frees it.- Mirrors :class:`ImageConditioner` for the audio modality.- """-- def __init__(- self,- checkpoint_path: str,- dtype: torch.dtype,- device: torch.device,- registry: Registry | None = None,- alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,- ) -> None:- self._dtype = dtype- self._device = device- self._alloc_trim_strategy = alloc_trim_strategy- self._encoder_builder = Builder(- model_path=checkpoint_path,- model_class_configurator=AudioEncoderConfigurator,- model_sd_ops=AUDIO_VAE_ENCODER_COMFY_KEYS_FILTER,- registry=registry or ModelRegistry(cache_models=True, cache_weights=False),- )-- def __call__(self, fn: Callable[[torch.nn.Module], T]) -> T:- """Build audio encoder → call *fn(encoder)* → free encoder."""- with gpu_model(- self._encoder_builder.build(device=self._device, dtype=self._dtype).eval(),- alloc_trim_strategy=self._alloc_trim_strategy,- ) as encoder:- return fn(encoder)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/__init__.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/__init__.pynew file mode 100644index 0000000..f79afe5--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/__init__.py@@ -0,0 +1,47 @@+"""Pipeline blocks — each block owns its model lifecycle.+Blocks build a model on each ``__call__``, use it, then free GPU memory.+This eliminates manual ``del model; cleanup_memory()`` in pipelines: each+block is self-contained, so no central model-coordinator object is needed.+"""++from ltx_core.components.noisers import Noiser+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder+from ltx_pipelines.utils.blocks.audio_conditioner import AudioConditioner+from ltx_pipelines.utils.blocks.audio_decoder import AudioDecoder+from ltx_pipelines.utils.blocks.diffusion_stage import (+ DenoisingLoop,+ DiffusionStage,+ ModelWrapper,+ RecordingDiffusionStage,+)+from ltx_pipelines.utils.blocks.duration_predictor import (+ DurationPredictor,+ require_num_frames_source,+ resolve_num_frames,+)+from ltx_pipelines.utils.blocks.image_conditioner import ImageConditioner+from ltx_pipelines.utils.blocks.prompt_encoder import PromptEncoder+from ltx_pipelines.utils.blocks.video_decoder import VideoDecoder+from ltx_pipelines.utils.blocks.video_upsampler import VideoUpsampler+from ltx_pipelines.utils.constants import detect_params+from ltx_pipelines.utils.helpers import cleanup_memory++__all__ = [+ "AudioConditioner",+ "AudioDecoder",+ "Builder",+ "DenoisingLoop",+ "DiffusionStage",+ "DurationPredictor",+ "ImageConditioner",+ "ModelWrapper",+ "Noiser",+ "PromptEncoder",+ "RecordingDiffusionStage",+ "VideoDecoder",+ "VideoUpsampler",+ "cleanup_memory",+ "detect_params",+ "require_num_frames_source",+ "resolve_num_frames",+]diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/audio_conditioner.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/audio_conditioner.pynew file mode 100644index 0000000..bd4775d--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/audio_conditioner.py@@ -0,0 +1,49 @@+from __future__ import annotations++from collections.abc import Callable+from typing import TypeVar++import torch++from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy+from ltx_core.loader.registry import ModelRegistry, Registry+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder+from ltx_core.model.audio_vae import AUDIO_VAE_ENCODER_COMFY_KEYS_FILTER, AudioEncoderConfigurator+from ltx_pipelines.utils.gpu_model import gpu_model++T = TypeVar("T")++__all__ = ["AudioConditioner"]+++class AudioConditioner:+ """Owns audio encoder lifecycle.+ Builds the encoder, passes it to the user-supplied callable, then frees it.+ Mirrors :class:`ImageConditioner` for the audio modality.+ """++ def __init__(+ self,+ checkpoint_path: str,+ dtype: torch.dtype,+ device: torch.device,+ registry: Registry | None = None,+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,+ ) -> None:+ self._dtype = dtype+ self._device = device+ self._alloc_trim_strategy = alloc_trim_strategy+ self._encoder_builder = Builder(+ model_path=checkpoint_path,+ model_class_configurator=AudioEncoderConfigurator,+ model_sd_ops=AUDIO_VAE_ENCODER_COMFY_KEYS_FILTER,+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),+ )++ def __call__(self, fn: Callable[[torch.nn.Module], T]) -> T:+ """Build audio encoder → call *fn(encoder)* → free encoder."""+ with gpu_model(+ self._encoder_builder.build(device=self._device, dtype=self._dtype).eval(),+ alloc_trim_strategy=self._alloc_trim_strategy,+ ) as encoder:+ return fn(encoder)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/audio_decoder.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/audio_decoder.pynew file mode 100644index 0000000..96a2624--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/audio_decoder.py@@ -0,0 +1,71 @@+from __future__ import annotations++import logging++import torch++from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy+from ltx_core.loader.registry import ModelRegistry, Registry+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder+from ltx_core.model.audio_vae import (+ AUDIO_VAE_DECODER_COMFY_KEYS_FILTER,+ VOCODER_COMFY_KEYS_FILTER,+ AudioDecoderConfigurator,+ VocoderConfigurator,+)+from ltx_core.model.audio_vae import decode_audio as vae_decode_audio+from ltx_core.types import Audio+from ltx_pipelines.utils.gpu_model import gpu_model++logger = logging.getLogger(__name__)++__all__ = ["AudioDecoder"]+++class AudioDecoder:+ """Owns audio decoder + vocoder lifecycle."""++ def __init__(+ self,+ checkpoint_path: str,+ dtype: torch.dtype,+ device: torch.device,+ registry: Registry | None = None,+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,+ ) -> None:+ self._checkpoint_path = checkpoint_path+ self._dtype = dtype+ self._device = device+ self._decoder_builder = Builder(+ model_path=checkpoint_path,+ model_class_configurator=AudioDecoderConfigurator,+ model_sd_ops=AUDIO_VAE_DECODER_COMFY_KEYS_FILTER,+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),+ )+ self._vocoder_builder = Builder(+ model_path=checkpoint_path,+ model_class_configurator=VocoderConfigurator,+ model_sd_ops=VOCODER_COMFY_KEYS_FILTER,+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),+ )+ self._alloc_trim_strategy = alloc_trim_strategy++ def __call__(self, latent: torch.Tensor) -> Audio:+ """Decode audio *latent* through VAE decoder + vocoder, then free both."""+ logger.info("Building audio decoder + vocoder from %s", self._checkpoint_path)+ # The vocoder always runs in fp32 (bf16 accumulation degrades spectral+ # metrics). On CUDA/CPU it is stored in bf16 and autocast upcasts per-op to+ # save memory; MPS has no fp32 autocast, so store it in fp32 directly and+ # avoid the per-call cast. Negligible footprint for this small model.+ vocoder_dtype = torch.float32 if self._device.type == "mps" else self._dtype+ with (+ gpu_model(+ self._decoder_builder.build(device=self._device, dtype=self._dtype).eval(),+ alloc_trim_strategy=self._alloc_trim_strategy,+ ) as decoder,+ gpu_model(+ self._vocoder_builder.build(device=self._device, dtype=vocoder_dtype).eval(),+ alloc_trim_strategy=self._alloc_trim_strategy,+ ) as vocoder,+ ):+ return vae_decode_audio(latent, decoder, vocoder)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/diffusion_stage.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/diffusion_stage.pynew file mode 100644index 0000000..1d51688--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/diffusion_stage.py@@ -0,0 +1,588 @@+from __future__ import annotations++import copy+import logging+from collections.abc import Callable, Iterator, Sequence+from contextlib import AbstractContextManager, ExitStack, contextmanager, nullcontext+from dataclasses import replace++import torch++from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy+from ltx_core.batch_split import BatchSplitAdapter+from ltx_core.block_streaming import DISK_CPU_SLOTS, StreamingModelBuilder+from ltx_core.components.noisers import Noiser+from ltx_core.components.patchifiers import AudioPatchifier, VideoLatentPatchifier+from ltx_core.conditioning import ConditioningItem, VideoGeneratedKeyframeSlots+from ltx_core.loader import SDOps+from ltx_core.loader.attention_ops import set_attention_module_op+from ltx_core.loader.fuse_loras import bf16_fuse_rule+from ltx_core.loader.helpers import to_device+from ltx_core.loader.module_ops import ModuleOps+from ltx_core.loader.primitives import LoraPathStrengthAndSDOps, ModelBuilderProtocol+from ltx_core.loader.registry import ModelRegistry, Registry+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder+from ltx_core.model.model_protocol import LTXModelProtocol, ModelConfigurator+from ltx_core.model.transformer import (+ LTXV_MODEL_COMFY_RENAMING_MAP,+ LTXModelConfigurator,+ X0Model,+)+from ltx_core.model.transformer.attention import AttentionCallable, AttentionFunction+from ltx_core.model.transformer.compiling import (+ CompilationConfig,+ build_compile_transformer_op,+ modify_sd_ops_for_compilation,+)+from ltx_core.quantization import QuantizationPolicy, fp8_cast_fuse_rule+from ltx_core.tools import AudioLatentTools, LatentTools, VideoLatentTools+from ltx_core.types import (+ VIDEO_SCALE_FACTORS,+ AudioLatentShape,+ DenoiseMask,+ LatentState,+ SpatioTemporalScaleFactors,+ VideoLatentShape,+)+from ltx_pipelines.utils.gpu_model import gpu_model+from ltx_pipelines.utils.helpers import assert_stage_supports_generated_keyframes, create_noised_state+from ltx_pipelines.utils.samplers import euler_denoising_loop+from ltx_pipelines.utils.streaming_model import streaming_model+from ltx_pipelines.utils.types import Denoiser, ModalitySpec, OffloadMode, VideoAudio++logger = logging.getLogger(__name__)++ModelWrapper = Callable[[torch.nn.Module, LatentTools | None], torch.nn.Module]+"""Wraps a built transformer for one :meth:`DiffusionStage.__call__` (see+:meth:`DiffusionStage.with_model_wrapper`). Must keep the+``(video, audio, perturbations) -> (video, audio)`` forward signature."""++DenoisingLoop = Callable[..., VideoAudio[LatentState]]++__all__ = ["DenoisingLoop", "DiffusionStage", "ModelWrapper", "RecordingDiffusionStage"]+++class _ResidentTransformer:+ """The transformer a :meth:`DiffusionStage.resident` block holds: built by its first call, released at its end."""++ def __init__(self) -> None:+ self.model: X0Model | None = None+ self.active = True+ self.lifetime = ExitStack()++ def release(self) -> None:+ self.active = False+ self.model = None+ self.lifetime.close()+++class DiffusionStage:+ """Owns transformer lifecycle. Builds on each call, frees on exit.+ Under ``AllocatorTrimStrategy.DEFER`` a builder whose weights the registry keeps GPU-resident+ (``keeps_gpu_resident_weights``) leaves its build assigned instead: every build reassigns those+ weights, so a dispose would release no memory.+ Replaces the manual build-transformer / ``del transformer`` pattern that+ every pipeline previously repeated.+ """++ def __init__(+ self,+ transformer_builder: ModelBuilderProtocol[LTXModelProtocol],+ dtype: torch.dtype,+ device: torch.device,+ *,+ quantization: QuantizationPolicy | None = None,+ compilation_config: CompilationConfig | None = None,+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,+ scale_factors: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,+ ) -> None:+ """Construct a stage from a single pre-built transformer ``builder``.+ Holds only that builder plus build-time configuration (dtype, device,+ quantization, compilation). Turning a checkpoint path + LoRA set into a+ builder -- and choosing a :class:`StreamingModelBuilder` when offloading --+ lives in :meth:`from_checkpoint`, which is how pipelines normally create a+ stage. A :class:`StreamingModelBuilder` selects the block-streaming build+ path; any other builder uses the standard (all-on-GPU) path.+ ``quantization`` and ``compilation_config`` are applied lazily by+ :meth:`_prepared_builder` on both the standard and streaming paths.+ ``scale_factors`` are the video VAE's spatiotemporal downscaling factors,+ defaulting to the 32x32x8 layout.+ """+ self._transformer_builder = transformer_builder+ self._dtype = dtype+ self._device = device+ self._quantization = quantization+ self._compilation_config = compilation_config+ self._alloc_trim_strategy = alloc_trim_strategy+ self.video_scale_factors = scale_factors+ self._model_wrapper: ModelWrapper | None = None+ self._resident: _ResidentTransformer | None = None++ @classmethod+ def from_checkpoint( # noqa: PLR0913+ cls,+ checkpoint_path: str,+ dtype: torch.dtype,+ device: torch.device,+ loras: tuple[LoraPathStrengthAndSDOps, ...] = (),+ quantization: QuantizationPolicy | None = None,+ registry: Registry | None = None,+ compilation_config: CompilationConfig | None = None,+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,+ offload_mode: OffloadMode = OffloadMode.NONE,+ model_configurator: type[ModelConfigurator] = LTXModelConfigurator,+ model_sd_ops: SDOps = LTXV_MODEL_COMFY_RENAMING_MAP,+ scale_factors: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,+ ) -> "DiffusionStage":+ """Build a stage from a checkpoint path and LoRA set.+ Constructs a single transformer builder from ``checkpoint_path`` ++ ``loras`` and delegates to ``__init__``. When ``offload_mode != OffloadMode.NONE``+ that builder is a :class:`StreamingModelBuilder` (fuse rule ++ ``cpu_slots_count`` for the mode); otherwise it is the standard single-GPU+ builder. Quantization and compilation stay on the stage and are applied+ lazily at build time. This is the high-level entry point used by pipelines;+ ``__init__`` itself takes an already-built builder.+ ``model_configurator`` / ``model_sd_ops`` let callers (e.g. the audio-only+ T2A pipeline) override the model class configurator and the state-dict key+ mapping.+ """+ transformer_builder: ModelBuilderProtocol[LTXModelProtocol]+ if offload_mode == OffloadMode.NONE:+ transformer_builder = Builder(+ model_path=checkpoint_path,+ model_class_configurator=model_configurator,+ model_sd_ops=model_sd_ops,+ loras=tuple(loras),+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),+ )+ else:+ transformer_builder = cls._build_streaming_builder(+ checkpoint_path=checkpoint_path,+ configurator=model_configurator,+ model_sd_ops=model_sd_ops,+ loras=tuple(loras),+ quantization=quantization,+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),+ offload_mode=offload_mode,+ )++ return cls(+ transformer_builder,+ dtype,+ device,+ quantization=quantization,+ compilation_config=compilation_config,+ alloc_trim_strategy=alloc_trim_strategy,+ scale_factors=scale_factors,+ )++ @staticmethod+ def _build_streaming_builder(+ *,+ checkpoint_path: str,+ configurator: type[ModelConfigurator],+ model_sd_ops: SDOps,+ loras: tuple[LoraPathStrengthAndSDOps, ...],+ quantization: QuantizationPolicy | None,+ registry: Registry,+ offload_mode: OffloadMode,+ ) -> StreamingModelBuilder:+ """Construct the streaming transformer builder for an offloading stage.+ Returns a :class:`StreamingModelBuilder` with the checkpoint path, LoRAs,+ fuse rule (bf16 or fp8_cast), and ``cpu_slots_count`` for ``offload_mode``.+ Compilation / quantization sd_ops and module_ops are not applied here --+ the stage applies them lazily via :meth:`_prepared_builder`.+ """+ # WeightsProvider currently only supports plain bf16 + fp8_cast LoRA fusion+ # (no companion-key emission). Quantization policies that emit+ # companion keys (e.g. ``.weight_scale``) cannot be streamed yet.+ if quantization is not None and quantization.fuse_rule is not fp8_cast_fuse_rule:+ raise ValueError(+ "Block streaming is not supported with this quantization policy "+ "(only bf16 and fp8_cast are currently supported)."+ )+ return StreamingModelBuilder(+ model_class_configurator=configurator,+ model_path=checkpoint_path,+ model_sd_ops=model_sd_ops,+ loras=loras,+ registry=registry,+ fuse_rule=quantization.fuse_rule if quantization is not None else bf16_fuse_rule,+ blocks_attr="transformer_blocks",+ blocks_prefix="transformer_blocks",+ cpu_slots_count=DISK_CPU_SLOTS if offload_mode == OffloadMode.DISK else None,+ )++ @property+ def supports_generated_keyframes(self) -> bool:+ """Whether this stage's checkpoint was trained with the keyframe absolute-position embedding.+ Read from the checkpoint's transformer config, so it is answered before any weights are+ built. Generated keyframe slots on a checkpoint without it would be denoised as unmarked+ tokens while still costing a full latent frame of tokens each, so pipelines refuse rather+ than silently degrade.+ """+ transformer_config = self._transformer_builder.model_config().get("transformer", {})+ return bool(transformer_config.get("use_keyframes_abs_pos_embedding", False))++ def _assert_supports_conditionings(self, conditionings: Sequence[ConditioningItem] | None) -> None:+ if conditionings is None or self.supports_generated_keyframes:+ return+ if any(isinstance(item, VideoGeneratedKeyframeSlots) for item in conditionings):+ assert_stage_supports_generated_keyframes(self)++ @property+ def dtype(self) -> torch.dtype:+ """Dtype this stage builds its transformer and sizes its latents in."""+ return self._dtype++ @property+ def device(self) -> torch.device:+ """Device this stage builds its transformer and sizes its latents on."""+ return self._device++ def with_attention(self, attention: AttentionFunction | AttentionCallable | None) -> "DiffusionStage":+ """Return a new ``DiffusionStage`` that pins the transformer build to ``attention``.+ Functional: never mutates ``self``. The returned stage shares all other+ configuration with the original; only the underlying builders' ``module_ops``+ gain a ``set_attention_module_op(attention)`` entry so subsequent transformer+ builds use that kernel. ``attention=None`` is a no-op (returns ``self``).+ """+ if attention is None:+ return self+ op = set_attention_module_op(attention)+ new = copy.copy(self)+ new._transformer_builder = self._transformer_builder.with_module_ops(+ (*self._transformer_builder.module_ops, op),+ )+ new._resident = None+ return new++ def with_builder(self, builder: ModelBuilderProtocol[LTXModelProtocol]) -> "DiffusionStage":+ """Return a new ``DiffusionStage`` that builds its transformer from ``builder``.+ Functional: never mutates ``self``; shares all other configuration (dtype, device,+ quantization, compilation). Affects the standard (non-offload) build path.+ """+ new = copy.copy(self)+ new._transformer_builder = builder+ new._resident = None+ return new++ def with_model_wrapper(self, wrapper: "ModelWrapper | None") -> "DiffusionStage":+ """Return a new stage that passes each built transformer through ``wrapper``.+ Functional: never mutates ``self``. The wrapper is applied inside the transformer+ context -- after the model is built, before the denoising loop -- on both the standard+ and block-streaming paths, and it receives the call's video tools. Wrapping *here*+ rather than around the builder is what lets a wrapper compose with offloading: a+ wrapping builder would hide a :class:`StreamingModelBuilder` from :attr:`_is_streaming`,+ so the stage would take the plain ``gpu_model`` path and never tear the stream down.+ ``wrapper=None`` clears any previously set wrapper.+ """+ new = copy.copy(self)+ new._model_wrapper = wrapper+ return new++ @property+ def quantization(self) -> QuantizationPolicy | None:+ """The quantization policy applied to every transformer build."""+ return self._quantization++ def with_quantization(self, quantization: QuantizationPolicy | None) -> "DiffusionStage":+ """Return a new ``DiffusionStage`` whose builds apply ``quantization`` instead.+ Functional: never mutates ``self``. A policy with the same ``sd_ops`` loads the same+ weights, so the two stages share the registry's tensors. Like a copy with another builder,+ the copy is outside any :meth:`resident` block of ``self``: its builds apply its own policy.+ """+ new = copy.copy(self)+ new._quantization = quantization+ new._resident = None+ return new++ def with_loras(self, loras: tuple[LoraPathStrengthAndSDOps, ...]) -> "DiffusionStage":+ """Return a new ``DiffusionStage`` built with exactly ``loras`` (replacing the current set)."""+ return self.with_builder(self._transformer_builder.with_loras(loras))++ def _prepared_builder(self) -> ModelBuilderProtocol[LTXModelProtocol]:+ """Return the configured builder with the stage's build-time ops applied.+ Compilation and quantization live on the stage (not on the builder) and are+ applied here, lazily, for both the standard and streaming paths. This keeps+ the builder holding only raw sd_ops/module_ops/LoRAs, so ``with_loras`` /+ ``with_builder`` swap them consistently regardless of the build path. The+ returned copy preserves the builder's concrete type (e.g. a+ ``StreamingModelBuilder`` stays one).+ """+ builder = self._transformer_builder+ _ensure_cudagraph_compatible_builder(builder, self._compilation_config)+ sd_ops = builder.model_sd_ops+ module_ops = builder.module_ops+ loras = builder.loras+ if self._quantization is not None:+ sd_ops, module_ops = _chain_quantization(sd_ops, module_ops, self._quantization)+ builder = builder.with_fuse_rule(self._quantization.fuse_rule)+ if self._compilation_config is not None:+ number_of_layers = builder.model_config()["transformer"]["num_layers"]+ sd_ops, module_ops, loras = _apply_compile_ops(+ sd_ops, module_ops, loras, number_of_layers, self._compilation_config+ )+ return builder.with_module_ops(module_ops).with_sd_ops(sd_ops).with_loras(loras)++ def _build_transformer(self, *, device: torch.device | None = None, **kwargs: object) -> X0Model:+ target = device or self._device+ return to_device(X0Model(self._prepared_builder().build(device=target, **kwargs)), target).eval()++ @property+ def _is_streaming(self) -> bool:+ """Whether the configured builder uses the block-streaming build path."""+ return isinstance(self._transformer_builder, StreamingModelBuilder)++ @contextmanager+ def _streaming_transformer_ctx(self) -> Iterator[X0Model]:+ builder = self._prepared_builder()+ assert isinstance(builder, StreamingModelBuilder)+ with streaming_model(builder, self._device, self._dtype, self._alloc_trim_strategy) as streaming_wrapper:+ yield X0Model(streaming_wrapper).eval()++ def _transformer_ctx(self, **kwargs: object) -> AbstractContextManager:+ if self._is_streaming:+ return self._streaming_transformer_ctx()+ if self._resident is None or not self._resident.active or self._transformer_builder.builds_per_shape:+ return self._transformer_lifetime(self._build_transformer(**kwargs))+ if self._resident.model is None:+ self._resident.model = self._resident.lifetime.enter_context(+ self._transformer_lifetime(self._build_transformer(**kwargs))+ )+ return nullcontext(self._resident.model)++ def _transformer_lifetime(self, transformer: X0Model) -> AbstractContextManager:+ if self._transformer_builder.keeps_gpu_resident_weights and (+ self._alloc_trim_strategy == AllocatorTrimStrategy.DEFER+ ):+ # The registry keeps these weights resident and every build reassigns them, so a+ # dispose would release no memory.+ return nullcontext(transformer)+ return gpu_model(transformer, alloc_trim_strategy=self._alloc_trim_strategy)++ @contextmanager+ def resident(self) -> Iterator[None]:+ """Hold one built transformer for every call on this stage inside the block.+ The first call builds; later calls -- on this stage or on the ``with_model_wrapper`` copies+ made of it inside the block, e.g. a tiling wrapper per window -- reuse that model, and the+ block's end releases it. A copy with another builder or policy (``with_builder``,+ ``with_loras``, ``with_attention``, ``with_quantization``) is outside the block and builds+ per call. Every call inside must want the stage's weights: the block does not track LoRA+ sets, so a call on a stage with other LoRAs must not happen inside it. A streaming stage, a+ builder that builds per latent shape (tiled data parallel), or a nested block just runs the+ body.+ """+ if self._is_streaming or self._resident is not None:+ yield+ return+ holder = _ResidentTransformer()+ self._resident = holder+ try:+ yield+ finally:+ self._resident = None+ holder.release()++ def __call__(+ self,+ denoiser: Denoiser,+ sigmas: torch.Tensor,+ noiser: Noiser,+ modalities: VideoAudio[ModalitySpec],+ loop: DenoisingLoop | None = None,+ max_batch_size: int = 1,+ ) -> VideoAudio[LatentState]:+ """Build transformer -> run denoising loop -> free transformer (see the class docstring for DEFER).+ Returns a :class:`VideoAudio` of unpatchified :class:`LatentState` with cleared+ conditionings for present modalities.+ """+ video, audio = modalities+ if video is None and audio is None:+ raise ValueError("At least one of `modalities.video` or `modalities.audio` must be provided")+ if video is not None:+ self._assert_supports_conditionings(video.conditionings)++ if loop is None:+ loop = euler_denoising_loop++ video_latent = video.latent if video is not None else None+ audio_latent = audio.latent if audio is not None else None+ conditioning_fps = video.conditioning_fps if video is not None else None++ # Build video_tools up front so it can be forwarded to the transformer+ # context (required by TiledDataParallelBuilder in multi-GPU mode).+ video_tools = (+ VideoLatentTools(+ VideoLatentPatchifier(patch_size=1),+ VideoLatentShape.from_torch_shape(video_latent.shape),+ conditioning_fps,+ scale_factors=self.video_scale_factors,+ )+ if video_latent is not None+ else None+ )++ mode = "streaming" if self._is_streaming else "standard"+ logger.info("Building transformer (%s) from %s", mode, self._transformer_builder.checkpoint)+ with self._transformer_ctx(video_tools=video_tools) as built:+ transformer = built if self._model_wrapper is None else self._model_wrapper(built, video_tools)+ logger.info(+ "Running denoising loop (%d steps, video latent %s, audio latent %s @ %.1f fps)",+ len(sigmas) - 1,+ tuple(video_latent.shape) if video_latent is not None else None,+ tuple(audio_latent.shape) if audio_latent is not None else None,+ conditioning_fps,+ )+ video_state: LatentState | None = None+ if video is not None and video_tools is not None:+ video_state = _build_state(video, video_tools, noiser, self._dtype, self._device)++ audio_tools = (+ AudioLatentTools(+ AudioPatchifier(patch_size=1),+ AudioLatentShape.from_torch_shape(audio_latent.shape),+ )+ if audio_latent is not None+ else None+ )+ audio_state: LatentState | None = None+ if audio is not None and audio_tools is not None:+ audio_state = _build_state(audio, audio_tools, noiser, self._dtype, self._device)++ wrapped = BatchSplitAdapter(transformer, max_batch_size=max_batch_size) # type: ignore[arg-type]+ video_state, audio_state = loop(+ sigmas=sigmas,+ video_state=video_state,+ audio_state=audio_state,+ transformer=wrapped,+ denoiser=denoiser,+ )++ if video_state is not None and video_tools is not None:+ video_state = video_tools.clear_conditioning(video_state)+ video_state = video_tools.unpatchify(video_state)+ if audio_state is not None and audio_tools is not None:+ audio_state = audio_tools.clear_conditioning(audio_state)+ audio_state = audio_tools.unpatchify(audio_state)++ return VideoAudio(video_state, audio_state)+++class RecordingDiffusionStage:+ """Wraps a :class:`DiffusionStage` and records the states each call produced.+ Diagnostics wrapper for a stage's intermediate ``LatentState``s. Use this to inspect+ per-stage states (including half-res stage-1 slots) without a lifecycle callback. Pipelines+ hold their stage in a plain attribute, so a caller substitutes this in::+ pipeline.stage = RecordingDiffusionStage(pipeline.stage)+ result = pipeline(...)+ keyframes = pipeline.stage.generated_keyframes+ Delegates every other attribute to the wrapped stage, so a substituted stage stays usable+ wherever the original was. Recording holds references to the returned states, which keeps+ their latents alive -- fine for diagnostics on one run, not something to leave enabled in a+ long-lived serving process.+ """++ def __init__(self, stage: DiffusionStage) -> None:+ self._stage = stage+ self.video_states: list[LatentState] = []+ self.audio_states: list[LatentState | None] = []++ def __call__(self, *args: object, **kwargs: object) -> VideoAudio[LatentState]:+ video_state, audio_state = self._stage(*args, **kwargs) # type: ignore[arg-type]+ if video_state is not None:+ self.video_states.append(video_state)+ self.audio_states.append(audio_state)+ return VideoAudio(video_state, audio_state)++ @property+ def generated_keyframes(self) -> torch.Tensor | None:+ """Generated keyframe latents from the first recorded call, if it produced any."""+ for state in self.video_states:+ if state.generated_keyframes is not None:+ return state.generated_keyframes+ return None++ def __getattr__(self, name: str) -> object:+ return getattr(self._stage, name)+++def _chain_quantization(+ sd_ops: SDOps,+ module_ops: tuple[ModuleOps, ...],+ quantization: QuantizationPolicy,+) -> tuple[SDOps, tuple[ModuleOps, ...]]:+ chained_sd_ops = sd_ops+ if quantization.sd_ops is not None:+ chained_sd_ops = SDOps(+ name=f"sd_ops_chain_{sd_ops.name}+{quantization.sd_ops.name}",+ mapping=(*sd_ops.mapping, *quantization.sd_ops.mapping),+ )+ return chained_sd_ops, (*module_ops, *quantization.module_ops)+++def _apply_compile_ops(+ sd_ops: SDOps,+ module_ops: tuple[ModuleOps, ...],+ loras: tuple[LoraPathStrengthAndSDOps, ...],+ number_of_layers: int,+ compilation_config: CompilationConfig,+) -> tuple[SDOps, tuple[ModuleOps, ...], tuple[LoraPathStrengthAndSDOps, ...]]:+ """Rewrite sd_ops/module_ops/LoRAs for compiled blocks (params land under ``_orig_mod``)."""+ sd_ops = modify_sd_ops_for_compilation(sd_ops, number_of_layers)+ compile_op = build_compile_transformer_op(compilation_config)+ module_ops = (*module_ops, compile_op)+ loras = tuple(+ LoraPathStrengthAndSDOps(+ lora.path,+ lora.strength,+ modify_sd_ops_for_compilation(lora.sd_ops, number_of_layers),+ )+ for lora in loras+ )+ return sd_ops, module_ops, loras+++def _ensure_cudagraph_compatible_builder(+ builder: ModelBuilderProtocol[LTXModelProtocol], compilation_config: CompilationConfig | None+) -> None:+ """CUDA-graph compile paths need GPU-resident weights across dispose/rebuild."""+ if compilation_config is None:+ return+ cudagraph_modes = frozenset({"reduce-overhead", "max-autotune"})+ needs_resident = compilation_config.capture or compilation_config.mode in cudagraph_modes+ if not needs_resident or builder.keeps_gpu_resident_weights:+ return+ raise ValueError(+ "CompilationConfig capture / mode=reduce-overhead|max-autotune requires a builder "+ "with keeps_gpu_resident_weights=True (SequenceParallelBuilder / "+ "TiledDataParallelBuilder with a weight tracker, or StreamingModelBuilder). "+ "Plain SingleGPUModelBuilder reloads weights onto fresh GPU storages each stage, "+ "which invalidates CUDA graphs (or repeatedly recaptures until Dynamo gives up)."+ )+++def _build_state(+ spec: ModalitySpec,+ tools: LatentTools,+ noiser: Noiser,+ dtype: torch.dtype,+ device: torch.device,+) -> LatentState:+ """Create a noised latent state from a modality spec and tools."""+ state = create_noised_state(+ tools=tools,+ conditionings=spec.conditionings,+ noiser=noiser,+ dtype=dtype,+ device=device,+ noise_scale=spec.noise_scale,+ initial_latent=spec.latent,+ )+ if spec.frozen:+ state = replace(+ state,+ denoise_mask=DenoiseMask(torch.zeros(state.denoise_mask.tensor.shape)).compact(device),+ frozen=True,+ )+ return statediff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/duration_predictor.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/duration_predictor.pynew file mode 100644index 0000000..75c75c1--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/duration_predictor.py@@ -0,0 +1,148 @@+from __future__ import annotations++import logging++import torch++from ltx_core.duration_head import (+ DURATION_HEAD_KEY_OPS,+ DurationHead,+ DurationHeadConfigurator,+)+from ltx_core.loader import SDOps+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder+from ltx_core.model.model_protocol import ModelConfigurator+from ltx_pipelines.utils.helpers import seconds_to_clamped_num_frames+from ltx_pipelines.utils.types import AutoDuration++logger = logging.getLogger(__name__)++__all__ = ["DurationPredictor", "require_num_frames_source", "resolve_num_frames"]+++class DurationPredictor:+ """Predicts shot duration (in frames) from ``PromptEncoder`` output.+ Unlike most blocks, the model is held directly rather than rebuilt on every call:+ DurationHead is a few MB, so there's no memory pressure motivating the+ build-on-call / free-on-exit pattern used for the large transformer/VAE blocks.+ Construct via :meth:`from_checkpoint`.+ """++ def __init__(self, head: DurationHead) -> None:+ """Construct from an already-built, already-loaded head."""+ self._head = head++ @classmethod+ def from_checkpoint(+ cls,+ checkpoint_path: str | None,+ dtype: torch.dtype,+ device: torch.device,+ model_configurator: type[ModelConfigurator[DurationHead]] = DurationHeadConfigurator,+ model_sd_ops: SDOps = DURATION_HEAD_KEY_OPS,+ ) -> "DurationPredictor | None":+ """Build a predictor from a checkpoint path, or ``None`` if unavailable.+ Returns ``None`` when ``checkpoint_path`` is ``None`` (split pack without a+ duration-head file), or when the file has no DurationHead weights (pre-2.5 /+ gemma3 monoliths). The underlying loader uses ``strict=False``, so a missing+ head does not raise -- it silently leaves parameters on the meta device;+ checked here so callers get ``None`` instead of a predictor that would crash+ later.+ No registry: the head is a few MB, so there's no benefit to caching its weights or+ shell across pipeline instances the way the large transformer/VAE builders do.+ """+ if checkpoint_path is None:+ return None+ builder = Builder(+ model_path=checkpoint_path,+ model_class_configurator=model_configurator,+ model_sd_ops=model_sd_ops,+ )+ head = builder.build(device=device, dtype=dtype).eval()+ if any(param.is_meta for param in head.parameters()):+ logger.info(+ "No DurationHead weights found in %s; auto-duration prediction unavailable.",+ checkpoint_path,+ )+ return None+ return cls(head)++ def __call__(+ self,+ video_encoding: torch.Tensor | None,+ audio_encoding: torch.Tensor | None,+ *,+ frame_rate: float,+ min_seconds: float = 1.0,+ max_seconds: float = 20.0,+ ) -> int:+ """Predict a frame count from caption connector tokens, snapped to the VAE's grid.+ ``min_seconds``/``max_seconds`` clamp the prediction so a misbehaving prediction can't+ request a degenerate or OOM-sized generation; the defaults are 1s and 20s. The result is+ a frame count snapped to the VAE's ``8k + 1`` causal temporal grid. Callers doing offline+ analysis of raw predictions (rather than feeding a real generation) can pass a much wider+ range to see the unclamped duration.+ """+ if video_encoding is None and audio_encoding is None:+ raise ValueError("DurationPredictor requires at least one of video_encoding / audio_encoding")+ seconds_pred = self._head(video_encoding, audio_encoding)+ if seconds_pred.shape != (1,):+ raise ValueError(+ f"DurationPredictor only supports a single-item batch, got prediction shape {tuple(seconds_pred.shape)}"+ )+ seconds = seconds_pred.item()+ min_frames = round(min_seconds * frame_rate)+ max_frames = round(max_seconds * frame_rate)+ num_frames = seconds_to_clamped_num_frames(+ seconds, frame_rate=frame_rate, min_frames=min_frames, max_frames=max_frames+ )+ if seconds > max_seconds or seconds < min_seconds:+ logger.warning(+ "DurationHead prediction clamped: raw %.2fs outside [%.2fs, %.2fs], using %.2fs (%d frames) @ %.2f fps",+ seconds,+ min_seconds,+ max_seconds,+ num_frames / frame_rate,+ num_frames,+ frame_rate,+ )+ else:+ logger.info("DurationHead predicted %.2fs (%d frames @ %.2f fps)", seconds, num_frames, frame_rate)+ return num_frames+++def require_num_frames_source(num_frames: int | AutoDuration, duration_predictor: DurationPredictor | None) -> None:+ """Guard against an unsatisfiable auto-duration request.+ Call at the very top of a pipeline's ``__call__`` -- before prompt encoding or any other+ work -- so a checkpoint without DurationHead weights (anything predating 2.5) fails fast+ with a clear message instead of after paying for work whose result would be discarded.+ """+ if isinstance(num_frames, AutoDuration) and duration_predictor is None:+ raise ValueError(+ "num_frames was AutoDuration but this checkpoint has no DurationHead weights to "+ "auto-predict duration from (DurationHead ships from LTX 2.5 / gemma4 onward). "+ "Pass num_frames explicitly."+ )+++def resolve_num_frames(+ num_frames: int | AutoDuration,+ duration_predictor: DurationPredictor | None,+ *,+ video_encoding: torch.Tensor | None,+ audio_encoding: torch.Tensor | None,+ frame_rate: float,+) -> int:+ """Resolve ``num_frames`` to a concrete frame count, predicting it if ``AutoDuration``.+ Call after prompt encoding (once ``video_encoding``/``audio_encoding`` exist) and after+ ``require_num_frames_source`` has already validated a predictor is available when needed.+ """+ if not isinstance(num_frames, AutoDuration):+ return num_frames+ return duration_predictor(+ video_encoding,+ audio_encoding,+ frame_rate=frame_rate,+ min_seconds=num_frames.min_seconds,+ max_seconds=num_frames.max_seconds,+ )diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/image_conditioner.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/image_conditioner.pynew file mode 100644index 0000000..cb7513c--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/image_conditioner.py@@ -0,0 +1,119 @@+from __future__ import annotations++from collections.abc import Callable, Sequence+from concurrent.futures import ThreadPoolExecutor+from dataclasses import replace+from typing import TypeVar++import torch++from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy+from ltx_core.loader import SDOps+from ltx_core.loader.registry import ModelRegistry, Registry+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder+from ltx_core.model.video_vae import (+ CHANNELS_LAST_3D_WEIGHTS,+ VAE_ENCODER_COMFY_KEYS_FILTER,+ VideoEncoder,+ VideoEncoderConfigurator,+)+from ltx_pipelines.utils.constants import detect_params+from ltx_pipelines.utils.gpu_model import gpu_model+from ltx_pipelines.utils.media_io.color_config import EXRColorSpace+from ltx_pipelines.utils.media_io.decode import BackgroundStill, check_conditioning_still, load_conditioning_still+from ltx_pipelines.utils.types import DecodedStill, ImageConditioningInput, LoadedImageConditioning++# Shared channels-last encoder sd_ops so the registry caches one copy across encoder builds.+VAE_ENCODER_SD_OPS = SDOps(+ name=f"sd_ops_chain_{VAE_ENCODER_COMFY_KEYS_FILTER.name}+{CHANNELS_LAST_3D_WEIGHTS.name}",+ mapping=(*VAE_ENCODER_COMFY_KEYS_FILTER.mapping, *CHANNELS_LAST_3D_WEIGHTS.mapping),+)++T = TypeVar("T")++__all__ = ["ImageConditioner"]+++class ImageConditioner:+ """Owns video encoder lifecycle, the CRF image conditionings are re-compressed at, and their stills.+ Builds the encoder, passes it to the user-supplied callable, then frees it.+ Also the single place that knows what H.264 CRF an image conditioning should be+ re-compressed at: that value is a property of the model generation, and this block+ already holds the checkpoint it belongs to. Pipelines call ``load_conditionings`` so callers may+ omit the CRF and still condition the way the model was trained, and so each still is+ decoded once per run however many stages, tiles or windows encode it.+ """++ def __init__(+ self,+ checkpoint_path: str,+ dtype: torch.dtype,+ device: torch.device,+ registry: Registry | None = None,+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,+ ) -> None:+ self._checkpoint_path = checkpoint_path+ self._default_image_crf: int | None = None+ self._dtype = dtype+ self._device = device+ self._encoder_builder = Builder(+ model_path=checkpoint_path,+ model_class_configurator=VideoEncoderConfigurator,+ model_sd_ops=VAE_ENCODER_SD_OPS,+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),+ )+ self._alloc_trim_strategy = alloc_trim_strategy+ self._decode_pool = ThreadPoolExecutor(max_workers=2, thread_name_prefix="image-decode")++ @property+ def default_image_crf(self) -> int:+ """The H.264 CRF image conditionings should be re-compressed at for this checkpoint.+ Read from the checkpoint's ``model_version`` metadata (see ``detect_params``) rather+ than hardcoded, because the value the model was trained with changed between+ generations. Resolved on first use and cached, so a run without image conditionings+ never reads the header.+ """+ if self._default_image_crf is None:+ self._default_image_crf = detect_params(self._checkpoint_path).default_image_crf+ return self._default_image_crf++ def resolve_crf(self, images: Sequence[ImageConditioningInput]) -> list[ImageConditioningInput]:+ """Fill in ``default_image_crf`` for every conditioning that left its CRF unset (``crf=None``).+ Call near the top of a pipeline's ``__call__``: omitting the CRF then means "use what+ matches this model", while an explicitly passed CRF (including ``0`` for no+ re-compression) is always honoured.+ """+ return [image if image.crf is not None else replace(image, crf=self.default_image_crf) for image in images]++ def load_conditionings(+ self, images: Sequence[ImageConditioningInput], color_space: EXRColorSpace | None = None+ ) -> list[LoadedImageConditioning]:+ """Resolve every CRF (see ``resolve_crf``) and start decoding each distinct still once for the run.+ Call near the top of a pipeline's ``__call__``. Every input comes back as a+ :class:`LoadedImageConditioning` whose still is shared by every input with the same path and+ CRF and decodes on a worker thread while the run goes on (prompt encoding, model builds);+ every stage, tile, window and chunk that encodes it (or a rebased copy) waits for that still+ and cuts its resolution from it instead of decoding the file again. ``.exr`` stills need+ ``color_space`` (from ``--hdr``). A missing file, or an EXR without ``color_space``, raises+ ``ValueError`` here, before any model is built; an undecodable one raises the same+ ``ValueError`` at its first encode.+ """+ stills: dict[tuple[str, int], DecodedStill] = {}+ loaded: list[LoadedImageConditioning] = []+ for image in self.resolve_crf(images):+ key = (image.path, image.crf)+ if key not in stills:+ check_conditioning_still(image.path, color_space)+ stills[key] = BackgroundStill(+ self._decode_pool.submit(load_conditioning_still, image.path, image.crf, color_space)+ )+ loaded.append(LoadedImageConditioning(still=stills[key], strength=image.strength, binding=image.binding))+ return loaded++ def _build_encoder(self) -> VideoEncoder:+ return self._encoder_builder.build(device=self._device, dtype=self._dtype).eval()++ def __call__(self, fn: Callable[[VideoEncoder], T]) -> T:+ """Build video encoder → call *fn(encoder)* → free encoder."""+ with gpu_model(self._build_encoder(), alloc_trim_strategy=self._alloc_trim_strategy) as encoder:+ return fn(encoder)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/prompt_encoder.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/prompt_encoder.pynew file mode 100644index 0000000..ec574a1--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/prompt_encoder.py@@ -0,0 +1,202 @@+from __future__ import annotations++import logging+from contextlib import AbstractContextManager++import torch++from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy+from ltx_core.block_streaming import DISK_CPU_SLOTS, StreamingModelBuilder+from ltx_core.loader.primitives import BuilderProtocol+from ltx_core.loader.registry import ModelRegistry, Registry+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder+from ltx_core.text_encoders.gemma import (+ EMBEDDINGS_PROCESSOR_KEY_OPS,+ EmbeddingsProcessorConfigurator,+ GemmaTextEncoderConfigurator,+ gemma_model_type,+ get_gemma_ops,+ resolve_gemma_weight_paths,+)+from ltx_core.text_encoders.gemma.embeddings_processor import EmbeddingsProcessor, EmbeddingsProcessorOutput+from ltx_pipelines.utils.gpu_model import gpu_model+from ltx_pipelines.utils.helpers import generate_enhanced_prompt+from ltx_pipelines.utils.model_paths import ModelPaths+from ltx_pipelines.utils.streaming_model import streaming_model+from ltx_pipelines.utils.types import OffloadMode++logger = logging.getLogger(__name__)++__all__ = ["PromptEncoder"]+++class PromptEncoder:+ """Owns text encoder + embeddings processor lifecycle.+ Loads Gemma, optionally enhances then encodes prompts, frees Gemma, then loads the+ embeddings processor. ``_enhancer_text_encoder_builder`` is always set: when enhance+ and encode share a checkpoint it aliases ``_text_encoder_builder`` (same object /+ residency); a distinct builder means sequential enhance-then-encode contexts.+ """++ def __init__(+ self,+ model_paths: ModelPaths,+ dtype: torch.dtype,+ device: torch.device,+ registry: Registry | None = None,+ offload_mode: OffloadMode = OffloadMode.NONE,+ text_encoder_builder: BuilderProtocol | None = None,+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,+ prompt_enhancer_gemma_root: str | None = None,+ enhancer_text_encoder_builder: BuilderProtocol | None = None,+ ) -> None:+ text_encoder_path = model_paths.text_encoder()+ embeddings_paths = model_paths.embeddings_weight_paths+ if not embeddings_paths:+ raise ValueError(+ "PromptEncoder requires ModelPaths.embeddings_weight_paths "+ "(transformer checkpoint and, in split mode, the text-encoder file)."+ )+ self._text_encoder_path = text_encoder_path+ self._embeddings_paths = embeddings_paths+ self._dtype = dtype+ self._device = device+ self._offload_mode = offload_mode+ self._alloc_trim_strategy = alloc_trim_strategy+ self._prompt_enhancer_gemma_root = prompt_enhancer_gemma_root++ registry = registry or ModelRegistry(cache_models=True, cache_weights=False)+ self._registry = registry++ if text_encoder_builder is not None:+ encode_type = text_encoder_builder.model_config().get("model_type") or gemma_model_type(text_encoder_path)+ else:+ encode_type = gemma_model_type(text_encoder_path)+ if encode_type not in _ENCODE_MODEL_TYPES:+ raise ValueError(+ f"Encode root model_type={encode_type!r} is not supported for encoding; "+ f"expected one of {sorted(_ENCODE_MODEL_TYPES)}."+ )+ self._encode_model_type = encode_type++ if text_encoder_builder is not None:+ if offload_mode != OffloadMode.NONE:+ raise ValueError(+ "text_encoder_builder cannot be used with offload_mode != OffloadMode.NONE "+ "because no streaming text encoder builder is available."+ )+ self._text_encoder_builder = text_encoder_builder+ self._streaming_text_encoder_builder = None+ else:+ gemma_sd_ops, gemma_module_ops = get_gemma_ops(text_encoder_path)+ weight_paths = resolve_gemma_weight_paths(text_encoder_path)+ self._text_encoder_builder = Builder(+ model_path=weight_paths,+ model_class_configurator=GemmaTextEncoderConfigurator.with_gemma_model_path(text_encoder_path),+ model_sd_ops=gemma_sd_ops,+ module_ops=gemma_module_ops,+ registry=registry,+ )+ self._streaming_text_encoder_builder = StreamingModelBuilder(+ model_path=weight_paths,+ model_class_configurator=GemmaTextEncoderConfigurator.with_gemma_model_path(text_encoder_path),+ model_sd_ops=gemma_sd_ops,+ module_ops=gemma_module_ops,+ registry=registry,+ blocks_attr="model.model.language_model.layers",+ blocks_prefix="model.model.language_model.layers",+ cpu_slots_count=DISK_CPU_SLOTS if offload_mode == OffloadMode.DISK else None,+ )++ # Always set: shared enhance aliases the encode builder (same object / residency).+ if enhancer_text_encoder_builder is not None:+ self._enhancer_text_encoder_builder = enhancer_text_encoder_builder+ elif prompt_enhancer_gemma_root is not None and prompt_enhancer_gemma_root != text_encoder_path:+ enhancer_sd_ops, enhancer_module_ops = get_gemma_ops(prompt_enhancer_gemma_root)+ enhancer_paths = resolve_gemma_weight_paths(prompt_enhancer_gemma_root)+ self._enhancer_text_encoder_builder = Builder(+ model_path=enhancer_paths,+ model_class_configurator=GemmaTextEncoderConfigurator.with_gemma_model_path(prompt_enhancer_gemma_root),+ model_sd_ops=enhancer_sd_ops,+ module_ops=enhancer_module_ops,+ registry=registry,+ )+ else:+ self._enhancer_text_encoder_builder = self._text_encoder_builder++ self._embeddings_processor_builder = Builder(+ model_path=embeddings_paths,+ model_class_configurator=EmbeddingsProcessorConfigurator.with_gemma_model_path(text_encoder_path),+ model_sd_ops=EMBEDDINGS_PROCESSOR_KEY_OPS,+ registry=registry,+ )++ def _build_text_encoder(self) -> torch.nn.Module:+ """Build the Gemma text encoder (non-streaming path)."""+ return self._text_encoder_builder.build(device=self._device, dtype=self._dtype).eval()++ def _build_enhancer_text_encoder(self) -> torch.nn.Module:+ return self._enhancer_text_encoder_builder.build(device=self._device, dtype=self._dtype).eval()++ def _build_embeddings_processor(self) -> EmbeddingsProcessor:+ """Build the embeddings processor on the target device."""+ return self._embeddings_processor_builder.build(device=self._device, dtype=self._dtype).eval()++ def _text_encoder_ctx(self) -> AbstractContextManager:+ if self._offload_mode != OffloadMode.NONE:+ return streaming_model(+ self._streaming_text_encoder_builder, self._device, self._dtype, self._alloc_trim_strategy+ )+ return gpu_model(self._build_text_encoder(), alloc_trim_strategy=self._alloc_trim_strategy)++ def __call__(+ self,+ prompts: list[str],+ *,+ enhance_first_prompt: bool = False,+ enhance_prompt_image: str | None = None,+ enhance_prompt_seed: int = 42,+ enhance_static_cache: bool = False,+ ) -> list[EmbeddingsProcessorOutput]:+ """Encode *prompts* through Gemma -> embeddings processor, freeing each model after use."""+ prompts = list(prompts)+ enhance_kwargs = {+ "image_path": enhance_prompt_image,+ "seed": enhance_prompt_seed,+ "static_cache": enhance_static_cache,+ }+ separate_enhancer = self._enhancer_text_encoder_builder is not self._text_encoder_builder++ if enhance_first_prompt and separate_enhancer:+ logger.info(+ "Enhancing with separate Gemma%s",+ f" from {self._prompt_enhancer_gemma_root}" if self._prompt_enhancer_gemma_root else "",+ )+ with gpu_model(+ self._build_enhancer_text_encoder(),+ alloc_trim_strategy=self._alloc_trim_strategy,+ ) as enhancer:+ prompts[0] = generate_enhanced_prompt(enhancer, prompts[0], **enhance_kwargs)+ elif enhance_first_prompt and self._encode_model_type != "gemma3":+ raise ValueError(+ f"Prompt enhancement with encode root model_type={self._encode_model_type!r} "+ "requires --prompt-enhancer-gemma-root pointing at a generative instruct checkpoint "+ "(e.g. gemma3 or gemma4 E2B-it)."+ )++ logger.info("Building text encoder from %s", self._text_encoder_path)+ with self._text_encoder_ctx() as text_encoder:+ if enhance_first_prompt and not separate_enhancer:+ prompts[0] = generate_enhanced_prompt(text_encoder, prompts[0], **enhance_kwargs)+ raw_outputs = text_encoder.encode(prompts)+ logger.info("Text encoder done, building embeddings processor from %s", self._embeddings_paths)++ with gpu_model(+ self._build_embeddings_processor(), alloc_trim_strategy=self._alloc_trim_strategy+ ) as embeddings_processor:+ result = [embeddings_processor.process_hidden_states(hs, mask) for hs, mask in raw_outputs]+ logger.info("Prompt encoding complete")+ return result+++_ENCODE_MODEL_TYPES = frozenset({"gemma3", "gemma4", "gemma4_unified"})diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/video_decoder.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/video_decoder.pynew file mode 100644index 0000000..9127487--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/video_decoder.py@@ -0,0 +1,224 @@+from __future__ import annotations++import copy+import logging+from collections.abc import Callable, Iterator, Sequence++import torch++from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy+from ltx_core.loader import SDOps+from ltx_core.loader.module_ops import ModuleOps+from ltx_core.loader.primitives import BuilderProtocol+from ltx_core.loader.registry import ModelRegistry, Registry+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder+from ltx_core.model.transformer.compiling import CompilationConfig+from ltx_core.model.video_vae import (+ CHANNELS_LAST_3D_WEIGHTS,+ MEMORY_EFFICIENT_DECODE,+ VAE_DECODER_COMFY_KEYS_FILTER,+ TilingConfig,+ VideoDecoderConfigurator,+ is_diffusion_video_vae,+ video_decoder_sd_ops_for_checkpoint,+)+from ltx_core.model.video_vae.keyframes import DecodeKeyframes+from ltx_core.model.video_vae.transformer import (+ DiffVAEMode,+ NAttentionKind,+ build_diffvae_mode_op,+ na_dsl_available,+)+from ltx_core.multigpu.vae.distributed_decoder import DistributedVideoDecoder, OwnedSingleFrames+from ltx_pipelines.utils.gpu_model import gpu_model++logger = logging.getLogger(__name__)++__all__ = ["VideoDecoder"]+++class VideoDecoder:+ """Owns video decoder lifecycle.+ Returns an iterator that cleans up the decoder after all chunks are consumed.+ """++ def __init__(+ self,+ checkpoint_path: str,+ dtype: torch.dtype,+ device: torch.device,+ registry: Registry | None = None,+ memory_efficient: bool = True,+ decoder_builder: BuilderProtocol | None = None,+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,+ diffvae_optimization: DiffVAEMode = DiffVAEMode.CHUNKED_EAGER,+ ) -> None:+ self._checkpoint_path = checkpoint_path+ self._dtype = dtype+ self._device = device+ self._diffvae_optimization = diffvae_optimization+ diffusion_vae = is_diffusion_video_vae(self._checkpoint_path)+ self._diffusion_vae = diffusion_vae+ cfg = diffvae_optimization.resolve() if diffusion_vae else None+ use_dsl = cfg is not None and cfg.attention is NAttentionKind.BLACKWELL_DSL+ if decoder_builder is not None:+ self._decoder_builder = decoder_builder+ else:+ if diffusion_vae and use_dsl and not na_dsl_available():+ raise RuntimeError(+ "diffvae_optimization=blackwell_dsl needs a datacenter Blackwell GPU and "+ "nvidia-cutlass-dsl (uv sync --group kernels)"+ )+ sd_ops = (+ video_decoder_sd_ops_for_checkpoint(+ self._checkpoint_path,+ diffusion_vae=True,+ na_dsl_kernel=use_dsl,+ )+ if diffusion_vae+ else VAE_DECODER_COMFY_KEYS_FILTER+ )+ module_ops: tuple[ModuleOps, ...] = ()+ # channels-last + memory-efficient decode apply to ConvVideoDecoder only+ if memory_efficient and not diffusion_vae:+ sd_ops = SDOps(+ name=f"sd_ops_chain_{sd_ops.name}+{CHANNELS_LAST_3D_WEIGHTS.name}",+ mapping=(*sd_ops.mapping, *CHANNELS_LAST_3D_WEIGHTS.mapping),+ )+ module_ops = (MEMORY_EFFICIENT_DECODE,)+ self._decoder_builder = Builder(+ model_path=self._checkpoint_path,+ model_class_configurator=VideoDecoderConfigurator,+ model_sd_ops=sd_ops,+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),+ module_ops=module_ops,+ )+ self._alloc_trim_strategy = alloc_trim_strategy++ @property+ def checkpoint_path(self) -> str:+ return self._checkpoint_path++ @property+ def diffvae_optimization(self) -> DiffVAEMode:+ return self._diffvae_optimization++ def with_builder(self, builder: BuilderProtocol) -> "VideoDecoder":+ """Return a new ``VideoDecoder`` that builds from ``builder``.+ Functional: never mutates ``self``; shares checkpoint, dtype, device, and+ trim strategy. Mirrors :meth:`DiffusionStage.with_builder`.+ """+ new = copy.copy(self)+ new._decoder_builder = builder+ return new++ def _prepared_builder(self) -> BuilderProtocol:+ """Return the builder with DiffVAE mode ops applied at build time.+ DiffVAE ``module_ops`` live on the decoder (not on the caller-held builder),+ matching :meth:`DiffusionStage._prepared_builder`. That keeps+ ``with_builder`` / Dist ``with_tiling`` swaps on the raw builder, while+ every build still installs the configured DiffVAE policy.+ """+ builder = self._decoder_builder+ if not self._diffusion_vae:+ return builder+ op = build_diffvae_mode_op(+ self._diffvae_optimization,+ CompilationConfig() if self._diffvae_optimization.resolve().compile_blocks else None,+ )+ return builder.with_module_ops((*builder.module_ops, op))++ def __call__(+ self,+ latent: torch.Tensor,+ tiling_config: TilingConfig | None = None,+ generator: torch.Generator | None = None,+ *,+ dtype: torch.dtype | None = None,+ keyframes: DecodeKeyframes | None = None,+ device_fn: Callable[[int], str | torch.device] | None = None,+ ) -> Iterator[torch.Tensor]:+ """Decode *latent* to pixel-space video chunks. Decoder freed after exhaustion.+ ``dtype`` overrides the constructor dtype for this call only (e.g. float32 for+ HDR raw-in/raw-out). Omitted → same dtype as on construction (SDR default).+ ``keyframes`` anchors the decode on already-encoded planes. Every decoder in this repo+ takes it, so this never branches on which VAE was built: a diffusion VAE uses the planes,+ a conv VAE logs that it is dropping them, and a distributed decoder splits them across+ its ranks. The keyword is only passed when there is something to pass, so a decoder+ written against the older signature still serves plain decodes.+ """+ build_dtype = self._dtype if dtype is None else dtype+ latent = latent.to(dtype=build_dtype)+ logger.info("Building video decoder from %s", self._checkpoint_path)+ decoder = self._prepared_builder().build(device=self._device, dtype=build_dtype).eval()+ extra = {} if keyframes is None else {"keyframes": keyframes}+ if device_fn is not None:+ extra["device_fn"] = device_fn+ chunks = decoder.decode_video(latent, tiling_config, generator, **extra)+ return _cleanup_iter(+ chunks,+ decoder,+ alloc_trim_strategy=self._alloc_trim_strategy,+ )++ def decode_single_frames(+ self,+ latents: Sequence[torch.Tensor],+ generator: torch.Generator | Sequence[torch.Generator | None] | None = None,+ *,+ dtype: torch.dtype | None = None,+ ) -> Iterator[torch.Tensor]:+ """Decode each latent as its own one-frame clip on one VAE build, then free it.+ A causal VAE cannot decode stacked independent planes without bleeding neighbours.+ A Dist build splits the list across its ranks and gathers the pixels back, so every+ rank returns the whole list; it does not go through the volume-splitting+ ``decode_video`` path, whose workers yield nothing.+ """+ if not latents:+ return iter(())+ build_dtype = self._dtype if dtype is None else dtype+ moved = [latent.to(dtype=build_dtype) for latent in latents]+ logger.info("Building video decoder from %s", self._checkpoint_path)+ decoder = self._prepared_builder().build(device=self._device, dtype=build_dtype).eval()+ chunks = decoder.decode_single_frames(moved, generator)+ return _cleanup_iter(+ chunks,+ decoder,+ alloc_trim_strategy=self._alloc_trim_strategy,+ )++ def decode_owned_single_frames(+ self,+ latents: Sequence[torch.Tensor],+ generator: torch.Generator | Sequence[torch.Generator | None] | None = None,+ *,+ dtype: torch.dtype | None = None,+ ) -> OwnedSingleFrames:+ """``decode_single_frames`` without the pixel gather: only the planes this build decodes.+ A Dist build yields its round-robin share and a collective ``gather`` every rank calls+ once with what it derived from its planes; an SGPU build owns every plane and its+ ``gather`` returns its argument. The decoder is freed once the planes are exhausted.+ """+ if not latents:+ return OwnedSingleFrames([], iter(()), dict)+ build_dtype = self._dtype if dtype is None else dtype+ moved = [latent.to(dtype=build_dtype) for latent in latents]+ logger.info("Building video decoder from %s", self._checkpoint_path)+ decoder = self._prepared_builder().build(device=self._device, dtype=build_dtype).eval()+ if isinstance(decoder, DistributedVideoDecoder):+ owned = decoder.owned_single_frames(moved, generator)+ else:+ owned = OwnedSingleFrames(list(range(len(moved))), decoder.decode_single_frames(moved, generator), dict)+ return owned._replace(+ planes=_cleanup_iter(owned.planes, decoder, alloc_trim_strategy=self._alloc_trim_strategy)+ )+++def _cleanup_iter(+ it: Iterator[torch.Tensor],+ model: torch.nn.Module,+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,+) -> Iterator[torch.Tensor]:+ """Wrap an iterator to release *model* memory once exhausted or abandoned."""+ with gpu_model(model, alloc_trim_strategy=alloc_trim_strategy):+ yield from itdiff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/video_upsampler.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/video_upsampler.pynew file mode 100644index 0000000..f434726--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/blocks/video_upsampler.py@@ -0,0 +1,61 @@+from __future__ import annotations++import logging++import torch++from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy+from ltx_core.loader.registry import ModelRegistry, Registry+from ltx_core.loader.single_gpu_model_builder import SingleGPUModelBuilder as Builder+from ltx_core.model.upsampler import LatentUpsamplerConfigurator, upsample_video+from ltx_core.model.video_vae import VideoEncoderConfigurator+from ltx_pipelines.utils.blocks.image_conditioner import VAE_ENCODER_SD_OPS+from ltx_pipelines.utils.gpu_model import gpu_model++logger = logging.getLogger(__name__)++__all__ = ["VideoUpsampler"]+++class VideoUpsampler:+ """Owns video encoder + spatial upsampler lifecycle."""++ def __init__(+ self,+ checkpoint_path: str,+ upsampler_path: str,+ dtype: torch.dtype,+ device: torch.device,+ registry: Registry | None = None,+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,+ ) -> None:+ self._upsampler_path = upsampler_path+ self._dtype = dtype+ self._device = device+ self._encoder_builder = Builder(+ model_path=checkpoint_path,+ model_class_configurator=VideoEncoderConfigurator,+ model_sd_ops=VAE_ENCODER_SD_OPS,+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),+ )+ self._upsampler_builder = Builder(+ model_path=upsampler_path,+ model_class_configurator=LatentUpsamplerConfigurator,+ registry=registry or ModelRegistry(cache_models=True, cache_weights=False),+ )+ self._alloc_trim_strategy = alloc_trim_strategy++ def __call__(self, latent: torch.Tensor) -> torch.Tensor:+ """Upsample *latent* using video encoder + spatial upsampler, then free both."""+ logger.info("Building video encoder + spatial upsampler from %s", self._upsampler_path)+ with (+ gpu_model(+ self._encoder_builder.build(device=self._device, dtype=self._dtype).eval(),+ alloc_trim_strategy=self._alloc_trim_strategy,+ ) as encoder,+ gpu_model(+ self._upsampler_builder.build(device=self._device, dtype=self._dtype).eval(),+ alloc_trim_strategy=self._alloc_trim_strategy,+ ) as upsampler,+ ):+ return upsample_video(latent=latent, video_encoder=encoder, upsampler=upsampler)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/denoisers.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/denoisers.pyindex bc41cfd..b734612 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/denoisers.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/denoisers.py@@ -47,7 +47,7 @@ def _repeat_state(state: LatentState, n: int) -> LatentState:return LatentState(latent=_repeat(state.latent),- denoise_mask=_repeat(state.denoise_mask),+ denoise_mask=state.denoise_mask.with_same_values(_repeat(state.denoise_mask.tensor)),positions=_repeat(state.positions),clean_latent=_repeat(state.clean_latent),attention_mask=_repeat(state.attention_mask) if state.attention_mask is not None else None,diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/helpers.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/helpers.pyindex bc54583..22e1d00 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/helpers.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/helpers.py@@ -2,6 +2,7 @@ from __future__ import annotationsimport loggingfrom collections.abc import Sequence+from dataclasses import replacefrom typing import TYPE_CHECKINGimport torch@@ -9,14 +10,12 @@ import torchfrom ltx_core.components.noisers import Noiserfrom ltx_core.conditioning import (ConditioningItem,- VideoConditionByKeyframeIndex,- VideoConditionByLatentIndex,VideoGeneratedKeyframeSlots,)from ltx_core.devices import activation_budget_bytes, cleanup_accelerator_memory, get_preferred_devicefrom ltx_core.loader.sft_loader import SafetensorsModelStateDictLoaderfrom ltx_core.model.audio_vae import encode_audio-from ltx_core.model.transformer import Modality+from ltx_core.model.transformer import Modality, TokenTimestepsfrom ltx_core.model.video_vae import AutoTiling, TileSizeConfig, TilingConfig, VideoEncoderfrom ltx_core.model.video_vae.diffusion_tiling import recommended_decode_tiling_configfrom ltx_core.model.video_vae.keyframes import DecodeKeyframes@@ -47,12 +46,11 @@ from ltx_pipelines.utils.media_io import (get_videostream_fps,is_exr_dir,load_exr_as_hdr_conditioning,- load_image_and_preprocess,resize_aspect_ratio_preserving,video_preprocess,)from ltx_pipelines.utils.media_io.color_config import EXRColorSpace-from ltx_pipelines.utils.types import ImageConditioningInput+from ltx_pipelines.utils.types import LoadedImageConditioningif TYPE_CHECKING:from ltx_pipelines.utils.blocks import DiffusionStage@@ -352,102 +350,16 @@ def audio_latent_from_file(return _conform_latent_length(latents, required_latent_frames)-def combined_image_conditionings(- images: list[ImageConditioningInput],+def image_conditionings(+ images: Sequence[LoadedImageConditioning],height: int,width: int,video_encoder: VideoEncoder,dtype: torch.dtype,device: torch.device,- color_space: EXRColorSpace | None = None,-) -> list[ConditioningItem]:- """Create a list of conditionings by replacing the latent at the first frame with the encoded image if present- and using other encoded images as the keyframe conditionings."""- conditionings = []- for img in images:- image = load_image_and_preprocess(- image_path=img.path,- height=height,- width=width,- dtype=dtype,- device=device,- crf=img.crf,- color_space=color_space,- )- encoded_image = video_encoder(image)- if img.frame_idx == 0:- conditioning = VideoConditionByLatentIndex(- latent=encoded_image,- strength=img.strength,- latent_idx=0,- )- else:- conditioning = VideoConditionByKeyframeIndex(- keyframes=encoded_image,- strength=img.strength,- frame_idx=img.frame_idx,- )- conditionings.append(conditioning)- return conditionings---def image_conditionings_by_replacing_latent(- images: list[ImageConditioningInput],- height: int,- width: int,- video_encoder: VideoEncoder,- dtype: torch.dtype,- device: torch.device,- color_space: EXRColorSpace | None = None,-) -> list[ConditioningItem]:- conditionings = []- for img in images:- image = load_image_and_preprocess(- image_path=img.path,- height=height,- width=width,- dtype=dtype,- device=device,- crf=img.crf,- color_space=color_space,- )- encoded_image = video_encoder(image)- conditionings.append(- VideoConditionByLatentIndex(- latent=encoded_image,- strength=img.strength,- latent_idx=img.frame_idx,- )- )-- return conditionings---def image_conditionings_by_adding_guiding_latent(- images: list[ImageConditioningInput],- height: int,- width: int,- video_encoder: VideoEncoder,- dtype: torch.dtype,- device: torch.device,- color_space: EXRColorSpace | None = None,) -> list[ConditioningItem]:- conditionings = []- for img in images:- image = load_image_and_preprocess(- image_path=img.path,- height=height,- width=width,- dtype=dtype,- device=device,- crf=img.crf,- color_space=color_space,- )- encoded_image = video_encoder(image)- conditionings.append(- VideoConditionByKeyframeIndex(keyframes=encoded_image, frame_idx=img.frame_idx, strength=img.strength)- )- return conditionings+ """Encode loaded stills at ``height`` x ``width`` into conditioning items using each ``binding``."""+ return [img.to_item(video_encoder(img.still.tensor(height, width, dtype, device))) for img in images]def evenly_spaced_keyframe_positions(num_keyframes: int, num_frames: int) -> list[int]:@@ -574,9 +486,8 @@ def create_noised_state("""state = tools.create_initial_state(device, dtype, initial_latent)state = state_with_conditionings(state, conditionings, tools)- state = noiser(state, noise_scale)-- return state+ state = replace(state, denoise_mask=state.denoise_mask.compact(device))+ return noiser(state, noise_scale)def state_with_conditionings(@@ -612,11 +523,18 @@ def modality_from_latent_state("""if state.frozen:sigma = torch.zeros_like(sigma)+ values = state.denoise_mask.values+ timesteps = timesteps_from_mask(values.reshape(1, -1), sigma)+ # Clamped because a mask value above the table's last entry would otherwise index one+ # past it, and that read happens inside a captured CUDA graph.+ timestep_indices = torch.searchsorted(+ values, state.denoise_mask.tensor.reshape(state.latent.shape[0], -1), out_int32=True+ ).clamp_(max=values.numel() - 1)return Modality(enabled=enabled,latent=state.latent,sigma=sigma,- timesteps=timesteps_from_mask(state.denoise_mask, sigma),+ timesteps=TokenTimesteps(timesteps, timestep_indices, state.denoise_mask.zero_index),positions=state.positions,context=context,context_mask=None,diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/decode.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/decode.pyindex 010248a..74e32d1 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/decode.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/decode.py@@ -4,8 +4,11 @@ from __future__ import annotationsimport loggingfrom collections.abc import Generator, Iterator+from concurrent.futures import Future+from dataclasses import dataclassfrom io import BytesIOfrom pathlib import Path+from typing import TYPE_CHECKINGimport avimport numpy as np@@ -15,10 +18,11 @@ from PIL import ExifTags, Image, ImageCms, UnidentifiedImageErrorfrom torch._prims_common import DeviceLikeTypefrom ltx_core.types import Audio, VideoPixelShape-from ltx_pipelines.utils.media_io.color_config import EXRColorSpace, srgb_to_acescct+from ltx_pipelines.utils.media_io.color_config import EXRColorSpace, srgb_to_acescct, to_working_spacefrom ltx_pipelines.utils.media_io.exr import (+ _exr_paths_for_conditioning,is_exr_dir,- load_exr_image_as_hdr_conditioning,+ read_exr,)from ltx_pipelines.utils.media_io.range_map import normalize_images, to_vae_rangefrom ltx_pipelines.utils.media_io.resize import (@@ -27,6 +31,9 @@ from ltx_pipelines.utils.media_io.resize import (resize_and_reflect_pad,)+if TYPE_CHECKING:+ from ltx_pipelines.utils.types import DecodedStill+logger = logging.getLogger(__name__)_ORIENTATION_EXIF_KEY = next(key for key, value in ExifTags.TAGS.items() if value == "Orientation")@@ -45,6 +52,9 @@ _INT_FORMAT_MAX: dict[str, float] = {}+_EXR_NEEDS_COLOR_SPACE = "EXR input requires --hdr {SRGB_LINEAR,ACESCG,ACESCCT} to declare the source colour space."++def load_image_and_preprocess(image_path: str,height: int,@@ -60,25 +70,70 @@ def load_image_and_preprocess(An ``.exr`` still requires ``color_space`` (from ``--hdr``) and is loaded via theHDR conditioning path instead of the SDR decode/normalize path."""+ return load_conditioning_still(image_path, crf, color_space).tensor(height, width, dtype, device)+++def load_conditioning_still(image_path: str, crf: int, color_space: EXRColorSpace | None = None) -> DecodedStill:+ """Decode an image conditioning's still once, for every resolution a run encodes it at.+ An SDR image is decoded and re-compressed at ``crf``; an ``.exr`` still is read as linear HDR and+ needs ``color_space`` (from ``--hdr``). A missing or undecodable file of either kind raises+ ``ValueError("Cannot decode image file '<path>'.")``.+ """if str(image_path).lower().endswith(".exr"):if color_space is None:- raise ValueError(- "EXR input requires --hdr {SRGB_LINEAR,ACESCG,ACESCCT} to declare the source colour space."- )- return load_exr_image_as_hdr_conditioning(- exr_path=image_path,- height=height,- width=width,- dtype=dtype,- device=device,- color_space=color_space,- )- image = decode_image(image_path=image_path)- image = preprocess(image=image, crf=crf)- image = torch.tensor(image, dtype=torch.float32, device=device)- image = resize_and_center_crop(image, height, width)- image = normalize_images(image, device, dtype)- return image+ raise ValueError(_EXR_NEEDS_COLOR_SPACE)+ try:+ pixels = read_exr(_exr_paths_for_conditioning(image_path)[0])+ except (OSError, RuntimeError, ValueError) as err:+ raise ValueError(f"Cannot decode image file '{image_path}'.") from err+ return ExrStill(pixels, color_space)+ image = preprocess(image=decode_image(image_path=image_path), crf=crf)+ image.setflags(write=False)+ return SdrStill(image)+++def check_conditioning_still(image_path: str, color_space: EXRColorSpace | None = None) -> None:+ """Raise what :func:`load_conditioning_still` would for a missing file or an EXR without ``color_space``.+ Cheap enough for a pipeline's prologue when the decode itself runs elsewhere.+ """+ if str(image_path).lower().endswith(".exr") and color_space is None:+ raise ValueError(_EXR_NEEDS_COLOR_SPACE)+ if not Path(image_path).exists():+ raise ValueError(f"Cannot decode image file '{image_path}'.")+++@dataclass(frozen=True, eq=False)+class BackgroundStill:+ """A still decoding on a worker thread; :meth:`tensor` waits for it and re-raises its decode error."""++ future: Future[DecodedStill]++ def tensor(self, height: int, width: int, dtype: torch.dtype, device: torch.device) -> torch.Tensor:+ return self.future.result().tensor(height, width, dtype, device)+++@dataclass(frozen=True, eq=False)+class SdrStill:+ """A decoded SDR still, re-compressed at its conditioning CRF: read-only ``(H, W, 3)`` uint8."""++ pixels: np.ndarray++ def tensor(self, height: int, width: int, dtype: torch.dtype, device: torch.device) -> torch.Tensor:+ image = torch.tensor(self.pixels, dtype=torch.float32, device=device)+ image = resize_and_center_crop(image, height, width)+ return normalize_images(image, device, dtype)+++@dataclass(frozen=True, eq=False)+class ExrStill:+ """A linear HDR still as read from EXR, ``(H, W, 3)`` float, mapped to the VAE working space per resolution."""++ pixels: torch.Tensor+ color_space: EXRColorSpace++ def tensor(self, height: int, width: int, dtype: torch.dtype, device: torch.device) -> torch.Tensor:+ frame = resize_and_center_crop(self.pixels.to(device=device, dtype=torch.float32), height, width)+ return to_vae_range(to_working_space(frame, self.color_space)).to(device=device, dtype=dtype)def video_preprocess(@@ -416,12 +471,12 @@ def preprocess(image: np.array, crf: int) -> np.array:``crf`` is required: the correct value is a property of the model generation, so acode-level default here would silently condition on the wrong compression. Reaching thiswith ``crf=None`` means a conditioning skipped resolution -- see- ``ImageConditioner.resolve_crf``, which every pipeline runs over its image inputs.+ ``ImageConditioner.load_conditionings``, which every pipeline runs over its image inputs."""if crf is None:raise ValueError("Image conditioning CRF is unresolved (crf=None). Resolve it against the checkpoint "- "first -- ImageConditioner.resolve_crf(images), or detect_params(checkpoint_path).default_image_crf."+ "first -- ImageConditioner.load_conditionings(images), or detect_params(checkpoint_path).default_image_crf.")if crf == 0:return imagediff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/encode.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/encode.pyindex bd70b14..5b25e66 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/encode.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/encode.py@@ -23,6 +23,7 @@ from ltx_core.color.hlg import encode_linear_hdr_frames_to_hlg_mp4from ltx_core.color.primaries import Primariesfrom ltx_core.color.yuv import FrameConverter, PixelFormat, yuv420p_bt709_converter_from ltx_core.types import Audio+from ltx_pipelines.marking.types import AudioMarker, VideoMarkerfrom ltx_pipelines.utils.media_io.color_config import EXRColorSpace, decode_hdr_videofrom ltx_pipelines.utils.media_io.exr import exr_colorspace_tag, exr_dir_label, save_exr_tensor@@ -130,6 +131,35 @@ def _encode_hdr_video_outputs(return exr_dir+# Frames converted and handed to the encoder thread at a time: x264 starts on the first slice+# while the rest are still being converted and copied off the GPU.+_HOST_SLICE_FRAMES = 8++# The encoder's conversion stream, one per device for the life of the process. cuBLAS keeps a workspace per+# (handle, stream) until the process exits, so every new stream a GEMM runs on stays charged to the allocator.+_CONVERT_STREAMS: dict[torch.device, torch.cuda.Stream] = {}+++def _convert_stream(device: torch.device) -> torch.cuda.Stream:+ stream = _CONVERT_STREAMS.get(device)+ if stream is None:+ stream = _CONVERT_STREAMS[device] = torch.cuda.Stream(device=device)+ return stream+++def _to_host(frames: torch.Tensor) -> np.ndarray:+ """Copy converted frames to the host through a pinned buffer.+ Pinned buffers come from torch's caching host allocator, so repeated encodes reuse them+ instead of page-faulting a fresh pageable allocation per clip. The array keeps its buffer+ alive until the encoder thread drops it.+ """+ if frames.device.type != "cuda":+ return frames.to("cpu").numpy()+ host = torch.empty(frames.shape, dtype=frames.dtype, pin_memory=True)+ host.copy_(frames)+ return host.numpy()++def _open_video_stream(container: av.container.Container,fps: int,@@ -153,7 +183,7 @@ def _open_video_stream(return stream-def encode_sdr_h264(+def encode_sdr_h264( # noqa: PLR0913video: Iterator[torch.Tensor],*,fps: int,@@ -165,26 +195,41 @@ def encode_sdr_h264(crf: int = 19,preset: str = "veryfast",thread_count: int = 0,+ chunks_complete: bool = False,+ video_marker: VideoMarker | None = None,) -> None:"""Mux RGB frame chunks to SDR H.264 (optional AAC).Pass :func:`~ltx_pipelines.chunks.deferred_stitch_audio` (and ``audio_sampling_rate``)when audio is assembled while ``video`` is consumed. Pull the iterator only after- video drain.+ video drain. ``chunks_complete`` says each CUDA chunk is finished on every stream when+ ``video`` yields it (see :func:`encode_video`). ``video_marker`` stamps each slice on the+ stream that converts it."""- def convert(chunk: torch.Tensor) -> torch.Tensor:- return frame_converter(chunk.movedim(-1, -3))+ def converted(piece: torch.Tensor) -> np.ndarray:+ if video_marker is not None:+ piece = video_marker.embed(piece)+ return _to_host(frame_converter(piece.movedim(-1, -3)))++ def host_frames(piece: torch.Tensor) -> np.ndarray:+ """Stamp, convert and copy ``piece`` to the host; on the encoder's own stream for complete chunks."""+ if not (chunks_complete and piece.is_cuda):+ return converted(piece)+ with torch.cuda.stream(_convert_stream(piece.device)):+ return converted(piece)first_raw_chunk = next(video, None)if first_raw_chunk is None:raise ValueError("video is empty; expected at least one frame chunk.")- first_chunk = convert(first_raw_chunk)+ # Converted a slice at a time, so the converter's float temporaries stay slice-sized.+ first_slices = iter(first_raw_chunk.split(_HOST_SLICE_FRAMES))+ first_slice = host_frames(next(first_slices))if frame_converter.pixel_format == PixelFormat.RGB24:- height, width = first_chunk.shape[-3], first_chunk.shape[-2]+ height, width = first_slice.shape[-3], first_slice.shape[-2]else:- height = first_chunk.shape[-2] * 2 // 3- width = first_chunk.shape[-1]+ height = first_slice.shape[-2] * 2 // 3+ width = first_slice.shape[-1]Path(output_path).parent.mkdir(parents=True, exist_ok=True)container = av.open(output_path, mode="w")@@ -199,16 +244,20 @@ def encode_sdr_h264(av_format = frame_converter.pixel_format.av_formatdef cpu_chunks() -> Generator[np.ndarray, None, None]:- yield first_chunk.to("cpu").numpy()+ progress = tqdm(total=video_chunks_number)+ yield first_slice+ yield from (host_frames(piece) for piece in first_slices)+ progress.update()for chunk in video:- yield convert(chunk).to("cpu").numpy()+ yield from (host_frames(piece) for piece in chunk.split(_HOST_SLICE_FRAMES))+ progress.update()+ progress.close()_encode_chunks_threaded(container=container,stream=stream,av_format=av_format,chunks=cpu_chunks(),- progress_total=video_chunks_number,)mux_audio = next(audio, None) if audio is not None else None@@ -237,6 +286,9 @@ def encode_video( # noqa: PLR0913*,color_space: EXRColorSpace | None = None,audio_sampling_rate: int | None = None,+ video_marker: VideoMarker | None = None,+ audio_marker: AudioMarker | None = None,+ chunks_complete: bool = False,) -> Path | None:"""Encode RGB frames to video; SDR H.264 or HDR EXR+HLG when ``color_space`` is set.Args:@@ -263,11 +315,21 @@ def encode_video( # noqa: PLR0913When set, writes half EXR frames + HLG master.audio_sampling_rate: AAC sample rate. Inferred when ``audio`` is an:class:`~ltx_core.types.Audio`; required for an audio iterator.+ video_marker: Optional pixel marker. ``None`` leaves pixels unchanged.+ HDR encode must not pass one.+ audio_marker: Optional audio marker. ``None`` leaves PCM unchanged.+ Each yielded ``Audio`` is stamped before AAC mux (SDR and HDR).+ chunks_complete: Each CUDA chunk ``video`` yields is already finished on every stream+ (e.g. the distributed decoder's batches). SDR conversion and host copies then run+ on a stream of their own, whose syncs do not wait for whatever else the device is+ still running -- such as the rest of a streaming decode. Leave ``False`` when a+ chunk may still be in flight on the current stream.Returns:EXR output directory when HDR, else ``None``.Raises:ValueError: On an empty ``video``, a non-stereo ``audio`` waveform, or conflicting- explicit and embedded audio sampling rates.+ explicit and embedded audio sampling rates. Also if ``video_marker`` is set+ together with ``color_space``."""if isinstance(audio, Audio):validate_audio_waveform(audio)@@ -281,38 +343,56 @@ def encode_video( # noqa: PLR0913if isinstance(video, torch.Tensor):video = iter([video])+ if video_marker is not None and color_space is not None:+ raise ValueError("video_marker is not supported on HDR encode")++ marked_audio = _mark_audio(audio, audio_marker) if audio_marker is not None and audio is not None else audioif color_space is not None:return _encode_hdr_video_outputs(video,Path(output_path),fps=float(fps),color_space=color_space,- audio=audio,+ audio=marked_audio,audio_sampling_rate=audio_sampling_rate,)-encode_sdr_h264(video,fps=fps,output_path=output_path,video_chunks_number=video_chunks_number,- audio=audio,+ audio=marked_audio,audio_sampling_rate=audio_sampling_rate,frame_converter=frame_converter,crf=crf,preset=preset,thread_count=thread_count,+ chunks_complete=chunks_complete,+ video_marker=video_marker,)return None-def encode_audio(audio: Audio, output_path: str) -> None:+def _mark_audio(audio: Iterator[Audio], marker: AudioMarker) -> Iterator[Audio]:+ """Stamp each yielded ``Audio`` independently. Never stitches tracks."""+ for track in audio:+ yield marker.embed(track)+++def encode_audio(+ audio: Audio,+ output_path: str,+ *,+ audio_marker: AudioMarker | None = None,+) -> None:"""Save an audio waveform as a 16-bit PCM ``.wav`` file at the source sampling rate.Reuses :func:`_write_audio` (the same muxing path used by :func:`encode_video`);the only difference is a PCM (``pcm_s16le``) stream in a WAV container instead ofthe AAC stream used for muxed video."""validate_audio_waveform(audio)+ if audio_marker is not None:+ audio = audio_marker.embed(audio)Path(output_path).parent.mkdir(parents=True, exist_ok=True)container = av.open(output_path, mode="w")audio_stream = container.add_stream("pcm_s16le", rate=audio.sampling_rate)@@ -331,7 +411,6 @@ def _encode_chunks_threaded(stream: av.video.stream.VideoStream,av_format: str,chunks: Iterator[np.ndarray],- progress_total: int,) -> None:"""Run libx264 frame.encode + container.mux on a background thread whilethe caller produces numpy chunks on the current thread. The 1-slot queue@@ -369,7 +448,7 @@ def _encode_chunks_threaded(encoder_thread = threading.Thread(target=encoder_worker, name="h264-encoder")encoder_thread.start()try:- for arr in tqdm(chunks, total=progress_total):+ for arr in chunks:chunk_queue.put(arr)finally:chunk_queue.put(None)diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/range_map.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/range_map.pyindex d6d1ded..31dc392 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/range_map.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/media_io/range_map.py@@ -12,10 +12,14 @@ def normalize_images(images: torch.Tensor, device: torch.device, dtype: torch.dtdef to_vae_range(x: torch.Tensor) -> torch.Tensor:"""Map [0, 1] working-space codes to [-1, 1] (VAE input convention).Raises:- ValueError: If ``x`` is outside ``[0, 1]`` (callers must compress or- already hold valid working-space codes — this does not clamp).+ ValueError: If a host ``x`` is outside ``[0, 1]`` (callers must compress or+ already hold valid working-space codes — this does not clamp). A CUDA ``x`` is+ checked asynchronously: a violation fails the stream with a device-side assert."""- if bool((x < 0).any() or (x > 1).any()):+ if x.is_cuda:+ # Checked on the device: reading the result back would wait for every queued kernel.+ torch._assert_async(((x >= 0) & (x <= 1)).all(), "to_vae_range expects values in [0, 1]")+ elif bool((x < 0).any() or (x > 1).any()):raise ValueError(f"to_vae_range expects values in [0, 1]; got min={float(x.min())}, max={float(x.max())}")return x * 2.0 - 1.0diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/samplers.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/samplers.pyindex 2130b55..b1e1ac7 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/samplers.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/samplers.py@@ -15,6 +15,7 @@ from ltx_core.components.diffusion_steps import (from ltx_core.components.protocols import DiffusionStepProtocolfrom ltx_core.devices import highest_precision_floatfrom ltx_core.model.transformer import X0Model+from ltx_core.types import HostDeviceTensorPairfrom ltx_core.utils import to_denoised, to_velocityfrom ltx_pipelines.utils.helpers import post_process_latent, timesteps_from_maskfrom ltx_pipelines.utils.res2s import get_res2s_coefficients@@ -23,6 +24,15 @@ from ltx_pipelines.utils.types import DenoisedLatentResult, Denoiser, LatentStatlogger = logging.getLogger(__name__)+def _present_state(video_state: LatentState | None, audio_state: LatentState | None) -> LatentState:+ """The video state, else the audio state; a loop needs at least one."""+ if video_state is not None:+ return video_state+ if audio_state is not None:+ return audio_state+ raise ValueError("At least one of video_state or audio_state must be provided")++def _step_state(state: LatentState | None,denoised: torch.Tensor | None,@@ -33,7 +43,7 @@ def _step_state("""Advance one diffusion step for a single modality, or return ``None`` if absent."""if state is None or denoised is None:return state- denoised = post_process_latent(denoised, state.denoise_mask, state.clean_latent)+ denoised = post_process_latent(denoised, state.denoise_mask.tensor, state.clean_latent)return replace(state, latent=stepper.step(state.latent, denoised, sigmas, step_idx))@@ -69,13 +79,14 @@ def euler_denoising_loop(Final video and audio :class:`LatentState` after the denoising loop."""stepper = EulerDiffusionStep()- for step_idx, _ in enumerate(tqdm(sigmas[:-1])):- video_result, audio_result = denoiser(transformer, video_state, audio_state, sigmas, step_idx)+ schedule = HostDeviceTensorPair.on_device(sigmas, _present_state(video_state, audio_state).latent.device)+ for step_idx in tqdm(range(len(schedule.device_tensor) - 1)):+ video_result, audio_result = denoiser(transformer, video_state, audio_state, schedule.device_tensor, step_idx)denoised_video = video_result.denoised if video_result is not None else Nonedenoised_audio = audio_result.denoised if audio_result is not None else None- video_state = _step_state(video_state, denoised_video, stepper, sigmas, step_idx)- audio_state = _step_state(audio_state, denoised_audio, stepper, sigmas, step_idx)+ video_state = _step_state(video_state, denoised_video, stepper, schedule.host_tensor, step_idx)+ audio_state = _step_state(audio_state, denoised_audio, stepper, schedule.host_tensor, step_idx)return VideoAudio(video_state, audio_state)@@ -104,6 +115,7 @@ def gradient_estimating_euler_denoising_loop(stepper = EulerDiffusionStep()previous_audio_velocity = Noneprevious_video_velocity = None+ schedule = HostDeviceTensorPair.on_device(sigmas, _present_state(video_state, audio_state).latent.device)def update_velocity_and_sample(noisy_sample: torch.Tensor, denoised_sample: torch.Tensor, sigma: float, previous_velocity: torch.Tensor | None@@ -115,17 +127,21 @@ def gradient_estimating_euler_denoising_loop(denoised_sample = to_denoised(noisy_sample, total_velocity, sigma)return current_velocity, denoised_sample- for step_idx, _ in enumerate(tqdm(sigmas[:-1])):- video_result, audio_result = denoiser(transformer, video_state, audio_state, sigmas, step_idx)+ for step_idx in tqdm(range(len(schedule.device_tensor) - 1)):+ video_result, audio_result = denoiser(transformer, video_state, audio_state, schedule.device_tensor, step_idx)denoised_video = video_result.denoised if video_result is not None else Nonedenoised_audio = audio_result.denoised if audio_result is not None else Noneif video_state is not None and denoised_video is not None:- denoised_video = post_process_latent(denoised_video, video_state.denoise_mask, video_state.clean_latent)+ denoised_video = post_process_latent(+ denoised_video, video_state.denoise_mask.tensor, video_state.clean_latent+ )if audio_state is not None and denoised_audio is not None:- denoised_audio = post_process_latent(denoised_audio, audio_state.denoise_mask, audio_state.clean_latent)+ denoised_audio = post_process_latent(+ denoised_audio, audio_state.denoise_mask.tensor, audio_state.clean_latent+ )- if sigmas[step_idx + 1] == 0:+ if schedule.host_tensor[step_idx + 1] == 0:if video_state is not None and denoised_video is not None:video_state = replace(video_state, latent=denoised_video)if audio_state is not None and denoised_audio is not None:@@ -134,18 +150,20 @@ def gradient_estimating_euler_denoising_loop(if video_state is not None and denoised_video is not None:previous_video_velocity, denoised_video = update_velocity_and_sample(- video_state.latent, denoised_video, sigmas[step_idx], previous_video_velocity+ video_state.latent, denoised_video, schedule.host_tensor[step_idx], previous_video_velocity)video_state = replace(- video_state, latent=stepper.step(video_state.latent, denoised_video, sigmas, step_idx)+ video_state,+ latent=stepper.step(video_state.latent, denoised_video, schedule.host_tensor, step_idx),)if audio_state is not None and denoised_audio is not None:previous_audio_velocity, denoised_audio = update_velocity_and_sample(- audio_state.latent, denoised_audio, sigmas[step_idx], previous_audio_velocity+ audio_state.latent, denoised_audio, schedule.host_tensor[step_idx], previous_audio_velocity)audio_state = replace(- audio_state, latent=stepper.step(audio_state.latent, denoised_audio, sigmas, step_idx)+ audio_state,+ latent=stepper.step(audio_state.latent, denoised_audio, schedule.host_tensor, step_idx),)return VideoAudio(video_state, audio_state)@@ -182,11 +200,11 @@ def _inject_sde_noise(eta: float = 0.5,) -> torch.Tensor:sigmas_copy = sigmas.clone()- hp = highest_precision_float(state.denoise_mask.device)+ hp = highest_precision_float(state.denoise_mask.tensor.device)new_noise = new_noise_fn(state.latent, step_noise_generator)if not legacy_mode:- timesteps = timesteps_from_mask(state.denoise_mask.to(hp), sigmas_copy[step_idx].to(hp))- next_timesteps = timesteps_from_mask(state.denoise_mask.to(hp), sigmas_copy[step_idx + 1].to(hp))+ timesteps = timesteps_from_mask(state.denoise_mask.tensor.to(hp), sigmas_copy[step_idx].to(hp))+ next_timesteps = timesteps_from_mask(state.denoise_mask.tensor.to(hp), sigmas_copy[step_idx + 1].to(hp))sigmas = torch.stack([timesteps, next_timesteps])step_idx = 0x_next = stepper.step(@@ -199,7 +217,7 @@ def _inject_sde_noise()if legacy_mode:- x_next = post_process_latent(x_next, state.denoise_mask, state.clean_latent)+ x_next = post_process_latent(x_next, state.denoise_mask.tensor, state.clean_latent)return x_next@@ -273,19 +291,24 @@ def res2s_audio_video_denoising_loop( # noqa: PLR0913,PLR0915,PLR0912substep_noise_injecting_fn = partial(sde_noise_injecting_fn, step_noise_generator=substep_noise_generator, eta=0.5)n_full_steps = len(sigmas) - 1+ host_sigmas = sigmas.cpu()+ terminal_sigma = bool(host_sigmas[-1] == 0)# inject minimal sigma value to avoid division by zero- if sigmas[-1] == 0:+ if terminal_sigma:sigmas = torch.cat([sigmas[:-1], torch.tensor([0.0011, 0.0], device=sigmas.device)], dim=0)+ host_sigmas = sigmas.cpu()+ schedule = HostDeviceTensorPair(host_tensor=host_sigmas, device_tensor=sigmas)# Compute step sizes in hyperbolic space- hs = -torch.log(sigmas[1:].to(hp).cpu() / (sigmas[:-1].to(hp).cpu()))+ host_sigmas_hp = schedule.host_tensor.to(hp)+ hs = -torch.log(host_sigmas_hp[1:] / host_sigmas_hp[:-1])# Initialize phi cache for reuse across loop iterationsphi_cache = {}c2 = 0.5 # Midpoint for res_2sfor step_idx in tqdm(range(n_full_steps)):- sigma = sigmas[step_idx].to(hp)- sigma_next = sigmas[step_idx + 1].to(hp)+ sigma = schedule.device_tensor[step_idx].to(hp)+ sigma_next = schedule.device_tensor[step_idx + 1].to(hp)# Initialize anchor pointx_anchor_video = video_state.latent.clone().to(hp) if video_state is not None else None@@ -294,13 +317,17 @@ def res2s_audio_video_denoising_loop( # noqa: PLR0913,PLR0915,PLR0912# ====================================================================# STAGE 1: Evaluate at current point# ====================================================================- video_result, audio_result = denoiser(transformer, video_state, audio_state, sigmas, step_idx)+ video_result, audio_result = denoiser(transformer, video_state, audio_state, schedule.device_tensor, step_idx)denoised_video_1 = video_result.denoised if video_result is not None else Nonedenoised_audio_1 = audio_result.denoised if audio_result is not None else Noneif video_state is not None and denoised_video_1 is not None:- denoised_video_1 = post_process_latent(denoised_video_1, video_state.denoise_mask, video_state.clean_latent)+ denoised_video_1 = post_process_latent(+ denoised_video_1, video_state.denoise_mask.tensor, video_state.clean_latent+ )if audio_state is not None and denoised_audio_1 is not None:- denoised_audio_1 = post_process_latent(denoised_audio_1, audio_state.denoise_mask, audio_state.clean_latent)+ denoised_audio_1 = post_process_latent(+ denoised_audio_1, audio_state.denoise_mask.tensor, audio_state.clean_latent+ )h = hs[step_idx].item()@@ -350,7 +377,7 @@ def res2s_audio_video_denoising_loop( # noqa: PLR0913,PLR0915,PLR0912# ====================================================================# ITERATIVE REFINEMENT (Bong Iteration)# ====================================================================- if bongmath and h < 0.5 and sigma > 0.03:+ if bongmath and h < 0.5 and schedule.host_tensor[step_idx] > 0.03:for _ in range(bongmath_max_iter):if x_mid_video is not None and eps_1_video is not None:x_anchor_video = x_mid_video - h * a21 * eps_1_video@@ -377,15 +404,19 @@ def res2s_audio_video_denoising_loop( # noqa: PLR0913,PLR0915,PLR0912transformer,video_state=mid_video_state,audio_state=mid_audio_state,- sigmas=torch.stack([sub_sigma]).to(sigmas.device),+ sigmas=torch.stack([sub_sigma]).to(schedule.device_tensor.device),step_index=0,)denoised_video_2 = video_result_2.denoised if video_result_2 is not None else Nonedenoised_audio_2 = audio_result_2.denoised if audio_result_2 is not None else Noneif video_state is not None and denoised_video_2 is not None:- denoised_video_2 = post_process_latent(denoised_video_2, video_state.denoise_mask, video_state.clean_latent)+ denoised_video_2 = post_process_latent(+ denoised_video_2, video_state.denoise_mask.tensor, video_state.clean_latent+ )if audio_state is not None and denoised_audio_2 is not None:- denoised_audio_2 = post_process_latent(denoised_audio_2, audio_state.denoise_mask, audio_state.clean_latent)+ denoised_audio_2 = post_process_latent(+ denoised_audio_2, audio_state.denoise_mask.tensor, audio_state.clean_latent+ )# ====================================================================# FINAL COMBINATION: Compute x_next using RK coefficients@@ -410,7 +441,7 @@ def res2s_audio_video_denoising_loop( # noqa: PLR0913,PLR0915,PLR0912state=video_state,sample=x_anchor_video,denoised_sample=x_next_video,- sigmas=sigmas,+ sigmas=schedule.device_tensor,step_idx=step_idx,)if x_next_audio is not None and audio_state is not None:@@ -418,7 +449,7 @@ def res2s_audio_video_denoising_loop( # noqa: PLR0913,PLR0915,PLR0912state=audio_state,sample=x_anchor_audio,denoised_sample=x_next_audio,- sigmas=sigmas,+ sigmas=schedule.device_tensor,step_idx=step_idx,)@@ -429,15 +460,21 @@ def res2s_audio_video_denoising_loop( # noqa: PLR0913,PLR0915,PLR0912audio_state = replace(audio_state, latent=x_next_audio.to(model_dtype))# Final step if we need to fully remove the noise- if sigmas[-1] == 0:- video_result_final, audio_result_final = denoiser(transformer, video_state, audio_state, sigmas, n_full_steps)+ if terminal_sigma:+ video_result_final, audio_result_final = denoiser(+ transformer, video_state, audio_state, schedule.device_tensor, n_full_steps+ )denoised_video_1 = video_result_final.denoised if video_result_final is not None else Nonedenoised_audio_1 = audio_result_final.denoised if audio_result_final is not None else Noneif video_state is not None and denoised_video_1 is not None:- denoised_video_1 = post_process_latent(denoised_video_1, video_state.denoise_mask, video_state.clean_latent)+ denoised_video_1 = post_process_latent(+ denoised_video_1, video_state.denoise_mask.tensor, video_state.clean_latent+ )video_state = replace(video_state, latent=denoised_video_1.to(model_dtype))if audio_state is not None and denoised_audio_1 is not None:- denoised_audio_1 = post_process_latent(denoised_audio_1, audio_state.denoise_mask, audio_state.clean_latent)+ denoised_audio_1 = post_process_latent(+ denoised_audio_1, audio_state.denoise_mask.tensor, audio_state.clean_latent+ )audio_state = replace(audio_state, latent=denoised_audio_1.to(model_dtype))return VideoAudio(video_state, audio_state)@@ -467,17 +504,18 @@ class _ModalityStep:modality: str,state: LatentState,result: DenoisedLatentResult | None,- make_step_fn: Callable[[DenoisedLatentResult | None, str], Callable[..., torch.Tensor]],+ schedule: HostDeviceTensorPair,+ make_step_fn: Callable[[DenoisedLatentResult | None, str, HostDeviceTensorPair], Callable[..., torch.Tensor]],) -> "_ModalityStep":"""Resolve one modality's step inputs from a denoiser result.``make_step_fn`` is invoked for every present modality, including onewhose result carries no denoised tensor, so that any validation itperforms fires in a stable order regardless of the denoiser's output."""- step_fn = make_step_fn(result, modality)+ step_fn = make_step_fn(result, modality, schedule)denoised = Noneif result is not None and result.denoised is not None:- denoised = post_process_latent(result.denoised.float(), state.denoise_mask, state.clean_latent)+ denoised = post_process_latent(result.denoised.float(), state.denoise_mask.tensor, state.clean_latent)return cls(modality=modality, state=state, step_fn=step_fn, denoised=denoised)@@ -491,7 +529,7 @@ def _ancestral_euler_denoising_loop(new_noise_fn: Callable[[torch.Tensor, torch.Generator], torch.Tensor],model_dtype: torch.dtype,draw_noise: bool,- make_step_fn: Callable[[DenoisedLatentResult | None, str], Callable[..., torch.Tensor]],+ make_step_fn: Callable[[DenoisedLatentResult | None, str, HostDeviceTensorPair], Callable[..., torch.Tensor]],) -> VideoAudio[LatentState]:"""Shared driver for the ancestral (noise-injecting) Euler denoising loops.@@ -505,10 +543,10 @@ def _ancestral_euler_denoising_loop(Whether to draw a noise tensor per modality per step. When ``False``,``noise=None`` is passed to the stepper.make_step_fn:- ``(result, modality) -> step_fn`` factory returning the step callable+ ``(result, modality, schedule) -> step_fn`` factory returning the step callablefor that modality, with any stepper-specific inputs already bound.``step_fn`` is invoked as- ``step_fn(sample=, denoised_sample=, sigmas=, step_index=, noise=)``.+ ``step_fn(sample=, denoised_sample=, step_index=, noise=)``.Called once per present modality per step right after the denoiserreturns (before the terminal-sigma check), so it can also validate thedenoiser output. ``modality`` is ``"video"`` or ``"audio"``.@@ -516,17 +554,17 @@ def _ancestral_euler_denoising_loop(VideoAudio[LatentState]Final ``(video_state, audio_state)`` after the denoising loop."""- if video_state is None and audio_state is None:- raise ValueError("At least one of video_state or audio_state must be provided")-- present_state = video_state if video_state is not None else audio_state+ present_state = _present_state(video_state, audio_state)generator = torch.Generator(device=present_state.latent.device).manual_seed(noise_seed)states: dict[str, LatentState | None] = {"video": video_state, "audio": audio_state}+ schedule = HostDeviceTensorPair.on_device(sigmas, present_state.latent.device)- for step_idx, _ in enumerate(tqdm(sigmas[:-1])):- video_result, audio_result = denoiser(transformer, states["video"], states["audio"], sigmas, step_idx)+ for step_idx in tqdm(range(len(schedule.device_tensor) - 1)):+ video_result, audio_result = denoiser(+ transformer, states["video"], states["audio"], schedule.device_tensor, step_idx+ )pending = [- _ModalityStep.from_modality_result(modality, state, result, make_step_fn)+ _ModalityStep.from_modality_result(modality, state, result, schedule, make_step_fn)for modality, state, result in (("video", states["video"], video_result),("audio", states["audio"], audio_result),@@ -534,7 +572,7 @@ def _ancestral_euler_denoising_loop(if state is not None]- terminal_step = sigmas[step_idx + 1] == 0+ terminal_step = bool(schedule.host_tensor[step_idx + 1] == 0)for step in pending:if step.denoised is None:continue@@ -545,12 +583,11 @@ def _ancestral_euler_denoising_loop(x_next = step.step_fn(sample=step.state.latent.float(),denoised_sample=step.denoised,- sigmas=sigmas,step_index=step_idx,noise=noise,)if draw_noise:- x_next = post_process_latent(x_next, step.state.denoise_mask, step.state.clean_latent)+ x_next = post_process_latent(x_next, step.state.denoise_mask.tensor, step.state.clean_latent)states[step.modality] = replace(step.state, latent=x_next.to(model_dtype))if terminal_step:@@ -625,7 +662,9 @@ def euler_ancestral_denoising_loop(# Gated on eta alone: at s_noise=0 the step still applies its variance-preserving# rescale, so it must not fall back to the noise-free branch.draw_noise=eta > 0,- make_step_fn=lambda _result, _modality: partial(stepper.step, eta=eta, s_noise=s_noise),+ make_step_fn=lambda _result, _modality, schedule: partial(+ stepper.step, sigmas=schedule.host_tensor, eta=eta, s_noise=s_noise+ ),)@@ -682,7 +721,9 @@ def euler_cfg_pp_denoising_loop("""stepper = EulerCfgPpDiffusionStep()- def make_step_fn(result: DenoisedLatentResult | None, modality: str) -> Callable[..., torch.Tensor]:+ def make_step_fn(+ result: DenoisedLatentResult | None, modality: str, schedule: HostDeviceTensorPair+ ) -> Callable[..., torch.Tensor]:"""Bind the modality's uncond prediction and the loop's noise scales into the CFG++ step."""uncond = result.uncond if result is not None else Noneif not isinstance(uncond, torch.Tensor):@@ -691,7 +732,13 @@ def euler_cfg_pp_denoising_loop("Use GuidedDenoiser or FactoryGuidedDenoiser with cfg_scale != 1 ""or force_uncond_pass=True and a negative_context.")- return partial(stepper.step, uncond_denoised=uncond, eta=eta, s_noise=s_noise)+ return partial(+ stepper.step,+ sigmas=schedule.device_tensor,+ uncond_denoised=uncond,+ eta=eta,+ s_noise=s_noise,+ )return _ancestral_euler_denoising_loop(sigmas=sigmas,diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/streaming_model.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/streaming_model.pynew file mode 100644index 0000000..6f9bfee--- /dev/null+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/streaming_model.py@@ -0,0 +1,36 @@+"""Context manager for block-streaming model builds (paired with :func:`~ltx_pipelines.utils.gpu_model.gpu_model`)."""++from __future__ import annotations++from collections.abc import Iterator+from contextlib import contextmanager++import torch++from ltx_core.allocator_trim_strategy import AllocatorTrimStrategy+from ltx_core.block_streaming import StreamingModelBuilder+from ltx_pipelines.utils.helpers import cleanup_memory+++@contextmanager+def streaming_model(+ builder: StreamingModelBuilder,+ target_device: torch.device,+ dtype: torch.dtype,+ alloc_trim_strategy: AllocatorTrimStrategy = AllocatorTrimStrategy.TRIM,+) -> Iterator[object]:+ """Build a streaming wrapper, yield it, then tear down and free memory.+ The builder's own ``cpu_slots_count`` selects RAM vs disk streaming.+ ``teardown()`` always runs -- it releases non-memory resources (forward+ hooks, the disk I/O worker thread, open file handles) that GC would not+ reclaim promptly. ``dispose`` always metas parameter storage; ``TRIM``+ also runs ``cleanup_memory()``, while ``DEFER`` leaves the CUDA cache warm.+ """+ wrapped = builder.build(device=target_device, dtype=dtype)+ try:+ yield wrapped+ finally:+ wrapped.teardown()+ wrapped.dispose()+ if alloc_trim_strategy == AllocatorTrimStrategy.TRIM:+ cleanup_memory()diff --git a/packages/ltx-pipelines/src/ltx_pipelines/utils/types.py b/packages/ltx-pipelines/src/ltx_pipelines/utils/types.pyindex e7b11a6..6f3f873 100644--- a/packages/ltx-pipelines/src/ltx_pipelines/utils/types.py+++ b/packages/ltx-pipelines/src/ltx_pipelines/utils/types.py@@ -1,17 +1,21 @@from __future__ import annotations-from collections.abc import Iterator-from dataclasses import dataclass, field+from collections.abc import Callable, Iterator+from dataclasses import dataclass, field, replacefrom enum import Enumfrom typing import Generic, NamedTuple, Protocol, TypeVarimport torchfrom ltx_core.components.patchifiers import AudioPatchifier, VideoLatentPatchifier-from ltx_core.conditioning import ConditioningItem+from ltx_core.conditioning import (+ ConditioningItem,+ VideoConditionByKeyframeIndex,+ VideoConditionByLatentIndex,+)from ltx_core.model.transformer import X0Modelfrom ltx_core.model.video_vae import TilingConfig-from ltx_core.types import Audio, LatentState+from ltx_core.types import Audio, LatentState, SpatioTemporalScaleFactorsfrom ltx_pipelines.utils.constants import VIDEO_LATENT_CHANNELS, VIDEO_SCALE_FACTORS_T = TypeVar("_T")@@ -32,19 +36,153 @@ class PipelineOutput(NamedTuple):tiling_config: TilingConfig | None-class ImageConditioningInput(NamedTuple):- """An image to condition on, with the H.264 CRF it should be re-compressed at.- Leaving ``crf`` unset means "use whatever matches the model": the pipeline fills it in- from the checkpoint it runs (``ImageConditioner.resolve_crf``), since the value the model- was trained with is a property of the model generation. Pass ``crf`` explicitly to- override it, including ``0`` to skip re-compression entirely.+@dataclass(frozen=True)+class ImageReplace:+ """Pin a still by replacing the latent that starts at a pixel-frame index.+ ``frame_idx`` is zero-based on the decoded video timeline, not a latent index. Because the+ video VAE is causal, valid values are frame 0 and the first pixel frame of each later latent:+ ``1 + VIDEO_SCALE_FACTORS.time * k`` (1, 9, 17, ... for the default temporal scale of 8)."""- path: strframe_idx: int++ def __post_init__(self) -> None:+ _assert_replacement_frame_idx(self.frame_idx)+++@dataclass(frozen=True)+class ImageGuide:+ """Guide the clip with a keyframe at a zero-based decoded pixel-frame index."""++ frame_idx: int+++ImageBinding = ImageReplace | ImageGuide+++@dataclass(frozen=True)+class ImageConditioningInput:+ """A still to condition on, by file, with the H.264 CRF it should be re-compressed at.+ All frame indices are zero-based pixel-frame indices. Leaving ``crf`` unset uses the model's+ checkpoint default; pass it explicitly to override that value, including ``0`` to skip+ re-compression.+ How the still binds is ``binding``: :class:`ImageReplace` or :class:`ImageGuide`.+ Construct with :meth:`at` (frame 0 replaces, later frames guide), :meth:`replace`, or+ :meth:`guide`. Pipelines load it with+ :meth:`~ltx_pipelines.utils.blocks.ImageConditioner.load_conditionings` into a+ :class:`LoadedImageConditioning` before encoding.+ """++ path: strstrength: float+ binding: ImageBindingcrf: int | None = None+ def __post_init__(self) -> None:+ if not isinstance(self.binding, (ImageReplace, ImageGuide)):+ raise TypeError(f"binding must be ImageReplace or ImageGuide, got {type(self.binding).__name__}")++ @classmethod+ def at(+ cls,+ path: str,+ frame_idx: int,+ strength: float,+ crf: int | None = None,+ ) -> ImageConditioningInput:+ """Bind at a pixel frame, replacing frame 0 and guiding at every later frame."""+ if frame_idx == 0:+ return cls.replace(path, 0, strength, crf)+ else:+ return cls.guide(path, frame_idx, strength, crf)++ @classmethod+ def replace(+ cls,+ path: str,+ frame_idx: int,+ strength: float,+ crf: int | None = None,+ ) -> ImageConditioningInput:+ """Replace at a zero-based pixel frame that starts a causal VAE latent."""+ return cls(path=path, strength=strength, binding=ImageReplace(frame_idx), crf=crf)++ @classmethod+ def guide(+ cls,+ path: str,+ frame_idx: int,+ strength: float,+ crf: int | None = None,+ ) -> ImageConditioningInput:+ """Guide generation with a still at a pixel-frame index."""+ return cls(path=path, strength=strength, binding=ImageGuide(frame_idx), crf=crf)+++class DecodedStill(Protocol):+ """A conditioning still decoded once for a run, cut to each resolution it is encoded at."""++ def tensor(self, height: int, width: int, dtype: torch.dtype, device: torch.device) -> torch.Tensor:+ """The still resized and cropped to ``height`` x ``width``, as a ``(1, C, 1, H, W)`` VAE input."""+ ...+++@dataclass(frozen=True)+class LoadedImageConditioning:+ """A decoded still bound to the clip: what a run encodes as an image conditioning.+ Built by :meth:`~ltx_pipelines.utils.blocks.ImageConditioner.load_conditionings` from an+ :class:`ImageConditioningInput`. Rebasing it onto a stage, tile, window or chunk timeline+ (:meth:`map_frame_index`) keeps ``still``, so the file is decoded once per run.+ """++ still: DecodedStill+ strength: float+ binding: ImageBinding++ @property+ def frame_idx(self) -> int:+ """Pixel-frame index carried by either supported image binding."""+ match self.binding:+ case ImageReplace(frame_idx):+ return frame_idx+ case ImageGuide(frame_idx):+ return frame_idx+ raise TypeError(f"unhandled image binding: {self.binding!r}")++ def assert_in_clip(self, num_frames: int) -> None:+ """Raise when this conditioning's pixel-frame index lies outside the planned stitched clip."""+ if self.frame_idx < 0 or self.frame_idx >= num_frames:+ raise ValueError(f"frame_idx={self.frame_idx} is outside the planned stitched range [0, {num_frames})")++ def map_frame_index(self, mapper: Callable[[int], int | None]) -> LoadedImageConditioning | None:+ """Map the pixel-frame index through ``mapper``, preserving replace vs guide binding and the still."""+ local = mapper(self.frame_idx)+ if local is None:+ return None+ match self.binding:+ case ImageReplace():+ return replace(self, binding=ImageReplace(local))+ case ImageGuide():+ return replace(self, binding=ImageGuide(local))+ raise TypeError(f"unhandled image binding: {self.binding!r}")++ def to_item(self, encoded: torch.Tensor) -> ConditioningItem:+ """Lower to a core item, converting a replacement's pixel frame to a latent index."""+ match self.binding:+ case ImageReplace(frame_idx):+ return VideoConditionByLatentIndex(+ latent=encoded,+ strength=self.strength,+ latent_idx=_replacement_frame_to_latent_index(frame_idx),+ )+ case ImageGuide(frame_idx):+ return VideoConditionByKeyframeIndex(+ keyframes=encoded,+ strength=self.strength,+ frame_idx=frame_idx,+ )+ raise TypeError(f"unhandled image binding: {self.binding!r}")+class PipelineComponents:"""@@ -201,3 +339,24 @@ class OffloadMode(Enum):NONE = "none"CPU = "cpu"DISK = "disk"+++def _assert_replacement_frame_idx(+ frame_idx: int,+ scale_factors: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,+) -> None:+ if frame_idx == 0:+ return+ temporal_scale = scale_factors.time+ if frame_idx < 1 or (frame_idx - 1) % temporal_scale != 0:+ raise ValueError(f"ImageReplace frame_idx must be 0 or 1 + {temporal_scale} * k, got {frame_idx}")+++def _replacement_frame_to_latent_index(+ frame_idx: int,+ scale_factors: SpatioTemporalScaleFactors = VIDEO_SCALE_FACTORS,+) -> int:+ if frame_idx == 0:+ return 0+ temporal_scale = scale_factors.time+ return 1 + (frame_idx - 1) // temporal_scalediff --git a/packages/ltx-trainer/AGENTS.md b/packages/ltx-trainer/AGENTS.mdindex be64409..2db140b 100644--- a/packages/ltx-trainer/AGENTS.md+++ b/packages/ltx-trainer/AGENTS.md@@ -152,13 +152,13 @@ The `configs/` directory is documented in two places that both contain a trainin**Model Interface (Modality-based):**```python-from ltx_core.model.transformer.modality import Modality+from ltx_core.model.transformer.modality import Modality, TokenTimestepsvideo = Modality(enabled=True,latent=video_latents, # [B, seq_len, 128] patchified latent tokenssigma=sigma, # [B,] current noise level (per-batch)- timesteps=video_timesteps, # [B, seq_len] per-token timestep embeddings+ timesteps=TokenTimesteps(video_timesteps), # [B, seq_len] per-token timesteps (dense form)positions=video_positions, # [B, 3, seq_len, 2] positional coordinatescontext=video_embeds, # text conditioning embeddingscontext_mask=None, # optional attention mask for text context@@ -167,7 +167,7 @@ audio = Modality(enabled=True,latent=audio_latents,sigma=sigma,- timesteps=audio_timesteps,+ timesteps=TokenTimesteps(audio_timesteps),positions=audio_positions, # [B, 1, seq_len, 2]context=audio_embeds,context_mask=None,@@ -548,21 +548,23 @@ If you do need a new strategy:```pythonfrom dataclasses import replace-from ltx_core.model.transformer.modality import Modality+from ltx_core.model.transformer.modality import Modality, TokenTimesteps# Create modality — all fields except enabled and masks are requiredvideo = Modality(enabled=True,latent=latents, # [B, seq_len, 128]sigma=sigma, # [B,] — the per-batch noise level- timesteps=timesteps, # [B, seq_len] — per-token (sigma * denoise_mask)+ timesteps=TokenTimesteps(timesteps), # [B, seq_len] — per-token (sigma * denoise_mask)positions=positions, # [B, 3, seq_len, 2]context=context, # text embeddings from embeddings_processorcontext_mask=None,)# Update (immutable — must use replace)-video = replace(video, latent=new_latent, sigma=new_sigma, timesteps=new_timesteps)+video = replace(+ video, latent=new_latent, sigma=new_sigma, timesteps=TokenTimesteps(new_timesteps)+)# Disable a modalityaudio = replace(audio, enabled=False)diff --git a/packages/ltx-trainer/docs/custom-training-strategies.md b/packages/ltx-trainer/docs/custom-training-strategies.mdindex 907eec9..8c3bacb 100644--- a/packages/ltx-trainer/docs/custom-training-strategies.md+++ b/packages/ltx-trainer/docs/custom-training-strategies.md@@ -145,7 +145,7 @@ import torchfrom pydantic import Fieldfrom torch import Tensor-from ltx_core.model.transformer.modality import Modality+from ltx_core.model.transformer.modality import Modality, TokenTimestepsfrom ltx_trainer.timestep_samplers import TimestepSamplerfrom ltx_trainer.training_strategies.base_strategy import (ModelInputs,@@ -287,7 +287,7 @@ class InpaintingStrategy(TrainingStrategy):enabled=True,latent=noisy_latents,sigma=sigmas,- timesteps=timesteps,+ timesteps=TokenTimesteps(timesteps),positions=positions,context=video_prompt_embeds,context_mask=prompt_attention_mask,diff --git a/packages/ltx-trainer/docs/utility-scripts.md b/packages/ltx-trainer/docs/utility-scripts.mdindex d456b35..a2b92db 100644--- a/packages/ltx-trainer/docs/utility-scripts.md+++ b/packages/ltx-trainer/docs/utility-scripts.md@@ -218,10 +218,9 @@ uv run python scripts/decode_latents.py /path/to/latents/dir /path/to/output \For inference with trained LoRAs, use the [`ltx-pipelines`](../../ltx-pipelines/) package which providesproduction-ready pipelines:-- **Text/Image-to-Video**: `TI2VidOneStagePipeline`, `TI2VidTwoStagesPipeline`+- **Text/Image-to-Video and keyframe interpolation**: `TI2VidOneStagePipeline`, `TI2VidTwoStagesPipeline`- **Distilled (fast) inference**: `DistilledPipeline`- **IC-LoRA video-to-video**: `ICLoraPipeline`-- **Keyframe interpolation**: `KeyframeInterpolationPipeline`All pipelines support loading custom LoRAs trained with this trainer.diff --git a/packages/ltx-trainer/pyproject.toml b/packages/ltx-trainer/pyproject.tomlindex 41549f4..dcc76d7 100644--- a/packages/ltx-trainer/pyproject.toml+++ b/packages/ltx-trainer/pyproject.toml@@ -1,6 +1,6 @@[project]name = "ltx-trainer"-version = "1.4.2"+version = "1.5.0"description = "LTX-2 training, democratized."readme = "README.md"authors = [diff --git a/packages/ltx-trainer/src/ltx_trainer/training_strategies/flexible.py b/packages/ltx-trainer/src/ltx_trainer/training_strategies/flexible.pyindex 0afd6e0..a46cb01 100644--- a/packages/ltx-trainer/src/ltx_trainer/training_strategies/flexible.py+++ b/packages/ltx-trainer/src/ltx_trainer/training_strategies/flexible.py@@ -15,7 +15,7 @@ import torchfrom pydantic import BaseModel, ConfigDict, Field, model_validatorfrom torch import Tensor-from ltx_core.model.transformer.modality import Modality+from ltx_core.model.transformer.modality import Modality, TokenTimestepsfrom ltx_core.types import SpatioTemporalScaleFactorsfrom ltx_trainer.timestep_samplers import TimestepSamplerfrom ltx_trainer.training_strategies.base_strategy import (@@ -433,7 +433,7 @@ class FlexibleStrategy(TrainingStrategy):enabled=True,latent=noisy_latents,sigma=sigmas,- timesteps=timesteps,+ timesteps=TokenTimesteps(timesteps),positions=positions,context=prompt_embeds,context_mask=prompt_attention_mask,diff --git a/packages/ltx-trainer/src/ltx_trainer/training_strategies/text_to_video.py b/packages/ltx-trainer/src/ltx_trainer/training_strategies/text_to_video.pyindex 66cf229..71d582e 100644--- a/packages/ltx-trainer/src/ltx_trainer/training_strategies/text_to_video.py+++ b/packages/ltx-trainer/src/ltx_trainer/training_strategies/text_to_video.py@@ -12,7 +12,7 @@ import torchfrom pydantic import Fieldfrom torch import Tensor-from ltx_core.model.transformer.modality import Modality+from ltx_core.model.transformer.modality import Modality, TokenTimestepsfrom ltx_trainer import loggerfrom ltx_trainer.timestep_samplers import TimestepSamplerfrom ltx_trainer.training_strategies.base_strategy import (@@ -155,7 +155,7 @@ class TextToVideoStrategy(TrainingStrategy):enabled=True,sigma=sigmas,latent=noisy_video,- timesteps=video_timesteps,+ timesteps=TokenTimesteps(video_timesteps),positions=video_positions,context=video_prompt_embeds,context_mask=prompt_attention_mask,@@ -242,7 +242,7 @@ class TextToVideoStrategy(TrainingStrategy):enabled=True,latent=noisy_audio,sigma=sigmas,- timesteps=audio_timesteps,+ timesteps=TokenTimesteps(audio_timesteps),positions=audio_positions,context=audio_prompt_embeds,context_mask=prompt_attention_mask,diff --git a/packages/ltx-trainer/src/ltx_trainer/training_strategies/video_to_video.py b/packages/ltx-trainer/src/ltx_trainer/training_strategies/video_to_video.pyindex 2d133b3..f734db2 100644--- a/packages/ltx-trainer/src/ltx_trainer/training_strategies/video_to_video.py+++ b/packages/ltx-trainer/src/ltx_trainer/training_strategies/video_to_video.py@@ -11,7 +11,7 @@ import torchfrom pydantic import Fieldfrom torch import Tensor-from ltx_core.model.transformer.modality import Modality+from ltx_core.model.transformer.modality import Modality, TokenTimestepsfrom ltx_trainer import loggerfrom ltx_trainer.timestep_samplers import TimestepSamplerfrom ltx_trainer.training_strategies.base_strategy import (@@ -208,7 +208,7 @@ class VideoToVideoStrategy(TrainingStrategy):enabled=True,latent=combined_latents,sigma=sigmas,- timesteps=timesteps,+ timesteps=TokenTimesteps(timesteps),positions=positions,context=prompt_embeds,context_mask=prompt_attention_mask,diff --git a/packages/ltx-trainer/src/ltx_trainer/validation_runner.py b/packages/ltx-trainer/src/ltx_trainer/validation_runner.pyindex a5f3fb0..cf6fcee 100644--- a/packages/ltx-trainer/src/ltx_trainer/validation_runner.py+++ b/packages/ltx-trainer/src/ltx_trainer/validation_runner.py@@ -36,7 +36,7 @@ from ltx_core.guidance.perturbations import ()from ltx_core.loader import SafetensorsModelStateDictLoaderfrom ltx_core.model.audio_vae.audio_vae import encode_audio as ltx_encode_audio-from ltx_core.model.transformer.modality import Modality+from ltx_core.model.transformer.modality import Modality, TokenTimestepsfrom ltx_core.model.transformer.model import X0Modelfrom ltx_core.model.video_vae import DimensionSizeConfig, TileSizeConfigfrom ltx_core.model.video_vae.diffusion_tiling import recommended_decode_tiling_config@@ -50,6 +50,7 @@ from ltx_core.tools import AudioLatentTools, VideoLatentToolsfrom ltx_core.types import (Audio,AudioLatentShape,+ DenoiseMask,LatentState,SpatioTemporalScaleFactors,VideoLatentShape,@@ -614,13 +615,14 @@ class ValidationRunner:)video_state = replace(video_state,- denoise_mask=torch.zeros_like(video_state.denoise_mask),+ denoise_mask=DenoiseMask(torch.zeros_like(video_state.denoise_mask.tensor)).compact(device),frozen=True,)video_clean = video_stateelse:video_state = video_tools.create_initial_state(device=device, dtype=torch.bfloat16)video_state = self._apply_video_conditionings(video_state, video_tools, sample, cached_media, device)+ video_state = replace(video_state, denoise_mask=video_state.denoise_mask.compact(device))video_clean = video_statevideo_state = noiser(video_state, noise_scale=1.0)@@ -642,7 +644,7 @@ class ValidationRunner:)audio_state = replace(audio_state,- denoise_mask=torch.zeros_like(audio_state.denoise_mask),+ denoise_mask=DenoiseMask(torch.zeros_like(audio_state.denoise_mask.tensor)).compact(device),frozen=True,)audio_clean = audio_state@@ -650,6 +652,7 @@ class ValidationRunner:audio_tools = self._create_audio_tools(num_frames, self._config.frame_rate)audio_state = audio_tools.create_initial_state(device=device, dtype=torch.bfloat16)audio_state = self._apply_audio_conditionings(audio_state, audio_tools, sample, cached_media, device)+ audio_state = replace(audio_state, denoise_mask=audio_state.denoise_mask.compact(device))audio_clean = audio_stateaudio_state = noiser(audio_state, noise_scale=1.0)@@ -754,7 +757,7 @@ class ValidationRunner:stop = start + num_cond_tokensstate.clean_latent[:, start:stop] = tokens- state.denoise_mask[:, start:stop] = 0.0+ state.denoise_mask.tensor[:, start:stop] = 0.0elif cond.type == "reference" and getattr(cond, "audio", None) is not None:tokens = tools.patchifier.patchify(latent)@@ -768,13 +771,12 @@ class ValidationRunner:denoise_mask = torch.zeros(*tokens.shape[:2],1,- device=device,dtype=torch.float32,)state = LatentState(latent=torch.cat([state.latent, torch.zeros_like(tokens, dtype=state.latent.dtype)], dim=1),- denoise_mask=torch.cat([state.denoise_mask, denoise_mask], dim=1),+ denoise_mask=DenoiseMask(torch.cat([state.denoise_mask.tensor, denoise_mask], dim=1)),positions=torch.cat([state.positions, positions], dim=2),clean_latent=torch.cat([state.clean_latent, tokens], dim=1),attention_mask=None,@@ -802,7 +804,7 @@ class ValidationRunner:state = LatentState(latent=state.latent,- denoise_mask=state.denoise_mask * inv,+ denoise_mask=DenoiseMask(state.denoise_mask.tensor * inv.cpu()),positions=state.positions,clean_latent=state.clean_latent * inv + tokens * m,attention_mask=state.attention_mask,@@ -955,13 +957,13 @@ class ValidationRunner:if denoised_video is not None and video_clean is not None:denoised_video = self._post_process_latent(denoised_video,- video_state.denoise_mask,+ video_state.denoise_mask.tensor,video_clean.clean_latent,)if denoised_audio is not None and audio_clean is not None:denoised_audio = self._post_process_latent(denoised_audio,- audio_state.denoise_mask,+ audio_state.denoise_mask.tensor,audio_clean.clean_latent,)@@ -1266,7 +1268,7 @@ class ValidationRunner:enabled=True,latent=state.latent,sigma=sigma,- timesteps=state.denoise_mask * sigma,+ timesteps=TokenTimesteps(state.denoise_mask.tensor * sigma),positions=state.positions,context=context,context_mask=None,diff --git a/pyproject.toml b/pyproject.tomlindex 1ca80f9..cb6551a 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -14,6 +14,34 @@ no-build-isolation-package = ["ltx-kernels"]# 9.25.0.15 is yanked. Linux-only: the wheel is CUDA and has no darwin build.override-dependencies = ["nvidia-cudnn-cu13==9.24.0.43; sys_platform == 'linux'"]+# Videoseal declares `decord` (no aarch64 wheels) and `opencv-python` (needs libGL).+# PixelSeal embed/detect does not use decord (try/except in videoseal). Replace OpenCV+# with the headless wheel so GPU CI / this host can import cv2.+[[tool.uv.dependency-metadata]]+name = "videoseal"+version = "1.0"+requires-dist = [+ "PyWavelets",+ "av",+ "calflops",+ "einops",+ "lpips",+ "omegaconf",+ "opencv-python-headless",+ "pandas",+ "pycocotools",+ "pytorch_msssim",+ "safetensors",+ "scikit-image",+ "scipy",+ "setuptools",+ "tensorboard",+ "timm==0.9.16",+ "torch>=2.3.1",+ "tqdm",+ "transformers",+]+[tool.uv.workspace]members = ["packages/*"]# ltx-kernels compiles CUDA extensions; keep it out of the workspace so a plain@@ -90,8 +118,6 @@ ignore = [[tool.ruff.lint.per-file-ignores]# CI gates: stand-alone CLIs whose whole job is printing a report / GitHub annotations.".github/actions/golden-pairing-gate/check_golden_pairing.py" = ["T201"]-# Emits `export …` lines for eval by kernels-build.-".github/actions/kernels-build/export_cuda_home.py" = ["T201"]# setup.py must set CUDA_HOME before importing torch.utils.cpp_extension."packages/ltx-kernels/setup.py" = ["E402"]# Tests for the CuTe DSL kernels. ltx-kernels is an optional, CUDA-host-only install,@@ -102,6 +128,10 @@ ignore = [# NVFP4 tests ``importorskip("ltx_kernels")`` before importing convert/linear/fuse# (which import the extension at module top level)."packages/ltx-core/tests/ltx_core/quantization/nvfp4/test_*.py" = ["E402"]+# PixelSeal / AudioSeal tests ``importorskip`` extras before importing seal classes+# (which import videoseal / audioseal at module top level).+"packages/ltx-pipelines/tests/ltx_pipelines/test_videoseal.py" = ["E402"]+"packages/ltx-pipelines/tests/ltx_pipelines/test_audioseal.py" = ["E402"]# CuTe DSL kernels. Device-side code written in CUTLASS's own idiom: tensor-algebra# names are conventionally uppercase (M, T, H, W, NH, TILER_N128), a `@cute.kernel`# entry point takes one parameter per compile-time constant, and its arguments are