Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

9.8 KB
Raw Blame
/**
 * Video Gen — how chunk N+1 of a chained render is conditioned on chunk N.
 *
 * Two strategies exist, and which one a request gets is decided here so the
 * route, `prepareParams.js` and the chain orchestrator in `local.js` can't
 * drift on the rule:
 *
 *   'frame'  — extract the prior chunk's LAST FRAME and run chunk N+1 as
 *              image-to-video off that still. Works on every runtime, but the
 *              model gets a pose with no velocity, so motion visibly stalls
 *              and restarts at every seam.
 *   'window' — hand the prior chunk's LAST N FRAMES to LTX-2's
 *              ExtendPipeline.extend_from_video, which conditions on that
 *              window's latent (motion AND appearance) instead of a still.
 *              Only the `ltx2` runtime has the pipeline, so every other
 *              runtime degrades to 'frame'.
 *
 * Why a WINDOW and not the whole prior clip. `extend_from_video` returns
 * `source + extension` (see `retake.py`: `F_total = F_source + extend_frames`),
 * so feeding it the previous chunk's full output makes chunk N+1 contain
 * everything chunk N did, and stitching the chunks then repeats that content
 * once per hop while per-chunk cost grows with every hop. A fixed-size tail
 * window makes the conditioning cost constant, keeps only the most recent
 * appearance state (so exposure/color drift doesn't compound from the clip's
 * opening frames), and — because we know how much of the output is echoed
 * context — lets the orchestrator trim the echo back off before stitching.
 *
 * Pure and dependency-free: `prepareParams.js` imports it without dragging in
 * `local.js`, which the module suites mock wholesale.
 */

/**
 * Default tail-window size, in pixel frames.
 *
 * ~1 second at 24fps, and just under 3 LTX latent frames (the VAE compresses
 * 8 pixel frames per latent), which is enough for the model to read a motion
 * vector out of the context rather than a single pose. Community reports on
 * long-form LTX chaining converge on roughly this size: smaller windows read
 * as a still and reintroduce the seam, larger ones cost more per hop without
 * visibly improving continuity.
 */
export const DEFAULT_CONTEXT_FRAMES = 22;

/**
 * `0` is a real, supported value — "condition on the last frame only", i.e.
 * opt back into the 'frame' strategy on a runtime that could do better. It is
 * NOT the same as an absent value, which takes DEFAULT_CONTEXT_FRAMES.
 */
export const MIN_CONTEXT_FRAMES = 0;

/**
 * Ceiling on the tail window. 121 frames is ~5s at 24fps and 16 LTX latent
 * frames — past that the context costs more to encode and denoise than the
 * extension itself, and LTX's own extend API caps context + extension at 505
 * frames total, which a chained render can reach from both ends at once.
 */
export const MAX_CONTEXT_FRAMES = 121;

/**
 * The LTX-2 VAE's temporal compression: one latent frame decodes to 8 pixel
 * frames (the first latent carries the extra anchor frame, which is why a
 * clip's pixel length is 8k+1 rather than 8k).
 *
 * This is a property of LTX-2's VAE, not a general one — everything derived
 * from it below is only valid for the runtime in CONTEXT_WINDOW_RUNTIMES. A
 * second runtime joining that set has to bring its own stride, and the failure
 * mode if it doesn't is a wrongly-sized trim rather than an error, so make the
 * stride per-runtime at that point instead of reusing this constant.
 */
export const LATENT_FRAME_STRIDE = 8;

/**
 * Runtimes whose helper can condition on a source *video* rather than a still.
 * `ltx2` / `ltx25` route to ExtendPipeline.extend_from_video; the `minimax_h3`
 * runtimes, `wan22`, `fastvideo` and `mlx_video` have no equivalent and take
 * the 'frame' path.
 *
 * Kept here rather than in `videoGen/modeContract.js` — that module's "declare
 * every runtime gate in one table" rule is about mode/source pairings that
 * REJECT a request at two boundaries, which is what had drifted. This never
 * rejects: an unsupported runtime silently gets the 'frame' path.
 *
 * And deliberately NOT derived from the registry's `supportedModes` (#3737),
 * even though every entry now resolves one: `mlx_video` declares `'extend'`
 * too, but implements it by extracting a last frame and running i2v — there is
 * no pipeline there to hand a video to. `supportedModes.includes('extend')`
 * would therefore route mlx_video chains down a path its helper cannot honor.
 * The two questions genuinely differ: "can the user pick Extend mode?" vs "can
 * this runtime condition on a source VIDEO?", and only the latter belongs here.
 */
export const CONTEXT_WINDOW_RUNTIMES = new Set(['ltx2', 'ltx25']);

/** True when the model's runtime exposes a video-conditioned extend path. */
export const supportsContextWindow = (model) => CONTEXT_WINDOW_RUNTIMES.has(model?.runtime);

/**
 * Normalize a caller-supplied context-frame count.
 *
 * Absent (`null`/`undefined`/`''`) → the default. An explicit `0` → 0, which
 * the strategy resolver reads as "last frame only" — conflating the two would
 * make it impossible to opt out of windowed continuation once it's the
 * default. Anything non-finite falls back to the default rather than
 * poisoning the arithmetic downstream; finite values clamp into range.
 */
export const resolveContextFrames = (requested) => {
  if (requested == null || requested === '') return DEFAULT_CONTEXT_FRAMES;
  const n = Number(requested);
  if (!Number.isFinite(n)) return DEFAULT_CONTEXT_FRAMES;
  return Math.min(MAX_CONTEXT_FRAMES, Math.max(MIN_CONTEXT_FRAMES, Math.floor(n)));
};

/**
 * The strategy a chained render will actually use.
 *
 * `contextFrames` is the already-resolved count (call `resolveContextFrames`
 * first). A model whose runtime has no extend pipeline always gets 'frame',
 * whatever the caller asked for — the count is simply ignored rather than
 * rejected, so switching models mid-form can't strand the request.
 */
export const resolveContinuityStrategy = ({ model, contextFrames }) => (
  contextFrames > 0 && supportsContextWindow(model) ? 'window' : 'frame'
);

/**
 * Latent frames to request from ExtendPipeline for a chunk that should add
 * `numFrames` pixel frames. No leading +1: the context window already supplies
 * the anchor frame. Floors at 1 so a too-small request still renders something.
 */
export const extendLatentFrames = (numFrames) => {
  const n = Number(numFrames);
  if (!Number.isFinite(n)) return 1;
  return Math.max(1, Math.floor(n / LATENT_FRAME_STRIDE));
};

/** Pixel frames an `extendLatents`-latent extension contributes to the output. */
export const extendedPixelFrames = (extendLatents) => Math.max(0, Math.floor(Number(extendLatents) || 0)) * LATENT_FRAME_STRIDE;

/**
 * How many leading frames of an extend render are echoed context that the
 * stitched timeline already contains.
 *
 * Derived from the RENDERED output rather than from the context clip we fed
 * in, because the VAE snaps the encoded source to a latent boundary — the
 * echo is generally a few frames longer than the window we supplied. Since
 * `extend()` appends exactly `extend_frames` latents, everything before the
 * last `extendLatents × 8` frames is echo.
 *
 * Returns 0 when the numbers don't support a trim (unprobeable output, an
 * output no longer than the extension, a nonsensical count) — the caller must
 * treat 0 as "leave the chunk alone", never as "trim everything".
 */
export const contextPrefixFrames = ({ totalFrames, extendLatents }) => {
  const total = Number(totalFrames);
  const newFrames = extendedPixelFrames(extendLatents);
  if (!Number.isFinite(total) || total <= 0 || newFrames <= 0) return 0;
  return Math.max(0, total - newFrames);
};

/**
 * First frame index of an `frames`-long tail window in a `totalFrames` clip.
 * Clamps at 0 so a window larger than the clip just keeps the whole clip.
 */
export const tailWindowStartFrame = ({ totalFrames, frames }) => {
  const total = Number(totalFrames);
  const want = Number(frames);
  if (!Number.isFinite(total) || !Number.isFinite(want)) return 0;
  return Math.max(0, Math.floor(total) - Math.floor(want));
};

/**
 * Where to join the ORIGINAL source clip to the model's new frames in a plain
 * (non-chained) native extend render, so the delivered file keeps the source
 * region as shot instead of the model's VAE decode of it.
 *
 * `extend_from_video` returns `source + extension`; the source half has been
 * encoded and decoded again (and its audio re-synthesised), and extending that
 * result again repeats it over the whole accumulated clip. Conditioning is left
 * alone — only delivery changes: frames `[0, keepFrames)` come from the source
 * file and frames `[newStartFrame, outputFrames)` from the render.
 *
 * LTX only conditions on the first `1 + 8k` source frames, so a source with a
 * ragged tail has `droppedSourceFrames` (< one latent) the model never saw. The
 * extension continues from the last frame the model DID see, so those frames
 * are dropped from the join exactly as they already are in the model's output.
 *
 * Returns `null` — the caller keeps the model output — whenever the measured
 * numbers don't match that layout: an unprobeable clip, an echo longer than the
 * source, or a source whose tail the model skipped by more than a latent (it
 * truncated the context, so which frames it kept is not something to guess at).
 */
export const planExtendSplice = ({ sourceFrames, outputFrames, extendLatents }) => {
  const source = Math.floor(Number(sourceFrames));
  const total = Number(outputFrames);
  const keepFrames = contextPrefixFrames({ totalFrames: total, extendLatents });
  if (!Number.isFinite(source) || source <= 0 || keepFrames <= 0) return null;
  if (keepFrames > source || source - keepFrames >= LATENT_FRAME_STRIDE) return null;
  return {
    keepFrames,
    newStartFrame: keepFrames,
    newFrames: total - keepFrames,
    droppedSourceFrames: source - keepFrames,
  };
};