Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: keep the original source clip in a native LTX Extend render instead of the model's re-decode (#11171)

Adam Eivybf95282373b8
diff --git a/server/lib/README.md b/server/lib/README.md
index 93d4e1e33c..b9c783a6ab 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -241,7 +241,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `maintenanceSequence.js` | The maintenance ladder shared by both runners: `MAINTENANCE_TASK_ORDER` / `MAINTENANCE_ORDER_GUIDANCE`, the interleaved `MAINTENANCE_SEQUENCE_TYPES` (a perpetual `claim-issue` drain between every pair of audits), `maintenanceStepParams` (audits file issues, documentation does the work), and `buildMaintenanceSteps` (fix mode runs audits directly with one final drain; issue filing can omit claim jobs) which emits quota-burn-shaped run-once steps so `quotaBurnInvoke.js` can run a step from the Schedule tab's manual run or a Quota Burn family plan alike. Pure. |
| `meetingUrl.js` | Selects the one cacheable join URL (`selectMeetingUrl`) out of a raw Google Calendar event — first http(s) `video` entry point, else `hangoutLink` — and owns the shared `MEETING_URL_MAX` bound. Three-state: `undefined` = the producer never described conferencing, `null` = described with none usable. |
| `minimaxH3Memory.js` | The declared weight-placement table for the three MiniMax H3 entries (#5420) — H3 is the one video model family whose components fit nowhere unassisted, so "does this box have enough" is a render gate, not a UI fact. `MINIMAX_H3_MEMORY_PROFILES` maps entry id → `{ shippedRepo, shippedRevision, profiles }`, each profile carrying an honest `minMemoryGb` host floor and (CUDA only) a `minVramGb` device floor, ordered best-first. Every capacity number is HOISTED from what already existed — the CUDA tiers are `resolve_offload_profile()`'s own thresholds in `scripts/generate_minimax_h3_cuda.py`, the host floors are the entries' `memoryGb` — the sole new number being `MINIMAX_H3_HOST_RESERVE_GB`, a policy reserve held back for the OS. `applyMiniMaxH3MemoryProfiles(list)` is the load-time backfill (twin of migration 317) and guards BOTH `repo` and `revision`, like the speed-profile decorator. `selectMiniMaxH3MemoryProfile({ model, totalMemoryGb })` picks the best profile the HOST can hold (VRAM is the runner's call — the server has no synchronous device view); `miniMaxH3MemoryDeclineReason()` RETURNS the fail-closed reason so the render path can 400 it and a status surface can show it, and returns `null` on an UNMEASURED host — "not measured" is a deferral to the runner, never the same as zero. `validateMiniMaxH3MemoryProfileTable` / `sanitizeMiniMaxH3MemoryProfiles` warn + strip a hand-edited table (NaN floor, duplicate/reserved id, mis-ordered tiers) at load. |
-| `videoContinuity.js` | How chunk N+1 of a chained video render is conditioned on chunk N. `resolveContextFrames(requested)` normalizes the tail-window size (absent → `DEFAULT_CONTEXT_FRAMES` = 22 ≈ 1s @ 24fps; an explicit `0` is preserved as "last frame only", NOT collapsed into the default) and clamps to `MIN_CONTEXT_FRAMES`..`MAX_CONTEXT_FRAMES`. `resolveContinuityStrategy({model, contextFrames})` picks `'window'` (LTX-2 `extend_from_video` conditioned on the prior chunk's last N frames — motion, not just a pose) or `'frame'` (extract the last frame, run i2v), degrading to `'frame'` on any runtime outside `CONTEXT_WINDOW_RUNTIMES` rather than rejecting. `extendLatentFrames` / `extendedPixelFrames` convert across the VAE's `LATENT_FRAME_STRIDE` (8 pixel frames per latent), `contextPrefixFrames({totalFrames, extendLatents})` measures how much of an extend render is echoed context to trim back off before stitching (0 = leave it alone), and `tailWindowStartFrame` gives the cut index for the window itself. Pure — importable from `prepareParams.js` without dragging in `local.js`. Mirrored for the picker in `client/src/lib/videoGenParams.js`, pinned by `videoContinuity.parity.test.js`. |
+| `videoContinuity.js` | How chunk N+1 of a chained video render is conditioned on chunk N. `resolveContextFrames(requested)` normalizes the tail-window size (absent → `DEFAULT_CONTEXT_FRAMES` = 22 ≈ 1s @ 24fps; an explicit `0` is preserved as "last frame only", NOT collapsed into the default) and clamps to `MIN_CONTEXT_FRAMES`..`MAX_CONTEXT_FRAMES`. `resolveContinuityStrategy({model, contextFrames})` picks `'window'` (LTX-2 `extend_from_video` conditioned on the prior chunk's last N frames — motion, not just a pose) or `'frame'` (extract the last frame, run i2v), degrading to `'frame'` on any runtime outside `CONTEXT_WINDOW_RUNTIMES` rather than rejecting. `extendLatentFrames` / `extendedPixelFrames` convert across the VAE's `LATENT_FRAME_STRIDE` (8 pixel frames per latent), `contextPrefixFrames({totalFrames, extendLatents})` measures how much of an extend render is echoed context to trim back off before stitching (0 = leave it alone), `tailWindowStartFrame` gives the cut index for the window itself, and `planExtendSplice({sourceFrames, outputFrames, extendLatents})` decides where a plain native Extend joins the ORIGINAL source to the render's new frames (`null` = layout doesn't verify, keep the model output). Pure — importable from `prepareParams.js` without dragging in `local.js`. Mirrored for the picker in `client/src/lib/videoGenParams.js`, pinned by `videoContinuity.parity.test.js`. |
| `videoPromptLinter.js` | Deterministic lint pass for continuous-video clip prompts (part of #6217, independent of `scriptVideoCompiler.js` #6225 by interface — lints already-built prompt strings plus caller-supplied framing/reference metadata rather than a compiler-shaped clip). `lintClipPrompt(clip, {bible, maxLength})` checks a single clip: a `cutType: 'continue'` clip must open with `"Hard cut to <framing>:"` and use a framing distinct from `previousFraming`; every `references` entry (`{kind: 'cast'|'locations', id}`) must resolve to a bible descriptor present VERBATIM in the prompt (never paraphrased); the prompt must not contain the banned cross-clip referents (same/still/again/continues/as before), negatives (no/without/never), or UI-overlay terms (text/caption/overlay), matched on word boundaries so substrings like "against" or "context" don't false-positive; and the prompt must be `<= MAX_CLIP_PROMPT_LENGTH` (800) characters. Returns `{pass, reasons[]}`, never throws. `lintClips(clips, {bible, maxLength})` lints an ordered chain, deriving each clip's `previousFraming` from the preceding array element's `framing` when not set explicitly. |
| `videoDisclosure.js` | Video Gen provenance/licensing + backend policy-scope facts (#3674). `VIDEO_MODEL_DISCLOSURES` maps each shipped video model id to `{ shippedRepo, disclosure }` (model card URL, weights license, runtime license, pinned-snapshot download size in decimal GB, review date) — every value checked against a primary upstream source, and any fact that could not be established is OMITTED so the UI renders "Unknown" instead of guessing. `applyVideoDisclosures(list)` is the load-time backfill (twin of migration 237) with the same preservation guards: an existing `disclosure` key wins, a custom id is skipped, and a `repo` pointed at a fork keeps Unknown. `VIDEO_BACKEND_DISCLOSURES` / `videoBackendDisclosure(id)` state where inference runs (`execution: local or hosted`) and whose policy applies — execution facts only, never a restrictiveness ranking. `APACHE_2` / `GEMMA_TERMS` are the shared license descriptors `videoTextEncoders.js` reuses, so a license-text correction reaches both tables from one edit. |
| `videoDraftDecoders.js` | Preview-fidelity ("draft") video decode (#5423). `VIDEO_DRAFT_DECODERS` declares each entry id → a separately downloaded decoder asset, pin-guarded on repo AND revision; `applyVideoDraftDecoders(list)` is the load-time backfill and `validateDraftDecoderTable` / `sanitizeDraftDecoders` warn + strip a hand-edited row (missing pin, a multi-shard or path-traversing file list, a runtime whose builder emits no draft flags) so a full decode can never report itself as a draft one. `DRAFT_DECODE_FULL` (`'full'`) is a deliberate NO-OP — `isFullDecode(id)` makes absence and that sentinel the same request, so a full-decode render builds byte-identical spawn args. `draftDecodeDeclineReason({ model, decodeId, models, runtimeRevision, assetCached })` RETURNS (never throws) the reason a draft decode does not apply — the model is a delivery target in the finish graph (`isDeliveryVideoModel`), it declares no decoder, the installed runner checkout is not the revision the asset was verified against, or the weights are not downloaded — because a knob that only makes a render cheaper must degrade rather than 400 a submitted job; `resolveVideoDraftDecoder()` returns the concrete asset or `null`. `publicVideoDraftDecodeOptions(model)` is the picker payload (empty for a model with no decoder, so the client renders no control) and `downloadableVideoDraftDecoders(list)` the download targets. The table ships NO entry, and as of 2026-08-30 that is a decision rather than a gap ([ADR](../../docs/decisions/2026-08-30-h3-draft-decoder-asset.md)): the shim substitutes one file into upstream’s checkpoint root under a STRICT key match, so a candidate must be a complete unquantized full-`VideoVAE` checkpoint — which every genuinely light decoder (TAE, quantized repack, decoder-only head) is not. |
@@ -431,7 +431,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `httpDrain.js` | `createHttpDrain(servers)` tracks every ordinary request the HTTP listeners accept; `begin(windowMs)` is the synchronous shutdown step that stops intake on all of them (`server.close()` plus a 503 `Connection: close` gate for requests arriving on still-open keep-alive sockets), turns off keep-alive on in-flight responses, drops open SSE streams, and resolves `{ remaining }` once accepted requests finish or the window ends (#8323). |
| `httpCompression.js` | `httpCompression` negotiates Brotli/gzip for API and static responses while preserving streaming, ranges, existing encodings, and `no-transform`. |
| `execGit.js` | `execGit(args, cwd, options)` and `execGitSafe(args, cwd, options)` shared process boundary for Git operations, worktrees, and `services/forgeAuth.js` credential resolution. The safe variant converts rejections into `{ exitCode: 1, stdout: '', stderr }`; `services/git.js` re-exports both for compatibility. `cwd` is REQUIRED — it rejects on a missing/blank one rather than letting `spawn` fall back to `process.cwd()` and run git against PortOS's own checkout (#4554). |
-| `ffmpeg.js` | Shared ffmpeg helpers (videoGen + videoTimeline). `probeVideoGeometry(videoPath)` measures a finished clip's `{ width, height, fps, numFrames, durationSec }` (null when any axis is unreadable) for hosted renders that record no frame count. `generateThumbnail(videoPath, jobId, { atSec } = {})` accepts a clamped poster seek while preserving the single-clip default. Includes `probeFrameCount(videoPath)` (metadata `nb_frames`, falling back to a real `-count_frames` pass — expensive, so call it once per file) and `trimVideoFromFrame(videoPath, outPath, {startFrame, fps})` — a frame-EXACT head cut via the `trim` filter (an `-ss` seek can drift a frame, which reads as a stutter at a stitch seam), keeping audio in sync with `atrim` when the clip has a soundtrack and taking `-an` when it doesn't. Re-encodes by necessity, so a later concat must re-encode too; `outPath` may equal the input (temp-file + rename install). Used by the chained-render context window — see `videoContinuity.js`. `buildTrimConcatArgs({inputs, outPath, width, height, fps, withAudio})` builds the argv for the other half of that job: a concat that drops leading frames from some of its inputs inside a `filter_complex` graph, so the cuts ride along with the timeline encode instead of costing one pre-encode per clip (pass `withAudio` only when EVERY input has an audio stream — check with `hasAudioStream`). `H264_ENCODE_ARGS` / `AAC_ENCODE_ARGS` are the shared encode profile: clips produced by different paths here get concatenated together, so a mismatch shows up as one segment graded differently from its neighbours — spread these rather than re-typing the flags. Every re-encode here also pins BT.709: `BT709_CONTAINER_ARGS` (the `colr` atom, always emitted) plus `bt709TagFilter()` → the `BT709_TAG_FILTER` `setparams=…` string, or `null` on an ffmpeg without that filter (`supportsSetparamsFilter()` probes `-filters` once per process; `null` = not probed, and a probe that couldn't run stays uncached). Both halves are required — from ffmpeg 8 the encoder reads color properties off the FRAMES, silently overriding the container flags, so a flags-only output decodes washed-out. `buildTrimConcatArgs` takes the filter as its `colorTagFilter` option rather than probing, to stay pure. `installEncodedVideo(tmpPath, targetPath, label)` is the shared in-place install: move the original aside to a `.bak` on Windows (where `fs.rename` refuses an existing destination), rename the new file over it, and restore the backup — plus unlink the temp — on any failure, returning `{ok:true,outPath}` / `{ok:false,reason}` so the worst case is "skipped", never "the video is gone". Use it for EVERY replace-a-video-with-its-re-encode path; `pipeline/audioMux.js` is the out-of-module caller. `edgeFadeFilter(durationSec)` is the `-af` fade-in/fade-out fragment a mid-song clip (a social cut) appends to its trim. Loudness mastering (#10249): `MASTER_LOUDNESS` (-14 LUFS / -1.5 dBTP / LRA 11), `planLoudnessMaster` (measure then build the linear apply filter; throws on a silent source), `reportMasteredLoudness` (post-encode before/after numbers). |
+| `ffmpeg.js` | Shared ffmpeg helpers (videoGen + videoTimeline). `probeVideoGeometry(videoPath)` measures a finished clip's `{ width, height, fps, numFrames, durationSec }` (null when any axis is unreadable) for hosted renders that record no frame count. `generateThumbnail(videoPath, jobId, { atSec } = {})` accepts a clamped poster seek while preserving the single-clip default. Includes `probeFrameCount(videoPath)` (metadata `nb_frames`, falling back to a real `-count_frames` pass — expensive, so call it once per file) and `trimVideoFromFrame(videoPath, outPath, {startFrame, fps})` — a frame-EXACT head cut via the `trim` filter (an `-ss` seek can drift a frame, which reads as a stutter at a stitch seam), keeping audio in sync with `atrim` when the clip has a soundtrack and taking `-an` when it doesn't. Re-encodes by necessity, so a later concat must re-encode too; `outPath` may equal the input (temp-file + rename install). Used by the chained-render context window — see `videoContinuity.js`. `buildTrimConcatArgs({inputs, outPath, width, height, fps, withAudio})` builds the argv for the other half of that job: a concat that drops leading frames from some of its inputs inside a `filter_complex` graph, so the cuts ride along with the timeline encode instead of costing one pre-encode per clip (pass `withAudio` only when EVERY input has an audio stream — check with `hasAudioStream`). `buildExtendSpliceArgs({sourcePath, generatedPath, outPath, keepFrames, newStartFrame, width, height, fps, audio})` is the plain-Extend counterpart: source frames `[0, keep)` + render frames from `newStartFrame`, one encode, audio crossfaded (~40 ms linear) at the seam — driven by `services/videoGen/extendSplice.js`. `H264_ENCODE_ARGS` / `AAC_ENCODE_ARGS` are the shared encode profile: clips produced by different paths here get concatenated together, so a mismatch shows up as one segment graded differently from its neighbours — spread these rather than re-typing the flags. Every re-encode here also pins BT.709: `BT709_CONTAINER_ARGS` (the `colr` atom, always emitted) plus `bt709TagFilter()` → the `BT709_TAG_FILTER` `setparams=…` string, or `null` on an ffmpeg without that filter (`supportsSetparamsFilter()` probes `-filters` once per process; `null` = not probed, and a probe that couldn't run stays uncached). Both halves are required — from ffmpeg 8 the encoder reads color properties off the FRAMES, silently overriding the container flags, so a flags-only output decodes washed-out. `buildTrimConcatArgs` takes the filter as its `colorTagFilter` option rather than probing, to stay pure. `installEncodedVideo(tmpPath, targetPath, label)` is the shared in-place install: move the original aside to a `.bak` on Windows (where `fs.rename` refuses an existing destination), rename the new file over it, and restore the backup — plus unlink the temp — on any failure, returning `{ok:true,outPath}` / `{ok:false,reason}` so the worst case is "skipped", never "the video is gone". Use it for EVERY replace-a-video-with-its-re-encode path; `pipeline/audioMux.js` is the out-of-module caller. `edgeFadeFilter(durationSec)` is the `-af` fade-in/fade-out fragment a mid-song clip (a social cut) appends to its trim. Loudness mastering (#10249): `MASTER_LOUDNESS` (-14 LUFS / -1.5 dBTP / LRA 11), `planLoudnessMaster` (measure then build the linear apply filter; throws on a silent source), `reportMasteredLoudness` (post-encode before/after numbers). |
| `forgeAccessErrors.js` | `isForgeNoAccessError(err)` / `isForgeNoAccessMessage(text)` / `logForgeNoAccessOnce(scope, repoSpec, detail)` / `forgeNoAccessRemedy(repoSpec)` — tells a PERMANENT `gh` no-access 404 (`Could not resolve to a Repository`, `HTTP 404`) apart from a transient blip. `execGh` stamps `ghNoAccess` on the rejection and keeps it OUT of the consecutive-failure backoff (which caps at 15 min and then re-fires forever), and the polling jobs (`branchReconcile`, `prWatcher`) narrate it exactly once per scope+repo per process with a remedy line instead of once per tick. `__resetForgeNoAccessLog()` is the test seam. |
| `forkHead.js` | `normalizeForkHead(value)` / `forkRemoteName(ownerLogin)` / `resolveTaskForkHead(metadata)` — the validated `{remoteUrl, ownerLogin}` address of a cross-repository PR's head branch, which has no `origin/<branch>` to attach to. Fail-closed: anything that isn't a usable pair (or that starts with `-`, which git would read as an option) is `null`, so every caller falls back to origin-only behavior instead of guessing. `forkRemoteName` is the deterministic `fork-<owner>` remote name that makes a re-attach reuse the remote it already added. Shared by `appPullRequests.js` (produces), `agentWorktreeCleanup.js`/`prRemediationFollowUp.js` (persist in task metadata), `agentWorkspacePrep.js` (threads), and `worktreeManager.js` (consumes — it has no forge client of its own). `resolveTaskForkHead` is the single reader of the generic `forkHead` task-metadata key, the fork-side counterpart to `taskTargetBranch.js`. |
| `frameQuality.js` | Scores decoded video frames so a chained render can CHOOSE its continuation anchor instead of taking whatever an end-seek lands on. `scoreFrame({width,height,data},{recency})` combines three terms over a raw single-channel buffer: log-compressed `gradientVariance` (focus), distance of `meanLuma` from mid-grey (exposure penalty, which rejects a fade-to-black or a blown highlight), and a small linear recency bonus that keeps the anchor near the cut when candidates are comparably clean. Usability and ranking are deliberately separate: `usable` gates on raw gradient variance against `MIN_SIGNAL_VARIANCE` alone, so only a mathematically degenerate frame is rejected and a legitimately dark low-key tail still qualifies — the same contract `imageFrameStats.js` keeps. Gating on the composite score instead would reject every night scene on the exposure term. `TAIL_WINDOW_SECONDS` is capped at the reach of the single `-sseof -1.0` seek this replaces: focus is systematically highest at the OLDEST candidate, so a wider window would widen the backward jump at each chain seam rather than tightening it. `pickBestFrame(paths)` decodes each candidate through `sharp` and returns the winner (or `null` when nothing decodes or every candidate is degenerate, so the caller degrades rather than failing a render); its `index` is the position in the array AS PASSED, because the caller turns that into the anchor's time offset. `MAX_CANDIDATES` is derived from the window, never set independently — the ffmpeg-side `-frames:v` cap truncates from the NEWEST end, exactly the frames recency prefers. `TAIL_WINDOW_SECONDS` / `CANDIDATE_FPS` / `MAX_CANDIDATES` define the window the caller decodes from. Consumer: `services/videoGen/local.js#extractLastFrame`. |
diff --git a/server/lib/ffmpeg.js b/server/lib/ffmpeg.js
index 49d2ac4b42..05104a2068 100644
--- a/server/lib/ffmpeg.js
+++ b/server/lib/ffmpeg.js
@@ -784,6 +784,75 @@ export const buildTrimConcatArgs = ({ inputs, outPath, width, height, fps, withA
];
};
+// Build the argv that joins the ORIGINAL source of a native extend render to the
+// model's new frames: input 0 is the source clip (kept for frames
+// `[0, keepFrames)`), input 1 is the render (kept from `newStartFrame` on).
+// Lets a plain Extend deliver the source as shot instead of the model's VAE
+// decode of it; `lib/videoContinuity.js#planExtendSplice` supplies the indices.
+//
+// Video is one single encode of source + new — never a per-round compounding of
+// the model's own decode. Geometry and rate normalize to the RENDER's, which is
+// what the extension was generated at. Audio follows the same split: the source
+// span from the source's track, the new span from the generated one, joined by a
+// short LINEAR crossfade (`crossfadeSec`) so the seam doesn't click. The source
+// leg runs `crossfadeSec` past the seam so the overlap is source-vs-generated at
+// the SAME timeline position, which keeps the output exactly as long as the render.
+//
+// `audio` is 'splice' (both inputs have a track), 'generated' (only the render
+// has one — the source region has nothing to keep, so the render's track stands)
+// or 'none'. Pure, like buildTrimConcatArgs: the caller probes and passes
+// `colorTagFilter` from `bt709TagFilter()`.
+export const buildExtendSpliceArgs = ({
+ sourcePath, generatedPath, outPath, keepFrames, newStartFrame, width, height, fps,
+ audio = 'none', crossfadeSec = 0.04, colorTagFilter = null,
+}) => {
+ const keep = Math.floor(Number(keepFrames));
+ const start = Math.floor(Number(newStartFrame));
+ const rate = Number(fps);
+ const canonW = Number(width);
+ const canonH = Number(height);
+ if (!sourcePath || !generatedPath || !outPath) return null;
+ if (!Number.isFinite(keep) || keep <= 0 || !Number.isFinite(start) || start <= 0) return null;
+ if (!Number.isFinite(rate) || rate <= 0) return null;
+ if (!Number.isFinite(canonW) || canonW <= 0 || !Number.isFinite(canonH) || canonH <= 0) return null;
+
+ const normalize = `scale=${canonW}:${canonH}:force_original_aspect_ratio=decrease,pad=${canonW}:${canonH}:(ow-iw)/2:(oh-ih)/2,setsar=1,fps=${rate}`;
+ const tag = typeof colorTagFilter === 'string' && colorTagFilter ? colorTagFilter : null;
+ const filters = [
+ `[0:v]trim=end_frame=${keep},setpts=PTS-STARTPTS,${normalize}[v0]`,
+ `[1:v]trim=start_frame=${start},setpts=PTS-STARTPTS,${normalize}[v1]`,
+ `[v0][v1]concat=n=2:v=1:a=0${tag ? '[cv]' : '[outv]'}`,
+ ...(tag ? [`[cv]${tag}[outv]`] : []),
+ ];
+
+ const canonAudio = 'aresample=48000,aformat=sample_fmts=fltp:channel_layouts=stereo';
+ const seamSec = keep / rate;
+ const fade = Math.max(0.001, Number(crossfadeSec) || 0);
+ if (audio === 'splice') {
+ filters.push(
+ // apad guards a source track that ends before the seam + overlap, which
+ // acrossfade would otherwise reject as shorter than the fade.
+ `[0:a]atrim=end=${(seamSec + fade).toFixed(6)},asetpts=PTS-STARTPTS,${canonAudio},apad=whole_dur=${(seamSec + fade).toFixed(6)}[a0]`,
+ `[1:a]atrim=start=${seamSec.toFixed(6)},asetpts=PTS-STARTPTS,${canonAudio}[a1]`,
+ `[a0][a1]acrossfade=d=${fade.toFixed(6)}:c1=tri:c2=tri[outa]`,
+ );
+ } else if (audio === 'generated') {
+ filters.push(`[1:a]asetpts=PTS-STARTPTS,${canonAudio}[outa]`);
+ }
+
+ return [
+ '-i', sourcePath, '-i', generatedPath,
+ '-filter_complex', filters.join(';'),
+ '-map', '[outv]',
+ ...(audio === 'none' ? [] : ['-map', '[outa]']),
+ ...H264_ENCODE_ARGS,
+ ...BT709_CONTAINER_ARGS,
+ ...(audio === 'none' ? ['-an'] : AAC_ENCODE_ARGS),
+ '-movflags', '+faststart',
+ '-y', outPath,
+ ];
+};
+
// 2× Lanczos upscale of an MP4 in place. Doubles width and height while
// preserving the exact aspect ratio and the audio track. Used as a quick
// post-render export option for LTX renders that come out at sub-720p
diff --git a/server/lib/videoContinuity.js b/server/lib/videoContinuity.js
index f9d67df801..7c56768a6d 100644
--- a/server/lib/videoContinuity.js
+++ b/server/lib/videoContinuity.js
@@ -166,3 +166,38 @@ export const tailWindowStartFrame = ({ totalFrames, frames }) => {
if (!Number.isFinite(total) || !Number.isFinite(want)) return 0;
return Math.max(0, Math.floor(total) - Math.floor(want));
};
+
+/**
+ * Where to join the ORIGINAL source clip to the model's new frames in a plain
+ * (non-chained) native extend render, so the delivered file keeps the source
+ * region as shot instead of the model's VAE decode of it.
+ *
+ * `extend_from_video` returns `source + extension`; the source half has been
+ * encoded and decoded again (and its audio re-synthesised), and extending that
+ * result again repeats it over the whole accumulated clip. Conditioning is left
+ * alone — only delivery changes: frames `[0, keepFrames)` come from the source
+ * file and frames `[newStartFrame, outputFrames)` from the render.
+ *
+ * LTX only conditions on the first `1 + 8k` source frames, so a source with a
+ * ragged tail has `droppedSourceFrames` (< one latent) the model never saw. The
+ * extension continues from the last frame the model DID see, so those frames
+ * are dropped from the join exactly as they already are in the model's output.
+ *
+ * Returns `null` — the caller keeps the model output — whenever the measured
+ * numbers don't match that layout: an unprobeable clip, an echo longer than the
+ * source, or a source whose tail the model skipped by more than a latent (it
+ * truncated the context, so which frames it kept is not something to guess at).
+ */
+export const planExtendSplice = ({ sourceFrames, outputFrames, extendLatents }) => {
+ const source = Math.floor(Number(sourceFrames));
+ const total = Number(outputFrames);
+ const keepFrames = contextPrefixFrames({ totalFrames: total, extendLatents });
+ if (!Number.isFinite(source) || source <= 0 || keepFrames <= 0) return null;
+ if (keepFrames > source || source - keepFrames >= LATENT_FRAME_STRIDE) return null;
+ return {
+ keepFrames,
+ newStartFrame: keepFrames,
+ newFrames: total - keepFrames,
+ droppedSourceFrames: source - keepFrames,
+ };
+};
diff --git a/server/lib/videoContinuity.test.js b/server/lib/videoContinuity.test.js
index 016a79c067..9eb5e0121b 100644
--- a/server/lib/videoContinuity.test.js
+++ b/server/lib/videoContinuity.test.js
@@ -2,7 +2,7 @@ import { describe, it, expect } from 'vitest';
import {
DEFAULT_CONTEXT_FRAMES, MIN_CONTEXT_FRAMES, MAX_CONTEXT_FRAMES, LATENT_FRAME_STRIDE,
resolveContextFrames, resolveContinuityStrategy, supportsContextWindow,
- extendLatentFrames, extendedPixelFrames, contextPrefixFrames, tailWindowStartFrame,
+ extendLatentFrames, extendedPixelFrames, contextPrefixFrames, tailWindowStartFrame, planExtendSplice,
} from './videoContinuity.js';
describe('resolveContextFrames', () => {
@@ -128,3 +128,29 @@ describe('tailWindowStartFrame', () => {
expect(tailWindowStartFrame({ totalFrames: 97, frames: NaN })).toBe(0);
});
});
+
+describe('planExtendSplice', () => {
+ it('keeps every source frame the model saw and takes the rest from the render', () => {
+ // 121-frame source (1 + 8*15) extended by 12 latents → 121 + 96 = 217.
+ expect(planExtendSplice({ sourceFrames: 121, outputFrames: 217, extendLatents: 12 }))
+ .toEqual({ keepFrames: 121, newStartFrame: 121, newFrames: 96, droppedSourceFrames: 0 });
+ });
+
+ it('drops the ragged tail LTX never conditioned on', () => {
+ // 125 source frames → the model saw 1 + 8*15 = 121 of them.
+ expect(planExtendSplice({ sourceFrames: 125, outputFrames: 121 + 24, extendLatents: 3 }))
+ .toEqual({ keepFrames: 121, newStartFrame: 121, newFrames: 24, droppedSourceFrames: 4 });
+ });
+
+ it('declines (null = keep the model output) when the layout is not the one it assumes', () => {
+ // Echo longer than the source: the render holds frames the source lacks.
+ expect(planExtendSplice({ sourceFrames: 20, outputFrames: 25 + 24, extendLatents: 3 })).toBeNull();
+ // The model skipped more than a latent of the source tail: it truncated the
+ // context, so which frames it kept is unknown.
+ expect(planExtendSplice({ sourceFrames: 200, outputFrames: 121 + 24, extendLatents: 3 })).toBeNull();
+ // Unmeasurable clips and a nonsensical extension never read as a plan.
+ expect(planExtendSplice({ sourceFrames: null, outputFrames: 145, extendLatents: 3 })).toBeNull();
+ expect(planExtendSplice({ sourceFrames: 121, outputFrames: null, extendLatents: 3 })).toBeNull();
+ expect(planExtendSplice({ sourceFrames: 121, outputFrames: 145, extendLatents: 0 })).toBeNull();
+ });
+});
diff --git a/server/services/videoGen/extendSplice.js b/server/services/videoGen/extendSplice.js
new file mode 100644
index 0000000000..88f5cee47d
--- /dev/null
+++ b/server/services/videoGen/extendSplice.js
@@ -0,0 +1,79 @@
+/**
+ * Video Gen — keep the ORIGINAL source in a native Extend render's output.
+ *
+ * `ExtendPipeline.extend_from_video` returns `source + extension`, decoded in
+ * one pass: the source half has been through the VAE again and its audio
+ * through the audio VAE/vocoder. Extending an already-extended clip repeats
+ * that over the whole accumulated clip, so picture and sound drift each round.
+ * Conditioning is untouched; this changes DELIVERY — the source span comes
+ * from the source file, the new span from the render, joined at the seam
+ * (layout math: `planExtendSplice` in lib/videoContinuity.js).
+ *
+ * Best-effort by design: any failure (no ffmpeg, an unprobeable clip, a layout
+ * that doesn't match, an encode that doesn't verify) leaves the model's own
+ * output in place and reports why. The render is never failed over this.
+ */
+
+import { existsSync } from 'fs';
+import { unlinkGuarded } from '../../lib/fileUtils.js';
+import {
+ findFfmpeg, runFfmpegProcess, probeFrameCount, probeVideoStreamInfo,
+ hasAudioStream, bt709TagFilter, buildExtendSpliceArgs, installEncodedVideo,
+} from '../../lib/ffmpeg.js';
+import { planExtendSplice } from '../../lib/videoContinuity.js';
+
+/** Linear crossfade at the source/generated audio seam (~one 25 Hz audio token). */
+export const SEAM_CROSSFADE_SEC = 0.04;
+
+/**
+ * Replace the source region of `outputPath` with the original `sourcePath`.
+ *
+ * Resolves `{ applied: true, keptFrames, droppedSourceFrames }` once the
+ * spliced file is verified and installed over `outputPath`, or
+ * `{ applied: false, reason }` with the render left exactly as the model wrote it.
+ */
+export async function spliceExtendSource({ sourcePath, outputPath, extendLatents }) {
+ const tmpPath = `${outputPath}.splice.mp4`;
+ const skip = (reason) => {
+ console.warn(`⚠️ Extend source splice skipped, keeping the model output: ${reason}`);
+ return { applied: false, reason };
+ };
+ if (!sourcePath || !existsSync(sourcePath)) return skip('source file missing');
+ if (!outputPath || !existsSync(outputPath)) return skip('rendered file missing');
+ const ffmpeg = await findFfmpeg();
+ if (!ffmpeg) return skip('ffmpeg not found');
+
+ const [sourceFrames, outputFrames, outputInfo] = await Promise.all([
+ probeFrameCount(sourcePath), probeFrameCount(outputPath), probeVideoStreamInfo(outputPath),
+ ]);
+ const plan = planExtendSplice({ sourceFrames, outputFrames, extendLatents });
+ if (!plan) return skip(`frame layout does not match (source ${sourceFrames ?? '?'}, output ${outputFrames ?? '?'}, +${extendLatents} latents)`);
+ if (!outputInfo.width || !outputInfo.height || !outputInfo.fps) return skip('render geometry unreadable');
+
+ const [sourceHasAudio, outputHasAudio] = await Promise.all([hasAudioStream(sourcePath), hasAudioStream(outputPath)]);
+ const audio = !outputHasAudio ? 'none' : sourceHasAudio ? 'splice' : 'generated';
+ const args = buildExtendSpliceArgs({
+ sourcePath, generatedPath: outputPath, outPath: tmpPath,
+ keepFrames: plan.keepFrames, newStartFrame: plan.newStartFrame,
+ width: outputInfo.width, height: outputInfo.height, fps: outputInfo.fps,
+ audio, crossfadeSec: SEAM_CROSSFADE_SEC, colorTagFilter: await bt709TagFilter(),
+ });
+ if (!args) return skip('could not build the splice graph');
+
+ const encoded = await runFfmpegProcess({ bin: ffmpeg, args });
+ if (!encoded.ok) {
+ await unlinkGuarded(tmpPath).catch(() => {});
+ return skip(encoded.reason);
+ }
+ // Verify before replacing anything: the join must be exactly the render's
+ // length (kept source frames + new frames), or the seam is off by a frame.
+ const splicedFrames = await probeFrameCount(tmpPath);
+ if (splicedFrames !== outputFrames) {
+ await unlinkGuarded(tmpPath).catch(() => {});
+ return skip(`spliced clip has ${splicedFrames ?? '?'} frames, expected ${outputFrames}`);
+ }
+ const installed = await installEncodedVideo(tmpPath, outputPath, 'extend-spliced');
+ if (!installed.ok) return skip(installed.reason);
+ console.log(`🧵 Extend kept the original source: ${plan.keepFrames} source + ${plan.newFrames} new frames${audio === 'splice' ? ' (audio crossfaded at the seam)' : ''}`);
+ return { applied: true, keptFrames: plan.keepFrames, droppedSourceFrames: plan.droppedSourceFrames };
+}
diff --git a/server/services/videoGen/extendSplice.test.js b/server/services/videoGen/extendSplice.test.js
new file mode 100644
index 0000000000..be57c33402
--- /dev/null
+++ b/server/services/videoGen/extendSplice.test.js
@@ -0,0 +1,91 @@
+/**
+ * Native Extend keeps the ORIGINAL source in the delivered file (#11171).
+ *
+ * Runs real ffmpeg on tiny synthetic clips: a black "source" and a white
+ * "render" whose opening frames stand in for the model's re-decode of the
+ * source. The splice must put source pixels before the seam and render pixels
+ * after it, at the render's exact length, and must leave the render untouched
+ * whenever the layout doesn't verify.
+ */
+import { describe, it, expect, beforeAll, afterAll } from 'vitest';
+import { mkdtempSync, rmSync, existsSync, readFileSync } from 'fs';
+import { tmpdir } from 'os';
+import { join } from 'path';
+import { execFile } from 'child_process';
+import { promisify } from 'util';
+import { findFfmpeg, runFfmpegProcess, probeFrameCount, hasAudioStream } from '../../lib/ffmpeg.js';
+import { spliceExtendSource } from './extendSplice.js';
+
+const execFileAsync = promisify(execFile);
+
+let tmpDir;
+let ffmpeg;
+
+const makeClip = (name, { color, frames, tone }) => runFfmpegProcess({
+ bin: ffmpeg,
+ args: [
+ '-f', 'lavfi', '-i', `color=c=${color}:size=64x48:rate=24`,
+ '-f', 'lavfi', '-i', `sine=frequency=${tone}:sample_rate=48000`,
+ '-frames:v', String(frames), '-shortest',
+ '-c:v', 'libx264', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-y', join(tmpDir, name),
+ ],
+ stderrTailBytes: 0,
+}).then((r) => (r.ok ? join(tmpDir, name) : null));
+
+// Mean luma of frame `n`, 0–255.
+const frameLuma = async (path, n) => {
+ const { stdout } = await execFileAsync(ffmpeg, [
+ '-v', 'error', '-i', path, '-vf', `select=eq(n\\,${n})`, '-frames:v', '1',
+ '-f', 'rawvideo', '-pix_fmt', 'gray', '-',
+ ], { encoding: 'buffer', maxBuffer: 1 << 20 });
+ return stdout.reduce((sum, v) => sum + v, 0) / stdout.length;
+};
+
+beforeAll(async () => {
+ tmpDir = mkdtempSync(join(tmpdir(), 'portos-extend-splice-'));
+ ffmpeg = await findFfmpeg();
+});
+afterAll(() => rmSync(tmpDir, { recursive: true, force: true }));
+
+describe('spliceExtendSource', () => {
+ it('delivers source frames before the seam and the render after it, at the render length', async () => {
+ if (!ffmpeg) return;
+ // 17 source frames (1 + 8*2) extended by 2 latents → a 33-frame render.
+ const source = await makeClip('source.mp4', { color: 'black', frames: 17, tone: 440 });
+ const output = await makeClip('render.mp4', { color: 'white', frames: 33, tone: 880 });
+ if (!source || !output) return;
+
+ const result = await spliceExtendSource({ sourcePath: source, outputPath: output, extendLatents: 2 });
+
+ expect(result).toMatchObject({ applied: true, keptFrames: 17, droppedSourceFrames: 0 });
+ expect(await probeFrameCount(output)).toBe(33);
+ expect(await frameLuma(output, 0)).toBeLessThan(60); // source region: original black
+ expect(await frameLuma(output, 16)).toBeLessThan(60);
+ expect(await frameLuma(output, 17)).toBeGreaterThan(190); // first new frame: the render
+ expect(await frameLuma(output, 32)).toBeGreaterThan(190);
+ expect(await hasAudioStream(output)).toBe(true);
+ expect(existsSync(`${output}.splice.mp4`)).toBe(false);
+ });
+
+ it('keeps the render byte-for-byte when the frame layout does not match', async () => {
+ if (!ffmpeg) return;
+ // 40 source frames against a render that only echoes 17: the model truncated
+ // its context, so there is no seam to trust.
+ const source = await makeClip('long-source.mp4', { color: 'black', frames: 40, tone: 440 });
+ const output = await makeClip('short-render.mp4', { color: 'white', frames: 33, tone: 880 });
+ if (!source || !output) return;
+ const before = readFileSync(output);
+
+ const result = await spliceExtendSource({ sourcePath: source, outputPath: output, extendLatents: 2 });
+
+ expect(result.applied).toBe(false);
+ expect(readFileSync(output).equals(before)).toBe(true);
+ });
+
+ it('declines without touching anything when the source file is gone', async () => {
+ const result = await spliceExtendSource({
+ sourcePath: join(tmpDir, 'missing.mp4'), outputPath: join(tmpDir, 'render.mp4'), extendLatents: 2,
+ });
+ expect(result).toEqual({ applied: false, reason: 'source file missing' });
+ });
+});
diff --git a/server/services/videoGen/generateVideo.js b/server/services/videoGen/generateVideo.js
index f75534e511..4c0705b84a 100644
--- a/server/services/videoGen/generateVideo.js
+++ b/server/services/videoGen/generateVideo.js
@@ -77,6 +77,7 @@ import {
resolveVideoDraftDecoder,
} from '../../lib/videoDraftDecoders.js';
import { isDefaultVideoStreamingMode } from '../../lib/videoStreamingMode.js';
+import { extendLatentFrames, supportsContextWindow } from '../../lib/videoContinuity.js';
// Re-export the extracted runtime + history surface so existing deep imports
// (`from '../videoGen/local.js'`) keep resolving every symbol they used to.
export * from './runtimes.js';
@@ -735,6 +736,12 @@ export async function generateVideo({ pythonPath, prompt, negativePrompt = '', m
...(await getSettings())?.videoGen,
...(displaySleep != null ? { displaySleep } : {}),
},
+ // A plain native Extend delivers the original source, not the model's
+ // re-decode of it. Chain chunks (`hidden`) are skipped: the chain trims the
+ // echoed context itself, and a window hop's "source" is a temp tail clip.
+ extendSplice: mode === 'extend' && extendFromVideoPath && !hidden && !batch && supportsContextWindow(model)
+ ? { sourcePath: extendFromVideoPath, extendLatents: extendLatentFrames(parsedNumFrames) }
+ : null,
});
return { jobId, generationId: jobId, filename, mode: 'local', model: modelId };
diff --git a/server/services/videoGen/local.test.js b/server/services/videoGen/local.test.js
index 988bec7e88..c38bd322c7 100644
--- a/server/services/videoGen/local.test.js
+++ b/server/services/videoGen/local.test.js
@@ -300,6 +300,12 @@ vi.mock('../../lib/ffmpeg.js', async () => ({
// hfChildEnv() carries the resolved token over the inherited child env; mocking here
// avoids touching the real settings layer (which would await an unmocked
// `getSettings()` chain and hang the spawn-mock-driven tests).
+// The splice itself (real ffmpeg) is covered by extendSplice.test.js; here it is
+// a spy so the suite can pin WHICH renders ask for it.
+vi.mock('./extendSplice.js', () => ({
+ spliceExtendSource: vi.fn(async () => ({ applied: false, reason: 'mocked' })),
+}));
+
vi.mock('../hfToken.js', () => ({
hfChildEnv: vi.fn(async () => ({})),
getHfToken: vi.fn(async () => null),
@@ -6483,3 +6489,42 @@ describe('generateVideo — post-completion native teardown abort', () => {
expect(failed.mock.calls[0][0].results).toHaveLength(1);
});
});
+
+describe('generateVideo — native Extend keeps the original source (#11171)', () => {
+ const extendRender = async (jobId, request) => {
+ const { spliceExtendSource } = await import('./extendSplice.js');
+ vi.mocked(spliceExtendSource).mockClear();
+ await renderToTerminal({
+ jobId,
+ pythonPath: '/usr/bin/python3',
+ modelId: 'ltx2_unified',
+ prompt: 'a quiet street at dusk',
+ width: 512, height: 512, numFrames: 25, fps: 24,
+ ...request,
+ });
+ return vi.mocked(spliceExtendSource);
+ };
+
+ it('splices a plain extend of a source clip, sized to the extension the helper was asked for', async () => {
+ const splice = await extendRender('splice-plain-extend', { mode: 'extend', extendFromVideoPath: '/mock/source.mp4' });
+ // numFrames 25 → 3 latents, the same count --extend-frames carries.
+ expect(splice).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
+ sourcePath: '/mock/source.mp4', extendLatents: 3, outputPath: expect.stringContaining('splice-plain-extend'),
+ }));
+ });
+
+ it('leaves a chain chunk alone — the chain trims its own echoed context', async () => {
+ const splice = await extendRender('splice-chain-chunk', { mode: 'extend', extendFromVideoPath: '/mock/tail-window.mp4', hidden: true });
+ expect(splice).not.toHaveBeenCalled();
+ });
+
+ it('leaves a runtime whose extend is last-frame i2v alone — its output holds no source', async () => {
+ const splice = await extendRender('splice-mlx-video', { modelId: 'ltx23_unified', mode: 'extend', extendFromVideoPath: '/mock/source.mp4' });
+ expect(splice).not.toHaveBeenCalled();
+ });
+
+ it('does not splice a render with no source clip', async () => {
+ const splice = await extendRender('splice-text', { mode: 'text' });
+ expect(splice).not.toHaveBeenCalled();
+ });
+});
diff --git a/server/services/videoGen/spawnWatch.js b/server/services/videoGen/spawnWatch.js
index bb8f475d7a..e2f00dc4c5 100644
--- a/server/services/videoGen/spawnWatch.js
+++ b/server/services/videoGen/spawnWatch.js
@@ -40,6 +40,7 @@ import {
import { isDisplaySleepEnabled, sleepDisplayForVideo, wakeDisplayForVideo } from './displayPower.js';
import { loadHistory, mutateVideoHistory } from './history.js';
import { estimateRenderMs } from './eta.js';
+import { spliceExtendSource } from './extendSplice.js';
import { videoJobState } from './jobState.js';
const MODULE_NOT_FOUND_RE = /ModuleNotFoundError: No module named ['"]([^'"]+)['"]/;
@@ -72,6 +73,9 @@ export async function spawnAndWatchVideo({
numFrames: parsedNumFrames,
steps: actualSteps,
videoGenSettings,
+ // `{ sourcePath, extendLatents }` for a plain native Extend render, whose
+ // delivered file should keep the original source (see ./extendSplice.js).
+ extendSplice = null,
}) {
const retainOwnership = () => {
// Chained renders use child job IDs; their ambient owner is the outer job.
@@ -534,6 +538,10 @@ export async function spawnAndWatchVideo({
broadcastSse(job, { type: 'complete', result });
videoGenEvents.emit('completed', { generationId: jobId, ...result });
} else {
+ // Best-effort: a failed splice leaves the model's own output in place.
+ if (extendSplice && (await spliceExtendSource({ ...extendSplice, outputPath })).applied) {
+ meta.extendSourceOriginal = true;
+ }
await finalizeGeneratedVideo({ job, jobId, outputPath, filename, meta, actualSeed, mutateHistory: mutateVideoHistory, publication: publications.get(outputPath) });
}
}
diff --git a/server/services/videoGen/spawnWatch.test.js b/server/services/videoGen/spawnWatch.test.js
index d0379370a9..42036453c6 100644
--- a/server/services/videoGen/spawnWatch.test.js
+++ b/server/services/videoGen/spawnWatch.test.js
@@ -11,6 +11,7 @@ const mocks = vi.hoisted(() => ({
closeWatcher: vi.fn(),
rm: vi.fn(),
optimize: vi.fn(),
+ splice: vi.fn(),
thumbnail: vi.fn(),
history: [],
}));
@@ -46,6 +47,7 @@ vi.mock('./runtimes.js', () => ({
invalidateByovReadyCache: vi.fn(),
pickDeathFingerprint: async () => null,
}));
+vi.mock('./extendSplice.js', () => ({ spliceExtendSource: mocks.splice }));
vi.mock('./displayPower.js', () => ({
isDisplaySleepEnabled: () => false,
sleepDisplayForVideo: vi.fn(),
@@ -237,4 +239,35 @@ describe('spawnAndWatchVideo claim lifecycle', () => {
expect(completed).not.toHaveBeenCalled();
expect(mocks.history).toEqual([]);
});
+
+ describe('native Extend source splice', () => {
+ const extendSplice = { sourcePath: '/mock/videos/source.mp4', extendLatents: 3 };
+
+ it('splices the original source into the render and records that on the history row', async () => {
+ mocks.splice.mockResolvedValue({ applied: true });
+ await spawnAndWatchVideo({ ...params, extendSplice });
+ await closeChild(0);
+
+ expect(mocks.splice).toHaveBeenCalledExactlyOnceWith({ ...extendSplice, outputPath: params.outputPath });
+ expect(mocks.history).toEqual([expect.objectContaining({ id: params.jobId, extendSourceOriginal: true })]);
+ });
+
+ it('publishes the model output unflagged when the splice declines', async () => {
+ mocks.splice.mockResolvedValue({ applied: false, reason: 'frame layout does not match' });
+ await spawnAndWatchVideo({ ...params, extendSplice });
+ await closeChild(0);
+
+ expect(completed).toHaveBeenCalledTimes(1);
+ expect(mocks.history).toHaveLength(1);
+ expect(mocks.history[0]).not.toHaveProperty('extendSourceOriginal');
+ });
+
+ it('never touches a render that did not ask for it', async () => {
+ await spawnAndWatchVideo(params);
+ await closeChild(0);
+
+ expect(mocks.splice).not.toHaveBeenCalled();
+ expect(mocks.history[0]).not.toHaveProperty('extendSourceOriginal');
+ });
+ });
});