Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

feat: keep Extend's source frames bit-identical and its audio a single AAC generation (#11179)

Adam Eivy00b0cdd98b39
diff --git a/server/lib/README.md b/server/lib/README.md
index cfab10679e..d2efe4be85 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -431,7 +431,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `httpDrain.js` | `createHttpDrain(servers)` tracks every ordinary request the HTTP listeners accept; `begin(windowMs)` is the synchronous shutdown step that stops intake on all of them (`server.close()` plus a 503 `Connection: close` gate for requests arriving on still-open keep-alive sockets), turns off keep-alive on in-flight responses, drops open SSE streams, and resolves `{ remaining }` once accepted requests finish or the window ends (#8323). |
| `httpCompression.js` | `httpCompression` negotiates Brotli/gzip for API and static responses while preserving streaming, ranges, existing encodings, and `no-transform`. |
| `execGit.js` | `execGit(args, cwd, options)` and `execGitSafe(args, cwd, options)` shared process boundary for Git operations, worktrees, and `services/forgeAuth.js` credential resolution. The safe variant converts rejections into `{ exitCode: 1, stdout: '', stderr }`; `services/git.js` re-exports both for compatibility. `cwd` is REQUIRED — it rejects on a missing/blank one rather than letting `spawn` fall back to `process.cwd()` and run git against PortOS's own checkout (#4554). |
-| `ffmpeg.js` | Shared ffmpeg helpers (videoGen + videoTimeline). `probeVideoGeometry(videoPath)` measures a finished clip's `{ width, height, fps, numFrames, durationSec }` (null when any axis is unreadable) for hosted renders that record no frame count. `generateThumbnail(videoPath, jobId, { atSec } = {})` accepts a clamped poster seek while preserving the single-clip default. Includes `probeFrameCount(videoPath)` (metadata `nb_frames`, falling back to a real `-count_frames` pass — expensive, so call it once per file) and `trimVideoFromFrame(videoPath, outPath, {startFrame, fps})` — a frame-EXACT head cut via the `trim` filter (an `-ss` seek can drift a frame, which reads as a stutter at a stitch seam), keeping audio in sync with `atrim` when the clip has a soundtrack and taking `-an` when it doesn't. Re-encodes by necessity, so a later concat must re-encode too; `outPath` may equal the input (temp-file + rename install). Used by the chained-render context window — see `videoContinuity.js`. `buildTrimConcatArgs({inputs, outPath, width, height, fps, withAudio})` builds the argv for the other half of that job: a concat that drops leading frames from some of its inputs inside a `filter_complex` graph, so the cuts ride along with the timeline encode instead of costing one pre-encode per clip (pass `withAudio` only when EVERY input has an audio stream — check with `hasAudioStream`). `buildExtendSpliceArgs({sourcePath, generatedPath, outPath, keepFrames, newStartFrame, width, height, fps, audio})` is the plain-Extend counterpart: source frames `[0, keep)` + render frames from `newStartFrame`, one encode, audio crossfaded (~40 ms linear) at the seam — driven by `services/videoGen/extendSplice.js`. `H264_ENCODE_ARGS` / `AAC_ENCODE_ARGS` are the shared encode profile: clips produced by different paths here get concatenated together, so a mismatch shows up as one segment graded differently from its neighbours — spread these rather than re-typing the flags. Every re-encode here also pins BT.709: `BT709_CONTAINER_ARGS` (the `colr` atom, always emitted) plus `bt709TagFilter()` → the `BT709_TAG_FILTER` `setparams=…` string, or `null` on an ffmpeg without that filter (`supportsSetparamsFilter()` probes `-filters` once per process; `null` = not probed, and a probe that couldn't run stays uncached). Both halves are required — from ffmpeg 8 the encoder reads color properties off the FRAMES, silently overriding the container flags, so a flags-only output decodes washed-out. `buildTrimConcatArgs` takes the filter as its `colorTagFilter` option rather than probing, to stay pure. `installEncodedVideo(tmpPath, targetPath, label)` is the shared in-place install: move the original aside to a `.bak` on Windows (where `fs.rename` refuses an existing destination), rename the new file over it, and restore the backup — plus unlink the temp — on any failure, returning `{ok:true,outPath}` / `{ok:false,reason}` so the worst case is "skipped", never "the video is gone". Use it for EVERY replace-a-video-with-its-re-encode path; `pipeline/audioMux.js` is the out-of-module caller. `edgeFadeFilter(durationSec)` is the `-af` fade-in/fade-out fragment a mid-song clip (a social cut) appends to its trim. Loudness mastering (#10249): `MASTER_LOUDNESS` (-14 LUFS / -1.5 dBTP / LRA 11), `planLoudnessMaster` (measure then build the linear apply filter; throws on a silent source), `reportMasteredLoudness` (post-encode before/after numbers). |
+| `ffmpeg.js` | Shared ffmpeg helpers (videoGen + videoTimeline). `probeVideoGeometry(videoPath)` measures a finished clip's `{ width, height, fps, numFrames, durationSec }` (null when any axis is unreadable) for hosted renders that record no frame count. `generateThumbnail(videoPath, jobId, { atSec } = {})` accepts a clamped poster seek while preserving the single-clip default. Includes `probeFrameCount(videoPath)` (metadata `nb_frames`, falling back to a real `-count_frames` pass — expensive, so call it once per file) and `trimVideoFromFrame(videoPath, outPath, {startFrame, fps})` — a frame-EXACT head cut via the `trim` filter (an `-ss` seek can drift a frame, which reads as a stutter at a stitch seam), keeping audio in sync with `atrim` when the clip has a soundtrack and taking `-an` when it doesn't. Re-encodes by necessity, so a later concat must re-encode too; `outPath` may equal the input (temp-file + rename install). Used by the chained-render context window — see `videoContinuity.js`. `buildTrimConcatArgs({inputs, outPath, width, height, fps, withAudio})` builds the argv for the other half of that job: a concat that drops leading frames from some of its inputs inside a `filter_complex` graph, so the cuts ride along with the timeline encode instead of costing one pre-encode per clip (pass `withAudio` only when EVERY input has an audio stream — check with `hasAudioStream`). `buildExtendSpliceArgs({sourcePath, generatedPath, outPath, keepFrames, newStartFrame, width, height, fps, audio, sourceAudioPath, masterOutPath})` is the plain-Extend counterpart: source frames `[0, keep)` + render frames from `newStartFrame`, one encode, audio crossfaded (~40 ms linear) at the seam; `sourceAudioPath` swaps the source span's audio for a lossless master and `masterOutPath` also writes the mix as a 24-bit FLAC master, so the delivered AAC is encoded once from PCM. Its stream-copy sibling is `buildExtendNewSpanArgs` (encode only the render's new span) + `buildExtendConcatList` + `buildExtendCopyArgs` (join it to the untouched source with `-c:v copy`); `probeVideoCodecSignature` gates it and `videoFrameHashes` (per-frame MD5) proves the join before anything is replaced — driven by `services/videoGen/extendSplice.js`. `H264_ENCODE_ARGS` / `AAC_ENCODE_ARGS` are the shared encode profile: clips produced by different paths here get concatenated together, so a mismatch shows up as one segment graded differently from its neighbours — spread these rather than re-typing the flags. Every re-encode here also pins BT.709: `BT709_CONTAINER_ARGS` (the `colr` atom, always emitted) plus `bt709TagFilter()` → the `BT709_TAG_FILTER` `setparams=…` string, or `null` on an ffmpeg without that filter (`supportsSetparamsFilter()` probes `-filters` once per process; `null` = not probed, and a probe that couldn't run stays uncached). Both halves are required — from ffmpeg 8 the encoder reads color properties off the FRAMES, silently overriding the container flags, so a flags-only output decodes washed-out. `buildTrimConcatArgs` takes the filter as its `colorTagFilter` option rather than probing, to stay pure. `installEncodedVideo(tmpPath, targetPath, label)` is the shared in-place install: move the original aside to a `.bak` on Windows (where `fs.rename` refuses an existing destination), rename the new file over it, and restore the backup — plus unlink the temp — on any failure, returning `{ok:true,outPath}` / `{ok:false,reason}` so the worst case is "skipped", never "the video is gone". Use it for EVERY replace-a-video-with-its-re-encode path; `pipeline/audioMux.js` is the out-of-module caller. `edgeFadeFilter(durationSec)` is the `-af` fade-in/fade-out fragment a mid-song clip (a social cut) appends to its trim. Loudness mastering (#10249): `MASTER_LOUDNESS` (-14 LUFS / -1.5 dBTP / LRA 11), `planLoudnessMaster` (measure then build the linear apply filter; throws on a silent source), `reportMasteredLoudness` (post-encode before/after numbers). |
| `forgeAccessErrors.js` | `isForgeNoAccessError(err)` / `isForgeNoAccessMessage(text)` / `logForgeNoAccessOnce(scope, repoSpec, detail)` / `forgeNoAccessRemedy(repoSpec)` — tells a PERMANENT `gh` no-access 404 (`Could not resolve to a Repository`, `HTTP 404`) apart from a transient blip. `execGh` stamps `ghNoAccess` on the rejection and keeps it OUT of the consecutive-failure backoff (which caps at 15 min and then re-fires forever), and the polling jobs (`branchReconcile`, `prWatcher`) narrate it exactly once per scope+repo per process with a remedy line instead of once per tick. `__resetForgeNoAccessLog()` is the test seam. |
| `forkHead.js` | `normalizeForkHead(value)` / `forkRemoteName(ownerLogin)` / `resolveTaskForkHead(metadata)` — the validated `{remoteUrl, ownerLogin}` address of a cross-repository PR's head branch, which has no `origin/<branch>` to attach to. Fail-closed: anything that isn't a usable pair (or that starts with `-`, which git would read as an option) is `null`, so every caller falls back to origin-only behavior instead of guessing. `forkRemoteName` is the deterministic `fork-<owner>` remote name that makes a re-attach reuse the remote it already added. Shared by `appPullRequests.js` (produces), `agentWorktreeCleanup.js`/`prRemediationFollowUp.js` (persist in task metadata), `agentWorkspacePrep.js` (threads), and `worktreeManager.js` (consumes — it has no forge client of its own). `resolveTaskForkHead` is the single reader of the generic `forkHead` task-metadata key, the fork-side counterpart to `taskTargetBranch.js`. |
| `frameQuality.js` | Scores decoded video frames so a chained render can CHOOSE its continuation anchor instead of taking whatever an end-seek lands on. `scoreFrame({width,height,data},{recency})` combines three terms over a raw single-channel buffer: log-compressed `gradientVariance` (focus), distance of `meanLuma` from mid-grey (exposure penalty, which rejects a fade-to-black or a blown highlight), and a small linear recency bonus that keeps the anchor near the cut when candidates are comparably clean. Usability and ranking are deliberately separate: `usable` gates on raw gradient variance against `MIN_SIGNAL_VARIANCE` alone, so only a mathematically degenerate frame is rejected and a legitimately dark low-key tail still qualifies — the same contract `imageFrameStats.js` keeps. Gating on the composite score instead would reject every night scene on the exposure term. `TAIL_WINDOW_SECONDS` is capped at the reach of the single `-sseof -1.0` seek this replaces: focus is systematically highest at the OLDEST candidate, so a wider window would widen the backward jump at each chain seam rather than tightening it. `pickBestFrame(paths)` decodes each candidate through `sharp` and returns the winner (or `null` when nothing decodes or every candidate is degenerate, so the caller degrades rather than failing a render); its `index` is the position in the array AS PASSED, because the caller turns that into the anchor's time offset. `MAX_CANDIDATES` is derived from the window, never set independently — the ffmpeg-side `-frames:v` cap truncates from the NEWEST end, exactly the frames recency prefers. `TAIL_WINDOW_SECONDS` / `CANDIDATE_FPS` / `MAX_CANDIDATES` define the window the caller decodes from. Consumer: `services/videoGen/local.js#extractLastFrame`. |
diff --git a/server/lib/ffmpeg.js b/server/lib/ffmpeg.js
index 05104a2068..1062bef158 100644
--- a/server/lib/ffmpeg.js
+++ b/server/lib/ffmpeg.js
@@ -8,7 +8,8 @@
import { execFile, spawn } from './childProcess.js';
import { existsSync, statSync } from 'fs';
-import { unlink, rename } from 'fs/promises';
+import { unlink, rename, readFile } from 'fs/promises';
+import { tmpdir } from 'os';
import { join, resolve as resolvePath, sep as PATH_SEP, dirname } from 'path';
import { randomUUID } from 'crypto';
import { promisify } from 'util';
@@ -305,6 +306,48 @@ export const probeVideoStreamInfo = async (videoPath) => {
};
};
+// What a stream copy needs two clips to share before their video can be joined
+// without a decode: codec, profile/level, pixel format and the colour tags the
+// decoder reads off the stream. Joined into one comparable string, or null when
+// the probe could not read the stream. Equal strings are necessary, not
+// sufficient — callers still verify the joined result.
+export const probeVideoCodecSignature = async (videoPath) => {
+ if (typeof videoPath !== 'string' || !videoPath) return null;
+ const stdout = await runFfprobe([
+ '-v', 'error',
+ '-select_streams', 'v:0',
+ '-show_entries', 'stream=codec_name,profile,level,pix_fmt,color_range,color_space,color_transfer,color_primaries,field_order',
+ '-of', 'default=noprint_wrappers=1',
+ videoPath,
+ ]);
+ const lines = stdout.split(/\r?\n/).filter(Boolean).sort();
+ return lines.some((line) => line.startsWith('codec_name=')) ? lines.join('|') : null;
+};
+
+// Per-frame MD5 of a clip's first video stream, in decode-output order — the
+// ground truth for "these frames are bit-identical". Null when the clip cannot
+// be fully decoded (any decode error fails the run), so a caller can never read
+// a broken join as a matching one.
+export const videoFrameHashes = async (videoPath) => {
+ const ffmpeg = await findFfmpeg();
+ if (!ffmpeg || typeof videoPath !== 'string' || !videoPath) return null;
+ const hashPath = join(tmpdir(), `framemd5-${randomUUID()}.txt`);
+ try {
+ const run = await runFfmpegProcess({
+ bin: ffmpeg,
+ args: ['-v', 'error', '-xerror', '-i', videoPath, '-map', '0:v:0', '-f', 'framemd5', '-y', hashPath],
+ });
+ if (!run.ok) return null;
+ const rows = (await readFile(hashPath, 'utf8')).split(/\r?\n/).filter((line) => line && !line.startsWith('#'));
+ const hashes = rows.map((row) => row.split(',').pop().trim());
+ return hashes.length ? hashes : null;
+ } catch {
+ return null;
+ } finally {
+ await unlink(hashPath).catch(() => {});
+ }
+};
+
// The playable geometry of a finished clip — `{ width, height, fps, numFrames,
// durationSec }` — or null when any axis cannot be measured. For entries whose
// producer never recorded a frame count (hosted Grok/fal/reactor renders), so a
@@ -784,6 +827,67 @@ export const buildTrimConcatArgs = ({ inputs, outPath, width, height, fps, withA
];
};
+// Lossless audio master kept beside an Extend output (see
+// `services/videoGen/extendSplice.js`): FLAC at 24-bit, 48 kHz, so the next
+// round's source span is decoded from PCM instead of from a prior AAC encode.
+const MASTER_AUDIO_ARGS = Object.freeze(['-c:a', 'flac', '-sample_fmt', 's32']);
+
+// The audio half of an Extend splice, shared by the re-encode and stream-copy
+// builders. `sourceIdx` is the input carrying the source span's audio (the
+// source clip, or its lossless master), `generatedIdx` the render's.
+//
+// The source span comes from `sourceIdx`, the new span from the render, joined by
+// a short LINEAR crossfade (`crossfadeSec`) so the seam doesn't click. The source
+// leg runs `crossfadeSec` past the seam so the overlap is source-vs-generated at
+// the SAME timeline position, which keeps the output exactly as long as the render.
+// Returns the filter chains plus whether a `[outm]` master pad was emitted.
+const extendSpliceAudioFilters = ({ audio, sourceIdx, generatedIdx, keep, rate, crossfadeSec, withMaster }) => {
+ if (audio !== 'splice' && audio !== 'generated') return [];
+ const canonAudio = 'aresample=48000,aformat=sample_fmts=fltp:channel_layouts=stereo';
+ const seamSec = keep / rate;
+ const fade = Math.max(0.001, Number(crossfadeSec) || 0);
+ const mixed = audio === 'splice'
+ ? [
+ // apad guards a source track that ends before the seam + overlap, which
+ // acrossfade would otherwise reject as shorter than the fade.
+ `[${sourceIdx}:a]atrim=end=${(seamSec + fade).toFixed(6)},asetpts=PTS-STARTPTS,${canonAudio},apad=whole_dur=${(seamSec + fade).toFixed(6)}[a0]`,
+ `[${generatedIdx}:a]atrim=start=${seamSec.toFixed(6)},asetpts=PTS-STARTPTS,${canonAudio}[a1]`,
+ `[a0][a1]acrossfade=d=${fade.toFixed(6)}:c1=tri:c2=tri`,
+ ]
+ : [`[${generatedIdx}:a]asetpts=PTS-STARTPTS,${canonAudio}`];
+ // One decoded mix feeds both outputs, so the delivered AAC is encoded once
+ // from the same samples the master keeps.
+ const last = `${mixed.pop()}${withMaster ? ',asplit=2[outa][outm]' : '[outa]'}`;
+ return [...mixed, last];
+};
+
+const validExtendSplice = ({ keepFrames, newStartFrame, fps, width, height }) => {
+ const keep = Math.floor(Number(keepFrames));
+ const start = Math.floor(Number(newStartFrame));
+ const rate = Number(fps);
+ const canonW = Number(width);
+ const canonH = Number(height);
+ const ok = Number.isFinite(keep) && keep > 0 && Number.isFinite(start) && start > 0
+ && Number.isFinite(rate) && rate > 0 && Number.isFinite(canonW) && canonW > 0
+ && Number.isFinite(canonH) && canonH > 0;
+ return ok ? { keep, start, rate, canonW, canonH } : null;
+};
+
+const extendNormalizeFilter = ({ canonW, canonH, rate }) =>
+ `scale=${canonW}:${canonH}:force_original_aspect_ratio=decrease,pad=${canonW}:${canonH}:(ow-iw)/2:(oh-ih)/2,setsar=1,fps=${rate}`;
+
+// The two outputs every Extend builder shares: the delivered mp4 (video codec
+// args vary by builder) and, when `masterOutPath` is given, the lossless master.
+const extendOutputs = ({ audio, videoLabel, videoCodecArgs, outPath, masterOutPath }) => [
+ '-map', videoLabel,
+ ...(audio === 'none' ? [] : ['-map', '[outa]']),
+ ...videoCodecArgs,
+ ...(audio === 'none' ? ['-an'] : AAC_ENCODE_ARGS),
+ '-movflags', '+faststart',
+ '-y', outPath,
+ ...(masterOutPath && audio !== 'none' ? ['-map', '[outm]', ...MASTER_AUDIO_ARGS, '-y', masterOutPath] : []),
+];
+
// Build the argv that joins the ORIGINAL source of a native extend render to the
// model's new frames: input 0 is the source clip (kept for frames
// `[0, keepFrames)`), input 1 is the render (kept from `newStartFrame` on).
@@ -792,67 +896,107 @@ export const buildTrimConcatArgs = ({ inputs, outPath, width, height, fps, withA
//
// Video is one single encode of source + new — never a per-round compounding of
// the model's own decode. Geometry and rate normalize to the RENDER's, which is
-// what the extension was generated at. Audio follows the same split: the source
-// span from the source's track, the new span from the generated one, joined by a
-// short LINEAR crossfade (`crossfadeSec`) so the seam doesn't click. The source
-// leg runs `crossfadeSec` past the seam so the overlap is source-vs-generated at
-// the SAME timeline position, which keeps the output exactly as long as the render.
+// what the extension was generated at. Audio follows the same split (see
+// `extendSpliceAudioFilters`).
//
// `audio` is 'splice' (both inputs have a track), 'generated' (only the render
// has one — the source region has nothing to keep, so the render's track stands)
-// or 'none'. Pure, like buildTrimConcatArgs: the caller probes and passes
-// `colorTagFilter` from `bt709TagFilter()`.
+// or 'none'. `sourceAudioPath` (optional) is a lossless master standing in for
+// the source clip's own track as the source span's audio; `masterOutPath`
+// (optional, must end in an audio extension like `.flac`) also writes this
+// output's mix as a lossless master. Pure, like buildTrimConcatArgs: the caller
+// probes and passes `colorTagFilter` from `bt709TagFilter()`.
export const buildExtendSpliceArgs = ({
sourcePath, generatedPath, outPath, keepFrames, newStartFrame, width, height, fps,
- audio = 'none', crossfadeSec = 0.04, colorTagFilter = null,
+ audio = 'none', crossfadeSec = 0.04, colorTagFilter = null, sourceAudioPath = null, masterOutPath = null,
}) => {
- const keep = Math.floor(Number(keepFrames));
- const start = Math.floor(Number(newStartFrame));
- const rate = Number(fps);
- const canonW = Number(width);
- const canonH = Number(height);
if (!sourcePath || !generatedPath || !outPath) return null;
- if (!Number.isFinite(keep) || keep <= 0 || !Number.isFinite(start) || start <= 0) return null;
- if (!Number.isFinite(rate) || rate <= 0) return null;
- if (!Number.isFinite(canonW) || canonW <= 0 || !Number.isFinite(canonH) || canonH <= 0) return null;
+ const geometry = validExtendSplice({ keepFrames, newStartFrame, fps, width, height });
+ if (!geometry) return null;
+ const { keep, start, rate } = geometry;
- const normalize = `scale=${canonW}:${canonH}:force_original_aspect_ratio=decrease,pad=${canonW}:${canonH}:(ow-iw)/2:(oh-ih)/2,setsar=1,fps=${rate}`;
+ const normalize = extendNormalizeFilter(geometry);
const tag = typeof colorTagFilter === 'string' && colorTagFilter ? colorTagFilter : null;
+ const withMaster = !!masterOutPath && audio !== 'none';
const filters = [
`[0:v]trim=end_frame=${keep},setpts=PTS-STARTPTS,${normalize}[v0]`,
`[1:v]trim=start_frame=${start},setpts=PTS-STARTPTS,${normalize}[v1]`,
`[v0][v1]concat=n=2:v=1:a=0${tag ? '[cv]' : '[outv]'}`,
...(tag ? [`[cv]${tag}[outv]`] : []),
+ ...extendSpliceAudioFilters({ audio, sourceIdx: sourceAudioPath ? 2 : 0, generatedIdx: 1, keep, rate, crossfadeSec, withMaster }),
];
- const canonAudio = 'aresample=48000,aformat=sample_fmts=fltp:channel_layouts=stereo';
- const seamSec = keep / rate;
- const fade = Math.max(0.001, Number(crossfadeSec) || 0);
- if (audio === 'splice') {
- filters.push(
- // apad guards a source track that ends before the seam + overlap, which
- // acrossfade would otherwise reject as shorter than the fade.
- `[0:a]atrim=end=${(seamSec + fade).toFixed(6)},asetpts=PTS-STARTPTS,${canonAudio},apad=whole_dur=${(seamSec + fade).toFixed(6)}[a0]`,
- `[1:a]atrim=start=${seamSec.toFixed(6)},asetpts=PTS-STARTPTS,${canonAudio}[a1]`,
- `[a0][a1]acrossfade=d=${fade.toFixed(6)}:c1=tri:c2=tri[outa]`,
- );
- } else if (audio === 'generated') {
- filters.push(`[1:a]asetpts=PTS-STARTPTS,${canonAudio}[outa]`);
- }
-
return [
'-i', sourcePath, '-i', generatedPath,
+ ...(sourceAudioPath && audio === 'splice' ? ['-i', sourceAudioPath] : []),
'-filter_complex', filters.join(';'),
+ ...extendOutputs({
+ audio, videoLabel: '[outv]', outPath, masterOutPath,
+ videoCodecArgs: [...H264_ENCODE_ARGS, ...BT709_CONTAINER_ARGS],
+ }),
+ ];
+};
+
+// Stream-copy counterpart of buildExtendSpliceArgs, in two steps so the source's
+// frames stay bit-identical (the re-encode path rewrites them every round).
+//
+// 1. `buildExtendNewSpanArgs` encodes ONLY the render's new span (frames from
+// `newStartFrame`) as a silent mp4 with the shared H.264 profile — a fresh
+// GOP, so it starts on a keyframe by construction.
+// 2. `buildExtendCopyArgs` joins the source (copied, never decoded) to that span
+// through the concat demuxer's `listPath`, pinning the source entry to
+// `keepFrames / fps` so the seam lands on the frame boundary, not on the
+// source file's (audio-padded) container duration. Audio is the same
+// crossfaded mix as the re-encode path.
+//
+// The caller proves the result (frame count and per-frame hashes) before it
+// replaces anything — stream copy is only valid when both halves carry
+// compatible codec parameters, which the encode cannot promise on its own.
+export const buildExtendNewSpanArgs = ({ generatedPath, outPath, newStartFrame, width, height, fps, colorTagFilter = null }) => {
+ if (!generatedPath || !outPath) return null;
+ const geometry = validExtendSplice({ keepFrames: 1, newStartFrame, fps, width, height });
+ if (!geometry) return null;
+ const tag = typeof colorTagFilter === 'string' && colorTagFilter ? colorTagFilter : null;
+ return [
+ '-i', generatedPath,
+ '-filter_complex', `[0:v]trim=start_frame=${geometry.start},setpts=PTS-STARTPTS,${extendNormalizeFilter(geometry)}${tag ? `,${tag}` : ''}[outv]`,
'-map', '[outv]',
- ...(audio === 'none' ? [] : ['-map', '[outa]']),
- ...H264_ENCODE_ARGS,
- ...BT709_CONTAINER_ARGS,
- ...(audio === 'none' ? ['-an'] : AAC_ENCODE_ARGS),
+ ...H264_ENCODE_ARGS, ...BT709_CONTAINER_ARGS, '-an',
'-movflags', '+faststart',
'-y', outPath,
];
};
+// One concat-demuxer list entry; single quotes inside a path are escaped per the
+// demuxer's own quoting rules.
+const concatEntry = (path) => `file '${String(path).replace(/'/g, "'\\''")}'`;
+
+/** Body of the concat-demuxer list for `buildExtendCopyArgs`. */
+export const buildExtendConcatList = ({ sourcePath, newSpanPath, keepFrames, fps }) =>
+ `${[concatEntry(sourcePath), `duration ${(Number(keepFrames) / Number(fps)).toFixed(6)}`, concatEntry(newSpanPath)].join('\n')}\n`;
+
+export const buildExtendCopyArgs = ({
+ listPath, generatedPath, outPath, keepFrames, fps, audio = 'none', crossfadeSec = 0.04,
+ sourceAudioPath = null, masterOutPath = null,
+}) => {
+ const keep = Math.floor(Number(keepFrames));
+ const rate = Number(fps);
+ if (!listPath || !generatedPath || !outPath) return null;
+ if (!Number.isFinite(keep) || keep <= 0 || !Number.isFinite(rate) || rate <= 0) return null;
+ // The concat input only carries the source's VIDEO, so a spliced mix names the
+ // clip (or its master) that supplies the source span's audio explicitly.
+ if (audio === 'splice' && !sourceAudioPath) return null;
+ const withMaster = !!masterOutPath && audio !== 'none';
+ const filters = extendSpliceAudioFilters({ audio, sourceIdx: 2, generatedIdx: 1, keep, rate, crossfadeSec, withMaster });
+ return [
+ '-f', 'concat', '-safe', '0', '-i', listPath,
+ '-i', generatedPath,
+ ...(audio === 'splice' ? ['-i', sourceAudioPath] : []),
+ ...(filters.length ? ['-filter_complex', filters.join(';')] : []),
+ ...extendOutputs({ audio, videoLabel: '0:v:0', outPath, masterOutPath, videoCodecArgs: ['-c:v', 'copy'] }),
+ ];
+};
+
// 2× Lanczos upscale of an MP4 in place. Doubles width and height while
// preserving the exact aspect ratio and the audio track. Used as a quick
// post-render export option for LTX renders that come out at sub-720p
diff --git a/server/services/videoGen/chainedVideo.js b/server/services/videoGen/chainedVideo.js
index 1848b8be36..ba6989ca6b 100644
--- a/server/services/videoGen/chainedVideo.js
+++ b/server/services/videoGen/chainedVideo.js
@@ -275,6 +275,9 @@ export async function generateChainedVideo({ chunks, chunkPrompts, contextFrames
uploadedTempPath: i === 0 ? rest.uploadedTempPath : null,
uploadedTempPaths: i === 0 ? (rest.uploadedTempPaths || []) : [],
hidden: true,
+ // Chunk 0 of an extend chain consumes the user's own clip, so its render
+ // keeps that clip instead of the model's decode of it (#11179).
+ keepExtendSource: i === 0 && firstMode === 'extend',
mode: isWindowHop ? 'extend' : (i === 0 ? firstMode : 'image'),
// After the first chunk, drop FFLF-style last image — chained
// continuation conditions on one thing, the tail of the chunk before it,
diff --git a/server/services/videoGen/deleteHistoryItem.test.js b/server/services/videoGen/deleteHistoryItem.test.js
index bea4e31ea3..36857729f8 100644
--- a/server/services/videoGen/deleteHistoryItem.test.js
+++ b/server/services/videoGen/deleteHistoryItem.test.js
@@ -94,6 +94,14 @@ describe('deleteHistoryItem → media asset index delete hook', () => {
expect(unindexVideo).not.toHaveBeenCalledWith('job-1.mp4');
});
+ it('removes the Extend audio master kept beside the clip', async () => {
+ await deleteHistoryItem('job-1');
+
+ const unlinked = unlink.mock.calls.map(([path]) => path);
+ expect(unlinked).toContain(join(MOCK_PATHS.videos, 'job-1.mp4'));
+ expect(unlinked).toContain(join(MOCK_PATHS.videos, 'job-1.master.flac'));
+ });
+
it('is non-fatal: a failing index removal still removes the entry and returns ok', async () => {
const errSpy = vi.spyOn(console, 'error').mockImplementation(() => {});
unindexVideo.mockRejectedValueOnce(new Error('db down'));
diff --git a/server/services/videoGen/extendMaster.js b/server/services/videoGen/extendMaster.js
new file mode 100644
index 0000000000..da452836d7
--- /dev/null
+++ b/server/services/videoGen/extendMaster.js
@@ -0,0 +1,10 @@
+/**
+ * Sidecar naming for the lossless audio master kept beside an Extend output.
+ *
+ * Its own import-free module: the history delete path and the unpublished-output
+ * discard both need the name, and neither should pull in the splice's ffmpeg graph.
+ */
+
+/** `<id>.mp4` → `<id>.master.flac`, in the same directory. Null for a non-mp4 path. */
+export const extendMasterPath = (videoPath) =>
+ typeof videoPath === 'string' && /\.mp4$/i.test(videoPath) ? videoPath.replace(/\.mp4$/i, '.master.flac') : null;
diff --git a/server/services/videoGen/extendSplice.js b/server/services/videoGen/extendSplice.js
index 68af452d61..3299ad2700 100644
--- a/server/services/videoGen/extendSplice.js
+++ b/server/services/videoGen/extendSplice.js
@@ -15,26 +15,120 @@
*/
import { existsSync } from 'fs';
+import { rename, writeFile } from 'fs/promises';
import { unlinkGuarded } from '../../lib/fileUtils.js';
import {
- findFfmpeg, runFfmpegProcess, probeFrameCount, probeVideoStreamInfo,
- hasAudioStream, bt709TagFilter, buildExtendSpliceArgs, installEncodedVideo,
+ findFfmpeg, runFfmpegProcess, probeFrameCount, probeVideoStreamInfo, probeVideoDuration,
+ probeVideoCodecSignature, videoFrameHashes,
+ hasAudioStream, bt709TagFilter, buildExtendSpliceArgs, buildExtendNewSpanArgs, buildExtendCopyArgs,
+ buildExtendConcatList, installEncodedVideo,
} from '../../lib/ffmpeg.js';
import { planExtendSplice } from '../../lib/videoContinuity.js';
+import { extendMasterPath } from './extendMaster.js';
/** Linear crossfade at the source/generated audio seam (~one 25 Hz audio token). */
export const SEAM_CROSSFADE_SEC = 0.04;
+/** How far a master's length may sit from its clip's before it is treated as stale (AAC priming/padding is ~tens of ms). */
+const MASTER_DURATION_TOLERANCE_SEC = 0.15;
+
+const partPath = (path) => path.replace(/\.flac$/i, '.part.flac');
+
+/**
+ * The lossless master to read the SOURCE span's audio from, or null to use the
+ * source clip's own track. A master only counts while its length still matches
+ * the clip beside it: a clip whose audio was later trimmed, replaced or
+ * re-muxed in place leaves a master that no longer describes it.
+ */
+async function usableSourceMaster(sourcePath) {
+ const masterPath = extendMasterPath(sourcePath);
+ if (!masterPath || !existsSync(masterPath)) return null;
+ const [masterSec, sourceSec] = await Promise.all([probeVideoDuration(masterPath), probeVideoDuration(sourcePath)]);
+ if (!masterSec || !sourceSec || Math.abs(masterSec - sourceSec) > MASTER_DURATION_TOLERANCE_SEC) {
+ console.warn('⚠️ Extend audio master does not match its clip, using the clip\'s own track');
+ return null;
+ }
+ return masterPath;
+}
+
+/**
+ * Join the untouched source video to the render's new frames with `-c:v copy`.
+ * Only the new span is encoded (a fresh GOP in the shared H.264 profile); the
+ * source is never decoded. Returns `{ ok: true }` once `tmpPath` holds a result
+ * whose frames are proven to be the source's exact frames followed by the new
+ * span's, or `{ ok: false, reason }` for the caller to fall back to a re-encode.
+ */
+async function copyJoin({ ffmpeg, plan, sourcePath, outputPath, tmpPath, geometry, audio, sourceAudioPath, masterTmp, colorTagFilter }) {
+ const spanPath = `${outputPath}.span.mp4`;
+ const listPath = `${outputPath}.concat.txt`;
+ const cleanup = () => Promise.all([spanPath, listPath].map((path) => unlinkGuarded(path).catch(() => {})));
+ try {
+ // A source cut mid-GOP can't be copied, and a source that is already longer
+ // than the kept span would need a keyframe-aligned cut; both take the re-encode.
+ if (plan.droppedSourceFrames !== 0) return { ok: false, reason: 'source span is not the whole source' };
+ const [sourceSig, outputSig] = await Promise.all([probeVideoCodecSignature(sourcePath), probeVideoCodecSignature(outputPath)]);
+ if (!sourceSig || sourceSig !== outputSig) return { ok: false, reason: 'source and render codec parameters differ' };
+ const sourceInfo = await probeVideoStreamInfo(sourcePath);
+ if (sourceInfo.width !== geometry.width || sourceInfo.height !== geometry.height
+ || Math.abs((sourceInfo.fps || 0) - geometry.fps) > 0.001) {
+ return { ok: false, reason: 'source and render canvas or frame rate differ' };
+ }
+
+ const spanArgs = buildExtendNewSpanArgs({
+ generatedPath: outputPath, outPath: spanPath, newStartFrame: plan.newStartFrame, ...geometry, colorTagFilter,
+ });
+ if (!spanArgs) return { ok: false, reason: 'could not build the new-span encode' };
+ const span = await runFfmpegProcess({ bin: ffmpeg, args: spanArgs });
+ if (!span.ok) return { ok: false, reason: span.reason };
+
+ await writeFile(listPath, buildExtendConcatList({
+ sourcePath, newSpanPath: spanPath, keepFrames: plan.keepFrames, fps: geometry.fps,
+ }));
+ const joinArgs = buildExtendCopyArgs({
+ listPath, generatedPath: outputPath, outPath: tmpPath, keepFrames: plan.keepFrames, fps: geometry.fps,
+ audio, crossfadeSec: SEAM_CROSSFADE_SEC, sourceAudioPath, masterOutPath: masterTmp,
+ });
+ if (!joinArgs) return { ok: false, reason: 'could not build the stream-copy join' };
+ const joined = await runFfmpegProcess({ bin: ffmpeg, args: joinArgs });
+ if (!joined.ok) return { ok: false, reason: joined.reason };
+
+ // Stream copy is only as good as the parameter sets the two halves share:
+ // prove every frame decodes to exactly what each half decodes to alone.
+ const [sourceHashes, spanHashes, joinedHashes] = await Promise.all([
+ videoFrameHashes(sourcePath), videoFrameHashes(spanPath), videoFrameHashes(tmpPath),
+ ]);
+ if (!sourceHashes || !spanHashes || !joinedHashes) return { ok: false, reason: 'stream-copy join did not decode cleanly' };
+ const expected = [...sourceHashes, ...spanHashes];
+ if (joinedHashes.length !== expected.length || joinedHashes.some((hash, i) => hash !== expected[i])) {
+ return { ok: false, reason: 'stream-copy join decodes differently from its parts' };
+ }
+ return { ok: true };
+ } finally {
+ await cleanup();
+ }
+}
+
/**
* Replace the source region of `outputPath` with the original `sourcePath`.
*
- * Resolves `{ applied: true, keptFrames, droppedSourceFrames }` once the
- * spliced file is verified and installed over `outputPath`, or
+ * Video: the source's frames are stream-copied (bit-identical) when it shares
+ * codec parameters, canvas and rate with the render and the join verifies;
+ * otherwise source + new are encoded once. Audio: the source span is read from
+ * the source's lossless master when it has one, the delivered AAC is encoded once
+ * from the mix, and (unless `keepMaster` is false) the mix is kept as this
+ * output's own master (`<id>.master.flac`) for the next round.
+ *
+ * Resolves `{ applied: true, keptFrames, droppedSourceFrames, streamCopied }` once
+ * the spliced file is verified and installed over `outputPath`, or
* `{ applied: false, reason }` with the render left exactly as the model wrote it.
*/
-export async function spliceExtendSource({ sourcePath, outputPath, extendLatents }) {
+export async function spliceExtendSource({ sourcePath, outputPath, extendLatents, keepMaster = true }) {
const tmpPath = `${outputPath}.splice.mp4`;
- const skip = (reason) => {
+ const masterPath = extendMasterPath(outputPath);
+ const masterTmp = keepMaster && masterPath ? partPath(masterPath) : null;
+ const discardTemps = () => Promise.all([tmpPath, masterTmp].filter(Boolean).map((path) => unlinkGuarded(path).catch(() => {})));
+ const skip = async (reason) => {
+ await discardTemps();
console.warn(`⚠️ Extend source splice skipped, keeping the model output: ${reason}`);
return { applied: false, reason };
};
@@ -42,6 +136,9 @@ export async function spliceExtendSource({ sourcePath, outputPath, extendLatents
if (!outputPath || !existsSync(outputPath)) return skip('rendered file missing');
const ffmpeg = await findFfmpeg();
if (!ffmpeg) return skip('ffmpeg not found');
+ // A relaunched render rewrites outputPath, so a master left by an earlier
+ // attempt no longer describes it.
+ if (masterPath) await unlinkGuarded(masterPath).catch(() => {});
const [sourceFrames, outputFrames, outputInfo] = await Promise.all([
probeFrameCount(sourcePath), probeFrameCount(outputPath), probeVideoStreamInfo(outputPath),
@@ -49,33 +146,51 @@ export async function spliceExtendSource({ sourcePath, outputPath, extendLatents
const plan = planExtendSplice({ sourceFrames, outputFrames, extendLatents });
if (!plan) return skip(`frame layout does not match (source ${sourceFrames ?? '?'}, output ${outputFrames ?? '?'}, +${extendLatents} latents)`);
if (!outputInfo.width || !outputInfo.height || !outputInfo.fps) return skip('render geometry unreadable');
+ const geometry = { width: outputInfo.width, height: outputInfo.height, fps: outputInfo.fps };
const [sourceHasAudio, outputHasAudio] = await Promise.all([hasAudioStream(sourcePath), hasAudioStream(outputPath)]);
// A render with no track is one the user asked to be silent (`disableAudio`),
// so the source's soundtrack is dropped with it, matching what Extend delivered before.
const audio = !outputHasAudio ? 'none' : sourceHasAudio ? 'splice' : 'generated';
- const args = buildExtendSpliceArgs({
- sourcePath, generatedPath: outputPath, outPath: tmpPath,
- keepFrames: plan.keepFrames, newStartFrame: plan.newStartFrame,
- width: outputInfo.width, height: outputInfo.height, fps: outputInfo.fps,
- audio, crossfadeSec: SEAM_CROSSFADE_SEC, colorTagFilter: await bt709TagFilter(),
- });
- if (!args) return skip('could not build the splice graph');
+ const sourceMaster = audio === 'splice' ? await usableSourceMaster(sourcePath) : null;
+ const sourceAudioPath = sourceMaster || sourcePath;
+ const colorTagFilter = await bt709TagFilter();
+ const wantMaster = audio !== 'none' ? masterTmp : null;
- const encoded = await runFfmpegProcess({ bin: ffmpeg, args });
- if (!encoded.ok) {
- await unlinkGuarded(tmpPath).catch(() => {});
- return skip(encoded.reason);
+ let streamCopied = false;
+ const copy = await copyJoin({
+ ffmpeg, plan, sourcePath, outputPath, tmpPath, geometry, audio, sourceAudioPath, masterTmp: wantMaster, colorTagFilter,
+ }).catch((err) => ({ ok: false, reason: err.message }));
+ if (copy.ok) {
+ streamCopied = true;
+ } else {
+ console.log(`ℹ️ Extend stream-copy not used, re-encoding the source span: ${copy.reason}`);
+ await discardTemps();
+ const args = buildExtendSpliceArgs({
+ sourcePath, generatedPath: outputPath, outPath: tmpPath,
+ keepFrames: plan.keepFrames, newStartFrame: plan.newStartFrame, ...geometry,
+ audio, crossfadeSec: SEAM_CROSSFADE_SEC, colorTagFilter,
+ sourceAudioPath: sourceMaster, masterOutPath: wantMaster,
+ });
+ if (!args) return skip('could not build the splice graph');
+ const encoded = await runFfmpegProcess({ bin: ffmpeg, args });
+ if (!encoded.ok) return skip(encoded.reason);
}
+
// Verify before replacing anything: the join must be exactly the render's
// length (kept source frames + new frames), or the seam is off by a frame.
const splicedFrames = await probeFrameCount(tmpPath);
- if (splicedFrames !== outputFrames) {
- await unlinkGuarded(tmpPath).catch(() => {});
- return skip(`spliced clip has ${splicedFrames ?? '?'} frames, expected ${outputFrames}`);
- }
+ if (splicedFrames !== outputFrames) return skip(`spliced clip has ${splicedFrames ?? '?'} frames, expected ${outputFrames}`);
const installed = await installEncodedVideo(tmpPath, outputPath, 'extend-spliced');
if (!installed.ok) return skip(installed.reason);
- console.log(`🧵 Extend kept the original source: ${plan.keepFrames} source + ${plan.newFrames} new frames${audio === 'splice' ? ' (audio crossfaded at the seam)' : ''}`);
- return { applied: true, keptFrames: plan.keepFrames, droppedSourceFrames: plan.droppedSourceFrames };
+ // The master is an optimization: a failed install leaves the clip as its own
+ // (AAC) source of audio for the next round, exactly as before masters existed.
+ if (wantMaster) {
+ await rename(wantMaster, masterPath).catch(async (err) => {
+ console.warn(`⚠️ Extend audio master not kept: ${err.message}`);
+ await unlinkGuarded(wantMaster).catch(() => {});
+ });
+ }
+ console.log(`🧵 Extend kept the original source${streamCopied ? ' (stream-copied)' : ''}: ${plan.keepFrames} source + ${plan.newFrames} new frames${audio === 'splice' ? ` (audio crossfaded at the seam${sourceMaster ? ', from the lossless master' : ''})` : ''}`);
+ return { applied: true, keptFrames: plan.keepFrames, droppedSourceFrames: plan.droppedSourceFrames, streamCopied };
}
diff --git a/server/services/videoGen/extendSplice.test.js b/server/services/videoGen/extendSplice.test.js
index be57c33402..2b78cc39c8 100644
--- a/server/services/videoGen/extendSplice.test.js
+++ b/server/services/videoGen/extendSplice.test.js
@@ -7,38 +7,39 @@
* after it, at the render's exact length, and must leave the render untouched
* whenever the layout doesn't verify.
*/
-import { describe, it, expect, beforeAll, afterAll } from 'vitest';
+import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest';
import { mkdtempSync, rmSync, existsSync, readFileSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { execFile } from 'child_process';
import { promisify } from 'util';
-import { findFfmpeg, runFfmpegProcess, probeFrameCount, hasAudioStream } from '../../lib/ffmpeg.js';
+import { findFfmpeg, runFfmpegProcess, probeFrameCount, hasAudioStream, probeVideoDuration, videoFrameHashes } from '../../lib/ffmpeg.js';
import { spliceExtendSource } from './extendSplice.js';
+import { extendMasterPath } from './extendMaster.js';
const execFileAsync = promisify(execFile);
let tmpDir;
let ffmpeg;
-const makeClip = (name, { color, frames, tone }) => runFfmpegProcess({
+const makeClip = (name, { color, frames, tone, x264 = [] }) => runFfmpegProcess({
bin: ffmpeg,
args: [
'-f', 'lavfi', '-i', `color=c=${color}:size=64x48:rate=24`,
'-f', 'lavfi', '-i', `sine=frequency=${tone}:sample_rate=48000`,
'-frames:v', String(frames), '-shortest',
- '-c:v', 'libx264', '-pix_fmt', 'yuv420p', '-c:a', 'aac', '-y', join(tmpDir, name),
+ '-c:v', 'libx264', '-pix_fmt', 'yuv420p', ...x264, '-c:a', 'aac', '-y', join(tmpDir, name),
],
stderrTailBytes: 0,
}).then((r) => (r.ok ? join(tmpDir, name) : null));
-// Mean luma of frame `n`, 0–255.
-const frameLuma = async (path, n) => {
+// Mean luma (0–255) of every decoded frame, in order. `select=eq(n,…)` on a
+// stream-copied join mis-indexes frames after the seam; per-frame stats do not.
+const frameLumas = async (path) => {
const { stdout } = await execFileAsync(ffmpeg, [
- '-v', 'error', '-i', path, '-vf', `select=eq(n\\,${n})`, '-frames:v', '1',
- '-f', 'rawvideo', '-pix_fmt', 'gray', '-',
- ], { encoding: 'buffer', maxBuffer: 1 << 20 });
- return stdout.reduce((sum, v) => sum + v, 0) / stdout.length;
+ '-v', 'error', '-i', path, '-vf', 'signalstats,metadata=print:key=lavfi.signalstats.YAVG:file=-', '-f', 'null', '-',
+ ], { maxBuffer: 1 << 20 });
+ return [...stdout.matchAll(/YAVG=([\d.]+)/g)].map((m) => Number(m[1]));
};
beforeAll(async () => {
@@ -59,10 +60,10 @@ describe('spliceExtendSource', () => {
expect(result).toMatchObject({ applied: true, keptFrames: 17, droppedSourceFrames: 0 });
expect(await probeFrameCount(output)).toBe(33);
- expect(await frameLuma(output, 0)).toBeLessThan(60); // source region: original black
- expect(await frameLuma(output, 16)).toBeLessThan(60);
- expect(await frameLuma(output, 17)).toBeGreaterThan(190); // first new frame: the render
- expect(await frameLuma(output, 32)).toBeGreaterThan(190);
+ const lumas = await frameLumas(output);
+ expect(lumas).toHaveLength(33);
+ expect(lumas.slice(0, 17).every((v) => v < 60)).toBe(true); // source region: original black
+ expect(lumas.slice(17).every((v) => v > 190)).toBe(true); // new region: the render
expect(await hasAudioStream(output)).toBe(true);
expect(existsSync(`${output}.splice.mp4`)).toBe(false);
});
@@ -88,4 +89,84 @@ describe('spliceExtendSource', () => {
});
expect(result).toEqual({ applied: false, reason: 'source file missing' });
});
+
+ it('stream-copies the source so its frames stay bit-identical, and keeps a lossless audio master', async () => {
+ if (!ffmpeg) return;
+ const source = await makeClip('copy-source.mp4', { color: 'black', frames: 17, tone: 440 });
+ const output = await makeClip('copy-render.mp4', { color: 'white', frames: 33, tone: 880 });
+ if (!source || !output) return;
+ const sourceHashes = await videoFrameHashes(source);
+
+ const result = await spliceExtendSource({ sourcePath: source, outputPath: output, extendLatents: 2 });
+
+ expect(result).toMatchObject({ applied: true, streamCopied: true });
+ expect((await videoFrameHashes(output)).slice(0, 17)).toEqual(sourceHashes);
+ const master = extendMasterPath(output);
+ expect(existsSync(master)).toBe(true);
+ // The master carries the whole mix: as long as the delivered clip, not just one span.
+ expect(Math.abs((await probeVideoDuration(master)) - (await probeVideoDuration(output)))).toBeLessThan(0.1);
+ expect(existsSync(`${master.replace(/\.flac$/, '.part.flac')}`)).toBe(false);
+ });
+
+ it('re-encodes when the source and render do not share codec parameters', async () => {
+ if (!ffmpeg) return;
+ const source = await makeClip('baseline-source.mp4', { color: 'black', frames: 17, tone: 440, x264: ['-profile:v', 'baseline'] });
+ const output = await makeClip('main-render.mp4', { color: 'white', frames: 33, tone: 880 });
+ if (!source || !output) return;
+
+ const result = await spliceExtendSource({ sourcePath: source, outputPath: output, extendLatents: 2 });
+
+ expect(result).toMatchObject({ applied: true, streamCopied: false });
+ expect(await probeFrameCount(output)).toBe(33);
+ const lumas = await frameLumas(output);
+ expect(lumas.slice(0, 17).every((v) => v < 60)).toBe(true);
+ expect(lumas.slice(17).every((v) => v > 190)).toBe(true);
+ });
+
+ it('chains: a second round reads its source audio from the first round\'s master and leaves its own', async () => {
+ if (!ffmpeg) return;
+ const source = await makeClip('round1-source.mp4', { color: 'black', frames: 17, tone: 440 });
+ const roundOne = await makeClip('round1-render.mp4', { color: 'white', frames: 33, tone: 880 });
+ if (!source || !roundOne) return;
+ expect((await spliceExtendSource({ sourcePath: source, outputPath: roundOne, extendLatents: 2 })).applied).toBe(true);
+ // 33 frames extended by 2 latents → 49.
+ const roundTwo = await makeClip('round2-render.mp4', { color: 'gray', frames: 49, tone: 660 });
+ if (!roundTwo) return;
+ const logs = [];
+ const spy = vi.spyOn(console, 'log').mockImplementation((line) => logs.push(String(line)));
+
+ const result = await spliceExtendSource({ sourcePath: roundOne, outputPath: roundTwo, extendLatents: 2 });
+ spy.mockRestore();
+
+ expect(result.applied).toBe(true);
+ expect(logs.some((line) => line.includes('lossless master'))).toBe(true);
+ expect(existsSync(extendMasterPath(roundTwo))).toBe(true);
+ expect(await probeFrameCount(roundTwo)).toBe(49);
+ });
+
+ it('ignores a stale master and honors keepMaster:false', async () => {
+ if (!ffmpeg) return;
+ const source = await makeClip('stale-source.mp4', { color: 'black', frames: 17, tone: 440 });
+ const output = await makeClip('stale-render.mp4', { color: 'white', frames: 33, tone: 880 });
+ const other = await makeClip('stale-other.mp4', { color: 'white', frames: 33, tone: 880 });
+ if (!source || !output || !other) return;
+ // A master whose length no longer matches its clip (audio was replaced in place).
+ const staleMaster = extendMasterPath(source);
+ const made = await runFfmpegProcess({
+ bin: ffmpeg,
+ args: ['-f', 'lavfi', '-i', 'sine=frequency=100:sample_rate=48000', '-t', '5', '-c:a', 'flac', '-y', staleMaster],
+ stderrTailBytes: 0,
+ });
+ if (!made.ok) return;
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+
+ const result = await spliceExtendSource({ sourcePath: source, outputPath: output, extendLatents: 2, keepMaster: false });
+ const warned = warn.mock.calls.some(([line]) => String(line).includes('does not match'));
+ warn.mockRestore();
+
+ expect(result.applied).toBe(true);
+ expect(warned).toBe(true);
+ expect(existsSync(extendMasterPath(output))).toBe(false);
+ expect(await probeFrameCount(output)).toBe(33);
+ });
});
diff --git a/server/services/videoGen/generateVideo.js b/server/services/videoGen/generateVideo.js
index 4c0705b84a..cdcbab5a6b 100644
--- a/server/services/videoGen/generateVideo.js
+++ b/server/services/videoGen/generateVideo.js
@@ -153,7 +153,7 @@ export const listVideoModels = () => getVideoModels().map(decorateVideoModel);
export const defaultVideoModelId = (capabilities, preferredId) => getDefaultVideoModelId(capabilities, preferredId);
-export async function generateVideo({ pythonPath, prompt, negativePrompt = '', modelId, width = null, height = null, numFrames = null, fps = 24, steps, guidanceScale, seed, batchSize = 1, tiling = 'auto', disableAudio = false, sourceImagePath = null, uploadedTempPath = null, uploadedTempPaths = [], lastImagePath = null, keyframes = null, extendFromVideoPath = null, audioFilePath = null, audioStartSec = null, audioConditioning = null, mode = null, imageStrength = null, i2vReferenceMode = null, loras = null, icReferencePaths = null, icStrength = null, icAttentionStrength = null, icSkipStage2 = false, textEncoderId = null, speedProfileId = null, draftDecode = null, streamingMode = null, visualConditioning = null, hidden = false, displaySleep = null, jobId: providedJobId = null }) {
+export async function generateVideo({ pythonPath, prompt, negativePrompt = '', modelId, width = null, height = null, numFrames = null, fps = 24, steps, guidanceScale, seed, batchSize = 1, tiling = 'auto', disableAudio = false, sourceImagePath = null, uploadedTempPath = null, uploadedTempPaths = [], lastImagePath = null, keyframes = null, extendFromVideoPath = null, audioFilePath = null, audioStartSec = null, audioConditioning = null, mode = null, imageStrength = null, i2vReferenceMode = null, loras = null, icReferencePaths = null, icStrength = null, icAttentionStrength = null, icSkipStage2 = false, textEncoderId = null, speedProfileId = null, draftDecode = null, streamingMode = null, visualConditioning = null, hidden = false, keepExtendSource = false, displaySleep = null, jobId: providedJobId = null }) {
const sourceAudioProvenance = audioConditioning == null ? null : federatedSourceAudioSchema.parse(audioConditioning);
uploadedTempPaths = Array.isArray(uploadedTempPaths) ? uploadedTempPaths : [];
if (!prompt?.trim()) throw new ServerError('Prompt is required', { status: 400, code: 'VALIDATION_ERROR' });
@@ -736,11 +736,15 @@ export async function generateVideo({ pythonPath, prompt, negativePrompt = '', m
...(await getSettings())?.videoGen,
...(displaySleep != null ? { displaySleep } : {}),
},
- // A plain native Extend delivers the original source, not the model's
- // re-decode of it. Chain chunks (`hidden`) are skipped: the chain trims the
- // echoed context itself, and a window hop's "source" is a temp tail clip.
- extendSplice: mode === 'extend' && extendFromVideoPath && !hidden && !batch && supportsContextWindow(model)
- ? { sourcePath: extendFromVideoPath, extendLatents: extendLatentFrames(parsedNumFrames) }
+ // A native Extend delivers the original source, not the model's re-decode of
+ // it. Chain chunks (`hidden`) are skipped — the chain trims the echoed context
+ // itself and a window hop's "source" is a temp tail clip — except a chain's
+ // FIRST chunk (`keepExtendSource`), whose source is the user's own clip. That
+ // chunk is spliced in place at its rendered length, so the chain's trim math
+ // and the tail window it cuts for chunk 1 are unaffected. Only a visible
+ // render keeps an audio master: a hidden chunk is never extended again.
+ extendSplice: mode === 'extend' && extendFromVideoPath && (!hidden || keepExtendSource) && !batch && supportsContextWindow(model)
+ ? { sourcePath: extendFromVideoPath, extendLatents: extendLatentFrames(parsedNumFrames), keepMaster: !hidden }
: null,
});
diff --git a/server/services/videoGen/generateVideoHelpers.js b/server/services/videoGen/generateVideoHelpers.js
index a18746c868..08c6e515a3 100644
--- a/server/services/videoGen/generateVideoHelpers.js
+++ b/server/services/videoGen/generateVideoHelpers.js
@@ -18,6 +18,7 @@ import { formatBytes, PATHS, unlinkGuarded } from '../../lib/fileUtils.js';
import { withBackupAssetPublication } from '../../lib/backupSnapshotBoundary.js';
import { renderTimingFields } from '../../lib/renderTiming.js';
import { videoGenEvents } from './events.js';
+import { extendMasterPath } from './extendMaster.js';
/**
* Parse byte size values from strings, returning bytes as a number.
@@ -780,7 +781,8 @@ export async function discardUnpublishedVideo({ jobId, outputPath }) {
return withBackupAssetPublication(async () => {
const results = await Promise.allSettled([
outputPath, `${outputPath}.fs.mp4`, join(PATHS.videoThumbnails, `${jobId}.jpg`),
- ].map(path => unlinkGuarded(path)));
+ extendMasterPath(outputPath),
+ ].filter(Boolean).map(path => unlinkGuarded(path)));
const failed = results.find(result => result.status === 'rejected' && result.reason?.code !== 'ENOENT');
if (failed) throw failed.reason;
});
diff --git a/server/services/videoGen/historyOps.js b/server/services/videoGen/historyOps.js
index aaee6954bd..55dfb8d774 100644
--- a/server/services/videoGen/historyOps.js
+++ b/server/services/videoGen/historyOps.js
@@ -5,6 +5,7 @@ import { ServerError } from '../../lib/errorHandler.js';
import { withBackupAssetPublication } from '../../lib/backupSnapshotBoundary.js';
import { safeUnder } from '../../lib/ffmpeg.js';
import { loadHistory, mutateVideoHistory } from './history.js';
+import { extendMasterPath } from './extendMaster.js';
export async function updateHistoryItemPrompt(id, prompt) {
let result;
@@ -51,6 +52,9 @@ export async function deleteHistoryItem(id) {
// filename resolves to inside the expected dir.
const videoFile = safeUnder(PATHS.videos, item.filename);
if (videoFile) await unlinkGuarded(videoFile).catch(() => {});
+ // The lossless audio master an Extend output keeps beside it (extendSplice.js).
+ const masterFile = videoFile && extendMasterPath(videoFile);
+ if (masterFile) await unlinkGuarded(masterFile).catch(() => {});
if (item.thumbnail) {
const thumbFile = safeUnder(PATHS.videoThumbnails, item.thumbnail);
if (thumbFile) await unlinkGuarded(thumbFile).catch(() => {});
diff --git a/server/services/videoGen/local.test.js b/server/services/videoGen/local.test.js
index 1afdee5478..24b273ab55 100644
--- a/server/services/videoGen/local.test.js
+++ b/server/services/videoGen/local.test.js
@@ -1539,6 +1539,20 @@ describe('generateChainedVideo — continuation strategy (context window vs last
.toBe(join(tmpdir(), `chaincontext-${innerJobIds[0]}.mp4`));
});
+ it('delivers the user clip, not the model\'s decode of it, from chunk 0 of an extend chain only (#11179)', async () => {
+ const sourceVideoPath = join(MOCK_PATHS.videos, 'original-video.mp4');
+ const { spliceExtendSource } = await import('./extendSplice.js');
+ vi.mocked(spliceExtendSource).mockClear();
+
+ const { innerJobIds } = await runChain({ mode: 'extend', extendFromVideoPath: sourceVideoPath }, 2);
+
+ // Chunk 1's "source" is a temp tail window the stitch already holds; only
+ // chunk 0 renders from the user's own clip. A hidden chunk keeps no audio master.
+ expect(spliceExtendSource).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
+ sourcePath: sourceVideoPath, keepMaster: false, outputPath: join(MOCK_PATHS.videos, `${innerJobIds[0]}.mp4`),
+ }));
+ });
+
// #4875 — a chained render is ONE clip, so a speed profile applies to every
// chunk or to none. Chunks 1+ re-enter as `extend` on a window-continuity
// chain (the default), and no two-stage profile is validated for that
@@ -6569,7 +6583,16 @@ describe('generateVideo — native Extend keeps the original source (#11171)', (
const splice = await extendRender('splice-plain-extend', { mode: 'extend', extendFromVideoPath: '/mock/source.mp4' });
// numFrames 25 → 3 latents, the same count --extend-frames carries.
expect(splice).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
- sourcePath: '/mock/source.mp4', extendLatents: 3, outputPath: expect.stringContaining('splice-plain-extend'),
+ sourcePath: '/mock/source.mp4', extendLatents: 3, keepMaster: true, outputPath: expect.stringContaining('splice-plain-extend'),
+ }));
+ });
+
+ it('splices the first chunk of an extend chain (the user\'s own clip) without keeping an audio master', async () => {
+ const splice = await extendRender('splice-chain-first', {
+ mode: 'extend', extendFromVideoPath: '/mock/source.mp4', hidden: true, keepExtendSource: true,
+ });
+ expect(splice).toHaveBeenCalledExactlyOnceWith(expect.objectContaining({
+ sourcePath: '/mock/source.mp4', extendLatents: 3, keepMaster: false,
}));
});