Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

fix: bind an Extend audio master to the soundtrack it was mixed for (#11179)

Adam Eivyf6749ffa1880
diff --git a/server/lib/README.md b/server/lib/README.md
index d2efe4be85..ae6805de47 100644
--- a/server/lib/README.md
+++ b/server/lib/README.md
@@ -431,7 +431,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub
| `httpDrain.js` | `createHttpDrain(servers)` tracks every ordinary request the HTTP listeners accept; `begin(windowMs)` is the synchronous shutdown step that stops intake on all of them (`server.close()` plus a 503 `Connection: close` gate for requests arriving on still-open keep-alive sockets), turns off keep-alive on in-flight responses, drops open SSE streams, and resolves `{ remaining }` once accepted requests finish or the window ends (#8323). |
| `httpCompression.js` | `httpCompression` negotiates Brotli/gzip for API and static responses while preserving streaming, ranges, existing encodings, and `no-transform`. |
| `execGit.js` | `execGit(args, cwd, options)` and `execGitSafe(args, cwd, options)` shared process boundary for Git operations, worktrees, and `services/forgeAuth.js` credential resolution. The safe variant converts rejections into `{ exitCode: 1, stdout: '', stderr }`; `services/git.js` re-exports both for compatibility. `cwd` is REQUIRED — it rejects on a missing/blank one rather than letting `spawn` fall back to `process.cwd()` and run git against PortOS's own checkout (#4554). |
-| `ffmpeg.js` | Shared ffmpeg helpers (videoGen + videoTimeline). `probeVideoGeometry(videoPath)` measures a finished clip's `{ width, height, fps, numFrames, durationSec }` (null when any axis is unreadable) for hosted renders that record no frame count. `generateThumbnail(videoPath, jobId, { atSec } = {})` accepts a clamped poster seek while preserving the single-clip default. Includes `probeFrameCount(videoPath)` (metadata `nb_frames`, falling back to a real `-count_frames` pass — expensive, so call it once per file) and `trimVideoFromFrame(videoPath, outPath, {startFrame, fps})` — a frame-EXACT head cut via the `trim` filter (an `-ss` seek can drift a frame, which reads as a stutter at a stitch seam), keeping audio in sync with `atrim` when the clip has a soundtrack and taking `-an` when it doesn't. Re-encodes by necessity, so a later concat must re-encode too; `outPath` may equal the input (temp-file + rename install). Used by the chained-render context window — see `videoContinuity.js`. `buildTrimConcatArgs({inputs, outPath, width, height, fps, withAudio})` builds the argv for the other half of that job: a concat that drops leading frames from some of its inputs inside a `filter_complex` graph, so the cuts ride along with the timeline encode instead of costing one pre-encode per clip (pass `withAudio` only when EVERY input has an audio stream — check with `hasAudioStream`). `buildExtendSpliceArgs({sourcePath, generatedPath, outPath, keepFrames, newStartFrame, width, height, fps, audio, sourceAudioPath, masterOutPath})` is the plain-Extend counterpart: source frames `[0, keep)` + render frames from `newStartFrame`, one encode, audio crossfaded (~40 ms linear) at the seam; `sourceAudioPath` swaps the source span's audio for a lossless master and `masterOutPath` also writes the mix as a 24-bit FLAC master, so the delivered AAC is encoded once from PCM. Its stream-copy sibling is `buildExtendNewSpanArgs` (encode only the render's new span) + `buildExtendConcatList` + `buildExtendCopyArgs` (join it to the untouched source with `-c:v copy`); `probeVideoCodecSignature` gates it and `videoFrameHashes` (per-frame MD5) proves the join before anything is replaced — driven by `services/videoGen/extendSplice.js`. `H264_ENCODE_ARGS` / `AAC_ENCODE_ARGS` are the shared encode profile: clips produced by different paths here get concatenated together, so a mismatch shows up as one segment graded differently from its neighbours — spread these rather than re-typing the flags. Every re-encode here also pins BT.709: `BT709_CONTAINER_ARGS` (the `colr` atom, always emitted) plus `bt709TagFilter()` → the `BT709_TAG_FILTER` `setparams=…` string, or `null` on an ffmpeg without that filter (`supportsSetparamsFilter()` probes `-filters` once per process; `null` = not probed, and a probe that couldn't run stays uncached). Both halves are required — from ffmpeg 8 the encoder reads color properties off the FRAMES, silently overriding the container flags, so a flags-only output decodes washed-out. `buildTrimConcatArgs` takes the filter as its `colorTagFilter` option rather than probing, to stay pure. `installEncodedVideo(tmpPath, targetPath, label)` is the shared in-place install: move the original aside to a `.bak` on Windows (where `fs.rename` refuses an existing destination), rename the new file over it, and restore the backup — plus unlink the temp — on any failure, returning `{ok:true,outPath}` / `{ok:false,reason}` so the worst case is "skipped", never "the video is gone". Use it for EVERY replace-a-video-with-its-re-encode path; `pipeline/audioMux.js` is the out-of-module caller. `edgeFadeFilter(durationSec)` is the `-af` fade-in/fade-out fragment a mid-song clip (a social cut) appends to its trim. Loudness mastering (#10249): `MASTER_LOUDNESS` (-14 LUFS / -1.5 dBTP / LRA 11), `planLoudnessMaster` (measure then build the linear apply filter; throws on a silent source), `reportMasteredLoudness` (post-encode before/after numbers). |
+| `ffmpeg.js` | Shared ffmpeg helpers (videoGen + videoTimeline). `probeVideoGeometry(videoPath)` measures a finished clip's `{ width, height, fps, numFrames, durationSec }` (null when any axis is unreadable) for hosted renders that record no frame count. `generateThumbnail(videoPath, jobId, { atSec } = {})` accepts a clamped poster seek while preserving the single-clip default. Includes `probeFrameCount(videoPath)` (metadata `nb_frames`, falling back to a real `-count_frames` pass — expensive, so call it once per file) and `trimVideoFromFrame(videoPath, outPath, {startFrame, fps})` — a frame-EXACT head cut via the `trim` filter (an `-ss` seek can drift a frame, which reads as a stutter at a stitch seam), keeping audio in sync with `atrim` when the clip has a soundtrack and taking `-an` when it doesn't. Re-encodes by necessity, so a later concat must re-encode too; `outPath` may equal the input (temp-file + rename install). Used by the chained-render context window — see `videoContinuity.js`. `buildTrimConcatArgs({inputs, outPath, width, height, fps, withAudio})` builds the argv for the other half of that job: a concat that drops leading frames from some of its inputs inside a `filter_complex` graph, so the cuts ride along with the timeline encode instead of costing one pre-encode per clip (pass `withAudio` only when EVERY input has an audio stream — check with `hasAudioStream`). `buildExtendSpliceArgs({sourcePath, generatedPath, outPath, keepFrames, newStartFrame, width, height, fps, audio, sourceAudioPath, masterOutPath})` is the plain-Extend counterpart: source frames `[0, keep)` + render frames from `newStartFrame`, one encode, audio crossfaded (~40 ms linear) at the seam; `sourceAudioPath` swaps the source span's audio for a lossless master and `masterOutPath` also writes the mix as a 24-bit FLAC master, so the delivered AAC is encoded once from PCM. Its stream-copy sibling is `buildExtendNewSpanArgs` (encode only the render's new span) + `buildExtendConcatList` + `buildExtendCopyArgs` (join it to the untouched source with `-c:v copy`); `probeVideoCodecSignature` gates it and `videoFrameHashes` (per-frame MD5) proves the join before anything is replaced; `audioFingerprint` / `sealAudioMaster` / `readAudioMasterFingerprint` bind a master to the exact decoded soundtrack it was mixed for so a replaced soundtrack invalidates it — driven by `services/videoGen/extendSplice.js`. `H264_ENCODE_ARGS` / `AAC_ENCODE_ARGS` are the shared encode profile: clips produced by different paths here get concatenated together, so a mismatch shows up as one segment graded differently from its neighbours — spread these rather than re-typing the flags. Every re-encode here also pins BT.709: `BT709_CONTAINER_ARGS` (the `colr` atom, always emitted) plus `bt709TagFilter()` → the `BT709_TAG_FILTER` `setparams=…` string, or `null` on an ffmpeg without that filter (`supportsSetparamsFilter()` probes `-filters` once per process; `null` = not probed, and a probe that couldn't run stays uncached). Both halves are required — from ffmpeg 8 the encoder reads color properties off the FRAMES, silently overriding the container flags, so a flags-only output decodes washed-out. `buildTrimConcatArgs` takes the filter as its `colorTagFilter` option rather than probing, to stay pure. `installEncodedVideo(tmpPath, targetPath, label)` is the shared in-place install: move the original aside to a `.bak` on Windows (where `fs.rename` refuses an existing destination), rename the new file over it, and restore the backup — plus unlink the temp — on any failure, returning `{ok:true,outPath}` / `{ok:false,reason}` so the worst case is "skipped", never "the video is gone". Use it for EVERY replace-a-video-with-its-re-encode path; `pipeline/audioMux.js` is the out-of-module caller. `edgeFadeFilter(durationSec)` is the `-af` fade-in/fade-out fragment a mid-song clip (a social cut) appends to its trim. Loudness mastering (#10249): `MASTER_LOUDNESS` (-14 LUFS / -1.5 dBTP / LRA 11), `planLoudnessMaster` (measure then build the linear apply filter; throws on a silent source), `reportMasteredLoudness` (post-encode before/after numbers). |
| `forgeAccessErrors.js` | `isForgeNoAccessError(err)` / `isForgeNoAccessMessage(text)` / `logForgeNoAccessOnce(scope, repoSpec, detail)` / `forgeNoAccessRemedy(repoSpec)` — tells a PERMANENT `gh` no-access 404 (`Could not resolve to a Repository`, `HTTP 404`) apart from a transient blip. `execGh` stamps `ghNoAccess` on the rejection and keeps it OUT of the consecutive-failure backoff (which caps at 15 min and then re-fires forever), and the polling jobs (`branchReconcile`, `prWatcher`) narrate it exactly once per scope+repo per process with a remedy line instead of once per tick. `__resetForgeNoAccessLog()` is the test seam. |
| `forkHead.js` | `normalizeForkHead(value)` / `forkRemoteName(ownerLogin)` / `resolveTaskForkHead(metadata)` — the validated `{remoteUrl, ownerLogin}` address of a cross-repository PR's head branch, which has no `origin/<branch>` to attach to. Fail-closed: anything that isn't a usable pair (or that starts with `-`, which git would read as an option) is `null`, so every caller falls back to origin-only behavior instead of guessing. `forkRemoteName` is the deterministic `fork-<owner>` remote name that makes a re-attach reuse the remote it already added. Shared by `appPullRequests.js` (produces), `agentWorktreeCleanup.js`/`prRemediationFollowUp.js` (persist in task metadata), `agentWorkspacePrep.js` (threads), and `worktreeManager.js` (consumes — it has no forge client of its own). `resolveTaskForkHead` is the single reader of the generic `forkHead` task-metadata key, the fork-side counterpart to `taskTargetBranch.js`. |
| `frameQuality.js` | Scores decoded video frames so a chained render can CHOOSE its continuation anchor instead of taking whatever an end-seek lands on. `scoreFrame({width,height,data},{recency})` combines three terms over a raw single-channel buffer: log-compressed `gradientVariance` (focus), distance of `meanLuma` from mid-grey (exposure penalty, which rejects a fade-to-black or a blown highlight), and a small linear recency bonus that keeps the anchor near the cut when candidates are comparably clean. Usability and ranking are deliberately separate: `usable` gates on raw gradient variance against `MIN_SIGNAL_VARIANCE` alone, so only a mathematically degenerate frame is rejected and a legitimately dark low-key tail still qualifies — the same contract `imageFrameStats.js` keeps. Gating on the composite score instead would reject every night scene on the exposure term. `TAIL_WINDOW_SECONDS` is capped at the reach of the single `-sseof -1.0` seek this replaces: focus is systematically highest at the OLDEST candidate, so a wider window would widen the backward jump at each chain seam rather than tightening it. `pickBestFrame(paths)` decodes each candidate through `sharp` and returns the winner (or `null` when nothing decodes or every candidate is degenerate, so the caller degrades rather than failing a render); its `index` is the position in the array AS PASSED, because the caller turns that into the anchor's time offset. `MAX_CANDIDATES` is derived from the window, never set independently — the ffmpeg-side `-frames:v` cap truncates from the NEWEST end, exactly the frames recency prefers. `TAIL_WINDOW_SECONDS` / `CANDIDATE_FPS` / `MAX_CANDIDATES` define the window the caller decodes from. Consumer: `services/videoGen/local.js#extractLastFrame`. |
diff --git a/server/lib/ffmpeg.js b/server/lib/ffmpeg.js
index 1062bef158..addd33eee8 100644
--- a/server/lib/ffmpeg.js
+++ b/server/lib/ffmpeg.js
@@ -348,6 +348,40 @@ export const videoFrameHashes = async (videoPath) => {
}
};
+// Identity of the soundtrack a clip plays: the MD5 of its decoded first audio
+// stream. Decoded rather than hashed as packets so a lossless remux (faststart)
+// keeps it, while any change to the samples — a replaced or re-muxed soundtrack,
+// even at the same length — does not. Null when the clip has no readable audio.
+export const audioFingerprint = async (mediaPath) => {
+ const ffmpeg = await findFfmpeg();
+ if (!ffmpeg || typeof mediaPath !== 'string' || !mediaPath) return null;
+ const { stdout } = await execFileAsync(
+ ffmpeg,
+ ['-v', 'error', '-i', mediaPath, '-map', '0:a:0', '-vn', '-f', 'md5', '-'],
+ safeChildProcessOptions({ timeout: 60000 }),
+ ).catch(() => ({ stdout: '' }));
+ return /^MD5=([0-9a-f]{32})/m.exec(stdout || '')?.[1] ?? null;
+};
+
+const MASTER_FINGERPRINT_TAG = 'portos_clip_audio_md5';
+
+/** Copy a lossless audio master to `outPath`, stamping the clip-audio fingerprint it was mixed for. */
+export const sealAudioMaster = async ({ inPath, outPath, fingerprint }) => {
+ const ffmpeg = await findFfmpeg();
+ if (!ffmpeg) return { ok: false, reason: 'ffmpeg not found' };
+ return runFfmpegProcess({
+ bin: ffmpeg,
+ args: ['-i', inPath, '-map', '0:a:0', '-c:a', 'copy', '-metadata', `${MASTER_FINGERPRINT_TAG}=${fingerprint}`, '-y', outPath],
+ });
+};
+
+/** The clip-audio fingerprint a master was sealed with, or null when absent or unreadable. */
+export const readAudioMasterFingerprint = async (masterPath) => {
+ const stdout = await runFfprobe(['-v', 'error', '-show_entries', 'format_tags', '-of', 'default=noprint_wrappers=1', masterPath]);
+ const line = stdout.split(/\r?\n/).find((row) => row.toLowerCase().startsWith(`tag:${MASTER_FINGERPRINT_TAG}=`));
+ return line ? line.slice(line.indexOf('=') + 1).trim() || null : null;
+};
+
// The playable geometry of a finished clip — `{ width, height, fps, numFrames,
// durationSec }` — or null when any axis cannot be measured. For entries whose
// producer never recorded a frame count (hosted Grok/fal/reactor renders), so a
diff --git a/server/services/videoGen/extendSplice.js b/server/services/videoGen/extendSplice.js
index 3299ad2700..e603d02d4c 100644
--- a/server/services/videoGen/extendSplice.js
+++ b/server/services/videoGen/extendSplice.js
@@ -15,10 +15,11 @@
*/
import { existsSync } from 'fs';
-import { rename, writeFile } from 'fs/promises';
+import { writeFile } from 'fs/promises';
import { unlinkGuarded } from '../../lib/fileUtils.js';
import {
- findFfmpeg, runFfmpegProcess, probeFrameCount, probeVideoStreamInfo, probeVideoDuration,
+ findFfmpeg, runFfmpegProcess, probeFrameCount, probeVideoStreamInfo,
+ audioFingerprint, sealAudioMaster, readAudioMasterFingerprint,
probeVideoCodecSignature, videoFrameHashes,
hasAudioStream, bt709TagFilter, buildExtendSpliceArgs, buildExtendNewSpanArgs, buildExtendCopyArgs,
buildExtendConcatList, installEncodedVideo,
@@ -29,22 +30,20 @@ import { extendMasterPath } from './extendMaster.js';
/** Linear crossfade at the source/generated audio seam (~one 25 Hz audio token). */
export const SEAM_CROSSFADE_SEC = 0.04;
-/** How far a master's length may sit from its clip's before it is treated as stale (AAC priming/padding is ~tens of ms). */
-const MASTER_DURATION_TOLERANCE_SEC = 0.15;
-
const partPath = (path) => path.replace(/\.flac$/i, '.part.flac');
/**
* The lossless master to read the SOURCE span's audio from, or null to use the
- * source clip's own track. A master only counts while its length still matches
- * the clip beside it: a clip whose audio was later trimmed, replaced or
- * re-muxed in place leaves a master that no longer describes it.
+ * source clip's own track. A master only counts while it is provably the mix the
+ * clip's soundtrack was encoded from: it carries the fingerprint of that decoded
+ * soundtrack, so a clip whose audio was later trimmed, replaced or re-muxed in
+ * place — even to the same length — no longer matches and is ignored.
*/
async function usableSourceMaster(sourcePath) {
const masterPath = extendMasterPath(sourcePath);
if (!masterPath || !existsSync(masterPath)) return null;
- const [masterSec, sourceSec] = await Promise.all([probeVideoDuration(masterPath), probeVideoDuration(sourcePath)]);
- if (!masterSec || !sourceSec || Math.abs(masterSec - sourceSec) > MASTER_DURATION_TOLERANCE_SEC) {
+ const [sealed, current] = await Promise.all([readAudioMasterFingerprint(masterPath), audioFingerprint(sourcePath)]);
+ if (!sealed || sealed !== current) {
console.warn('⚠️ Extend audio master does not match its clip, using the clip\'s own track');
return null;
}
@@ -183,13 +182,16 @@ export async function spliceExtendSource({ sourcePath, outputPath, extendLatents
if (splicedFrames !== outputFrames) return skip(`spliced clip has ${splicedFrames ?? '?'} frames, expected ${outputFrames}`);
const installed = await installEncodedVideo(tmpPath, outputPath, 'extend-spliced');
if (!installed.ok) return skip(installed.reason);
- // The master is an optimization: a failed install leaves the clip as its own
- // (AAC) source of audio for the next round, exactly as before masters existed.
+ // The master is an optimization: without one the clip stays its own (AAC)
+ // source of audio for the next round, exactly as before masters existed.
if (wantMaster) {
- await rename(wantMaster, masterPath).catch(async (err) => {
- console.warn(`⚠️ Extend audio master not kept: ${err.message}`);
- await unlinkGuarded(wantMaster).catch(() => {});
- });
+ const fingerprint = await audioFingerprint(outputPath);
+ const sealed = fingerprint ? await sealAudioMaster({ inPath: wantMaster, outPath: masterPath, fingerprint }) : { ok: false, reason: 'clip audio unreadable' };
+ if (!sealed.ok) {
+ console.warn(`⚠️ Extend audio master not kept: ${sealed.reason}`);
+ await unlinkGuarded(masterPath).catch(() => {});
+ }
+ await unlinkGuarded(wantMaster).catch(() => {});
}
console.log(`🧵 Extend kept the original source${streamCopied ? ' (stream-copied)' : ''}: ${plan.keepFrames} source + ${plan.newFrames} new frames${audio === 'splice' ? ` (audio crossfaded at the seam${sourceMaster ? ', from the lossless master' : ''})` : ''}`);
return { applied: true, keptFrames: plan.keepFrames, droppedSourceFrames: plan.droppedSourceFrames, streamCopied };
diff --git a/server/services/videoGen/extendSplice.test.js b/server/services/videoGen/extendSplice.test.js
index 2b78cc39c8..a40287a51a 100644
--- a/server/services/videoGen/extendSplice.test.js
+++ b/server/services/videoGen/extendSplice.test.js
@@ -8,7 +8,7 @@
* whenever the layout doesn't verify.
*/
import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest';
-import { mkdtempSync, rmSync, existsSync, readFileSync } from 'fs';
+import { mkdtempSync, rmSync, existsSync, readFileSync, renameSync } from 'fs';
import { tmpdir } from 'os';
import { join } from 'path';
import { execFile } from 'child_process';
@@ -144,6 +144,34 @@ describe('spliceExtendSource', () => {
expect(await probeFrameCount(roundTwo)).toBe(49);
});
+ it('ignores a master once the clip\'s soundtrack was replaced in place, even at the same length', async () => {
+ if (!ffmpeg) return;
+ const source = await makeClip('swap-source.mp4', { color: 'black', frames: 17, tone: 440 });
+ const roundOne = await makeClip('swap-render.mp4', { color: 'white', frames: 33, tone: 880 });
+ const roundTwo = await makeClip('swap-render-2.mp4', { color: 'gray', frames: 49, tone: 660 });
+ if (!source || !roundOne || !roundTwo) return;
+ expect((await spliceExtendSource({ sourcePath: source, outputPath: roundOne, extendLatents: 2 })).applied).toBe(true);
+ const master = extendMasterPath(roundOne);
+ expect(existsSync(master)).toBe(true);
+ // Swap the soundtrack, keeping the picture and the length.
+ const swapped = join(tmpDir, 'swap-render.swapped.mp4');
+ const remux = await runFfmpegProcess({
+ bin: ffmpeg,
+ args: ['-i', roundOne, '-f', 'lavfi', '-i', 'sine=frequency=1500:sample_rate=48000', '-map', '0:v', '-map', '1:a', '-c:v', 'copy', '-c:a', 'aac', '-shortest', '-y', swapped],
+ stderrTailBytes: 0,
+ });
+ if (!remux.ok) return;
+ renameSync(swapped, roundOne);
+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});
+
+ const result = await spliceExtendSource({ sourcePath: roundOne, outputPath: roundTwo, extendLatents: 2 });
+ const warned = warn.mock.calls.some(([line]) => String(line).includes('does not match'));
+ warn.mockRestore();
+
+ expect(result.applied).toBe(true);
+ expect(warned).toBe(true);
+ });
+
it('ignores a stale master and honors keepMaster:false', async () => {
if (!ffmpeg) return;
const source = await makeClip('stale-source.mp4', { color: 'black', frames: 17, tone: 440 });