Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
fix: bind an Extend audio master to the soundtrack it was mixed for (#11179)
Adam Eivyf6749ffa1880
diff --git a/server/lib/README.md b/server/lib/README.mdindex d2efe4be85..ae6805de47 100644--- a/server/lib/README.md+++ b/server/lib/README.md@@ -431,7 +431,7 @@ The barrel `server/lib/index.js` is a machine-checkable enumeration of every pub| `httpDrain.js` | `createHttpDrain(servers)` tracks every ordinary request the HTTP listeners accept; `begin(windowMs)` is the synchronous shutdown step that stops intake on all of them (`server.close()` plus a 503 `Connection: close` gate for requests arriving on still-open keep-alive sockets), turns off keep-alive on in-flight responses, drops open SSE streams, and resolves `{ remaining }` once accepted requests finish or the window ends (#8323). || `httpCompression.js` | `httpCompression` negotiates Brotli/gzip for API and static responses while preserving streaming, ranges, existing encodings, and `no-transform`. || `execGit.js` | `execGit(args, cwd, options)` and `execGitSafe(args, cwd, options)` shared process boundary for Git operations, worktrees, and `services/forgeAuth.js` credential resolution. The safe variant converts rejections into `{ exitCode: 1, stdout: '', stderr }`; `services/git.js` re-exports both for compatibility. `cwd` is REQUIRED — it rejects on a missing/blank one rather than letting `spawn` fall back to `process.cwd()` and run git against PortOS's own checkout (#4554). |-| `ffmpeg.js` | Shared ffmpeg helpers (videoGen + videoTimeline). `probeVideoGeometry(videoPath)` measures a finished clip's `{ width, height, fps, numFrames, durationSec }` (null when any axis is unreadable) for hosted renders that record no frame count. `generateThumbnail(videoPath, jobId, { atSec } = {})` accepts a clamped poster seek while preserving the single-clip default. Includes `probeFrameCount(videoPath)` (metadata `nb_frames`, falling back to a real `-count_frames` pass — expensive, so call it once per file) and `trimVideoFromFrame(videoPath, outPath, {startFrame, fps})` — a frame-EXACT head cut via the `trim` filter (an `-ss` seek can drift a frame, which reads as a stutter at a stitch seam), keeping audio in sync with `atrim` when the clip has a soundtrack and taking `-an` when it doesn't. Re-encodes by necessity, so a later concat must re-encode too; `outPath` may equal the input (temp-file + rename install). Used by the chained-render context window — see `videoContinuity.js`. `buildTrimConcatArgs({inputs, outPath, width, height, fps, withAudio})` builds the argv for the other half of that job: a concat that drops leading frames from some of its inputs inside a `filter_complex` graph, so the cuts ride along with the timeline encode instead of costing one pre-encode per clip (pass `withAudio` only when EVERY input has an audio stream — check with `hasAudioStream`). `buildExtendSpliceArgs({sourcePath, generatedPath, outPath, keepFrames, newStartFrame, width, height, fps, audio, sourceAudioPath, masterOutPath})` is the plain-Extend counterpart: source frames `[0, keep)` + render frames from `newStartFrame`, one encode, audio crossfaded (~40 ms linear) at the seam; `sourceAudioPath` swaps the source span's audio for a lossless master and `masterOutPath` also writes the mix as a 24-bit FLAC master, so the delivered AAC is encoded once from PCM. Its stream-copy sibling is `buildExtendNewSpanArgs` (encode only the render's new span) + `buildExtendConcatList` + `buildExtendCopyArgs` (join it to the untouched source with `-c:v copy`); `probeVideoCodecSignature` gates it and `videoFrameHashes` (per-frame MD5) proves the join before anything is replaced — driven by `services/videoGen/extendSplice.js`. `H264_ENCODE_ARGS` / `AAC_ENCODE_ARGS` are the shared encode profile: clips produced by different paths here get concatenated together, so a mismatch shows up as one segment graded differently from its neighbours — spread these rather than re-typing the flags. Every re-encode here also pins BT.709: `BT709_CONTAINER_ARGS` (the `colr` atom, always emitted) plus `bt709TagFilter()` → the `BT709_TAG_FILTER` `setparams=…` string, or `null` on an ffmpeg without that filter (`supportsSetparamsFilter()` probes `-filters` once per process; `null` = not probed, and a probe that couldn't run stays uncached). Both halves are required — from ffmpeg 8 the encoder reads color properties off the FRAMES, silently overriding the container flags, so a flags-only output decodes washed-out. `buildTrimConcatArgs` takes the filter as its `colorTagFilter` option rather than probing, to stay pure. `installEncodedVideo(tmpPath, targetPath, label)` is the shared in-place install: move the original aside to a `.bak` on Windows (where `fs.rename` refuses an existing destination), rename the new file over it, and restore the backup — plus unlink the temp — on any failure, returning `{ok:true,outPath}` / `{ok:false,reason}` so the worst case is "skipped", never "the video is gone". Use it for EVERY replace-a-video-with-its-re-encode path; `pipeline/audioMux.js` is the out-of-module caller. `edgeFadeFilter(durationSec)` is the `-af` fade-in/fade-out fragment a mid-song clip (a social cut) appends to its trim. Loudness mastering (#10249): `MASTER_LOUDNESS` (-14 LUFS / -1.5 dBTP / LRA 11), `planLoudnessMaster` (measure then build the linear apply filter; throws on a silent source), `reportMasteredLoudness` (post-encode before/after numbers). |+| `ffmpeg.js` | Shared ffmpeg helpers (videoGen + videoTimeline). `probeVideoGeometry(videoPath)` measures a finished clip's `{ width, height, fps, numFrames, durationSec }` (null when any axis is unreadable) for hosted renders that record no frame count. `generateThumbnail(videoPath, jobId, { atSec } = {})` accepts a clamped poster seek while preserving the single-clip default. Includes `probeFrameCount(videoPath)` (metadata `nb_frames`, falling back to a real `-count_frames` pass — expensive, so call it once per file) and `trimVideoFromFrame(videoPath, outPath, {startFrame, fps})` — a frame-EXACT head cut via the `trim` filter (an `-ss` seek can drift a frame, which reads as a stutter at a stitch seam), keeping audio in sync with `atrim` when the clip has a soundtrack and taking `-an` when it doesn't. Re-encodes by necessity, so a later concat must re-encode too; `outPath` may equal the input (temp-file + rename install). Used by the chained-render context window — see `videoContinuity.js`. `buildTrimConcatArgs({inputs, outPath, width, height, fps, withAudio})` builds the argv for the other half of that job: a concat that drops leading frames from some of its inputs inside a `filter_complex` graph, so the cuts ride along with the timeline encode instead of costing one pre-encode per clip (pass `withAudio` only when EVERY input has an audio stream — check with `hasAudioStream`). `buildExtendSpliceArgs({sourcePath, generatedPath, outPath, keepFrames, newStartFrame, width, height, fps, audio, sourceAudioPath, masterOutPath})` is the plain-Extend counterpart: source frames `[0, keep)` + render frames from `newStartFrame`, one encode, audio crossfaded (~40 ms linear) at the seam; `sourceAudioPath` swaps the source span's audio for a lossless master and `masterOutPath` also writes the mix as a 24-bit FLAC master, so the delivered AAC is encoded once from PCM. Its stream-copy sibling is `buildExtendNewSpanArgs` (encode only the render's new span) + `buildExtendConcatList` + `buildExtendCopyArgs` (join it to the untouched source with `-c:v copy`); `probeVideoCodecSignature` gates it and `videoFrameHashes` (per-frame MD5) proves the join before anything is replaced; `audioFingerprint` / `sealAudioMaster` / `readAudioMasterFingerprint` bind a master to the exact decoded soundtrack it was mixed for so a replaced soundtrack invalidates it — driven by `services/videoGen/extendSplice.js`. `H264_ENCODE_ARGS` / `AAC_ENCODE_ARGS` are the shared encode profile: clips produced by different paths here get concatenated together, so a mismatch shows up as one segment graded differently from its neighbours — spread these rather than re-typing the flags. Every re-encode here also pins BT.709: `BT709_CONTAINER_ARGS` (the `colr` atom, always emitted) plus `bt709TagFilter()` → the `BT709_TAG_FILTER` `setparams=…` string, or `null` on an ffmpeg without that filter (`supportsSetparamsFilter()` probes `-filters` once per process; `null` = not probed, and a probe that couldn't run stays uncached). Both halves are required — from ffmpeg 8 the encoder reads color properties off the FRAMES, silently overriding the container flags, so a flags-only output decodes washed-out. `buildTrimConcatArgs` takes the filter as its `colorTagFilter` option rather than probing, to stay pure. `installEncodedVideo(tmpPath, targetPath, label)` is the shared in-place install: move the original aside to a `.bak` on Windows (where `fs.rename` refuses an existing destination), rename the new file over it, and restore the backup — plus unlink the temp — on any failure, returning `{ok:true,outPath}` / `{ok:false,reason}` so the worst case is "skipped", never "the video is gone". Use it for EVERY replace-a-video-with-its-re-encode path; `pipeline/audioMux.js` is the out-of-module caller. `edgeFadeFilter(durationSec)` is the `-af` fade-in/fade-out fragment a mid-song clip (a social cut) appends to its trim. Loudness mastering (#10249): `MASTER_LOUDNESS` (-14 LUFS / -1.5 dBTP / LRA 11), `planLoudnessMaster` (measure then build the linear apply filter; throws on a silent source), `reportMasteredLoudness` (post-encode before/after numbers). || `forgeAccessErrors.js` | `isForgeNoAccessError(err)` / `isForgeNoAccessMessage(text)` / `logForgeNoAccessOnce(scope, repoSpec, detail)` / `forgeNoAccessRemedy(repoSpec)` — tells a PERMANENT `gh` no-access 404 (`Could not resolve to a Repository`, `HTTP 404`) apart from a transient blip. `execGh` stamps `ghNoAccess` on the rejection and keeps it OUT of the consecutive-failure backoff (which caps at 15 min and then re-fires forever), and the polling jobs (`branchReconcile`, `prWatcher`) narrate it exactly once per scope+repo per process with a remedy line instead of once per tick. `__resetForgeNoAccessLog()` is the test seam. || `forkHead.js` | `normalizeForkHead(value)` / `forkRemoteName(ownerLogin)` / `resolveTaskForkHead(metadata)` — the validated `{remoteUrl, ownerLogin}` address of a cross-repository PR's head branch, which has no `origin/<branch>` to attach to. Fail-closed: anything that isn't a usable pair (or that starts with `-`, which git would read as an option) is `null`, so every caller falls back to origin-only behavior instead of guessing. `forkRemoteName` is the deterministic `fork-<owner>` remote name that makes a re-attach reuse the remote it already added. Shared by `appPullRequests.js` (produces), `agentWorktreeCleanup.js`/`prRemediationFollowUp.js` (persist in task metadata), `agentWorkspacePrep.js` (threads), and `worktreeManager.js` (consumes — it has no forge client of its own). `resolveTaskForkHead` is the single reader of the generic `forkHead` task-metadata key, the fork-side counterpart to `taskTargetBranch.js`. || `frameQuality.js` | Scores decoded video frames so a chained render can CHOOSE its continuation anchor instead of taking whatever an end-seek lands on. `scoreFrame({width,height,data},{recency})` combines three terms over a raw single-channel buffer: log-compressed `gradientVariance` (focus), distance of `meanLuma` from mid-grey (exposure penalty, which rejects a fade-to-black or a blown highlight), and a small linear recency bonus that keeps the anchor near the cut when candidates are comparably clean. Usability and ranking are deliberately separate: `usable` gates on raw gradient variance against `MIN_SIGNAL_VARIANCE` alone, so only a mathematically degenerate frame is rejected and a legitimately dark low-key tail still qualifies — the same contract `imageFrameStats.js` keeps. Gating on the composite score instead would reject every night scene on the exposure term. `TAIL_WINDOW_SECONDS` is capped at the reach of the single `-sseof -1.0` seek this replaces: focus is systematically highest at the OLDEST candidate, so a wider window would widen the backward jump at each chain seam rather than tightening it. `pickBestFrame(paths)` decodes each candidate through `sharp` and returns the winner (or `null` when nothing decodes or every candidate is degenerate, so the caller degrades rather than failing a render); its `index` is the position in the array AS PASSED, because the caller turns that into the anchor's time offset. `MAX_CANDIDATES` is derived from the window, never set independently — the ffmpeg-side `-frames:v` cap truncates from the NEWEST end, exactly the frames recency prefers. `TAIL_WINDOW_SECONDS` / `CANDIDATE_FPS` / `MAX_CANDIDATES` define the window the caller decodes from. Consumer: `services/videoGen/local.js#extractLastFrame`. |diff --git a/server/lib/ffmpeg.js b/server/lib/ffmpeg.jsindex 1062bef158..addd33eee8 100644--- a/server/lib/ffmpeg.js+++ b/server/lib/ffmpeg.js@@ -348,6 +348,40 @@ export const videoFrameHashes = async (videoPath) => {}};+// Identity of the soundtrack a clip plays: the MD5 of its decoded first audio+// stream. Decoded rather than hashed as packets so a lossless remux (faststart)+// keeps it, while any change to the samples — a replaced or re-muxed soundtrack,+// even at the same length — does not. Null when the clip has no readable audio.+export const audioFingerprint = async (mediaPath) => {+ const ffmpeg = await findFfmpeg();+ if (!ffmpeg || typeof mediaPath !== 'string' || !mediaPath) return null;+ const { stdout } = await execFileAsync(+ ffmpeg,+ ['-v', 'error', '-i', mediaPath, '-map', '0:a:0', '-vn', '-f', 'md5', '-'],+ safeChildProcessOptions({ timeout: 60000 }),+ ).catch(() => ({ stdout: '' }));+ return /^MD5=([0-9a-f]{32})/m.exec(stdout || '')?.[1] ?? null;+};++const MASTER_FINGERPRINT_TAG = 'portos_clip_audio_md5';++/** Copy a lossless audio master to `outPath`, stamping the clip-audio fingerprint it was mixed for. */+export const sealAudioMaster = async ({ inPath, outPath, fingerprint }) => {+ const ffmpeg = await findFfmpeg();+ if (!ffmpeg) return { ok: false, reason: 'ffmpeg not found' };+ return runFfmpegProcess({+ bin: ffmpeg,+ args: ['-i', inPath, '-map', '0:a:0', '-c:a', 'copy', '-metadata', `${MASTER_FINGERPRINT_TAG}=${fingerprint}`, '-y', outPath],+ });+};++/** The clip-audio fingerprint a master was sealed with, or null when absent or unreadable. */+export const readAudioMasterFingerprint = async (masterPath) => {+ const stdout = await runFfprobe(['-v', 'error', '-show_entries', 'format_tags', '-of', 'default=noprint_wrappers=1', masterPath]);+ const line = stdout.split(/\r?\n/).find((row) => row.toLowerCase().startsWith(`tag:${MASTER_FINGERPRINT_TAG}=`));+ return line ? line.slice(line.indexOf('=') + 1).trim() || null : null;+};+// The playable geometry of a finished clip — `{ width, height, fps, numFrames,// durationSec }` — or null when any axis cannot be measured. For entries whose// producer never recorded a frame count (hosted Grok/fal/reactor renders), so adiff --git a/server/services/videoGen/extendSplice.js b/server/services/videoGen/extendSplice.jsindex 3299ad2700..e603d02d4c 100644--- a/server/services/videoGen/extendSplice.js+++ b/server/services/videoGen/extendSplice.js@@ -15,10 +15,11 @@*/import { existsSync } from 'fs';-import { rename, writeFile } from 'fs/promises';+import { writeFile } from 'fs/promises';import { unlinkGuarded } from '../../lib/fileUtils.js';import {- findFfmpeg, runFfmpegProcess, probeFrameCount, probeVideoStreamInfo, probeVideoDuration,+ findFfmpeg, runFfmpegProcess, probeFrameCount, probeVideoStreamInfo,+ audioFingerprint, sealAudioMaster, readAudioMasterFingerprint,probeVideoCodecSignature, videoFrameHashes,hasAudioStream, bt709TagFilter, buildExtendSpliceArgs, buildExtendNewSpanArgs, buildExtendCopyArgs,buildExtendConcatList, installEncodedVideo,@@ -29,22 +30,20 @@ import { extendMasterPath } from './extendMaster.js';/** Linear crossfade at the source/generated audio seam (~one 25 Hz audio token). */export const SEAM_CROSSFADE_SEC = 0.04;-/** How far a master's length may sit from its clip's before it is treated as stale (AAC priming/padding is ~tens of ms). */-const MASTER_DURATION_TOLERANCE_SEC = 0.15;-const partPath = (path) => path.replace(/\.flac$/i, '.part.flac');/*** The lossless master to read the SOURCE span's audio from, or null to use the- * source clip's own track. A master only counts while its length still matches- * the clip beside it: a clip whose audio was later trimmed, replaced or- * re-muxed in place leaves a master that no longer describes it.+ * source clip's own track. A master only counts while it is provably the mix the+ * clip's soundtrack was encoded from: it carries the fingerprint of that decoded+ * soundtrack, so a clip whose audio was later trimmed, replaced or re-muxed in+ * place — even to the same length — no longer matches and is ignored.*/async function usableSourceMaster(sourcePath) {const masterPath = extendMasterPath(sourcePath);if (!masterPath || !existsSync(masterPath)) return null;- const [masterSec, sourceSec] = await Promise.all([probeVideoDuration(masterPath), probeVideoDuration(sourcePath)]);- if (!masterSec || !sourceSec || Math.abs(masterSec - sourceSec) > MASTER_DURATION_TOLERANCE_SEC) {+ const [sealed, current] = await Promise.all([readAudioMasterFingerprint(masterPath), audioFingerprint(sourcePath)]);+ if (!sealed || sealed !== current) {console.warn('⚠️ Extend audio master does not match its clip, using the clip\'s own track');return null;}@@ -183,13 +182,16 @@ export async function spliceExtendSource({ sourcePath, outputPath, extendLatentsif (splicedFrames !== outputFrames) return skip(`spliced clip has ${splicedFrames ?? '?'} frames, expected ${outputFrames}`);const installed = await installEncodedVideo(tmpPath, outputPath, 'extend-spliced');if (!installed.ok) return skip(installed.reason);- // The master is an optimization: a failed install leaves the clip as its own- // (AAC) source of audio for the next round, exactly as before masters existed.+ // The master is an optimization: without one the clip stays its own (AAC)+ // source of audio for the next round, exactly as before masters existed.if (wantMaster) {- await rename(wantMaster, masterPath).catch(async (err) => {- console.warn(`⚠️ Extend audio master not kept: ${err.message}`);- await unlinkGuarded(wantMaster).catch(() => {});- });+ const fingerprint = await audioFingerprint(outputPath);+ const sealed = fingerprint ? await sealAudioMaster({ inPath: wantMaster, outPath: masterPath, fingerprint }) : { ok: false, reason: 'clip audio unreadable' };+ if (!sealed.ok) {+ console.warn(`⚠️ Extend audio master not kept: ${sealed.reason}`);+ await unlinkGuarded(masterPath).catch(() => {});+ }+ await unlinkGuarded(wantMaster).catch(() => {});}console.log(`🧵 Extend kept the original source${streamCopied ? ' (stream-copied)' : ''}: ${plan.keepFrames} source + ${plan.newFrames} new frames${audio === 'splice' ? ` (audio crossfaded at the seam${sourceMaster ? ', from the lossless master' : ''})` : ''}`);return { applied: true, keptFrames: plan.keepFrames, droppedSourceFrames: plan.droppedSourceFrames, streamCopied };diff --git a/server/services/videoGen/extendSplice.test.js b/server/services/videoGen/extendSplice.test.jsindex 2b78cc39c8..a40287a51a 100644--- a/server/services/videoGen/extendSplice.test.js+++ b/server/services/videoGen/extendSplice.test.js@@ -8,7 +8,7 @@* whenever the layout doesn't verify.*/import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest';-import { mkdtempSync, rmSync, existsSync, readFileSync } from 'fs';+import { mkdtempSync, rmSync, existsSync, readFileSync, renameSync } from 'fs';import { tmpdir } from 'os';import { join } from 'path';import { execFile } from 'child_process';@@ -144,6 +144,34 @@ describe('spliceExtendSource', () => {expect(await probeFrameCount(roundTwo)).toBe(49);});+ it('ignores a master once the clip\'s soundtrack was replaced in place, even at the same length', async () => {+ if (!ffmpeg) return;+ const source = await makeClip('swap-source.mp4', { color: 'black', frames: 17, tone: 440 });+ const roundOne = await makeClip('swap-render.mp4', { color: 'white', frames: 33, tone: 880 });+ const roundTwo = await makeClip('swap-render-2.mp4', { color: 'gray', frames: 49, tone: 660 });+ if (!source || !roundOne || !roundTwo) return;+ expect((await spliceExtendSource({ sourcePath: source, outputPath: roundOne, extendLatents: 2 })).applied).toBe(true);+ const master = extendMasterPath(roundOne);+ expect(existsSync(master)).toBe(true);+ // Swap the soundtrack, keeping the picture and the length.+ const swapped = join(tmpDir, 'swap-render.swapped.mp4');+ const remux = await runFfmpegProcess({+ bin: ffmpeg,+ args: ['-i', roundOne, '-f', 'lavfi', '-i', 'sine=frequency=1500:sample_rate=48000', '-map', '0:v', '-map', '1:a', '-c:v', 'copy', '-c:a', 'aac', '-shortest', '-y', swapped],+ stderrTailBytes: 0,+ });+ if (!remux.ok) return;+ renameSync(swapped, roundOne);+ const warn = vi.spyOn(console, 'warn').mockImplementation(() => {});++ const result = await spliceExtendSource({ sourcePath: roundOne, outputPath: roundTwo, extendLatents: 2 });+ const warned = warn.mock.calls.some(([line]) => String(line).includes('does not match'));+ warn.mockRestore();++ expect(result.applied).toBe(true);+ expect(warned).toBe(true);+ });+it('ignores a stale master and honors keepMaster:false', async () => {if (!ffmpeg) return;const source = await makeClip('stale-source.mp4', { color: 'black', frames: 17, tone: 440 });