Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS
JavaScript ai-agentsdashboarddeveloper-toolsdigital-twinquantified-selfsecond-brainself-hostedtailscale
feat: include uploaded audio in code animation MP4 exports
[._.]/ Adam Eivy036d6e4797e3
diff --git a/client/src/components/codeAnimation/CodeAnimationPreview.jsx b/client/src/components/codeAnimation/CodeAnimationPreview.jsxindex 1b046782f9..4296b6ded3 100644--- a/client/src/components/codeAnimation/CodeAnimationPreview.jsx+++ b/client/src/components/codeAnimation/CodeAnimationPreview.jsx@@ -318,7 +318,7 @@ export default function CodeAnimationPreview({ html, audioUrl, messages, audioGltype="button"onClick={handleExport}disabled={exporting || exportStarting}- title="Render every frame server-side through renderFrame(t) — no dropped frames, H.264 MP4 in Media History"+ title="Render every frame with the selected audio track — no dropped frames, H.264 MP4 in Media History"className={BUTTON_PRIMARY}>{exporting || exportStarting ? <LoaderCircle className="h-4 w-4 animate-spin" /> : <Clapperboard className="h-4 w-4" />}diff --git a/docs/HTML_COMPOSITIONS.md b/docs/HTML_COMPOSITIONS.mdindex 45f22003ce..f374305583 100644--- a/docs/HTML_COMPOSITIONS.md+++ b/docs/HTML_COMPOSITIONS.md@@ -44,6 +44,10 @@ Submit `POST /api/html-composition/render` with:shorter than the video, is trimmed to duration, and fades out over the final halfsecond. A missing requested track fails the job.+Use `audioUpload: "example.wav"` instead to include a file from `data/uploads/`+with the same loop, trim and fade behavior. Choose one audio source per render;+missing files and unsupported audio formats fail the job.+For procedural music, submit `synthesizeMusic: true` instead of `musicTrack`.Define `async portosComposition.renderAudio({ sampleRate, durationSec })` to returnan ordinary array containing exactly `Math.round(sampleRate * durationSec)` mono@@ -60,7 +64,7 @@ write is required. Missing or invalid audio fails the job.### Loudness masteringAudio PortOS itself produces or mixes, a `synthesizeMusic` score or a-`musicTrack` bed, is mastered by default (#10249). The encode first runs an+`musicTrack` or `audioUpload` bed, is mastered by default (#10249). The encode first runs anffmpeg `loudnorm` measurement pass over the trimmed bed, then applies it inlinear mode inside the same encode to **-14 LUFS integrated, -1.5 dBTP, LRA 11**(`MASTER_LOUDNESS` in `server/lib/ffmpeg.js`). A bed that measures below -50@@ -183,7 +187,7 @@ silent contact sheet instead of a video. `proof.format` (`landscape`,a proof request cannot also carry `formats`. The renderer applies the same contractand launch-video gates, seeks one frame per interval (at most 60), and tilesthem six across at phone size (360px wide, 240px for vertical) into one PNG.-A proof is silent, so the request must omit `musicTrack` and+A proof is silent, so the request must omit `musicTrack`, `audioUpload` and`synthesizeMusic`. Nothing is registered in Media History and nolaunch-video artifacts are delivered. The completed job's result carries`proof: { file, url, times, columns, width, height }` (plus `format` when one was requested), where `file` is@@ -192,6 +196,13 @@ it under `/data/`; tile *n* shows `times[n]`.Launch runs write `proofs/contact-<jobId>.png` beside the run; othercompositions write `data/composition-proofs/`. Both are excluded from backups. Code Animation exports stage `data/code-animation-exports/<job-id>/<export-id>/` (also excluded): the stored HTML with a shim that maps `ANIMATION_META`/`renderFrame(t)` onto `portosComposition`, stops the page's own `requestAnimationFrame` clock, and hides everything but the film canvas.+Code Animation includes the saved audio automatically: Music-library and generated+soundtrack files use `musicTrack`, while uploaded audio uses `audioUpload` (a+filename in `data/uploads/`). Both loop and trim to the video length with the+same half-second tail fade and loudness mastering. Missing or unsupported audio+fails the export. Procedural Web Audio playback is still excluded from+frame-exact exports; its export note explains how to attach a soundtrack.+The 202 response contains the media queue's `jobId`. Subscribe to`GET /api/html-composition/:jobId/events` for the usual queued, started, progress,complete, error and canceled SSE frames. Cancel throughdiff --git a/server/lib/validation.js b/server/lib/validation.jsindex 41baec6096..6da1b7e1a9 100644--- a/server/lib/validation.js+++ b/server/lib/validation.js@@ -2915,20 +2915,21 @@ export const launchVideoStoryboardSchema = z.object({export const htmlCompositionRenderSchema = z.object({launchVideo: launchVideoOptionsSchema.optional(),synthesizeMusic: z.boolean().optional(),- // Master synthesized and library music to -14 LUFS / -1.5 dBTP (#10249); false keeps the bed as supplied.+ // Master synthesized, library and uploaded audio to -14 LUFS / -1.5 dBTP; false keeps the bed as supplied.masterLoudness: z.boolean().optional(),// Overrides the page's own portosComposition.motionBlur when present (#9080).motionBlur: htmlCompositionMotionBlurChoiceSchema.optional(),directory: z.string().min(1).max(1024).refine(value => !value.startsWith('/') && !value.includes('\\') && !value.includes(':') && !value.split('/').some(part => part === '..' || part === '.' || !part), 'directory must be a relative path inside data'),musicTrack: z.string().min(1).max(255).regex(/^[^/\\]+$/, 'musicTrack must be a Music-library filename').optional(),+ audioUpload: z.string().min(1).max(255).regex(/^[^/\\\0]+$/, 'audioUpload must be an uploaded audio filename').refine(value => value !== '.' && value !== '..', 'audioUpload must be an uploaded audio filename').optional(),// Several aspect ratios of one timeline, rendered in sequence on one page.// Omitted, the job renders the composition's own width/height as before.formats: launchVideoFormatListSchema.optional(),// A proof renders a silent contact sheet for review instead of the video;// `format` checks one declared framing, defaulting to the composition's own.proof: z.object({ everySec: z.number().min(0.25).max(10).default(1), format: launchVideoFormatSchema.optional() }).strict().optional(),-}).refine(value => !(value.synthesizeMusic && value.musicTrack), 'Choose synthesized music or a library track, not both')- .refine(value => !(value.proof && (value.synthesizeMusic || value.musicTrack)), 'A proof is silent; omit synthesizeMusic and musicTrack')+}).refine(value => [value.synthesizeMusic, value.musicTrack, value.audioUpload].filter(Boolean).length <= 1, 'Choose synthesized music, a library track, or uploaded audio')+ .refine(value => !(value.proof && (value.synthesizeMusic || value.musicTrack || value.audioUpload)), 'A proof is silent; omit synthesizeMusic, musicTrack and audioUpload').refine(value => !(value.proof && value.formats), 'A proof checks one framing; use proof.format instead of formats');// GET /api/html-composition/beats — a Music-library filename, same shape asdiff --git a/server/services/codeAnimation/export.js b/server/services/codeAnimation/export.jsindex 7594b7fbb7..7c4e8cc8cb 100644--- a/server/services/codeAnimation/export.js+++ b/server/services/codeAnimation/export.js@@ -19,6 +19,8 @@ import { mkdir, readdir, rm, stat } from 'fs/promises';import { ServerError } from '../../lib/errorHandler.js';import { PATHS } from '../../lib/paths.js';import { atomicWrite } from '../../lib/fileUtils.js';+import { isTopLevelEntryName, makePathResolver } from '../../lib/pathSafety.js';+import { UPLOAD_AUDIO_EXTENSIONS } from '../../lib/mimeTypes.js';import { getCodeAnimationJobRecord, isCodeAnimationJobId, readCodeAnimationHtml } from './jobStore.js';import { CODE_ANIMATION_SONG_GLOBAL } from './prompt.js';import { importsThree, injectThreeImportMap, threeStagedFiles } from './threeVendor.js';@@ -30,6 +32,7 @@ export const EXPORT_MAX_DURATION_SEC = 120;export const EXPORT_DIRECTORY_ROOT = 'code-animation-exports';// A staged export older than this has long since been snapshotted by its render.const STALE_STAGING_MS = 24 * 60 * 60 * 1000;+const resolveAudioUpload = makePathResolver(() => PATHS.uploads, { extensions: UPLOAD_AUDIO_EXTENSIONS });// Remove this animation's earlier stagings, keeping any a queued render may// not have snapshotted yet.@@ -105,12 +108,15 @@ export function injectExportShim(html, shim) {return `${html.slice(0, at)}${tag}${html.slice(at)}`;}-// Only a Music-library track can be muxed; `/data/music/<file>` is the URL the-// job stored for it.-function musicTrackOf(job) {- const match = /^\/data\/music\/([^/]+)$/.exec(job.audioUrl || '');- const name = match ? decodeURIComponent(match[1]) : null;- return name && !/[/\\]/.test(name) && name !== '..' && name !== '.' ? name : null;+function exportAudioOf(job) {+ if (!job.audioUrl) return {};+ const match = /^\/(data\/music|api\/uploads)\/([^/]+)$/.exec(job.audioUrl);+ let name;+ try { name = match ? decodeURIComponent(match[2]) : null; } catch { /* Invalid stored URL. */ }+ if (!isTopLevelEntryName(name)) {+ throw new ServerError('The animation\'s audio must be a Music-library track or uploaded audio file', { status: 400, code: 'AUDIO_NOT_FOUND' });+ }+ return { [match[1] === 'data/music' ? 'musicTrack' : 'audioUpload']: name };}/**@@ -124,10 +130,10 @@ export async function startCodeAnimationExport(id, deps = {}) {async function startAdmittedExport(id, deps) {const {enqueueJob = (await import('../mediaJobQueue/index.js')).enqueueJob,- beatGrid = async (track) => {+ beatGrid = async (track, source = 'music') => {const [{ resolveMusicTrackPath }, { getBeatGrid }] = await Promise.all([import('../pipeline/audioMux.js'), import('../../lib/beatGrid.js')]);- const path = await resolveMusicTrackPath(track);- if (!path) throw new ServerError('The animation\'s soundtrack is missing from the Music library', { status: 400, code: 'AUDIO_NOT_FOUND' });+ const path = source === 'upload' ? resolveAudioUpload(track) : await resolveMusicTrackPath(track);+ if (!path) throw new ServerError('The animation\'s soundtrack is missing or is not a supported audio file', { status: 400, code: 'AUDIO_NOT_FOUND' });return getBeatGrid(path).catch(() => null);},} = deps;@@ -141,9 +147,8 @@ async function startAdmittedExport(id, deps) {throw new ServerError(`Frame-exact export supports ${EXPORT_FRAME_SIZES.join(', ')}; this animation is ${width}x${height} — use Record (real-time) instead`, { status: 400, code: 'EXPORT_SIZE_UNSUPPORTED' });}const html = await readCodeAnimationHtml(id);- const musicTrack = musicTrackOf(job);+ const audio = exportAudioOf(job);const notes = [];- if (job.audioUrl && !musicTrack) notes.push('Uploaded audio is not muxed; the export is silent. Attach a Music-library track to include it.');if (!job.audioUrl && job.input?.soundtrack === 'procedural') {notes.push('Procedural Web Audio is not rendered offline; the frame-exact export is silent. Attach a Music-library track to include audio.');}@@ -151,7 +156,8 @@ async function startAdmittedExport(id, deps) {notes.push('No soundtrack was requested; the export is intentionally silent.');}if (durationSeconds > EXPORT_MAX_DURATION_SEC) notes.push(`The export is capped at ${EXPORT_MAX_DURATION_SEC}s; this animation runs ${durationSeconds}s.`);- const song = musicTrack ? await beatGrid(musicTrack) : null;+ const song = audio.musicTrack ? await beatGrid(audio.musicTrack)+ : audio.audioUpload ? await beatGrid(audio.audioUpload, 'upload') : null;// One staging directory per export: a queued render snapshots its directory// only when it starts, so a later export must never rewrite an earlier one's.const directory = `${EXPORT_DIRECTORY_ROOT}/${id}/${randomUUID()}`;@@ -168,7 +174,7 @@ async function startAdmittedExport(id, deps) {if (three) {for (const { rel, data } of await threeStagedFiles()) await atomicWrite(join(dir, rel), data);}- queued = await enqueueJob({ kind: 'html-composition', params: { directory, ...(musicTrack ? { musicTrack } : {}) } });+ queued = await enqueueJob({ kind: 'html-composition', params: { directory, ...audio } });} catch (error) {// Nothing references this staging yet (the enqueue failed or never ran), so it is litter.await rm(dir, { recursive: true, force: true }).catch(() => {});diff --git a/server/services/codeAnimation/export.test.js b/server/services/codeAnimation/export.test.jsindex 0ef50603a8..a81d81886e 100644--- a/server/services/codeAnimation/export.test.js+++ b/server/services/codeAnimation/export.test.js@@ -4,17 +4,18 @@ import { tmpdir } from 'os';import { join } from 'path';import vm from 'vm';-const state = vi.hoisted(() => ({ dataRoot: null, job: null, html: '' }));+const state = vi.hoisted(() => ({ dataRoot: null, job: null, html: '', track: null }));vi.mock('../../lib/paths.js', async () => {const actual = await vi.importActual('../../lib/paths.js');- return { ...actual, PATHS: new Proxy(actual.PATHS, { get: (target, prop) => (prop === 'data' ? state.dataRoot : target[prop]) }) };+ return { ...actual, PATHS: new Proxy(actual.PATHS, { get: (target, prop) => (prop === 'data' ? state.dataRoot : prop === 'uploads' ? join(state.dataRoot, 'uploads') : target[prop]) }) };});vi.mock('./jobStore.js', () => ({isCodeAnimationJobId: (id) => /^[0-9a-f-]{36}$/.test(id),getCodeAnimationJobRecord: async () => state.job,readCodeAnimationHtml: async () => state.html,}));+vi.mock('../tracks/index.js', () => ({ getTrack: async () => state.track }));const { startCodeAnimationExport } = await import('./export.js');@@ -121,11 +122,42 @@ describe('startCodeAnimationExport', () => {await expect(readFile(join(state.dataRoot, enqueueDirectory(), 'dependencies.json'), 'utf8')).rejects.toMatchObject({ code: 'ENOENT' });});- it('exports uploaded-audio and over-long films with explanatory notes', async () => {- state.job = { ...state.job, audioUrl: '/api/uploads/take.wav', frame: { ...state.job.frame, durationSeconds: 180 } };- const result = await startCodeAnimationExport(JOB_ID, { enqueueJob, beatGrid: vi.fn() });- expect(result.notes).toHaveLength(2);- expect(Object.keys(enqueueJob.mock.calls[0][0].params)).toEqual(['directory']);+ it('includes uploaded audio and its beat grid, retaining the duration-cap note', async () => {+ state.job = { ...state.job, audioUrl: '/api/uploads/Take%20A.wav', frame: { ...state.job.frame, durationSeconds: 180 } };+ const beatGrid = vi.fn(async () => ({ bpm: 120, beats: [0.5] }));+ const result = await startCodeAnimationExport(JOB_ID, { enqueueJob, beatGrid });+ expect(result.notes).toEqual(['The export is capped at 120s; this animation runs 180s.']);+ expect(beatGrid).toHaveBeenCalledWith('Take A.wav', 'upload');+ expect(enqueueJob).toHaveBeenCalledWith({ kind: 'html-composition', params: { directory: lastDirectory, audioUpload: 'Take A.wav' } });+ expect(await readFile(join(state.dataRoot, lastDirectory, 'index.html'), 'utf8')).toContain('"bpm":120');+ });++ it('uses a generated soundtrack in preference to the original uploaded audio', async () => {+ state.job = { ...state.job, audioUrl: '/api/uploads/original.wav', soundtrack: { trackId: 'soundtrack-track' } };+ state.track = { id: 'soundtrack-track', audioFilename: 'Generated Score.wav' };+ await startCodeAnimationExport(JOB_ID, { enqueueJob, beatGrid: vi.fn(async () => null) });+ expect(enqueueJob).toHaveBeenCalledWith({ kind: 'html-composition', params: { directory: lastDirectory, musicTrack: 'Generated Score.wav' } });+ });++ it('refuses a missing upload instead of queueing a silent export', async () => {+ state.job = { ...state.job, audioUrl: '/api/uploads/missing.wav' };+ await expect(startCodeAnimationExport(JOB_ID, { enqueueJob })).rejects.toMatchObject({ code: 'AUDIO_NOT_FOUND' });+ expect(enqueueJob).not.toHaveBeenCalled();+ });++ it.each(['/api/uploads/%2e%2e%2fprivate.wav', '/api/uploads/%00.wav', '/api/uploads/%zz.wav', 'https://example.com/audio.wav'])(+ 'refuses an invalid saved audio URL: %s', async audioUrl => {+ state.job = { ...state.job, audioUrl };+ await expect(startCodeAnimationExport(JOB_ID, { enqueueJob })).rejects.toMatchObject({ code: 'AUDIO_NOT_FOUND' });+ expect(enqueueJob).not.toHaveBeenCalled();+ },+ );++ it('keeps an animation without audio silent', async () => {+ state.job = { ...state.job, audioUrl: null, input: { soundtrack: 'none' } };+ const result = await startCodeAnimationExport(JOB_ID, { enqueueJob });+ expect(result.notes).toEqual(['No soundtrack was requested; the export is intentionally silent.']);+ expect(enqueueJob).toHaveBeenCalledWith({ kind: 'html-composition', params: { directory: lastDirectory } });});it('removes the staging directory when the render cannot be queued', async () => {diff --git a/server/services/htmlComposition/index.js b/server/services/htmlComposition/index.jsindex ef2f0b3e60..9c79ba06f5 100644--- a/server/services/htmlComposition/index.js+++ b/server/services/htmlComposition/index.js@@ -15,8 +15,11 @@ import { resolveMusicTrackPath } from '../pipeline/audioMux.js';import { openComposition } from './browser.js';import { encodeComposition, encodeContactSheet, proofTimes, synthesizeCompositionMusic } from './encode.js';import { validateLaunchVideoAssets } from '../../lib/launchVideoValidation.js';+import { makePathResolver } from '../../lib/pathSafety.js';+import { UPLOAD_AUDIO_EXTENSIONS } from '../../lib/mimeTypes.js';const active = new Map();+const resolveAudioUpload = makePathResolver(() => PATHS.uploads, { extensions: UPLOAD_AUDIO_EXTENSIONS });const lifecycle = channel('portos.composition-render.lifecycle');// A music-video owner may run as long as its song, and no longer than this.const MUSIC_VIDEO_DURATION_CAP_SEC = 900;@@ -178,14 +181,18 @@ async function renderCompositionAdmitted({ jobId, owner, audio, maxDurationSec,const parsedInput = validateRequest(htmlCompositionRenderSchema, input);let { directory } = parsedInput;const sourceDirectory = directory;- const { musicTrack, launchVideo, synthesizeMusic, proof, formats, motionBlur: blurChoice, masterLoudness } = parsedInput;+ const { musicTrack, audioUpload, launchVideo, synthesizeMusic, proof, formats, motionBlur: blurChoice, masterLoudness } = parsedInput;// A music-video render may still ask for the extra frames its contract// declares in portosComposition.formats (renderTargets enforces that).- if (musicVideo && (launchVideo || synthesizeMusic || musicTrack || proof)) {- throw new Error('A music-video composition render cannot use launch-video, proof, or library-music options');+ if (musicVideo && (launchVideo || synthesizeMusic || musicTrack || audioUpload || proof)) {+ throw new Error('A music-video composition render cannot use launch-video, proof, or soundtrack options');}let musicPath = musicTrack ? await resolveMusicTrackPath(musicTrack) : null;if (musicTrack && !musicPath) throw new Error('musicTrack is missing from the Music library');+ if (audioUpload) {+ musicPath = resolveAudioUpload(audioUpload);+ if (!musicPath) throw new Error('audioUpload is missing or is not a supported audio file');+ }signal.throwIfAborted();let launchPlan;let launchAssets;diff --git a/server/services/htmlComposition/index.test.js b/server/services/htmlComposition/index.test.jsindex 87c0293eda..0868abfbdb 100644--- a/server/services/htmlComposition/index.test.js+++ b/server/services/htmlComposition/index.test.js@@ -358,10 +358,11 @@ globalThis.portosComposition = { durationSec:1, fps:12, width:1280, height:720,await expect(renderComposition({ ...await composition(fixture('', 'renderAudio: async () => Array(24000).fill(2),')), synthesizeMusic: true })).rejects.toThrow('finite mono PCM');}, 30000);- it('loops and trims library music to the video with a half-second tail fade', async () => {- await mkdir(PATHS.music, { recursive: true });- execFileSync(ffmpeg, ['-v', 'error', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=0.3', '-y', join(PATHS.music, 'example.wav')]);- const result = await renderComposition({ ...await composition(), musicTrack: 'example.wav' });+ it.each(['musicTrack', 'audioUpload'])('loops and trims %s to the video with a half-second tail fade', async source => {+ const root = source === 'musicTrack' ? PATHS.music : PATHS.uploads;+ await mkdir(root, { recursive: true });+ execFileSync(ffmpeg, ['-v', 'error', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=0.3', '-y', join(root, 'example.wav')]);+ const result = await renderComposition({ ...await composition(), [source]: 'example.wav' });const pcm = execFileSync(ffmpeg, ['-v', 'error', '-i', join(PATHS.videos, result.filename), '-vn', '-f', 'f32le', '-ac', '1', '-ar', '48000', '-']);expect(pcm.length / 4 / 48000).toBeGreaterThanOrEqual(1);expect(pcm.length / 4 / 48000).toBeLessThan(1.05); // AAC packet padding@@ -370,6 +371,7 @@ globalThis.portosComposition = { durationSec:1, fps:12, width:1280, height:720,for (let n = start; n < end; n++) sum += pcm.readFloatLE(n * 4) ** 2;return Math.sqrt(sum / (end - start));};+ expect(rms(5000, 10000)).toBeGreaterThan(0.01);expect(rms(45000, 47500)).toBeLessThan(rms(5000, 10000) * 0.2);}, 30000);diff --git a/server/services/htmlComposition/renderComposition.preflight.test.js b/server/services/htmlComposition/renderComposition.preflight.test.jsindex ad90e342e2..425218e372 100644--- a/server/services/htmlComposition/renderComposition.preflight.test.js+++ b/server/services/htmlComposition/renderComposition.preflight.test.js@@ -38,6 +38,18 @@ const { PATHS } = await import('../../lib/fileUtils.js');const { renderComposition } = await import('./index.js');describe('renderComposition delivery preflight', () => {+ it.each([+ [{ audioUpload: 'missing.wav' }, 'audioUpload is missing'],+ [{ audioUpload: '../private.wav' }, 'Validation failed'],+ [{ audioUpload: 'not-audio.txt' }, 'audioUpload is missing'],+ [{ audioUpload: 'example.wav', musicTrack: 'example.wav' }, 'Validation failed'],+ [{ audioUpload: 'example.wav', synthesizeMusic: true }, 'Validation failed'],+ [{ audioUpload: 'example.wav', proof: { everySec: 1 } }, 'Validation failed'],+ ])('refuses unavailable, unsafe or conflicting uploaded audio before capture: %j', async (audio, message) => {+ await expect(renderComposition({ directory: 'compositions/example', jobId: randomUUID(), ...audio })).rejects.toThrow(message);+ expect(encodeComposition).not.toHaveBeenCalled();+ });+// Regression: the production-stage signal used to be ignored, allowing// fixture teardown to delete source/output while its renderer still ran.it.each([false, true])('drains an aborted renderer before settlement (cleanup refusal: %s)', async cleanupFails => {