Skip to content

Self-hosted, Self-Improving, Federated Everything App — dev management, agent orchestration, persistent mind, digital twin, second brain, and creative pipeline from a single dashboard over Tailscale to port the user into the OS

feat: include uploaded audio in code animation MP4 exports

[._.]/ Adam Eivy036d6e4797e3
diff --git a/client/src/components/codeAnimation/CodeAnimationPreview.jsx b/client/src/components/codeAnimation/CodeAnimationPreview.jsx
index 1b046782f9..4296b6ded3 100644
--- a/client/src/components/codeAnimation/CodeAnimationPreview.jsx
+++ b/client/src/components/codeAnimation/CodeAnimationPreview.jsx
@@ -318,7 +318,7 @@ export default function CodeAnimationPreview({ html, audioUrl, messages, audioGl
type="button"
onClick={handleExport}
disabled={exporting || exportStarting}
- title="Render every frame server-side through renderFrame(t) — no dropped frames, H.264 MP4 in Media History"
+ title="Render every frame with the selected audio track — no dropped frames, H.264 MP4 in Media History"
className={BUTTON_PRIMARY}
>
{exporting || exportStarting ? <LoaderCircle className="h-4 w-4 animate-spin" /> : <Clapperboard className="h-4 w-4" />}
diff --git a/docs/HTML_COMPOSITIONS.md b/docs/HTML_COMPOSITIONS.md
index 45f22003ce..f374305583 100644
--- a/docs/HTML_COMPOSITIONS.md
+++ b/docs/HTML_COMPOSITIONS.md
@@ -44,6 +44,10 @@ Submit `POST /api/html-composition/render` with:
shorter than the video, is trimmed to duration, and fades out over the final half
second. A missing requested track fails the job.
+Use `audioUpload: "example.wav"` instead to include a file from `data/uploads/`
+with the same loop, trim and fade behavior. Choose one audio source per render;
+missing files and unsupported audio formats fail the job.
+
For procedural music, submit `synthesizeMusic: true` instead of `musicTrack`.
Define `async portosComposition.renderAudio({ sampleRate, durationSec })` to return
an ordinary array containing exactly `Math.round(sampleRate * durationSec)` mono
@@ -60,7 +64,7 @@ write is required. Missing or invalid audio fails the job.
### Loudness mastering
Audio PortOS itself produces or mixes, a `synthesizeMusic` score or a
-`musicTrack` bed, is mastered by default (#10249). The encode first runs an
+`musicTrack` or `audioUpload` bed, is mastered by default (#10249). The encode first runs an
ffmpeg `loudnorm` measurement pass over the trimmed bed, then applies it in
linear mode inside the same encode to **-14 LUFS integrated, -1.5 dBTP, LRA 11**
(`MASTER_LOUDNESS` in `server/lib/ffmpeg.js`). A bed that measures below -50
@@ -183,7 +187,7 @@ silent contact sheet instead of a video. `proof.format` (`landscape`,
a proof request cannot also carry `formats`. The renderer applies the same contract
and launch-video gates, seeks one frame per interval (at most 60), and tiles
them six across at phone size (360px wide, 240px for vertical) into one PNG.
-A proof is silent, so the request must omit `musicTrack` and
+A proof is silent, so the request must omit `musicTrack`, `audioUpload` and
`synthesizeMusic`. Nothing is registered in Media History and no
launch-video artifacts are delivered. The completed job's result carries
`proof: { file, url, times, columns, width, height }` (plus `format` when one was requested), where `file` is
@@ -192,6 +196,13 @@ it under `/data/`; tile *n* shows `times[n]`.
Launch runs write `proofs/contact-<jobId>.png` beside the run; other
compositions write `data/composition-proofs/`. Both are excluded from backups. Code Animation exports stage `data/code-animation-exports/<job-id>/<export-id>/` (also excluded): the stored HTML with a shim that maps `ANIMATION_META`/`renderFrame(t)` onto `portosComposition`, stops the page's own `requestAnimationFrame` clock, and hides everything but the film canvas.
+Code Animation includes the saved audio automatically: Music-library and generated
+soundtrack files use `musicTrack`, while uploaded audio uses `audioUpload` (a
+filename in `data/uploads/`). Both loop and trim to the video length with the
+same half-second tail fade and loudness mastering. Missing or unsupported audio
+fails the export. Procedural Web Audio playback is still excluded from
+frame-exact exports; its export note explains how to attach a soundtrack.
+
The 202 response contains the media queue's `jobId`. Subscribe to
`GET /api/html-composition/:jobId/events` for the usual queued, started, progress,
complete, error and canceled SSE frames. Cancel through
diff --git a/server/lib/validation.js b/server/lib/validation.js
index 41baec6096..6da1b7e1a9 100644
--- a/server/lib/validation.js
+++ b/server/lib/validation.js
@@ -2915,20 +2915,21 @@ export const launchVideoStoryboardSchema = z.object({
export const htmlCompositionRenderSchema = z.object({
launchVideo: launchVideoOptionsSchema.optional(),
synthesizeMusic: z.boolean().optional(),
- // Master synthesized and library music to -14 LUFS / -1.5 dBTP (#10249); false keeps the bed as supplied.
+ // Master synthesized, library and uploaded audio to -14 LUFS / -1.5 dBTP; false keeps the bed as supplied.
masterLoudness: z.boolean().optional(),
// Overrides the page's own portosComposition.motionBlur when present (#9080).
motionBlur: htmlCompositionMotionBlurChoiceSchema.optional(),
directory: z.string().min(1).max(1024).refine(value => !value.startsWith('/') && !value.includes('\\') && !value.includes(':') && !value.split('/').some(part => part === '..' || part === '.' || !part), 'directory must be a relative path inside data'),
musicTrack: z.string().min(1).max(255).regex(/^[^/\\]+$/, 'musicTrack must be a Music-library filename').optional(),
+ audioUpload: z.string().min(1).max(255).regex(/^[^/\\\0]+$/, 'audioUpload must be an uploaded audio filename').refine(value => value !== '.' && value !== '..', 'audioUpload must be an uploaded audio filename').optional(),
// Several aspect ratios of one timeline, rendered in sequence on one page.
// Omitted, the job renders the composition's own width/height as before.
formats: launchVideoFormatListSchema.optional(),
// A proof renders a silent contact sheet for review instead of the video;
// `format` checks one declared framing, defaulting to the composition's own.
proof: z.object({ everySec: z.number().min(0.25).max(10).default(1), format: launchVideoFormatSchema.optional() }).strict().optional(),
-}).refine(value => !(value.synthesizeMusic && value.musicTrack), 'Choose synthesized music or a library track, not both')
- .refine(value => !(value.proof && (value.synthesizeMusic || value.musicTrack)), 'A proof is silent; omit synthesizeMusic and musicTrack')
+}).refine(value => [value.synthesizeMusic, value.musicTrack, value.audioUpload].filter(Boolean).length <= 1, 'Choose synthesized music, a library track, or uploaded audio')
+ .refine(value => !(value.proof && (value.synthesizeMusic || value.musicTrack || value.audioUpload)), 'A proof is silent; omit synthesizeMusic, musicTrack and audioUpload')
.refine(value => !(value.proof && value.formats), 'A proof checks one framing; use proof.format instead of formats');
// GET /api/html-composition/beats — a Music-library filename, same shape as
diff --git a/server/services/codeAnimation/export.js b/server/services/codeAnimation/export.js
index 7594b7fbb7..7c4e8cc8cb 100644
--- a/server/services/codeAnimation/export.js
+++ b/server/services/codeAnimation/export.js
@@ -19,6 +19,8 @@ import { mkdir, readdir, rm, stat } from 'fs/promises';
import { ServerError } from '../../lib/errorHandler.js';
import { PATHS } from '../../lib/paths.js';
import { atomicWrite } from '../../lib/fileUtils.js';
+import { isTopLevelEntryName, makePathResolver } from '../../lib/pathSafety.js';
+import { UPLOAD_AUDIO_EXTENSIONS } from '../../lib/mimeTypes.js';
import { getCodeAnimationJobRecord, isCodeAnimationJobId, readCodeAnimationHtml } from './jobStore.js';
import { CODE_ANIMATION_SONG_GLOBAL } from './prompt.js';
import { importsThree, injectThreeImportMap, threeStagedFiles } from './threeVendor.js';
@@ -30,6 +32,7 @@ export const EXPORT_MAX_DURATION_SEC = 120;
export const EXPORT_DIRECTORY_ROOT = 'code-animation-exports';
// A staged export older than this has long since been snapshotted by its render.
const STALE_STAGING_MS = 24 * 60 * 60 * 1000;
+const resolveAudioUpload = makePathResolver(() => PATHS.uploads, { extensions: UPLOAD_AUDIO_EXTENSIONS });
// Remove this animation's earlier stagings, keeping any a queued render may
// not have snapshotted yet.
@@ -105,12 +108,15 @@ export function injectExportShim(html, shim) {
return `${html.slice(0, at)}${tag}${html.slice(at)}`;
}
-// Only a Music-library track can be muxed; `/data/music/<file>` is the URL the
-// job stored for it.
-function musicTrackOf(job) {
- const match = /^\/data\/music\/([^/]+)$/.exec(job.audioUrl || '');
- const name = match ? decodeURIComponent(match[1]) : null;
- return name && !/[/\\]/.test(name) && name !== '..' && name !== '.' ? name : null;
+function exportAudioOf(job) {
+ if (!job.audioUrl) return {};
+ const match = /^\/(data\/music|api\/uploads)\/([^/]+)$/.exec(job.audioUrl);
+ let name;
+ try { name = match ? decodeURIComponent(match[2]) : null; } catch { /* Invalid stored URL. */ }
+ if (!isTopLevelEntryName(name)) {
+ throw new ServerError('The animation\'s audio must be a Music-library track or uploaded audio file', { status: 400, code: 'AUDIO_NOT_FOUND' });
+ }
+ return { [match[1] === 'data/music' ? 'musicTrack' : 'audioUpload']: name };
}
/**
@@ -124,10 +130,10 @@ export async function startCodeAnimationExport(id, deps = {}) {
async function startAdmittedExport(id, deps) {
const {
enqueueJob = (await import('../mediaJobQueue/index.js')).enqueueJob,
- beatGrid = async (track) => {
+ beatGrid = async (track, source = 'music') => {
const [{ resolveMusicTrackPath }, { getBeatGrid }] = await Promise.all([import('../pipeline/audioMux.js'), import('../../lib/beatGrid.js')]);
- const path = await resolveMusicTrackPath(track);
- if (!path) throw new ServerError('The animation\'s soundtrack is missing from the Music library', { status: 400, code: 'AUDIO_NOT_FOUND' });
+ const path = source === 'upload' ? resolveAudioUpload(track) : await resolveMusicTrackPath(track);
+ if (!path) throw new ServerError('The animation\'s soundtrack is missing or is not a supported audio file', { status: 400, code: 'AUDIO_NOT_FOUND' });
return getBeatGrid(path).catch(() => null);
},
} = deps;
@@ -141,9 +147,8 @@ async function startAdmittedExport(id, deps) {
throw new ServerError(`Frame-exact export supports ${EXPORT_FRAME_SIZES.join(', ')}; this animation is ${width}x${height} — use Record (real-time) instead`, { status: 400, code: 'EXPORT_SIZE_UNSUPPORTED' });
}
const html = await readCodeAnimationHtml(id);
- const musicTrack = musicTrackOf(job);
+ const audio = exportAudioOf(job);
const notes = [];
- if (job.audioUrl && !musicTrack) notes.push('Uploaded audio is not muxed; the export is silent. Attach a Music-library track to include it.');
if (!job.audioUrl && job.input?.soundtrack === 'procedural') {
notes.push('Procedural Web Audio is not rendered offline; the frame-exact export is silent. Attach a Music-library track to include audio.');
}
@@ -151,7 +156,8 @@ async function startAdmittedExport(id, deps) {
notes.push('No soundtrack was requested; the export is intentionally silent.');
}
if (durationSeconds > EXPORT_MAX_DURATION_SEC) notes.push(`The export is capped at ${EXPORT_MAX_DURATION_SEC}s; this animation runs ${durationSeconds}s.`);
- const song = musicTrack ? await beatGrid(musicTrack) : null;
+ const song = audio.musicTrack ? await beatGrid(audio.musicTrack)
+ : audio.audioUpload ? await beatGrid(audio.audioUpload, 'upload') : null;
// One staging directory per export: a queued render snapshots its directory
// only when it starts, so a later export must never rewrite an earlier one's.
const directory = `${EXPORT_DIRECTORY_ROOT}/${id}/${randomUUID()}`;
@@ -168,7 +174,7 @@ async function startAdmittedExport(id, deps) {
if (three) {
for (const { rel, data } of await threeStagedFiles()) await atomicWrite(join(dir, rel), data);
}
- queued = await enqueueJob({ kind: 'html-composition', params: { directory, ...(musicTrack ? { musicTrack } : {}) } });
+ queued = await enqueueJob({ kind: 'html-composition', params: { directory, ...audio } });
} catch (error) {
// Nothing references this staging yet (the enqueue failed or never ran), so it is litter.
await rm(dir, { recursive: true, force: true }).catch(() => {});
diff --git a/server/services/codeAnimation/export.test.js b/server/services/codeAnimation/export.test.js
index 0ef50603a8..a81d81886e 100644
--- a/server/services/codeAnimation/export.test.js
+++ b/server/services/codeAnimation/export.test.js
@@ -4,17 +4,18 @@ import { tmpdir } from 'os';
import { join } from 'path';
import vm from 'vm';
-const state = vi.hoisted(() => ({ dataRoot: null, job: null, html: '' }));
+const state = vi.hoisted(() => ({ dataRoot: null, job: null, html: '', track: null }));
vi.mock('../../lib/paths.js', async () => {
const actual = await vi.importActual('../../lib/paths.js');
- return { ...actual, PATHS: new Proxy(actual.PATHS, { get: (target, prop) => (prop === 'data' ? state.dataRoot : target[prop]) }) };
+ return { ...actual, PATHS: new Proxy(actual.PATHS, { get: (target, prop) => (prop === 'data' ? state.dataRoot : prop === 'uploads' ? join(state.dataRoot, 'uploads') : target[prop]) }) };
});
vi.mock('./jobStore.js', () => ({
isCodeAnimationJobId: (id) => /^[0-9a-f-]{36}$/.test(id),
getCodeAnimationJobRecord: async () => state.job,
readCodeAnimationHtml: async () => state.html,
}));
+vi.mock('../tracks/index.js', () => ({ getTrack: async () => state.track }));
const { startCodeAnimationExport } = await import('./export.js');
@@ -121,11 +122,42 @@ describe('startCodeAnimationExport', () => {
await expect(readFile(join(state.dataRoot, enqueueDirectory(), 'dependencies.json'), 'utf8')).rejects.toMatchObject({ code: 'ENOENT' });
});
- it('exports uploaded-audio and over-long films with explanatory notes', async () => {
- state.job = { ...state.job, audioUrl: '/api/uploads/take.wav', frame: { ...state.job.frame, durationSeconds: 180 } };
- const result = await startCodeAnimationExport(JOB_ID, { enqueueJob, beatGrid: vi.fn() });
- expect(result.notes).toHaveLength(2);
- expect(Object.keys(enqueueJob.mock.calls[0][0].params)).toEqual(['directory']);
+ it('includes uploaded audio and its beat grid, retaining the duration-cap note', async () => {
+ state.job = { ...state.job, audioUrl: '/api/uploads/Take%20A.wav', frame: { ...state.job.frame, durationSeconds: 180 } };
+ const beatGrid = vi.fn(async () => ({ bpm: 120, beats: [0.5] }));
+ const result = await startCodeAnimationExport(JOB_ID, { enqueueJob, beatGrid });
+ expect(result.notes).toEqual(['The export is capped at 120s; this animation runs 180s.']);
+ expect(beatGrid).toHaveBeenCalledWith('Take A.wav', 'upload');
+ expect(enqueueJob).toHaveBeenCalledWith({ kind: 'html-composition', params: { directory: lastDirectory, audioUpload: 'Take A.wav' } });
+ expect(await readFile(join(state.dataRoot, lastDirectory, 'index.html'), 'utf8')).toContain('"bpm":120');
+ });
+
+ it('uses a generated soundtrack in preference to the original uploaded audio', async () => {
+ state.job = { ...state.job, audioUrl: '/api/uploads/original.wav', soundtrack: { trackId: 'soundtrack-track' } };
+ state.track = { id: 'soundtrack-track', audioFilename: 'Generated Score.wav' };
+ await startCodeAnimationExport(JOB_ID, { enqueueJob, beatGrid: vi.fn(async () => null) });
+ expect(enqueueJob).toHaveBeenCalledWith({ kind: 'html-composition', params: { directory: lastDirectory, musicTrack: 'Generated Score.wav' } });
+ });
+
+ it('refuses a missing upload instead of queueing a silent export', async () => {
+ state.job = { ...state.job, audioUrl: '/api/uploads/missing.wav' };
+ await expect(startCodeAnimationExport(JOB_ID, { enqueueJob })).rejects.toMatchObject({ code: 'AUDIO_NOT_FOUND' });
+ expect(enqueueJob).not.toHaveBeenCalled();
+ });
+
+ it.each(['/api/uploads/%2e%2e%2fprivate.wav', '/api/uploads/%00.wav', '/api/uploads/%zz.wav', 'https://example.com/audio.wav'])(
+ 'refuses an invalid saved audio URL: %s', async audioUrl => {
+ state.job = { ...state.job, audioUrl };
+ await expect(startCodeAnimationExport(JOB_ID, { enqueueJob })).rejects.toMatchObject({ code: 'AUDIO_NOT_FOUND' });
+ expect(enqueueJob).not.toHaveBeenCalled();
+ },
+ );
+
+ it('keeps an animation without audio silent', async () => {
+ state.job = { ...state.job, audioUrl: null, input: { soundtrack: 'none' } };
+ const result = await startCodeAnimationExport(JOB_ID, { enqueueJob });
+ expect(result.notes).toEqual(['No soundtrack was requested; the export is intentionally silent.']);
+ expect(enqueueJob).toHaveBeenCalledWith({ kind: 'html-composition', params: { directory: lastDirectory } });
});
it('removes the staging directory when the render cannot be queued', async () => {
diff --git a/server/services/htmlComposition/index.js b/server/services/htmlComposition/index.js
index ef2f0b3e60..9c79ba06f5 100644
--- a/server/services/htmlComposition/index.js
+++ b/server/services/htmlComposition/index.js
@@ -15,8 +15,11 @@ import { resolveMusicTrackPath } from '../pipeline/audioMux.js';
import { openComposition } from './browser.js';
import { encodeComposition, encodeContactSheet, proofTimes, synthesizeCompositionMusic } from './encode.js';
import { validateLaunchVideoAssets } from '../../lib/launchVideoValidation.js';
+import { makePathResolver } from '../../lib/pathSafety.js';
+import { UPLOAD_AUDIO_EXTENSIONS } from '../../lib/mimeTypes.js';
const active = new Map();
+const resolveAudioUpload = makePathResolver(() => PATHS.uploads, { extensions: UPLOAD_AUDIO_EXTENSIONS });
const lifecycle = channel('portos.composition-render.lifecycle');
// A music-video owner may run as long as its song, and no longer than this.
const MUSIC_VIDEO_DURATION_CAP_SEC = 900;
@@ -178,14 +181,18 @@ async function renderCompositionAdmitted({ jobId, owner, audio, maxDurationSec,
const parsedInput = validateRequest(htmlCompositionRenderSchema, input);
let { directory } = parsedInput;
const sourceDirectory = directory;
- const { musicTrack, launchVideo, synthesizeMusic, proof, formats, motionBlur: blurChoice, masterLoudness } = parsedInput;
+ const { musicTrack, audioUpload, launchVideo, synthesizeMusic, proof, formats, motionBlur: blurChoice, masterLoudness } = parsedInput;
// A music-video render may still ask for the extra frames its contract
// declares in portosComposition.formats (renderTargets enforces that).
- if (musicVideo && (launchVideo || synthesizeMusic || musicTrack || proof)) {
- throw new Error('A music-video composition render cannot use launch-video, proof, or library-music options');
+ if (musicVideo && (launchVideo || synthesizeMusic || musicTrack || audioUpload || proof)) {
+ throw new Error('A music-video composition render cannot use launch-video, proof, or soundtrack options');
}
let musicPath = musicTrack ? await resolveMusicTrackPath(musicTrack) : null;
if (musicTrack && !musicPath) throw new Error('musicTrack is missing from the Music library');
+ if (audioUpload) {
+ musicPath = resolveAudioUpload(audioUpload);
+ if (!musicPath) throw new Error('audioUpload is missing or is not a supported audio file');
+ }
signal.throwIfAborted();
let launchPlan;
let launchAssets;
diff --git a/server/services/htmlComposition/index.test.js b/server/services/htmlComposition/index.test.js
index 87c0293eda..0868abfbdb 100644
--- a/server/services/htmlComposition/index.test.js
+++ b/server/services/htmlComposition/index.test.js
@@ -358,10 +358,11 @@ globalThis.portosComposition = { durationSec:1, fps:12, width:1280, height:720,
await expect(renderComposition({ ...await composition(fixture('', 'renderAudio: async () => Array(24000).fill(2),')), synthesizeMusic: true })).rejects.toThrow('finite mono PCM');
}, 30000);
- it('loops and trims library music to the video with a half-second tail fade', async () => {
- await mkdir(PATHS.music, { recursive: true });
- execFileSync(ffmpeg, ['-v', 'error', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=0.3', '-y', join(PATHS.music, 'example.wav')]);
- const result = await renderComposition({ ...await composition(), musicTrack: 'example.wav' });
+ it.each(['musicTrack', 'audioUpload'])('loops and trims %s to the video with a half-second tail fade', async source => {
+ const root = source === 'musicTrack' ? PATHS.music : PATHS.uploads;
+ await mkdir(root, { recursive: true });
+ execFileSync(ffmpeg, ['-v', 'error', '-f', 'lavfi', '-i', 'sine=frequency=440:duration=0.3', '-y', join(root, 'example.wav')]);
+ const result = await renderComposition({ ...await composition(), [source]: 'example.wav' });
const pcm = execFileSync(ffmpeg, ['-v', 'error', '-i', join(PATHS.videos, result.filename), '-vn', '-f', 'f32le', '-ac', '1', '-ar', '48000', '-']);
expect(pcm.length / 4 / 48000).toBeGreaterThanOrEqual(1);
expect(pcm.length / 4 / 48000).toBeLessThan(1.05); // AAC packet padding
@@ -370,6 +371,7 @@ globalThis.portosComposition = { durationSec:1, fps:12, width:1280, height:720,
for (let n = start; n < end; n++) sum += pcm.readFloatLE(n * 4) ** 2;
return Math.sqrt(sum / (end - start));
};
+ expect(rms(5000, 10000)).toBeGreaterThan(0.01);
expect(rms(45000, 47500)).toBeLessThan(rms(5000, 10000) * 0.2);
}, 30000);
diff --git a/server/services/htmlComposition/renderComposition.preflight.test.js b/server/services/htmlComposition/renderComposition.preflight.test.js
index ad90e342e2..425218e372 100644
--- a/server/services/htmlComposition/renderComposition.preflight.test.js
+++ b/server/services/htmlComposition/renderComposition.preflight.test.js
@@ -38,6 +38,18 @@ const { PATHS } = await import('../../lib/fileUtils.js');
const { renderComposition } = await import('./index.js');
describe('renderComposition delivery preflight', () => {
+ it.each([
+ [{ audioUpload: 'missing.wav' }, 'audioUpload is missing'],
+ [{ audioUpload: '../private.wav' }, 'Validation failed'],
+ [{ audioUpload: 'not-audio.txt' }, 'audioUpload is missing'],
+ [{ audioUpload: 'example.wav', musicTrack: 'example.wav' }, 'Validation failed'],
+ [{ audioUpload: 'example.wav', synthesizeMusic: true }, 'Validation failed'],
+ [{ audioUpload: 'example.wav', proof: { everySec: 1 } }, 'Validation failed'],
+ ])('refuses unavailable, unsafe or conflicting uploaded audio before capture: %j', async (audio, message) => {
+ await expect(renderComposition({ directory: 'compositions/example', jobId: randomUUID(), ...audio })).rejects.toThrow(message);
+ expect(encodeComposition).not.toHaveBeenCalled();
+ });
+
// Regression: the production-stage signal used to be ignored, allowing
// fixture teardown to delete source/output while its renderer still ran.
it.each([false, true])('drains an aborted renderer before settlement (cleanup refusal: %s)', async cleanupFails => {