Code-first video & image, rendered natively on WebGPU.
feat(preview): exact soundtrack blocks by continuing effect state
okanasl2cfb609573eb
diff --git a/.changeset/preview-streamed-audio.md b/.changeset/preview-streamed-audio.mdindex 44a629db..6658f079 100644--- a/.changeset/preview-streamed-audio.md+++ b/.changeset/preview-streamed-audio.md@@ -2,4 +2,4 @@"framefields": minor----The preview mixes its soundtrack in the browser, around the playhead. The page runs the export's mixer (same decoders, audio effects on its GPU, signals) on 2-second blocks as it plays, each mixed from 2 s earlier so reverbs, delays and compressors are warmed up, and plays them back to back on the AudioContext clock; the rest of the film mixes in the background for the waveform and the master gain. Sources are read in byte ranges, so pinned assets stream from their bucket and the server sends no WAV. The mixing runs in a module worker (the project's bundle, loaded as one) with its own GPU device, so it never stalls the picture; the page mixes if the worker can't, and the server only if neither can. `mixAudioTracks` and `renderAudio` take a `window` (the same samples as that stretch of a full mix) and `master: false`.+The preview mixes its soundtrack in the browser, around the playhead. The page runs the export's mixer (same decoders, audio effects on its GPU, signals) on 2-second blocks as it plays, each continuing the effect state of the block before, so from the film's start it is the export's mix sample for sample (after a seek the first block warms its effects up from 2 s earlier until the background pass replaces it), and plays them back to back on the AudioContext clock; the rest of the film mixes in the background for the waveform and the master gain. Sources are read in byte ranges, so pinned assets stream from their bucket and the server sends no WAV. The mixing runs in a module worker (the project's bundle, loaded as one) with its own GPU device, so it never stalls the picture; the page mixes if the worker can't, and the server only if neither can. `mixAudioTracks` and `renderAudio` take a `window` (the same samples as that stretch of a full mix), `continue` (carry the last mix's effect state, so back-to-back windows are exact) and `master: false`.diff --git a/packages/compositions/src/utils/audio-extractor.ts b/packages/compositions/src/utils/audio-extractor.tsindex 44414103..5090d59e 100644--- a/packages/compositions/src/utils/audio-extractor.ts+++ b/packages/compositions/src/utils/audio-extractor.ts@@ -615,6 +615,13 @@ export interface MixAudioOptions {* mixes from earlier and drops it).*/window?: { startSec: number; endSec: number };+ /**+ * Continue the effect state the last mix under this render id left,+ * instead of starting from silence. A caller mixing back-to-back windows+ * ([a, b) then [b, c)) gets exactly the samples of a full mix, effects+ * with memory included. Default false.+ */+ continue?: boolean;/*** Apply the master stage (whole-mix auto gain, soft limiting). Default* true. A windowed caller that masters once it knows the film's peak@@ -685,7 +692,8 @@ export async function mixAudioTracks(// Effects keep state on the GPU (delay lines, reverb tails) that carries// across the chunks of one mix. A mix starts from silence, also when a// caller mixes again under the same render id (the preview, block by block).- if (device) WebGPUAudioProcessor.resetState(renderId ?? "global", device);+ if (device && !options.continue)+ WebGPUAudioProcessor.resetState(renderId ?? "global", device);// Register all signals from the composition treeshaderStore.registerSignalsFromComposition(renderId, virtualMedia);diff --git a/packages/framefields/src/audio-mix-window.test.ts b/packages/framefields/src/audio-mix-window.test.tsindex 66f25f0d..810e81ea 100644--- a/packages/framefields/src/audio-mix-window.test.ts+++ b/packages/framefields/src/audio-mix-window.test.ts@@ -162,4 +162,46 @@ describe("windowed audio mix", () => {const exported = await mix(comp);expect(maxDiff(exported.channels, first.channels, 0)).toBe(0);}, 60_000);++ it("continues effect state across back-to-back windows, exactly", async () => {+ // A long, strong delay: its echoes outlast any lead-in. Windows that+ // continue the last one's state reproduce the full mix sample for sample.+ const comp = new Composition({+ width: 64,+ height: 64,+ fps: FPS,+ durationMs: 4000,+ });+ comp.addAudio(+ Layer.audio(source, { trimStartSec: 1, durationFrames: 120 }).apply(+ new AudioDelay({ delayTime: 0.3, feedback: 0.9, pingPong: true }),+ ),+ );+ const vm = comp.toVirtualMedia();+ await mix(comp);+ const device = await ensureDevice();+ const full = await mixAudioTracks(vm, FPS, SR, device, "chain", {+ master: false,+ });+ const edges = [0, 0.7, 2, 2.05, 3.3, 4];+ const parts: Float32Array[][] = [];+ for (let k = 0; k + 1 < edges.length; k++) {+ const part = await mixAudioTracks(vm, FPS, SR, device, "chain", {+ window: { startSec: edges[k], endSec: edges[k + 1] },+ master: false,+ continue: k > 0,+ });+ parts.push(part.channels);+ }+ const joined = [0, 1].map((c) => {+ const out = new Float32Array(full.channels[c].length);+ let at = 0;+ for (const p of parts) {+ out.set(p[c], at);+ at += p[c].length;+ }+ return out;+ });+ expect(maxDiff(full.channels, joined, 0)).toBe(0);+ }, 60_000);});diff --git a/packages/framefields/src/preview/audio-mixer.ts b/packages/framefields/src/preview/audio-mixer.tsindex 505c9ae1..7f37f5de 100644--- a/packages/framefields/src/preview/audio-mixer.ts+++ b/packages/framefields/src/preview/audio-mixer.ts@@ -18,6 +18,12 @@ export interface PreviewMix {sampleRate: number;/** Its loudest sample: the page masters with the film's loudest. */peak: number;+ /**+ * The same samples as that stretch of a full mix: mixed from the film's+ * start, or continuing a chain of back-to-back stretches that began+ * there. Otherwise effects with memory start from a lead-in.+ */+ exact: boolean;}export interface MixRange {@@ -39,17 +45,30 @@ export interface AudioTarget {*/const AUDIO_LEAD_IN_SEC = 2;+/** The mixing function `createAudioMixer` returns. */+export type AudioMixer = (+ range: MixRange,+ chain?: string,+) => Promise<PreviewMix>;+/*** Mixes [startSec, endSec) of the target's soundtrack with the export's* mixer: the same decoders (reading only the stretch it plays, so pinned* assets stream from their bucket), audio effects on `device` and signals.+ *+ * Each `chain` keeps its own effect state. A stretch that starts where the+ * chain's last one ended continues that state, so a chain of back-to-back+ * stretches is the full mix, sample for sample; any other starts fresh from+ * a lead-in. The page plays on one chain and fills the film in on another,+ * so neither breaks the other.*/export function createAudioMixer(target: AudioTarget,fps: number,device: GPUDevice,renderId: string,-): (range: MixRange) => Promise<PreviewMix> {+): AudioMixer {+ const chains = new Map<string, { endSec: number; exact: boolean }>();// Prepared once, as the export prepares it: a fresh tree, live signal// time finished, media timing preflighted.let prepared: Promise<VirtualMediaData> | undefined;@@ -62,31 +81,45 @@ export function createAudioMixer(await preflightMediaTime(tree, fps);return tree;};- return async (range) => {+ return async (range, chain = "main") => {prepared ??= prepare();const tree = await prepared;- const from = Math.max(0, range.startSec - AUDIO_LEAD_IN_SEC);+ const last = chains.get(chain);+ chains.delete(chain); // until this stretch is mixed+ const contiguous =+ last !== undefined && Math.abs(range.startSec - last.endSec) < 1e-6;+ const from = contiguous+ ? range.startSec+ : Math.max(0, range.startSec - AUDIO_LEAD_IN_SEC);+ const exact = contiguous ? last.exact : from === 0;const { channels, sampleRate } = await mixAudioTracks(tree,fps,48_000,device,- renderId,- { window: { startSec: from, endSec: range.endSec }, master: false },+ `${renderId}-${chain}`,+ {+ window: { startSec: from, endSec: range.endSec },+ master: false,+ continue: contiguous,+ },);+ chains.set(chain, { endSec: range.endSec, exact });const drop = Math.round((range.startSec - from) * sampleRate);const left = (channels[0] ?? new Float32Array(0)).slice(drop);const right = (channels[1] ?? channels[0] ?? left).slice(drop);let peak = 0;for (let i = 0; i < left.length; i++)peak = Math.max(peak, Math.abs(left[i]), Math.abs(right[i]));- return { left, right, sampleRate, peak };+ return { left, right, sampleRate, peak, exact };};}export interface MixRequest {id: number;range: MixRange;+ /** The effect-state chain to mix on (see `createAudioMixer`). */+ chain: string;}export type MixReply =@@ -131,10 +164,10 @@ export function startAudioMixer(// One stretch at a time, in the order asked: the page queues them.let queue: Promise<void> = Promise.resolve();scope.onmessage = (event: MessageEvent<MixRequest>) => {- const { id, range } = event.data;+ const { id, range, chain } = event.data;queue = queue.then(async () => {try {- const mix = await (await ready)(range);+ const mix = await (await ready)(range, chain);scope.postMessage({ id, mix } satisfies MixReply, [mix.left.buffer,mix.right.buffer,diff --git a/packages/framefields/src/preview/page.ts b/packages/framefields/src/preview/page.tsindex 5954457e..53c41172 100644--- a/packages/framefields/src/preview/page.ts+++ b/packages/framefields/src/preview/page.ts@@ -556,13 +556,13 @@ export const previewPage = /* html */ `<!doctype html>};});return {- mix: function (range) {+ mix: function (range, chain) {return ready.then(function () {if (failed) throw failed;return new Promise(function (resolve, reject) {var id = ++next;calls.set(id, { resolve: resolve, reject: reject });- worker.postMessage({ id: id, range: range });+ worker.postMessage({ id: id, range: range, chain: chain });});});},@@ -575,7 +575,7 @@ export const previewPage = /* html */ `<!doctype html>audio.buffer = null;audio.peaks = new Array(PEAK_BINS).fill(0);stream = { player: p, worker: mixerWorker(), blocks: new Map(), order: [], pending: new Map(), queue: [], busy: false,- seen: new Set(), peak: 0, gain: 1, fill: 0, total: Math.max(1, Math.ceil(filmSec() / BLOCK_SEC)) };+ seen: new Set(), peak: 0, gain: 1, fill: 0, filling: false, total: Math.max(1, Math.ceil(filmSec() / BLOCK_SEC)) };ensureAudioContext();setAudioStatus("mixing");var s = stream;@@ -583,65 +583,81 @@ export const previewPage = /* html */ `<!doctype html>if (stream === s && audio.state === "mixing") setAudioStatus("ready");}, function () {});}+ // Blocks mix on two chains, each continuing its own effect state from the+ // block before: "fill" runs through the film from its start, so its blocks+ // are exactly the export's; "play" starts at a seek with a lead-in and+ // carries on from there. An exact block is never replaced by a lead-in one.function streamBlock(s, i, urgent) {var have = s.blocks.get(i);if (have) return Promise.resolve(have);var waiting = s.pending.get(i);if (waiting) {- if (urgent) {- var at = s.queue.findIndex(function (j) { return j.i === i; });- if (at > 0) s.queue.unshift(s.queue.splice(at, 1)[0]);- }+ if (urgent) promote(s, i);return waiting;}- waiting = new Promise(function (resolve, reject) {- var job = { i: i, resolve: resolve, reject: reject };+ waiting = enqueue(s, i, i === s.fill ? "fill" : "play", urgent, true);+ pumpStream(s);+ return waiting;+ }+ function enqueue(s, i, chain, urgent, own) {+ if (chain === "fill") s.filling = true;+ var waiting = new Promise(function (resolve, reject) {+ var job = { i: i, chain: chain, own: own, resolve: resolve, reject: reject };if (urgent) s.queue.unshift(job); else s.queue.push(job);});- s.pending.set(i, waiting);- pumpStream(s);+ if (own) s.pending.set(i, waiting);+ else waiting.catch(function () {});return waiting;}+ function promote(s, i) {+ var at = s.queue.findIndex(function (j) { return j.i === i; });+ if (at > 0) s.queue.unshift(s.queue.splice(at, 1)[0]);+ }function pumpStream(s) {if (s.busy || stream !== s) return;if (!s.queue.length) { fillStream(s); if (!s.queue.length) return; }s.busy = true;var job = s.queue.shift();var range = { startSec: job.i * BLOCK_SEC, endSec: Math.min(filmSec(), (job.i + 1) * BLOCK_SEC) };- mixBlock(s, range).then(function (mix) {- var block = { left: mix.left, right: mix.right, sampleRate: mix.sampleRate };- if (stream === s) keepBlock(s, job.i, block, mix.peak);- job.resolve(block);+ mixBlock(s, range, job.chain).then(function (mix) {+ if (stream === s) keepBlock(s, job.i, { left: mix.left, right: mix.right, sampleRate: mix.sampleRate, exact: mix.exact }, mix.peak);+ if (job.chain === "fill") { s.fill = job.i + 1; s.filling = false; }+ job.resolve(s.blocks.get(job.i));}, function (err) {+ if (job.chain === "fill") s.filling = false;job.reject(err);streamFailed(s, err);}).then(function () {- s.pending.delete(job.i);+ if (job.own) s.pending.delete(job.i);s.busy = false;pumpStream(s);});}// In the worker; on the page itself once the worker has failed.- function mixBlock(s, range) {- if (!s.worker) return s.player.mixAudio(range);- return s.worker.mix(range).catch(function (err) {+ function mixBlock(s, range, chain) {+ if (!s.worker) return s.player.mixAudio(range, chain);+ return s.worker.mix(range, chain).catch(function (err) {if (s.worker) {console.warn("The soundtrack's mixing worker failed, so the page mixes it:", err);s.worker.stop();s.worker = null;}- return s.player.mixAudio(range);+ return s.player.mixAudio(range, chain);});}- // Idle: mix the next block not mixed yet, for the waveform and the gain.+ // Idle: the fill chain mixes its next block, every block in order, so+ // its effect state runs on unbroken from the film's start.function fillStream(s) {- while (s.fill < s.total && s.seen.has(s.fill)) s.fill++;- if (s.fill < s.total && !s.pending.has(s.fill)) streamBlock(s, s.fill, false);+ if (s.filling || s.fill >= s.total) return;+ enqueue(s, s.fill, "fill", false, !s.pending.has(s.fill));}function keepBlock(s, i, block, peak) {- s.blocks.set(i, block);- s.order.push(i);- while (s.order.length > KEEP_BLOCKS) s.blocks.delete(s.order.shift());+ var had = s.blocks.get(i);+ if (!had || !had.exact || block.exact) {+ s.blocks.set(i, block);+ s.order.push(i);+ while (s.order.length > KEEP_BLOCKS) s.blocks.delete(s.order.shift());+ }if (s.seen.has(i)) return;s.seen.add(i);s.peak = Math.max(s.peak, peak);@@ -1357,7 +1373,7 @@ export const previewPage = /* html */ `<!doctype html>buffering: stalled,audio: audio.state,// Soundtrack blocks mixed in this page so far, of the film's.- audioMixed: stream ? { blocks: stream.seen.size, of: stream.total } : null,+ audioMixed: stream ? { blocks: stream.seen.size, of: stream.total, exact: Array.from(stream.blocks.values()).filter(function (b) { return b.exact; }).length } : null,// Where the soundtrack mixes: "worker" (off the page's thread), "page", or "server".audioMixer: stream ? (stream.worker ? "worker" : "page") : audio.mode === "server" ? "server" : null,connected: !offline,diff --git a/packages/framefields/src/preview/player.ts b/packages/framefields/src/preview/player.tsindex d33b57d2..42e22913 100644--- a/packages/framefields/src/preview/player.ts+++ b/packages/framefields/src/preview/player.ts@@ -68,9 +68,13 @@ export interface PreviewPlayer {* Mixes [startSec, endSec) of the soundtrack here, with the export's* mixer: the same decoders (reading only the stretch it plays, so pinned* assets stream from their bucket), audio effects on this page's GPU and- * signals. The page mixes the blocks around its playhead as it plays.+ * signals, on `chain`'s effect state (see `createAudioMixer`). The page+ * mixes the blocks around its playhead as it plays.*/- mixAudio(range: { startSec: number; endSec: number }): Promise<PreviewMix>;+ mixAudio(+ range: { startSec: number; endSec: number },+ chain?: string,+ ): Promise<PreviewMix>;/** A frame at full size as an image, e.g. to attach to a note. */snapshot(frame: number): Promise<ImageBitmap>;/**diff --git a/plugins/framefields/skills/framefields-render/SKILL.md b/plugins/framefields/skills/framefields-render/SKILL.mdindex 64fa80e4..5c5927ee 100644--- a/plugins/framefields/skills/framefields-render/SKILL.md+++ b/plugins/framefields/skills/framefields-render/SKILL.md@@ -123,7 +123,7 @@ describe("Hero Scene Conformance", () => {---### Step 4.5: Show the User Before Exporting (`preview`)-When a person is in the loop, let them watch the composition before you spend time on the final render. `startPreview` serves a localhost player with the soundtrack, a timeline with the waveform, and frame stepping. The page loads the composition's own code and renders it with WebGPU in the browser, in real time: a device that can't draw every frame in time skips some, and the sound keeps its speed; the server only serves that code and the project's files. The page mixes the soundtrack itself with the export's mixer, in 2-second blocks around the playhead (each mixed from 2 s earlier so effects with memory are warmed up), so sound starts at once; the rest mixes in the background for the waveform and the master gain. It mixes in a background worker with its own GPU device, so it doesn't stall drawing. `framefieldsPreview.state()` reports `audioMixed` (blocks mixed) and `audioMixer` (`worker`, `page` or `server`, the fallbacks in order). Point it at the module that builds the composition:+When a person is in the loop, let them watch the composition before you spend time on the final render. `startPreview` serves a localhost player with the soundtrack, a timeline with the waveform, and frame stepping. The page loads the composition's own code and renders it with WebGPU in the browser, in real time: a device that can't draw every frame in time skips some, and the sound keeps its speed; the server only serves that code and the project's files. The page mixes the soundtrack itself with the export's mixer, in 2-second blocks around the playhead, each continuing the effect state of the block before (exactly the export's from the film's start; after a seek the first block warms its effects up from 2 s earlier until the background pass replaces it), so sound starts at once; the rest mixes in the background for the waveform and the master gain. It mixes in a background worker with its own GPU device, so it doesn't stall drawing. `framefieldsPreview.state()` reports `audioMixed` (blocks mixed, and how many are exact) and `audioMixer` (`worker`, `page` or `server`, the fallbacks in order). Point it at the module that builds the composition:```typescriptimport { startPreview } from "framefields";