Code-first video & image, rendered natively on WebGPU.
feat(preview): mix the soundtrack in the browser, around the playhead
okanasl786503b9a37f
diff --git a/.changeset/preview-streamed-audio.md b/.changeset/preview-streamed-audio.mdnew file mode 100644index 00000000..e085bd6b--- /dev/null+++ b/.changeset/preview-streamed-audio.md@@ -0,0 +1,5 @@+---+"framefields": minor+---++The preview mixes its soundtrack in the browser, around the playhead. The page runs the export's mixer (same decoders, audio effects on its GPU, signals) on 2-second blocks as it plays, each mixed from 2 s earlier so reverbs, delays and compressors are warmed up, and plays them back to back on the AudioContext clock; the rest of the film mixes in the background for the waveform and the master gain. Sources are read in byte ranges, so pinned assets stream from their bucket and the server sends no WAV. The server mixes only when the browser can't. `mixAudioTracks` and `renderAudio` take a `window` (the same samples as that stretch of a full mix) and `master: false`.diff --git a/CLAUDE.md b/CLAUDE.mdindex 590d6099..f4a2f0d3 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -15,7 +15,7 @@ Code-first video engine on native WebGPU (Dawn) in Node, no browser. pnpm + turb## Repo map- `packages/framefields`: public SDK. `Composition`, `Layer.*`, `LayerAnimation`, `Signal`, effects (`src/effects/classes.ts`), `FontManager`, `renderFrameGrid`. `src/index.ts` is the export surface. CLI in `src/cli/` (`init`, `add`, `list`, `fonts add|list`, `assets add|push|pull|status`, `check`, `renders`, `render [--cloud]`, `preview`; local render and preview run `cli/local-worker.ts` with the project's own tsx and framefields); `framefields.json` (`cli/config.ts`) holds the components folder, fonts, named `renders` that the CLI, `renderCloud` and the dashboard pick by name, and `storage` remotes plus sha256-pinned `assets`; `framefields/cloud` (`renderCloud`) is the GPU-free client for api.framefields.dev; `init` never prompts or overwrites, and its file templates are strings in `templates.ts` (a test parses them).- `packages/components`: the component registry `framefields add` copies into user projects (shadcn-style). `defineComponent`/`bind` contract in `src/core/contract.ts`, shared helpers in `src/core/` (keys, color, layout, surface, theme). Components import only `framefields` and `../core`. Registry items declare font needs (e.g. emoji); CLI fonts are pinned by URL, size and SHA-256 in `cli/fonts.ts`. Prop docs in `components.json` are generated (drift test). Tree-shake size budgets are tested.-- `packages/compositions`: Yoga layout (`utils/layout/resolve-layout.ts`), program schema (`program/schema.ts`), lint (`program/lint.ts`), animation/evaluator.+- `packages/compositions`: Yoga layout (`utils/layout/resolve-layout.ts`), program schema (`program/schema.ts`), lint (`program/lint.ts`), animation/evaluator, the audio mixer (`utils/audio-extractor.ts`; a `window` mixes one stretch, which the preview streams around its playhead).- `packages/webgpu-renderers`: WGSL shaders, 2D/3D pipelines, Slug text (`src/slug/`), audio DSP.- `packages/renderer`: headless Node renderer (Dawn, WebCodecs/mediabunny).- `packages/vision-sam`: SAM 2.1 Tiny/Small and SAM 3 trackers (Apache-2.0 code; SAM 3 weights need runtime SAM License acceptance), private; `framefields` bundles it as a lazy chunk (`clip-vision.ts` imports it dynamically for `comp.vision(..., { sam2 | sam3 })`; `index.ts` re-exports types only) and exposes the trackers as `framefields/sam`. No weights ship; `packages/framefields/NOTICE.md` covers their licenses. Pins in `src/models/registry.ts`.diff --git a/packages/compositions/src/utils/audio-extractor.ts b/packages/compositions/src/utils/audio-extractor.tsindex 5acb7f45..8991a6c6 100644--- a/packages/compositions/src/utils/audio-extractor.ts+++ b/packages/compositions/src/utils/audio-extractor.ts@@ -8,7 +8,11 @@ import {type VirtualMediaData,} from "@framefields/core";import { type AudioProcessor, audioRegistry } from "@framefields/node-sdk";-import { inputStore, shaderStore } from "@framefields/webgpu-renderers";+import {+ inputStore,+ resolveInputUrl,+ shaderStore,+} from "@framefields/webgpu-renderers";import { AudioSampleSink } from "mediabunny";import { computeRenderParams } from "./apply-operations.js";import { normalizeTimeline } from "./normalization.js";@@ -146,6 +150,55 @@ interface DecodedAudioRange {validToSec: number;}+/**+ * The browser's whole-file decoder, for codecs WebCodecs can't decode+ * (mediabunny reads byte ranges and decodes with WebCodecs first).+ */+async function decodeWithAudioContext(url: string): Promise<DecodedAudioRange> {+ const AudioContextClass =+ window.AudioContext ||+ (window as Window & { webkitAudioContext?: typeof AudioContext })+ .webkitAudioContext;+ if (!AudioContextClass) {+ throw new Error("AudioContext is not supported in this browser");+ }+ const audioCtx = new AudioContextClass();+ try {+ // Local paths load through the preview server, as the input store does.+ const response = await fetch(resolveInputUrl(url, false));+ mediaLogger.debug(+ `[AudioExtractor] Fetch response status for ${url}: ${response.status} (${response.statusText})`,+ );+ if (!response.ok) {+ throw new Error(`Failed to fetch audio source: ${response.statusText}`);+ }+ const arrayBuffer = await response.arrayBuffer();+ mediaLogger.debug(+ `[AudioExtractor] Fetched ${arrayBuffer.byteLength} bytes. Decoding audio data...`,+ );+ const audioBuffer = await audioCtx.decodeAudioData(arrayBuffer);+ mediaLogger.debug(+ `[AudioExtractor] Decoded successfully: ${audioBuffer.numberOfChannels} channels, sampleRate: ${audioBuffer.sampleRate}, duration: ${audioBuffer.duration}s`,+ );++ const channels: Float32Array[] = [];+ for (let c = 0; c < audioBuffer.numberOfChannels; c++) {+ channels.push(new Float32Array(audioBuffer.getChannelData(c)));+ }+ // decodeAudioData only decodes whole files.+ return {+ channels,+ sampleRate: audioBuffer.sampleRate,+ startIndex: 0,+ validFromSec: 0,+ validToSec: Number.POSITIVE_INFINITY,+ };+ } finally {+ void audioCtx.close();+ mediaLogger.debug(`[AudioExtractor] AudioContext closed for ${url}`);+ }+}+/*** Decodes [range.startSec, range.endSec) of a source (the whole file without* a range), so a short clip from a long file decodes only what it plays.@@ -166,57 +219,10 @@ async function decodeAudioRange(process.versions &&process.versions.node);- if (isBrowser) {- mediaLogger.debug(- `[AudioExtractor] Browser detected. Fetching arrayBuffer for ${url}...`,- );- const AudioContextClass =- window.AudioContext || (window as any).webkitAudioContext;- if (!AudioContextClass) {- throw new Error("AudioContext is not supported in this browser");- }- const audioCtx = new AudioContextClass();- try {- const response = await fetch(url);- mediaLogger.debug(- `[AudioExtractor] Fetch response status for ${url}: ${response.status} (${response.statusText})`,- );- if (!response.ok) {- throw new Error(- `Failed to fetch audio source: ${response.statusText}`,- );- }- const arrayBuffer = await response.arrayBuffer();- mediaLogger.debug(- `[AudioExtractor] Fetched ${arrayBuffer.byteLength} bytes. Decoding audio data...`,- );- const audioBuffer = await audioCtx.decodeAudioData(arrayBuffer);- mediaLogger.debug(- `[AudioExtractor] Decoded successfully: ${audioBuffer.numberOfChannels} channels, sampleRate: ${audioBuffer.sampleRate}, duration: ${audioBuffer.duration}s`,- );-- const channels: Float32Array[] = [];- for (let c = 0; c < audioBuffer.numberOfChannels; c++) {- channels.push(new Float32Array(audioBuffer.getChannelData(c)));- }- // decodeAudioData only decodes whole files.- return {- channels,- sampleRate: audioBuffer.sampleRate,- startIndex: 0,- validFromSec: 0,- validToSec: Number.POSITIVE_INFINITY,- };- } finally {- void audioCtx.close();- mediaLogger.debug(`[AudioExtractor] AudioContext closed for ${url}`);- }- }-let inputAcquired = false;try {mediaLogger.debug(- `[AudioExtractor] Server / Node.js detected. Acquiring from inputStore for ${url}...`,+ `[AudioExtractor] Acquiring from inputStore for ${url}...`,);const input = await inputStore.acquire(url);inputAcquired = true;@@ -359,6 +365,8 @@ async function decodeAudioRange() {throw err;}+ // Codecs the browser's WebCodecs can't decode: its own whole-file decoder.+ if (isBrowser) return decodeWithAudioContext(url);if (typeof process !== "undefined" && process.versions?.node) {try {const { execFileSync } = await import("node:child_process");@@ -481,6 +489,7 @@ async function resampleAndStretchPCM(outputLength: number,trimOffsetSamples: number,inputStartIndex = 0,+ firstOutputSample = 0,): Promise<Float32Array> {const output = new Float32Array(outputLength);const resampleRatio = sourceSampleRate / targetSampleRate;@@ -494,7 +503,8 @@ async function resampleAndStretchPCM(}// Subtracting the integer start after the sum keeps the position exact,// so a range resamples bit-for-bit as the whole file would.- const srcIndex = trimOffsetSamples + i * factor - inputStartIndex;+ const srcIndex =+ trimOffsetSamples + (firstOutputSample + i) * factor - inputStartIndex;const indexFloor = Math.floor(srcIndex);const indexCeil = Math.ceil(srcIndex);@@ -589,6 +599,46 @@ export interface AudioMixProgress {export interface MixAudioOptions {onProgress?: (progress: AudioMixProgress) => void;+ /**+ * Mixes only [startSec, endSec) of the timeline, reading and processing+ * only what plays there: the same samples as that stretch of a full mix+ * (effects that carry state need lead-in before `startSec`; the caller+ * mixes from earlier and drops it).+ */+ window?: { startSec: number; endSec: number };+ /**+ * Apply the master stage (whole-mix auto gain, soft limiting). Default+ * true. A windowed caller that masters once it knows the film's peak+ * passes false and reads `peak`.+ */+ master?: boolean;+}++export interface AudioMix {+ channels: Float32Array[];+ sampleRate: number;+ /** The loudest sample before the master stage. */+ peak: number;+}++/**+ * The master stage, in place: `gain` (the automatic ceiling: a mix whose+ * summed tracks exceed -0.5 dB, 0.95, is scaled down smoothly), then soft+ * limiting and non-finite sanitation (transparent saturation instead of+ * harsh square-wave clipping).+ */+export function applyMaster(channels: Float32Array[], gain: number): void {+ for (const channel of channels) {+ for (let i = 0; i < channel.length; i++) {+ const v = (channel[i] ?? 0) * gain;+ channel[i] = Number.isFinite(v) ? Math.tanh(v) : 0;+ }+ }+}++/** The master stage's gain for a mix whose loudest sample is `peak`. */+export function masterGain(peak: number): number {+ return peak > 0.95 ? 0.92 / peak : 1.0;}/** A clip's file name for progress, without paths or signed-URL queries. */@@ -617,7 +667,7 @@ export async function mixAudioTracks(device?: GPUDevice,renderId?: string,options: MixAudioOptions = {},-): Promise<{ channels: Float32Array[]; sampleRate: number }> {+): Promise<AudioMix> {mediaLogger.debug(`[AudioExtractor] mixAudioTracks started. FPS: ${fps}, targetSampleRate: ${targetSampleRate}`,);@@ -629,6 +679,24 @@ export async function mixAudioTracks(const durationMs = getActiveMediaMetadata(virtualMedia)?.durationMs || 1000;const totalDurationSec = durationMs / 1000;const totalTargetSamples = Math.ceil(totalDurationSec * targetSampleRate);+ // The window, in timeline samples: each node mixes only its overlap.+ const winStart = options.window+ ? Math.max(0, Math.round(options.window.startSec * targetSampleRate))+ : 0;+ const winEnd = options.window+ ? Math.min(+ totalTargetSamples,+ Math.round(options.window.endSec * targetSampleRate),+ )+ : Number.POSITIVE_INFINITY;+ const outputSamples = options.window+ ? Math.max(0, winEnd - winStart)+ : totalTargetSamples;+ /** The part [lo, hi) of a node's local samples that the window covers. */+ const visible = (absStart: number, length: number) => ({+ lo: Math.max(0, winStart - absStart),+ hi: Math.min(length, winEnd - absStart),+ });mediaLogger.debug(`[AudioExtractor] Composition total duration: ${durationMs}ms (${totalDurationSec}s). Total mixed samples to allocate: ${totalTargetSamples}`,);@@ -672,12 +740,13 @@ export async function mixAudioTracks(length: number,map: ResolvedMediaTimeRemap,startFrame: number,+ skip = 0,): Promise<Float32Array[] | null> {const left = new Float32Array(length),right = new Float32Array(length);for (let offset = 0; offset < length; ) {let count = Math.min(length - offset, targetSampleRate * 2);- const first = startFrame + (offset / targetSampleRate) * fps;+ const first = startFrame + ((skip + offset) / targetSampleRate) * fps;let range = mediaSourceRange(map,first,@@ -764,6 +833,7 @@ export async function mixAudioTracks(inheritedSpeed: number,inheritedDurationFrames: number | null,inheritedRemap?: { map: ResolvedMediaTimeRemap; offsetFrames: number },+ absStart = 0,): Promise<Float32Array[] | null> {if (!node || !isAudible(node)) return null;const op = node.operation;@@ -783,10 +853,18 @@ export async function mixAudioTracks((actualDurationFrames / fps) * targetSampleRate,);if (localSamplesCount <= 0) return null;+ const { lo, hi } = visible(absStart, localSamplesCount);+ if (hi <= lo) return null;- const bus = new MixBus(localSamplesCount);+ const bus = new MixBus(hi - lo);for (const child of node.children) {+ const childOp = child.operation;+ const childStartFrame =+ childOp?.startFrame ?? childOp?.timeline?.startFrame ?? 0;+ const childStartSample = Math.round(+ (childStartFrame / fps) * targetSampleRate,+ );const childChannels = await traverse(child,inheritedSeekSec,@@ -795,16 +873,11 @@ export async function mixAudioTracks(inheritedSpeed,actualDurationFrames,inheritedRemap,+ absStart + childStartSample,);- if (childChannels) {- const childOp = child.operation;- const childStartFrame =- childOp?.startFrame ?? childOp?.timeline?.startFrame ?? 0;- const childStartSample = Math.round(- (childStartFrame / fps) * targetSampleRate,- );- bus.add(childChannels, childStartSample);- }+ // A child's buffer starts where the window enters it.+ if (childChannels)+ bus.add(childChannels, Math.max(childStartSample, lo) - lo);}return bus.channels;}@@ -864,8 +937,10 @@ export async function mixAudioTracks((nodeActualDurationFrames / fps) * targetSampleRate,);if (localSamplesCount <= 0) return null;+ const { lo, hi } = visible(absStart, localSamplesCount);+ if (hi <= lo) return null;- const bus = new MixBus(localSamplesCount);+ const bus = new MixBus(hi - lo);const processSegment = async (seg: { startSec: number; endSec?: number } | null,@@ -919,25 +994,31 @@ export async function mixAudioTracks(localSamplesCount - segmentStartSample,Math.ceil(clipDurationSec * targetSampleRate),);+ // The part of the clip inside the window: `skip` samples in,+ // `count` long, written at `writeAt` in the bus.+ const clipFrom = Math.max(segmentStartSample, lo);+ const clipTo = Math.min(segmentStartSample + clipSamplesCount, hi);+ const count = clipTo - clipFrom;+ const skip = clipFrom - segmentStartSample;+ const writeAt = clipFrom - lo;const mapped =- remap && clipSamplesCount > 0+ remap && count > 0? await mappedChannels(params.sourceUrl,- clipSamplesCount,+ count,remap.map,remap.offsetFrames + windowStartFrameOffset,+ skip,): null;const decoded =- !remap &&- clipSamplesCount > 0 &&- segmentStartSample < localSamplesCount+ !remap && count > 0 && segmentStartSample < localSamplesCount? await getDecodedRange(params.sourceUrl,- effectiveTrimSec,+ effectiveTrimSec + (skip / targetSampleRate) * finalSpeed,effectiveTrimSec +- (clipSamplesCount / targetSampleRate) * finalSpeed ++ ((skip + count) / targetSampleRate) * finalSpeed +RANGE_MARGIN_SEC,): null;@@ -955,9 +1036,10 @@ export async function mixAudioTracks(decoded.sampleRate,targetSampleRate,finalSpeed,- clipSamplesCount,+ count,trimOffsetSamples,decoded.startIndex,+ skip,);stretchedRight =decoded.channels.length > 1@@ -966,23 +1048,24 @@ export async function mixAudioTracks(decoded.sampleRate,targetSampleRate,finalSpeed,- clipSamplesCount,+ count,trimOffsetSamples,decoded.startIndex,+ skip,): stretchedLeft;} else {throw new Error("No decoded audio");}const stretched = [stretchedLeft, stretchedRight];- if (bus.canAdopt(stretched, segmentStartSample)) {+ if (bus.canAdopt(stretched, writeAt)) {// The clip fills the silent bus: scale it in place and keep itif (volume !== 1) {- for (let i = 0; i < clipSamplesCount; i++) {+ for (let i = 0; i < count; i++) {stretchedLeft[i] *= volume;}if (stretchedRight !== stretchedLeft) {- for (let i = 0; i < clipSamplesCount; i++) {+ for (let i = 0; i < count; i++) {stretchedRight[i] *= volume;}}@@ -996,8 +1079,8 @@ export async function mixAudioTracks(} else {// Add to local mix with volume scalingconst [localLeft, localRight] = bus.get();- for (let i = 0; i < clipSamplesCount; i++) {- const outIndex = segmentStartSample + i;+ for (let i = 0; i < count; i++) {+ const outIndex = writeAt + i;localLeft[outIndex] += stretchedLeft[i] * volume;localRight[outIndex] += stretchedRight[i] * volume;}@@ -1018,6 +1101,16 @@ export async function mixAudioTracks(// Traverse children recursivelyif (node.children) {for (const child of node.children) {+ const childOp = child.operation;+ const childStartFrame =+ childOp?.startFrame ?? childOp?.timeline?.startFrame ?? 0;+ const relativeStartFrame =+ op.op === "CompositorLayer" && childOp?.op === "CompositorLayer"+ ? childStartFrame - nodeStartFrame+ : childStartFrame;+ const childStart =+ segmentStartSample ++ Math.round((relativeStartFrame / fps) * targetSampleRate);const childChannels = await traverse(child,totalSeekSec,@@ -1026,20 +1119,10 @@ export async function mixAudioTracks(speedMultiplier,actualDurationFrames,remap,+ absStart + childStart,);- if (childChannels) {- const childOp = child.operation;- const childStartFrame =- childOp?.startFrame ?? childOp?.timeline?.startFrame ?? 0;- const relativeStartFrame =- op.op === "CompositorLayer" && childOp?.op === "CompositorLayer"- ? childStartFrame - nodeStartFrame- : childStartFrame;- const childStartSample = Math.round(- (relativeStartFrame / fps) * targetSampleRate,- );- bus.add(childChannels, segmentStartSample + childStartSample);- }+ if (childChannels)+ bus.add(childChannels, Math.max(childStart, lo) - lo);}}@@ -1053,34 +1136,35 @@ export async function mixAudioTracks(Math.ceil(clipDurationSec * targetSampleRate),);+ // The part of the segment inside the window, `skip` samples in.+ const from = Math.max(segmentStartSample, lo);+ const to = Math.min(segmentStartSample + clipSamplesCount, hi);+ const skip = from - segmentStartSample;if (- clipSamplesCount > 0 &&+ to > from &&segmentStartSample < localSamplesCount &&segmentStartSample >= 0) {const slicedChannels = bus.get()- .map((ch) =>- ch.subarray(- segmentStartSample,- segmentStartSample + clipSamplesCount,- ),- );+ .map((ch) => ch.subarray(from - lo, to - lo));// The layer's own clock: its in-point moves the source, not its// animation, and a CompositorLayer's keyframes count from its// own first frame (as its picture's do), not the composition's.const layerClockFrames =op.op === "CompositorLayer" ? localClockFrames : 0;+ const skipSec = skip / targetSampleRate;const elapsedMs =((totalClockFrames - layerClockFrames) / fps +totalSeekSec -- layerTrimSec) *+ layerTrimSec ++ skipSec) *1000;const durationMs =getActiveMediaMetadata(node)?.durationMs ?? undefined;await pluginProcessor(slicedChannels, targetSampleRate, node, {device,- frame: totalClockFrames,+ frame: totalClockFrames + skipSec * fps,fps,renderId,elapsedMs,@@ -1136,13 +1220,10 @@ export async function mixAudioTracks(// Output mixed buffer is stereo; a root mix of the same length is used as isconst mixedChannels =- rootChannels?.every((ch) => ch.length === totalTargetSamples) &&+ rootChannels?.every((ch) => ch.length === outputSamples) &&rootChannels[0] !== rootChannels[1]? rootChannels- : [- new Float32Array(totalTargetSamples),- new Float32Array(totalTargetSamples),- ];+ : [new Float32Array(outputSamples), new Float32Array(outputSamples)];const [outputLeft, outputRight] = mixedChannels;if (rootChannels && mixedChannels !== rootChannels) {for (let c = 0; c < 2; c++) {@@ -1160,26 +1241,18 @@ export async function mixAudioTracks(// Calculate master mix peak across all channels to prevent multi-track summation distortionlet mixPeak = 0;- for (let i = 0; i < totalTargetSamples; i++) {+ for (let i = 0; i < outputSamples; i++) {const l = Math.abs(outputLeft[i] ?? 0);const r = Math.abs(outputRight[i] ?? 0);if (l > mixPeak) mixPeak = l;if (r > mixPeak) mixPeak = r;}- // Automatic gain ceiling: if multi-track summation exceeds -0.5 dB (0.95), scale smoothly- const autoGain = mixPeak > 0.95 ? 0.92 / mixPeak : 1.0;-- // Soft-knee limiting & non-finite float sanitation (transparent saturation instead of harsh square-wave clipping)- for (let i = 0; i < totalTargetSamples; i++) {- const l = (outputLeft[i] ?? 0) * autoGain;- const r = (outputRight[i] ?? 0) * autoGain;- outputLeft[i] = Number.isFinite(l) ? Math.tanh(l) : 0;- outputRight[i] = Number.isFinite(r) ? Math.tanh(r) : 0;- }+ if (options.master !== false) applyMaster(mixedChannels, masterGain(mixPeak));return {channels: mixedChannels,sampleRate: targetSampleRate,+ peak: mixPeak,};}diff --git a/packages/framefields/src/audio-mix-window.test.ts b/packages/framefields/src/audio-mix-window.test.tsnew file mode 100644index 00000000..675ac64f--- /dev/null+++ b/packages/framefields/src/audio-mix-window.test.ts@@ -0,0 +1,135 @@+/**+ * A windowed mix (what the preview plays around its playhead) is the same+ * samples as that stretch of a full mix. Effects that carry state match once+ * the window is mixed from far enough before it (the lead-in).+ */+import fs from "node:fs/promises";+import os from "node:os";+import path from "node:path";+import { beforeAll, describe, expect, it } from "vitest";+import { AudioFade, AudioReverb } from "./effects/index.js";+import { Composition, encodeStereoWav, Layer } from "./index.js";+import { HeadlessMediaRenderer } from "./renderer/index.js";++const SR = 48000;+const FPS = 30;++function noise(seconds: number, seed: number): Float32Array {+ const out = new Float32Array(Math.round(seconds * SR));+ let s = seed;+ for (let i = 0; i < out.length; i++) {+ s = (s * 1103515245 + 12345) >>> 0;+ out[i] = ((s / 2 ** 32) * 2 - 1) * 0.3;+ }+ return out;+}++let source = "";+beforeAll(async () => {+ const dir = await fs.mkdtemp(path.join(os.tmpdir(), "audio-mix-window-"));+ source = path.join(dir, "source.wav");+ await fs.writeFile(+ source,+ encodeStereoWav([noise(12, 1), noise(12, 2)], { sampleRate: SR }),+ );+});++function film(effects: "none" | "fade" | "reverb" = "none") {+ const comp = new Composition({+ width: 64,+ height: 64,+ fps: FPS,+ durationMs: 4000,+ });+ const first = Layer.audio(source, {+ trimStartSec: 2.25,+ startFrame: 7,+ durationFrames: 70,+ });+ if (effects === "fade")+ first.apply(new AudioFade({ fadeInDuration: 0.8, fadeOutDuration: 0.6 }));+ if (effects === "reverb") first.apply(new AudioReverb({}));+ comp.addAudio(first);+ comp.addAudio(+ Layer.audio(source, {+ trimStartSec: 6,+ startFrame: 40,+ durationFrames: 60,+ volume: 0.5,+ }),+ );+ comp.addAudio(+ Layer.audio(source, {+ startFrame: 85,+ durationFrames: 30,+ timeRemap: { kind: "rate", sourceStartSec: 9, rate: -0.75 },+ }),+ );+ return comp;+}++const renderer = new HeadlessMediaRenderer();+const mix = (+ comp: Composition,+ window?: { startSec: number; endSec: number },+) =>+ renderer.renderAudio(comp.toVirtualMedia(), {+ fps: FPS,+ sampleRate: SR,+ window,+ master: false,+ });++/** The largest difference between a window and the same stretch of `full`. */+function maxDiff(full: Float32Array[], part: Float32Array[], startSec: number) {+ const offset = Math.round(startSec * SR);+ let max = 0;+ for (let c = 0; c < 2; c++)+ for (let i = 0; i < part[c].length; i++)+ max = Math.max(max, Math.abs(part[c][i] - full[c][offset + i]));+ return max;+}++describe("windowed audio mix", () => {+ it("is the same samples as that stretch of the full mix", async () => {+ const comp = film();+ const full = await mix(comp);+ for (const [startSec, endSec] of [+ [0, 4],+ [0.37, 1.13],+ [1.2, 1.4],+ [2.9, 3.8],+ [3.5, 4],+ ]) {+ const part = await mix(comp, { startSec, endSec });+ expect(part.channels[0].length).toBe(+ Math.round(endSec * SR) - Math.round(startSec * SR),+ );+ expect(maxDiff(full.channels, part.channels, startSec)).toBeLessThan(+ 1e-6,+ );+ }+ }, 60_000);++ it("keeps time-based effects on the clip's own clock", async () => {+ const comp = film("fade");+ const full = await mix(comp);+ const part = await mix(comp, { startSec: 0.5, endSec: 2.5 });+ expect(maxDiff(full.channels, part.channels, 0.5)).toBeLessThan(1e-6);+ }, 60_000);++ it("matches effects with state after a lead-in", async () => {+ const comp = film("reverb");+ const full = await mix(comp);+ const lead = 2;+ const [startSec, endSec] = [2.5, 3.5];+ const part = await mix(comp, { startSec: startSec - lead, endSec });+ const tail = part.channels.map((ch) => ch.subarray(Math.round(lead * SR)));+ expect(maxDiff(full.channels, tail, startSec)).toBeLessThan(1e-6);+ // Without it the reverb starts cold at the window's edge.+ const cold = await mix(comp, { startSec, endSec });+ expect(maxDiff(full.channels, cold.channels, startSec)).toBeGreaterThan(+ 1e-3,+ );+ }, 60_000);+});diff --git a/packages/framefields/src/preview/index.ts b/packages/framefields/src/preview/index.tsindex a377e56b..6f8febe6 100644--- a/packages/framefields/src/preview/index.ts+++ b/packages/framefields/src/preview/index.ts@@ -19,6 +19,7 @@ import {} from "./bundle.js";import { describeNote, type NewNote, NoteStore } from "./notes.js";import { previewPage } from "./page.js";+import { computePeaks, PEAK_BINS, SILENT_PEAK } from "./peaks.js";/** Where the composition comes from: the module that builds it. */export interface PreviewSource {@@ -122,12 +123,13 @@ interface PreviewMeta {}type AudioStatus =+ /** The page mixes it (the default); the server only when the page asks. */+ | { state: "browser" }| { state: "mixing" }| { state: "ready"; durationSec: number; peaks: number[] }| { state: "none" }| { state: "error"; error: string };-const PEAK_BINS = 1200;/** Requests carrying this header come from another preview process, not a page. */const TAKEOVER_HEADER = "x-framefields-preview";const PLAYER_PATH = "/@framefields/player.js";@@ -232,9 +234,11 @@ export async function startPreview(const engine = engineDir();engine.catch(() => {});- // ── Audio: mixed once, by the export engine ──+ // ── Audio: the page mixes it with the export's mixer, reading sources+ // itself (straight from their bucket, for pinned assets). The server mixes+ // only when the page can't, and sends the mix as a WAV. ──let audio: AudioStatus = {- state: options.audio === false ? "none" : "mixing",+ state: options.audio === false ? "none" : "browser",};let audioWav: Buffer | undefined;@@ -268,7 +272,7 @@ export async function startPreview(const left = channels[0] ?? new Float32Array(0);const right = channels[1] ?? left;const peaks = computePeaks(left, right, PEAK_BINS);- if (left.length === 0 || Math.max(...peaks) < 0.001) {+ if (left.length === 0 || Math.max(...peaks) < SILENT_PEAK) {setAudio({ state: "none" });return;}@@ -468,6 +472,15 @@ export async function startPreview(return;}}+ // The page couldn't mix the soundtrack: mix it here instead.+ if (req.method === "POST" && url.pathname === "/audio/mix") {+ if (audio.state === "browser" || audio.state === "error") {+ audio = { state: "mixing" };+ startAudio();+ }+ sendJson(res, 202, audio);+ return;+ }if (req.method === "GET" && url.pathname === "/audio") {if (audio.state !== "ready" || !audioWav) {sendJson(res, 404, { error: `audio is ${audio.state}` });@@ -558,9 +571,8 @@ export async function startPreview(const { port: actualPort } = server.address() as AddressInfo;const url = `http://${host.includes(":") ? `[${host}]` : host}:${actualPort}/`;- // Build the page's bundle and mix the soundtrack while the browser opens.+ // Build the page's bundle while the browser opens.void playerBundle().catch(() => {});- startAudio();if (options.open === true) {// A tab from the last preview reconnects within ~1 s and reloads itself;// only open a new one when nobody comes back.@@ -836,27 +848,6 @@ async function requestTakeover(host: string, port: number): Promise<boolean> {}/** Max absolute amplitude of both channels per bin, 0..1. */-function computePeaks(- left: Float32Array,- right: Float32Array,- bins: number,-): number[] {- const peaks = new Array<number>(bins).fill(0);- if (left.length === 0) return peaks;- const per = left.length / bins;- for (let b = 0; b < bins; b++) {- const start = Math.floor(b * per);- const end = Math.min(left.length, Math.floor((b + 1) * per));- let max = 0;- for (let i = start; i < end; i++) {- const v = Math.max(Math.abs(left[i] ?? 0), Math.abs(right[i] ?? 0));- if (v > max) max = v;- }- peaks[b] = Math.min(1, max);- }- return peaks;-}-function sendJson(res: http.ServerResponse, status: number, body: unknown) {res.writeHead(status, {"content-type": "application/json",diff --git a/packages/framefields/src/preview/page.ts b/packages/framefields/src/preview/page.tsindex 3fa752e7..3733acdb 100644--- a/packages/framefields/src/preview/page.ts+++ b/packages/framefields/src/preview/page.ts@@ -484,7 +484,7 @@ export const previewPage = /* html */ `<!doctype html>}// ── Audio ──- var audio = { state: "mixing", error: "", buffer: null, peaks: null, ctx: null, gain: null, src: null, muted: false, volume: 0.9 };+ var audio = { mode: "browser", state: "mixing", error: "", buffer: null, peaks: null, ctx: null, gain: null, src: null, muted: false, volume: 0.9 };var AUDIO_LABEL = { mixing: "Mixing audio", loading: "Loading audio", ready: "Audio ready", none: "No audio", error: "Audio failed" };var offline = false;@@ -505,26 +505,181 @@ export const previewPage = /* html */ `<!doctype html>ws.replaceChildren();if (busy) ws.append(el("span", "spinner"));ws.append(document.createTextNode(state === "none" ? "No audio in this composition" : state === "error" ? "Audio could not be mixed" : AUDIO_LABEL[state]));- var usable = state === "ready" && !!audio.buffer;+ var usable = state === "ready" && (!!audio.buffer || !!stream);$("mute").disabled = !usable;$("volume").disabled = !usable;renderChip();drawWave();}+ function ensureAudioContext() {+ if (audio.ctx) return;+ var Ctx = window.AudioContext || window.webkitAudioContext;+ audio.ctx = new Ctx();+ audio.gain = audio.ctx.createGain();+ audio.gain.connect(audio.ctx.destination);+ applyVolume();+ }+ // ── Soundtrack, streamed. The page mixes it with the export's mixer in+ // BLOCK_SEC blocks: the ones at the playhead first, as it plays, the rest+ // in the background (which draws the waveform and finds the film's loudest+ // sample for the master gain). Blocks play back to back on the+ // AudioContext clock. Sources stream straight from where they live, pinned+ // assets from their bucket. If mixing fails here, the server mixes it. ──+ var BLOCK_SEC = 2, AHEAD_SEC = 4, KEEP_BLOCKS = 64, PEAK_BINS = 1200, SILENT = 0.001;+ var stream = null, playback = null;+ function filmSec() { return meta.frameCount / meta.fps; }+ function beginStream(p) {+ stopClock();+ audio.buffer = null;+ audio.peaks = new Array(PEAK_BINS).fill(0);+ stream = { player: p, blocks: new Map(), order: [], pending: new Map(), queue: [], busy: false,+ seen: new Set(), peak: 0, gain: 1, fill: 0, total: Math.max(1, Math.ceil(filmSec() / BLOCK_SEC)) };+ ensureAudioContext();+ setAudioStatus("mixing");+ var s = stream;+ streamBlock(s, Math.floor(frame / meta.fps / BLOCK_SEC), true).then(function () {+ if (stream === s && audio.state === "mixing") setAudioStatus("ready");+ }, function () {});+ }+ function streamBlock(s, i, urgent) {+ var have = s.blocks.get(i);+ if (have) return Promise.resolve(have);+ var waiting = s.pending.get(i);+ if (waiting) {+ if (urgent) {+ var at = s.queue.findIndex(function (j) { return j.i === i; });+ if (at > 0) s.queue.unshift(s.queue.splice(at, 1)[0]);+ }+ return waiting;+ }+ waiting = new Promise(function (resolve, reject) {+ var job = { i: i, resolve: resolve, reject: reject };+ if (urgent) s.queue.unshift(job); else s.queue.push(job);+ });+ s.pending.set(i, waiting);+ pumpStream(s);+ return waiting;+ }+ function pumpStream(s) {+ if (s.busy || stream !== s) return;+ if (!s.queue.length) { fillStream(s); if (!s.queue.length) return; }+ s.busy = true;+ var job = s.queue.shift();+ var range = { startSec: job.i * BLOCK_SEC, endSec: Math.min(filmSec(), (job.i + 1) * BLOCK_SEC) };+ s.player.mixAudio(range).then(function (mix) {+ var block = { left: mix.left, right: mix.right, sampleRate: mix.sampleRate };+ if (stream === s) keepBlock(s, job.i, block, mix.peak);+ job.resolve(block);+ }, function (err) {+ job.reject(err);+ streamFailed(s, err);+ }).then(function () {+ s.pending.delete(job.i);+ s.busy = false;+ pumpStream(s);+ });+ }+ // Idle: mix the next block not mixed yet, for the waveform and the gain.+ function fillStream(s) {+ while (s.fill < s.total && s.seen.has(s.fill)) s.fill++;+ if (s.fill < s.total && !s.pending.has(s.fill)) streamBlock(s, s.fill, false);+ }+ function keepBlock(s, i, block, peak) {+ s.blocks.set(i, block);+ s.order.push(i);+ while (s.order.length > KEEP_BLOCKS) s.blocks.delete(s.order.shift());+ if (s.seen.has(i)) return;+ s.seen.add(i);+ s.peak = Math.max(s.peak, peak);+ var dur = filmSec(), from = Math.floor((i * BLOCK_SEC / dur) * PEAK_BINS);+ var to = Math.min(PEAK_BINS, Math.ceil((Math.min(dur, (i + 1) * BLOCK_SEC) / dur) * PEAK_BINS));+ var perBin = (dur / PEAK_BINS) * block.sampleRate, origin = i * BLOCK_SEC * block.sampleRate;+ for (var b = from; b < to; b++) {+ var lo = Math.max(0, Math.floor(b * perBin - origin)), hi = Math.min(block.left.length, Math.floor((b + 1) * perBin - origin));+ var m = audio.peaks[b];+ for (var k = lo; k < hi; k++) m = Math.max(m, Math.abs(block.left[k]), Math.abs(block.right[k]));+ audio.peaks[b] = Math.min(1, m);+ }+ drawWave();+ if (s.seen.size < s.total) return;+ // The whole film is mixed: master as the export does, or say it's silent.+ s.gain = s.peak > 0.95 ? 0.92 / s.peak : 1;+ if (s.peak < SILENT) setAudioStatus("none");+ }+ function streamFailed(s, err) {+ if (stream !== s) return;+ stream = null;+ stopClock();+ console.warn("Mixing the soundtrack in the browser failed, so the preview server mixes it:", err);+ audio.mode = "server";+ setAudioStatus("mixing");+ fetch("/audio/mix", { method: "POST" }).catch(function () {});+ }+ // The master stage on a block: the film's gain, then the export's soft limiter.+ function mastered(channel, gain) {+ var out = new Float32Array(channel.length);+ for (var k = 0; k < channel.length; k++) {+ var v = Math.tanh(channel[k] * gain);+ out[k] = v === v ? v : 0;+ }+ return out;+ }+ function startStream(f0) {+ var s = stream, ctx = audio.ctx;+ ctx.resume();+ var pb = { t0: f0 / meta.fps, startAt: ctx.currentTime + 0.08, next: Math.floor(f0 / meta.fps / BLOCK_SEC), sources: [], timer: 0 };+ playback = pb;+ function tick() {+ if (playback !== pb || stream !== s) return;+ var now = ctx.currentTime - pb.startAt + pb.t0;+ while (pb.next < s.total && pb.next * BLOCK_SEC < now + AHEAD_SEC) {+ var have = s.blocks.get(pb.next);+ if (!have) { streamBlock(s, pb.next, true).then(tick, function () {}); return; }+ scheduleBlock(pb, pb.next, have, s.gain);+ pb.next++;+ }+ }+ pb.timer = setInterval(tick, 250);+ tick();+ return function () { return Math.max(0, ctx.currentTime - pb.startAt); };+ }+ // Block i starts at film second i * BLOCK_SEC: on the clock, that is+ // startAt + (i * BLOCK_SEC - t0). A block that comes late starts part way+ // in, so sound and picture stay on the same clock.+ function scheduleBlock(pb, i, block, gain) {+ var ctx = audio.ctx, buf = ctx.createBuffer(2, block.left.length, block.sampleRate);+ buf.copyToChannel(mastered(block.left, gain), 0);+ buf.copyToChannel(mastered(block.right, gain), 1);+ var src = ctx.createBufferSource();+ src.buffer = buf;+ src.connect(audio.gain);+ var when = pb.startAt + (i * BLOCK_SEC - pb.t0), at = Math.max(when, ctx.currentTime + 0.02), offset = at - when;+ if (offset >= buf.duration) return;+ src.start(at, offset);+ pb.sources.push(src);+ src.onended = function () { pb.sources = pb.sources.filter(function (x) { return x !== src; }); };+ }+ function stopStream() {+ var pb = playback;+ playback = null;+ if (!pb) return;+ clearInterval(pb.timer);+ pb.sources.forEach(function (src) { try { src.stop(); } catch (e) {} src.disconnect(); });+ }function loadAudio(status) {+ if (status && status.state === "browser") {+ audio.mode = "browser";+ setAudioStatus("mixing");+ if (player) beginStream(player);+ return;+ }stopClock();audio.buffer = null;audio.peaks = null;if (!status || status.state !== "ready") { setAudioStatus(status ? status.state : "none", status && status.error); return; }audio.peaks = status.peaks;setAudioStatus("loading");- if (!audio.ctx) {- var Ctx = window.AudioContext || window.webkitAudioContext;- audio.ctx = new Ctx();- audio.gain = audio.ctx.createGain();- audio.gain.connect(audio.ctx.destination);- applyVolume();- }+ ensureAudioContext();fetch("/audio").then(function (r) { if (!r.ok) throw new Error("HTTP " + r.status); return r.arrayBuffer(); }).then(function (data) { return audio.ctx.decodeAudioData(data); })@@ -541,6 +696,7 @@ export const previewPage = /* html */ `<!doctype html>// The clock for real-time playback: the soundtrack when there is one,// otherwise the wall clock. Audio as master keeps picture and sound in sync.function startClock(f0) {+ if (stream && audio.ctx && audio.state === "ready") return startStream(f0);if (audio.buffer && audio.ctx) {audio.ctx.resume();var src = audio.ctx.createBufferSource();@@ -555,6 +711,7 @@ export const previewPage = /* html */ `<!doctype html>return function () { return (performance.now() - t0) / 1000; };}function stopClock() {+ stopStream();if (!audio.src) return;try { audio.src.stop(); } catch (e) {}audio.src.disconnect();@@ -647,6 +804,14 @@ export const previewPage = /* html */ `<!doctype html>async function playFrames(alive) {meter.start();renderFps();+ // Sound starts with picture: wait for the block at the playhead.+ if (stream && audio.state !== "none") {+ setStalled(true);+ await streamBlock(stream, Math.floor(frame / meta.fps / BLOCK_SEC), true).catch(function () {});+ if (!alive()) return;+ setStalled(false);+ if (stream && audio.state === "mixing") setAudioStatus("ready");+ }// A soundtrack still downloading would start late: wait for it.if (audio.state === "loading") {setStalled(true);@@ -1136,6 +1301,8 @@ export const previewPage = /* html */ `<!doctype html>playing: playing,buffering: stalled,audio: audio.state,+ // Soundtrack blocks mixed in this page so far, of the film's.+ audioMixed: stream ? { blocks: stream.seen.size, of: stream.total } : null,connected: !offline,// Frames shown per second over the last 2 s of playback (the last// playback's while paused), and how long the last frame took to draw.@@ -1215,6 +1382,7 @@ export const previewPage = /* html */ `<!doctype html>if (!window.framefieldsPlayer) return;window.framefieldsPlayer.then(function (p) {player = p;+ if (audio.mode === "browser") beginStream(p);$("play").disabled = false;show(frame).then(function () {pageState = "ready";diff --git a/packages/framefields/src/preview/peaks.ts b/packages/framefields/src/preview/peaks.tsnew file mode 100644index 00000000..ef5e1fa5--- /dev/null+++ b/packages/framefields/src/preview/peaks.ts@@ -0,0 +1,27 @@+/** Waveform peaks of a stereo mix: the loudest sample of either channel per bin, 0–1. */+export function computePeaks(+ left: Float32Array,+ right: Float32Array,+ bins: number,+): number[] {+ const peaks = new Array<number>(bins).fill(0);+ if (left.length === 0) return peaks;+ const per = left.length / bins;+ for (let b = 0; b < bins; b++) {+ const start = Math.floor(b * per);+ const end = Math.min(left.length, Math.floor((b + 1) * per));+ let max = 0;+ for (let i = start; i < end; i++) {+ const v = Math.max(Math.abs(left[i] ?? 0), Math.abs(right[i] ?? 0));+ if (v > max) max = v;+ }+ peaks[b] = Math.min(1, max);+ }+ return peaks;+}++/** Bins the page's waveform draws. */+export const PEAK_BINS = 1200;++/** A mix counts as silent below this peak. */+export const SILENT_PEAK = 0.001;diff --git a/packages/framefields/src/preview/player.ts b/packages/framefields/src/preview/player.tsindex 84ac2fcc..7ab0423a 100644--- a/packages/framefields/src/preview/player.ts+++ b/packages/framefields/src/preview/player.ts@@ -10,6 +10,8 @@ import { BUILTIN_NODE_RENDERERS } from "framefields-preview:renderers";import {compositionStateStore,drawCompositionTree,+ mixAudioTracks,+ preflightMediaTime,} from "@framefields/compositions";import { updateClockSignals, type VirtualMediaData } from "@framefields/core";import {@@ -46,6 +48,21 @@ export interface PlayerMeta {durationMs: number;}+/** A stretch of the soundtrack, before the master stage. */+export interface PreviewMix {+ left: Float32Array;+ right: Float32Array;+ sampleRate: number;+ /** Its loudest sample: the page masters with the film's loudest. */+ peak: number;+}++/**+ * Seconds mixed before a window and dropped, so effects that carry state+ * (reverb tails, delays, compressors) enter it warmed up, as in a full mix.+ */+const AUDIO_LEAD_IN_SEC = 2;+export interface PreviewPlayer {meta: PlayerMeta;canvas: HTMLCanvasElement;@@ -65,6 +82,13 @@ export interface PreviewPlayer {* before it plays. Skipped (resolves false) while a frame is being drawn.*/warm(frame: number): Promise<boolean>;+ /**+ * Mixes [startSec, endSec) of the soundtrack here, with the export's+ * mixer: the same decoders (reading only the stretch it plays, so pinned+ * assets stream from their bucket), audio effects on this page's GPU and+ * signals. The page mixes the blocks around its playhead as it plays.+ */+ mixAudio(range: { startSec: number; endSec: number }): Promise<PreviewMix>;/** A frame at full size as an image, e.g. to attach to a note. */snapshot(frame: number): Promise<ImageBitmap>;/**@@ -79,6 +103,8 @@ export interface PreviewPlayer {type Renderable = {toVirtualMediaAsync?: () => Promise<VirtualMediaData>;toVirtualMedia?: () => VirtualMediaData;+ /** Finishes live signal time, recorded frame by frame, before a mix. */+ prepareMediaTime?: () => Promise<void>;fps?: number;};@@ -390,6 +416,18 @@ async function createPlayer(})();}+ // The tree the soundtrack mixes from, prepared once as the export prepares+ // it: a fresh tree, live signal time finished, media timing preflighted.+ let audioTree: Promise<VirtualMediaData> | undefined;+ async function prepareAudioTree(): Promise<VirtualMediaData> {+ const tree = target.toVirtualMediaAsync+ ? await target.toVirtualMediaAsync()+ : (target.toVirtualMedia?.() ?? (vm as VirtualMediaData));+ await target.prepareMediaTime?.();+ await preflightMediaTime(tree, fps);+ return tree;+ }+const blitter = new Blitter(device, surface.colorFormat);const player: PreviewPlayer = {@@ -415,6 +453,26 @@ async function createPlayer(pump();return true;},+ async mixAudio(range) {+ audioTree ??= prepareAudioTree();+ const tree = await audioTree;+ const from = Math.max(0, range.startSec - AUDIO_LEAD_IN_SEC);+ const { channels, sampleRate } = await mixAudioTracks(+ tree,+ fps,+ 48_000,+ device,+ `${renderId}-audio`,+ { window: { startSec: from, endSec: range.endSec }, master: false },+ );+ const drop = Math.round((range.startSec - from) * sampleRate);+ const left = (channels[0] ?? new Float32Array(0)).subarray(drop);+ const right = (channels[1] ?? channels[0] ?? left).subarray(drop);+ let peak = 0;+ for (let i = 0; i < left.length; i++)+ peak = Math.max(peak, Math.abs(left[i]), Math.abs(right[i]));+ return { left, right, sampleRate, peak };+ },async snapshot(frame) {if (disposed) throw new Error("the player was disposed");while (drawing) await drawing;diff --git a/packages/framefields/src/preview/preview.test.ts b/packages/framefields/src/preview/preview.test.tsindex ffe66b11..eb7ffac8 100644--- a/packages/framefields/src/preview/preview.test.ts+++ b/packages/framefields/src/preview/preview.test.ts@@ -50,8 +50,17 @@ describe("startPreview", () => {fps: 10,frameCount: 10,width: 64,- audio: { state: "none" },+ // The page mixes the soundtrack around its playhead.+ audio: { state: "browser" },});+ // When it can't, the server mixes the whole of it on request.+ await fetch(`${session.url}audio/mix`, { method: "POST" });+ let audio: { state: string } = { state: "mixing" };+ for (let i = 0; i < 100 && audio.state === "mixing"; i++) {+ await new Promise((r) => setTimeout(r, 50));+ audio = (await (await fetch(`${session.url}meta`)).json()).audio;+ }+ expect(audio).toEqual({ state: "none" });const page = await (await fetch(session.url)).text();expect(page).toContain('id="canvas"');diff --git a/packages/renderer/src/headless-webgpu-renderer.ts b/packages/renderer/src/headless-webgpu-renderer.tsindex 6a79219b..d54f2323 100644--- a/packages/renderer/src/headless-webgpu-renderer.ts+++ b/packages/renderer/src/headless-webgpu-renderer.ts@@ -4,6 +4,7 @@ import fs from "node:fs/promises";import os from "node:os";import path from "node:path";import {+ type AudioMix,compositionStateStore,drawCompositionTree,mixAudioTracks,@@ -970,10 +971,20 @@ export class HeadlessWebGPURenderer {* Mixes the composition's full audio timeline (clips, effects, synthesized* tracks) to stereo PCM, exactly as `renderVideo` muxes it.*/+ /**+ * Mixes the timeline's audio. `window` mixes only that stretch (the same+ * samples as that part of a full mix); `master: false` skips the master+ * stage, for callers that master with the whole film's `peak` later.+ */public async renderAudio(virtualMedia: VirtualMediaData,- options: { fps?: number; sampleRate?: number } = {},- ): Promise<{ channels: Float32Array[]; sampleRate: number }> {+ options: {+ fps?: number;+ sampleRate?: number;+ window?: { startSec: number; endSec: number };+ master?: boolean;+ } = {},+ ): Promise<AudioMix> {await HeadlessWebGPURenderer.initialize();const fps = options.fps ?? (virtualMedia.metadata?.fps || 30);await preflightMediaTime(virtualMedia, fps);@@ -986,6 +997,7 @@ export class HeadlessWebGPURenderer {options.sampleRate ?? 48_000,device,`aud-${randomUUID()}`,+ { window: options.window, master: options.master },);} finally {renderSemaphore.release();diff --git a/packages/webgpu-renderers/src/input-store.ts b/packages/webgpu-renderers/src/input-store.tsindex 081840d8..092e9ed9 100644--- a/packages/webgpu-renderers/src/input-store.ts+++ b/packages/webgpu-renderers/src/input-store.ts@@ -2,7 +2,7 @@ import { ALL_FORMATS, FilePathSource, Input, UrlSource } from "mediabunny";import { isNodeRuntime, toLocalPath } from "./local-path.js";/** In the browser, local paths are served through the dev server's `/@fs`. */-function resolveInputUrl(url: string, isNode: boolean): string {+export function resolveInputUrl(url: string, isNode: boolean): string {let resolvedUrl = url;if (!isNode && typeof window !== "undefined") {if (resolvedUrl.startsWith("file://")) {diff --git a/plugins/framefields/skills/framefields-render/SKILL.md b/plugins/framefields/skills/framefields-render/SKILL.mdindex 46a6cbe6..011b8bda 100644--- a/plugins/framefields/skills/framefields-render/SKILL.md+++ b/plugins/framefields/skills/framefields-render/SKILL.md@@ -123,7 +123,7 @@ describe("Hero Scene Conformance", () => {---### Step 4.5: Show the User Before Exporting (`preview`)-When a person is in the loop, let them watch the composition before you spend time on the final render. `startPreview` serves a localhost player with the soundtrack, a timeline with the waveform, and frame stepping. The page loads the composition's own code and renders it with WebGPU in the browser, in real time: a device that can't draw every frame in time skips some, and the sound keeps its speed; the server only serves that code, the project's files, and the soundtrack mixed by the export engine. Point it at the module that builds the composition:+When a person is in the loop, let them watch the composition before you spend time on the final render. `startPreview` serves a localhost player with the soundtrack, a timeline with the waveform, and frame stepping. The page loads the composition's own code and renders it with WebGPU in the browser, in real time: a device that can't draw every frame in time skips some, and the sound keeps its speed; the server only serves that code and the project's files. The page mixes the soundtrack itself with the export's mixer, in 2-second blocks around the playhead (each mixed from 2 s earlier so effects with memory are warmed up), so sound starts at once; the rest mixes in the background for the waveform and the master gain. `framefieldsPreview.state().audioMixed` reports how many blocks are mixed. If the browser can't mix, the server mixes the whole soundtrack. Point it at the module that builds the composition:```typescriptimport { startPreview } from "framefields";