Code-first video & image, rendered natively on WebGPU.
Play the preview in real time: draw the frame due now, skip the rest
Silvio Tomatisd70a58b9ac3c
diff --git a/.changeset/preview-realtime-playback.md b/.changeset/preview-realtime-playback.mdnew file mode 100644index 00000000..48d1e775--- /dev/null+++ b/.changeset/preview-realtime-playback.md@@ -0,0 +1,5 @@+---+"framefields": minor+---++Preview: Play runs in real time. It draws the frame due on the soundtrack's clock and skips the ones it missed, so playback starts at once and the sound never waits, however slow the device; it no longer renders 3 s ahead first and stalls whenever rendering falls behind.diff --git a/README.md b/README.mdindex 6915c8b4..f7daf8b8 100644--- a/README.md+++ b/README.md@@ -262,12 +262,13 @@ Project normalized landmarks to screen space with a configurable camera FOV, the- **FrameGrid contact sheets** — `comp.renderFrameGrid(...)` outputs sequential-frame contact sheets for instant review of easing, kinetic type, and transitions.### 8. Live Preview in the Browser-- **Runs your composition, not a video** — `startPreview({ entry, export })` serves a localhost WebGPU player that loads the composition's own module and renders every frame live in the browser. Nothing is streamed: the server only hands over the bundle, the project's assets, and the soundtrack mixed by the export engine.+- **Runs your composition, not a video** — `startPreview({ entry, export })` serves a localhost WebGPU player that loads the composition's own module and renders its frames live in the browser. Nothing is streamed: the server only hands over the bundle, the project's assets, and the soundtrack mixed by the export engine.- **Timeline, waveform & frame stepping** — play/pause, scrub, step frame by frame, and read resolution, FPS, duration, and audio status at a glance.- **One stable URL per project** — the port is derived from the working directory, so re-running the preview replaces the running server and any open tab reloads into the new version by itself. Close the tab and the server shuts down about five seconds later. A reload comes back at the same moment, and `#t=12.5` links to one.- **Notes pinned on the picture** — pause, click a spot (or drag an area) and write what should change. Each note keeps its moment, its spot and a picture of the frame with the spot marked, in `.framefields/preview-notes/`, and prints where the preview runs, so an agent can apply exactly what you meant. Tick notes off as you check them.- **Tells you what happened** — the page's errors (a failed build, a frame that threw, a missing font) and why the server stopped are printed where the preview runs, so whoever started it, person or agent, sees them without opening devtools. Scripts can drive the page through `window.framefieldsPreview` (`seek`, `play`, `pause`, `state`).- **Shown where you are** — `startPreview` serves the page and returns its URL instead of opening a browser, so an agent can show it in its own pane (Claude Code, Codex); pass `open: true` to open the system browser.+- **Plays in real time, even on a slow device** — Play draws the frame due on the soundtrack's clock and skips the ones it missed, so playback starts at once and the sound never waits. A device that can't keep up shows fewer frames.```typescriptimport { startPreview } from "framefields";diff --git a/packages/framefields/src/preview/blit.ts b/packages/framefields/src/preview/blit.tsindex 51b605ac..7a68ed8d 100644--- a/packages/framefields/src/preview/blit.ts+++ b/packages/framefields/src/preview/blit.ts@@ -2,8 +2,8 @@/// <reference types="webgpu" />/*** Copies one texture onto another of any size, with linear filtering. Kept- * apart from the 2D renderer on purpose: buffered frames are shown while the- * next frame is being drawn, and the renderer's pools must not be touched then.+ * apart from the 2D renderer on purpose: showing a cached frame must not+ * touch the renderer's pools.*/const WGSL = /* wgsl */ `diff --git a/packages/framefields/src/preview/index.ts b/packages/framefields/src/preview/index.tsindex 50a4f238..26b3c324 100644--- a/packages/framefields/src/preview/index.ts+++ b/packages/framefields/src/preview/index.ts@@ -131,7 +131,7 @@ const PLAYER_PATH = "/@framefields/player.js";/*** Serves a localhost player for a composition. The page loads the- * composition's own code, bundled with the engine, and renders every frame+ * composition's own code, bundled with the engine, and renders its frames* with WebGPU in the browser, like the editor's player; nothing is streamed.* The server provides the bundle, the project's files, and the soundtrack,* which it mixes with the export engine.diff --git a/packages/framefields/src/preview/page.ts b/packages/framefields/src/preview/page.tsindex 0819d618..c9cbcade 100644--- a/packages/framefields/src/preview/page.ts+++ b/packages/framefields/src/preview/page.ts@@ -148,11 +148,6 @@ export const previewPage = /* html */ `<!doctype html>}.tl-inner { position: relative; }.ruler { position: relative; height: 22px; border-bottom: 1px solid var(--line); }- /* Rendered ahead of the playhead while playing, like a video's loaded range. */- .buffered {- position: absolute; top: 20px; height: 3px; border-radius: 2px; background: var(--ink-3);- opacity: .55; pointer-events: none; z-index: 2; display: none;- }.tick { position: absolute; bottom: 0; width: 1px; height: 4px; background: var(--line-2); }.tick.major { height: 8px; background: var(--ink-3); }.tick span { position: absolute; bottom: 9px; left: 4px; font-size: 10.5px; line-height: 1; color: var(--ink-3); white-space: nowrap; font-variant-numeric: tabular-nums; }@@ -351,7 +346,6 @@ export const previewPage = /* html */ `<!doctype html><div class="timeline" id="timeline" aria-label="Timeline"><div class="tl-inner" id="tlInner"><div class="ruler" id="ruler"></div>- <div class="buffered" id="buffered"></div><div class="note-marks" id="noteMarks"></div><div class="wave-lane"><canvas id="wave"></canvas>@@ -441,9 +435,6 @@ export const previewPage = /* html */ `<!doctype html>}function seek(f) {stop();- // The playhead moved: frames rendered ahead of the old spot are stale.- if (player) player.buffer.clear();- $("buffered").style.display = "none";var done = show(clampFrame(f));keepPosition();scheduleWarm();@@ -594,7 +585,7 @@ export const previewPage = /* html */ `<!doctype html>if (draft && !draft.text.trim()) cancelDraft();if (notesOn) renderPins();if (frame >= last()) frame = 0;- playRealtime(alive).finally(function () { if (token === playToken && playing) pause(); });+ playFrames(alive).finally(function () { if (token === playToken && playing) pause(); });}function stop() {playing = false;@@ -606,64 +597,30 @@ export const previewPage = /* html */ `<!doctype html>function pause() { stop(); keepPosition(); show(frame).then(scheduleWarm); }function togglePlay() { playing ? pause() : play(); }- // Like a video player: frames are rendered ahead into a buffer, in order,- // and every one is shown at the film's frame rate. Playback starts once- // three seconds (or the rest of the film) are ready; when rendering falls- // behind, the clock and the sound hold until the buffer is full again.- // Pausing keeps the buffer (and it keeps filling), so playing on starts at- // once; moving the playhead clears it.- var BUFFER_SEC = 3;- async function playRealtime(alive) {- player.buffer.start(frame);- while (alive() && frame < last()) {- if (!(await fillBuffer(alive))) return;- if (!(await runClock(alive))) return;- }- }- // While paused, the loaded range keeps growing on the timeline.- setInterval(function () {- if (!player || playing) return;- if (player.buffer.ahead(frame) > 0) showBufferedRange();- else $("buffered").style.display = "none";- }, 250);- function showBufferedRange() {- var bar = $("buffered"), end = Math.max(frame, Math.min(last(), player.buffer.end - 1));- bar.style.display = "block";- bar.style.left = pct(frame) + "%";- bar.style.width = Math.max(0, pct(end) - pct(frame)) + "%";- }- async function fillBuffer(alive) {- var need = Math.max(1, Math.min(Math.round(meta.fps * BUFFER_SEC), last() - frame + 1, player.buffer.capacity - 2));- if (player.buffer.ahead(frame) >= need) return true;- setStalled(true);- while (alive() && player.buffer.ahead(frame) < need) {- showBufferedRange();- await sleep(30);+ // Real time: nothing is rendered ahead. Each turn draws the frame due on+ // the clock straight to the screen, and frames that drawing missed are+ // skipped. Playback starts at once and the sound keeps its speed; a device+ // that can't keep up shows fewer frames.+ async function playFrames(alive) {+ // A soundtrack still downloading would start late: wait for it.+ if (audio.state === "loading") {+ setStalled(true);+ while (alive() && audio.state === "loading") await sleep(50);+ // A newer playback may be waiting too: only this one's own stall ends here.+ if (!alive()) return;+ setStalled(false);}- if (!alive()) return false;- setStalled(false);- return true;- }- // Resolves true when the buffer ran dry, false at the end or when stopped.- async function runClock(alive) {- var f0 = frame, clock = startClock(f0), onScreen = -1;+ var f0 = frame, clock = startClock(f0), presented = -1;while (alive()) {var due = Math.min(last(), f0 + Math.floor(clock() * meta.fps));- if (due !== onScreen) {- if (!player.buffer.show(due)) {- stopClock();- setTime(onScreen >= 0 ? onScreen : f0);- return true;- }- onScreen = shown = due;- }+ if (due === presented) { await sleep(4); continue; }setTime(due);- showBufferedRange();- if (due >= last()) return false;- // A timer, not rAF: embedded or occluded views throttle rAF to ~1 Hz.- await sleep(4);+ // Each frame is drawn once; caching it would cost a texture and a copy per frame.+ var drawn = await player.render(due, { cache: false });+ if (!alive()) return;+ presented = shown = drawn;+ if (drawn >= last()) return;}- return false;}// ── Timeline ──diff --git a/packages/framefields/src/preview/player.ts b/packages/framefields/src/preview/player.tsindex 7d3be032..dd027bac 100644--- a/packages/framefields/src/preview/player.ts+++ b/packages/framefields/src/preview/player.ts@@ -49,8 +49,13 @@ export interface PlayerMeta {export interface PreviewPlayer {meta: PlayerMeta;canvas: HTMLCanvasElement;- /** Draws a frame. Calls made while one is drawing collapse to the latest. */- render(frame: number): Promise<number>;+ /**+ * Draws a frame. Calls made while one is drawing collapse to the latest.+ * `cache: false` leaves the drawn frame out of the step/scrub cache: for+ * real-time playback, which draws every frame once and would otherwise pay+ * a texture and a copy per frame.+ */+ render(frame: number, options?: { cache?: boolean }): Promise<number>;/** Milliseconds the last frame took to draw. */lastRenderMs: number;/** True when the program runs vision models (worth analysing ahead). */@@ -60,30 +65,10 @@ export interface PreviewPlayer {* before it plays. Skipped (resolves false) while a frame is being drawn.*/warm(frame: number): Promise<boolean>;- /** Frames rendered ahead in order, for playback at full frame rate. */- buffer: PlaybackBuffer;/** A frame at full size as an image, e.g. to attach to a note. */snapshot(frame: number): Promise<ImageBitmap>;}-export interface PlaybackBuffer {- /** Most frames it holds at once. */- readonly capacity: number;- /**- * Plays on from `frame`: frames already rendered from there are kept and- * rendering continues after them; anywhere else starts over.- */- start(frame: number): void;- /** Drops every rendered frame, e.g. when the playhead moves elsewhere. */- clear(): void;- /** Frames ready in a row from `frame` (0 when `frame` isn't buffered). */- ahead(frame: number): number;- /** The first frame not rendered yet. */- readonly end: number;- /** Shows a buffered frame; false when it isn't ready. */- show(frame: number): boolean;-}-type Renderable = {toVirtualMediaAsync?: () => Promise<VirtualMediaData>;toVirtualMedia?: () => VirtualMediaData;@@ -189,14 +174,15 @@ async function createPlayer(let first = true;let drawing: Promise<void> | null = null;let wanted: number | null = null;+ let wantedCached = true;let waiters: Array<(f: number) => void> = [];let offscreen: GPUTexture | undefined;// Frames drawn on screen are kept so stepping or scrubbing back to one is a- // blit, not a re-render. Full resolution, bounded by a byte budget; playback- // still uses the reduced-res ring below. Vision compositions are excluded:- // their frames depend on the one before, so a hit would skip that work.+ // blit, not a re-render. Full resolution, bounded by a byte budget. Vision+ // compositions are excluded: their frames depend on the one before, so a+ // hit would skip that work.const renderCache = new Map<number, { tex: GPUTexture; used: number }>();const renderCacheMax = Math.max(1,@@ -256,7 +242,11 @@ async function createPlayer(}}- async function draw(frame: number, toScreen = true): Promise<void> {+ async function draw(+ frame: number,+ toScreen = true,+ cache = toScreen,+ ): Promise<void> {const renderer = getRenderer2D(device, surface.colorFormat);const ctx: RenderContextValue = { device, renderer, surface };const w = surface.width;@@ -270,7 +260,7 @@ async function createPlayer(containerWidth: w,containerHeight: h,isVideoMode,- // Every drawn frame is cached or buffered, so it waits for its LUTs+ // A drawn frame can be cached, so it waits for its LUTs// and exact video frame as the export does, never a stand-in.forceWait: true,};@@ -329,7 +319,7 @@ async function createPlayer(props,);let cacheTex: GPUTexture | undefined;- if (toScreen) {+ if (cache) {cacheTex = newRenderCacheTexture();encoder.copyTextureToTexture({ texture: tex }, { texture: cacheTex }, [w,@@ -355,6 +345,7 @@ async function createPlayer(void (async () => {while (wanted !== null) {const f = wanted;+ const cache = wantedCached;wanted = null;const started = performance.now();try {@@ -363,7 +354,7 @@ async function createPlayer(blitter.blit(cached, surface.getCurrentTexture());surface.present();} else {- await draw(f);+ await draw(f, true, cache);}} catch (err) {console.error("[framefields] frame", f, err);@@ -380,130 +371,16 @@ async function createPlayer(})();}- // ── Playback buffer: a ring of frames rendered ahead, shown at full rate.- // Frames are kept below native size to bound memory (~400 MB at most);- // a paused frame is drawn at full resolution.- const lastFrame = meta.frameCount - 1;- const capacity = Math.ceil(fps * 3.5) + 2;- const budget = 400 * 1024 * 1024;- const scale = Math.min(- 1,- Math.sqrt(budget / capacity / 4 / (width * height)),- );- const even = (n: number) => Math.max(2, Math.round(n / 2) * 2);- const ringSize = { w: even(width * scale), h: even(height * scale) };- const ring: GPUTexture[] = [];- const ringFrames = new Int32Array(capacity).fill(-1);const blitter = new Blitter(device, surface.colorFormat);- let bufferToken = 0;- let producing = false;- let produced = 0;- let shown = -1;-- function ringTexture(slot: number): GPUTexture {- ring[slot] ??= device.createTexture({- size: [ringSize.w, ringSize.h],- format: surface.colorFormat,- usage:- GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING,- label: `preview_buffer_${slot}`,- });- return ring[slot];- }-- async function produce(token: number): Promise<void> {- producing = true;- try {- await produceFrames(token);- } finally {- if (token === bufferToken) producing = false;- }- }-- async function produceFrames(token: number): Promise<void> {- while (token === bufferToken && produced <= lastFrame) {- // The slot holds a frame not shown yet, or the one on screen (kept,- // so playing on after a pause starts from the buffer).- if (produced - capacity >= shown) {- await new Promise((r) => setTimeout(r, 4));- continue;- }- while (drawing) await drawing;- if (token !== bufferToken) return;- const frame = produced;- const slot = frame % capacity;- const cached = takeRenderCache(frame);- if (cached) {- blitter.blit(cached, ringTexture(slot));- ringFrames[slot] = frame;- produced = frame + 1;- // Yield so a run of cached frames can't starve the page.- await new Promise((r) => setTimeout(r, 0));- continue;- }- drawing = draw(frame, false)- .then(() => {- if (offscreen) blitter.blit(offscreen, ringTexture(slot));- })- .catch((err: unknown) =>- console.error("[framefields] frame", frame, err),- );- await drawing;- drawing = null;- if (token !== bufferToken) {- // Stopped mid-frame: a frame asked for meanwhile still draws.- pump();- return;- }- ringFrames[slot] = frame;- produced = frame + 1;- pump();- }- }-- const buffer: PlaybackBuffer = {- capacity,- get end() {- return produced;- },- start(frame) {- shown = frame - 1;- // Resuming where playback paused: keep what is rendered ahead.- if (ringFrames[frame % capacity] === frame && frame < produced) {- if (!producing) void produce(++bufferToken);- return;- }- ringFrames.fill(-1);- produced = frame;- void produce(++bufferToken);- },- clear() {- bufferToken++;- producing = false;- ringFrames.fill(-1);- produced = 0;- },- ahead(frame) {- if (ringFrames[frame % capacity] !== frame) return 0;- return produced - frame;- },- show(frame) {- const slot = frame % capacity;- if (ringFrames[slot] !== frame) return false;- blitter.blit(ringTexture(slot), surface.getCurrentTexture());- surface.present();- shown = frame;- return true;- },- };const player: PreviewPlayer = {meta,canvas,lastRenderMs: 0,hasVision,- render(frame) {+ render(frame, options) {wanted = frame;+ wantedCached = options?.cache ?? true;const done = new Promise<number>((resolve) => waiters.push(resolve));pump();return done;@@ -518,7 +395,6 @@ async function createPlayer(pump();return true;},- buffer,async snapshot(frame) {while (drawing) await drawing;const done = (async () => {diff --git a/plugins/framefields/skills/framefields-render/SKILL.md b/plugins/framefields/skills/framefields-render/SKILL.mdindex 958fec5b..5c51f89c 100644--- a/plugins/framefields/skills/framefields-render/SKILL.md+++ b/plugins/framefields/skills/framefields-render/SKILL.md@@ -112,7 +112,7 @@ describe("Hero Scene Conformance", () => {---### Step 4.5: Show the User Before Exporting (`preview`)-When a person is in the loop, let them watch the composition before you spend time on the final render. `startPreview` serves a localhost player with the soundtrack, a timeline with the waveform, and frame stepping. The page loads the composition's own code and renders it with WebGPU in the browser, so playback runs at or near full frame rate; the server only serves that code, the project's files, and the soundtrack mixed by the export engine. Point it at the module that builds the composition:+When a person is in the loop, let them watch the composition before you spend time on the final render. `startPreview` serves a localhost player with the soundtrack, a timeline with the waveform, and frame stepping. The page loads the composition's own code and renders it with WebGPU in the browser, in real time: a device that can't draw every frame in time skips some, and the sound keeps its speed; the server only serves that code, the project's files, and the soundtrack mixed by the export engine. Point it at the module that builds the composition:```typescriptimport { startPreview } from "framefields";@@ -132,7 +132,8 @@ await session.closed; // keep the process alive while the user watches- **Point at a moment.** `#t=12.5` in the URL opens the player at 12.5 s (`#frame=750` by frame), and the address follows the playhead when it stops, so a link you give the user lands where you mean.- **Drive it from a script, not with clicks.** If you can run JavaScript in the page (an agent browser), use `window.framefieldsPreview`: `await seek(seconds)` or `await seekFrame(n)` resolve once that frame is drawn (take the screenshot then), `play()` and `pause()` control playback, `notes()` lists the notes, and `state()` returns `{ state: "loading" | "ready" | "error", error, frame, time, playing, buffering, audio, connected, meta }`. `<html data-preview="ready">` marks the first frame drawn. A hidden tab pauses playback, since browsers throttle it.- **Show it where the user is.** `startPreview` only serves the page and returns the URL. If your app has a built-in browser (Claude Code's browser pane, Codex's), open the URL there. Otherwise give the user the link, or pass `open: true` to open their default browser (skipped when an open tab reconnects on its own). The page needs WebGPU (current Chrome, Edge or Safari).-- Audio is the same mix `renderVideo` muxes (`audio: false` skips it). Playback uses the soundtrack as its clock; when a frame takes long (a video seek, a heavy shot), the clock and the sound hold until the picture catches up.+- **Playback runs in real time, on any device.** Play draws the frame due on the clock and skips the ones it missed, so it starts at once and the sound keeps its speed; a slow device shows fewer frames. Paused, the page draws the exact frame under the playhead. Vision effects analyse each frame from the one drawn before it: while paused, the page analyses up to 20 s ahead so that stretch plays exactly, but past it a device that skips frames shows cutouts and tracking from the last frame it drew. Check them paused, or in the export.+- Audio is the same mix `renderVideo` muxes (`audio: false` skips it). Playback uses the soundtrack as its clock; when a frame takes long (a video seek, a heavy shot), the picture skips ahead to catch up with the sound.---diff --git a/plugins/framefields/skills/framefields/SKILL.md b/plugins/framefields/skills/framefields/SKILL.mdindex 0d17534a..f6ed3f6d 100644--- a/plugins/framefields/skills/framefields/SKILL.md+++ b/plugins/framefields/skills/framefields/SKILL.md@@ -228,7 +228,7 @@ Show the user the frame PNGs or the grid before the full render. `framefields-re### Live preview-`npm run preview` serves a localhost player (picture, sound, timeline) and prints its URL. The page runs `film.ts` itself and renders with WebGPU, so the user sees the real film at full frame rate.+`npm run preview` serves a localhost player (picture, sound, timeline) and prints its URL. The page runs `film.ts` itself and renders with WebGPU in real time, so the user sees the real film; a slow device skips frames to keep up with the sound.- Run it as a background command and read its output: it says when the page opened, prints errors from the page (a failed build, a frame that threw, a missing font), and says why it stopped. It stops 5 s after its last tab closes; just run it again.- It doesn't open anything. If your app has a built-in browser (Claude Code, Codex), open the URL there; otherwise give the user the link. `startPreview(..., { open: true })` opens their default browser.