Code-first video & image, rendered natively on WebGPU.
Show the frame rate preview playback reaches
Silvio Tomatis7ce675db1e38
diff --git a/.changeset/preview-frame-rate.md b/.changeset/preview-frame-rate.mdnew file mode 100644index 00000000..f28bb67e--- /dev/null+++ b/.changeset/preview-frame-rate.md@@ -0,0 +1,5 @@+---+"framefields": minor+---++Preview: the page shows the frame rate playback reaches next to the timecode, and `window.framefieldsPreview.state()` reports it as `playbackFps`, with `renderMs` for the last frame drawn.diff --git a/README.md b/README.mdindex f7daf8b8..4e3fe8f9 100644--- a/README.md+++ b/README.md@@ -268,7 +268,7 @@ Project normalized landmarks to screen space with a configurable camera FOV, the- **Notes pinned on the picture** — pause, click a spot (or drag an area) and write what should change. Each note keeps its moment, its spot and a picture of the frame with the spot marked, in `.framefields/preview-notes/`, and prints where the preview runs, so an agent can apply exactly what you meant. Tick notes off as you check them.- **Tells you what happened** — the page's errors (a failed build, a frame that threw, a missing font) and why the server stopped are printed where the preview runs, so whoever started it, person or agent, sees them without opening devtools. Scripts can drive the page through `window.framefieldsPreview` (`seek`, `play`, `pause`, `state`).- **Shown where you are** — `startPreview` serves the page and returns its URL instead of opening a browser, so an agent can show it in its own pane (Claude Code, Codex); pass `open: true` to open the system browser.-- **Plays in real time, even on a slow device** — Play draws the frame due on the soundtrack's clock and skips the ones it missed, so playback starts at once and the sound never waits. A device that can't keep up shows fewer frames.+- **Plays in real time, even on a slow device** — Play draws the frame due on the soundtrack's clock and skips the ones it missed, so playback starts at once and the sound never waits. A device that can't keep up shows fewer frames, and the page shows the frame rate it reaches.```typescriptimport { startPreview } from "framefields";diff --git a/packages/framefields/src/preview/page.ts b/packages/framefields/src/preview/page.tsindex c9cbcade..3fa752e7 100644--- a/packages/framefields/src/preview/page.ts+++ b/packages/framefields/src/preview/page.ts@@ -136,6 +136,7 @@ export const previewPage = /* html */ `<!doctype html>.timecode { display: flex; align-items: baseline; gap: 5px; margin-left: 8px; font-size: 13px; white-space: nowrap; }.timecode .dim { color: var(--ink-3); }.timecode .frame { color: var(--ink-3); font-size: 11.5px; margin-left: 6px; }+ .fps { margin-left: 10px; color: var(--ink-3); font-size: 11.5px; white-space: nowrap; }input[type="range"].slider { -webkit-appearance: none; appearance: none; width: 72px; height: 4px; border-radius: 2px; background: var(--line-2); margin: 0 8px 0 0; }input[type="range"].slider::-webkit-slider-thumb { -webkit-appearance: none; width: 12px; height: 12px; border-radius: 50%; background: var(--ink); border: 0; }input[type="range"].slider::-moz-range-thumb { width: 12px; height: 12px; border-radius: 50%; background: var(--ink); border: 0; }@@ -319,6 +320,7 @@ export const previewPage = /* html */ `<!doctype html><span id="tcNow">0:00.00</span><span class="dim">/</span><span class="dim" id="tcDur">0:00.00</span><span class="frame" id="tcFrame">frame 0</span></div>+ <span class="fps tnum" id="fps" role="status" hidden></span><span class="grow"></span><button class="btn notes-btn" id="notesBtn" title="Notes" aria-haspopup="dialog" aria-expanded="false" hidden><svg viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.4" stroke-linejoin="round"><path d="M3 2.75h10a1.25 1.25 0 0 1 1.25 1.25v6.5A1.25 1.25 0 0 1 13 11.75H7.5L4.5 14v-2.25H3A1.25 1.25 0 0 1 1.75 10.5V4A1.25 1.25 0 0 1 3 2.75z"/></svg>@@ -592,7 +594,48 @@ export const previewPage = /* html */ `<!doctype html>stopClock();setStalled(false);setPlayingUi(false);+ meter.stop();+ renderFps();}++ // ── Frame rate reached: distinct frames shown per second while playing,+ // over the last 2 s, stalls included. The film's own rate is in meta ──+ var meter = (function () {+ var started = null, lastFrame = null, samples = [], lastFps = null;+ function read() {+ if (started === null) return { state: "paused", fps: lastFps };+ var now = performance.now(), elapsed = Math.min(2000, now - started);+ samples = samples.filter(function (t) { return t > now - 2000; });+ if (elapsed < 1000) return { state: "measuring", fps: null };+ // Counting whole frames in the window can overshoot the film's rate.+ return { state: "playing", fps: Math.min(meta ? meta.fps : Infinity, (samples.length * 1000) / elapsed) };+ }+ return {+ read: read,+ start: function () { started = performance.now(); lastFrame = null; samples = []; },+ stop: function () {+ if (started === null) return;+ // A playback too short to measure leaves no rate, not an older one.+ var r = read();+ lastFps = r.state === "playing" ? r.fps : null;+ started = null;+ samples = [];+ },+ record: function (f) {+ if (started === null || f === lastFrame) return;+ lastFrame = f;+ samples.push(performance.now());+ }+ };+ })();+ function renderFps() {+ var r = meter.read(), out = $("fps");+ out.hidden = r.state === "paused" && r.fps === null;+ if (out.hidden) return;+ out.textContent = r.state === "measuring" ? "… fps" : r.fps.toFixed(1) + " fps";+ out.title = (r.state === "paused" ? "Frame rate the last playback reached" : "Frame rate playback reaches") + (meta ? "; the film runs at " + meta.fps + " fps" : "");+ }+ setInterval(function () { if (playing) renderFps(); }, 500);// Pausing settles on exactly the frame the timecode shows.function pause() { stop(); keepPosition(); show(frame).then(scheduleWarm); }function togglePlay() { playing ? pause() : play(); }@@ -600,8 +643,10 @@ export const previewPage = /* html */ `<!doctype html>// Real time: nothing is rendered ahead. Each turn draws the frame due on// the clock straight to the screen, and frames that drawing missed are// skipped. Playback starts at once and the sound keeps its speed; a device- // that can't keep up shows fewer frames.+ // that can't keep up shows fewer frames, and the frame rate says how many.async function playFrames(alive) {+ meter.start();+ renderFps();// A soundtrack still downloading would start late: wait for it.if (audio.state === "loading") {setStalled(true);@@ -619,6 +664,7 @@ export const previewPage = /* html */ `<!doctype html>var drawn = await player.render(due, { cache: false });if (!alive()) return;presented = shown = drawn;+ meter.record(drawn);if (drawn >= last()) return;}}@@ -1081,6 +1127,7 @@ export const previewPage = /* html */ `<!doctype html>pause: function () { if (playing) pause(); },/** Where the preview is: loading, ready or error, and what it shows. */state: function () {+ var rate = meter.read().fps;return {state: pageState,error: pageState === "error" ? $("stageMsg").textContent : null,@@ -1090,6 +1137,10 @@ export const previewPage = /* html */ `<!doctype html>buffering: stalled,audio: audio.state,connected: !offline,+ // Frames shown per second over the last 2 s of playback (the last+ // playback's while paused), and how long the last frame took to draw.+ playbackFps: rate == null ? null : Number(rate.toFixed(1)),+ renderMs: player ? Math.round(player.lastRenderMs) : null,meta: meta && { title: meta.title, width: meta.width, height: meta.height, fps: meta.fps, frameCount: meta.frameCount, duration: Number((last() / meta.fps).toFixed(3)) }};}diff --git a/plugins/framefields/skills/framefields-render/SKILL.md b/plugins/framefields/skills/framefields-render/SKILL.mdindex 5c51f89c..0ad249f3 100644--- a/plugins/framefields/skills/framefields-render/SKILL.md+++ b/plugins/framefields/skills/framefields-render/SKILL.md@@ -130,9 +130,9 @@ await session.closed; // keep the process alive while the user watches- **One tab for the whole loop.** The port is fixed per project directory. Running the preview again (after you've made changes) takes over from the one already running, and a tab left open reconnects and reloads into the new version by itself, at the moment it was showing. Don't ask the user to reopen anything. When the user closes the tab, the preview stops by itself about 5 s later (a reload doesn't stop it; `idleCloseMs` changes the delay). If the preview was stopped (tab closed, time limit, or you killed it), just start it again.- **Notes pinned on the picture are the user's precise feedback.** Paused, the user clicks the frame (or drags an area) and writes what should change; the note keeps the frame, the spot as fractions of the frame (`x`, `y`, and `w`, `h` for an area) and the text. Each note prints in the output (`note 3 at 0:12.50 (frame 750), spot 42%,31%: …`) with the path of `note-<id>.jpg`, the frame with the spot marked, and all of them are in `.framefields/preview-notes/notes.json` (`.framefields` ignores itself in git; `notesDir` moves it, `false` turns notes off). To act on them: read `notes.json`, open each open note's picture to see exactly what it points at, find the layer in the code from the moment and the spot, make the change, and run the preview again. The tab comes back with the notes, so the user can check them and tick them off. You can also set `"done": true` in `notes.json` yourself. A chat that can't read files gets the same list from the **Copy** button in the Notes panel.- **Point at a moment.** `#t=12.5` in the URL opens the player at 12.5 s (`#frame=750` by frame), and the address follows the playhead when it stops, so a link you give the user lands where you mean.-- **Drive it from a script, not with clicks.** If you can run JavaScript in the page (an agent browser), use `window.framefieldsPreview`: `await seek(seconds)` or `await seekFrame(n)` resolve once that frame is drawn (take the screenshot then), `play()` and `pause()` control playback, `notes()` lists the notes, and `state()` returns `{ state: "loading" | "ready" | "error", error, frame, time, playing, buffering, audio, connected, meta }`. `<html data-preview="ready">` marks the first frame drawn. A hidden tab pauses playback, since browsers throttle it.+- **Drive it from a script, not with clicks.** If you can run JavaScript in the page (an agent browser), use `window.framefieldsPreview`: `await seek(seconds)` or `await seekFrame(n)` resolve once that frame is drawn (take the screenshot then), `play()` and `pause()` control playback, `notes()` lists the notes, and `state()` returns `{ state: "loading" | "ready" | "error", error, frame, time, playing, buffering, audio, connected, playbackFps, renderMs, meta }`. `<html data-preview="ready">` marks the first frame drawn. A hidden tab pauses playback, since browsers throttle it.- **Show it where the user is.** `startPreview` only serves the page and returns the URL. If your app has a built-in browser (Claude Code's browser pane, Codex's), open the URL there. Otherwise give the user the link, or pass `open: true` to open their default browser (skipped when an open tab reconnects on its own). The page needs WebGPU (current Chrome, Edge or Safari).-- **Playback runs in real time, on any device.** Play draws the frame due on the clock and skips the ones it missed, so it starts at once and the sound keeps its speed; a slow device shows fewer frames. Paused, the page draws the exact frame under the playhead. Vision effects analyse each frame from the one drawn before it: while paused, the page analyses up to 20 s ahead so that stretch plays exactly, but past it a device that skips frames shows cutouts and tracking from the last frame it drew. Check them paused, or in the export.+- **Playback runs in real time, on any device.** Play draws the frame due on the clock and skips the ones it missed, so it starts at once and the sound keeps its speed; a slow device shows fewer frames. Next to the timecode the page shows the frame rate playback reaches, apart from the film's own `fps`; `state()` reports it as `playbackFps`, and `renderMs` is how long the last frame took to draw. Paused, the page draws the exact frame under the playhead. Vision effects analyse each frame from the one drawn before it: while paused, the page analyses up to 20 s ahead so that stretch plays exactly, but past it a device that skips frames shows cutouts and tracking from the last frame it drew. Check them paused, or in the export.- Audio is the same mix `renderVideo` muxes (`audio: false` skips it). Playback uses the soundtrack as its clock; when a frame takes long (a video seek, a heavy shot), the picture skips ahead to catch up with the sound.---