Code-first video & image, rendered natively on WebGPU.
feat(physics): live worlds and on-demand clip vision, no up-front tracking
okanasl119b397db38b
diff --git a/.changeset/physics-rapier.md b/.changeset/physics-rapier.mdindex 3672d828..f570bbbc 100644--- a/.changeset/physics-rapier.md+++ b/.changeset/physics-rapier.md@@ -4,4 +4,6 @@Add rigid-body physics: `Physics.world2d` / `Physics.world3d` (Rapier, deterministic) simulate once while the composition is built and replay as per-frame signals. Bodies come from layers (boxes, circles, convex hulls of shapes, measured text, Layer3D primitives and models); `simulate()` binds each layer, and every body exposes `impact`, `speed`, velocity and pose signals for driving any prop or effect. Includes ground/walls, pins, hinges, welds, springs, ropes, signal-driven forces and kinematic bodies, collision lists, and `bakeKeyframes()` for plain keyframes.-`world.field()` adds position-dependent forces (wind, vortices, flow fields). `comp.trackPerson(source)` tracks a performer before rendering — smoothed pose keypoints and a soft silhouette per frame, placeable into composition pixels and cacheable as JSON — so limbs can be kinematic colliders and the silhouette a force field.+`world.field()` adds position-dependent forces (wind, vortices, flow fields). `world.live(comp)` steps a world as the composition renders instead of all at once: before each frame draws it advances to it, after that frame's inputs are ready, giving the same result as `simulate()`.++`comp.vision(clip)` analyses a clip's own frames on demand (pose keypoints and a soft silhouette) and shares the results with everything that reads them; every render path and the preview await it before drawing a frame, and results persist on disk in Node. `comp.onPrepareFrame()` registers async per-frame work, and the renderer awaits an `onRequestFrame` that returns a promise. `colorSignal` takes `multiplier` and `offset`.diff --git a/examples/28_physics_dancer/README.md b/examples/28_physics_dancer/README.mdindex 86e9e6d6..ba7ca65a 100644--- a/examples/28_physics_dancer/README.md+++ b/examples/28_physics_dancer/README.md@@ -6,8 +6,9 @@ her as the silk dress sweeps through it.```bashpnpm fonts # Syne from google/fonts → assets/fonts/-pnpm render sim # tracks the dancer (once, ~30 s, cached in output/) and runs the physics+pnpm render sim # analyses the clip and steps the physics without drawing (~35 s cold, ~6 s cached)pnpm render # output/physics-dancer.mp4+pnpm render preview # live player; vision and physics fill in as it playspnpm render:src # same, on the engine's source (no package builds)```@@ -17,12 +18,14 @@ The clip is example 19's `assets/dancer.mp4`; generate it there with `pnpm asset| Piece | API ||---|---|-| Pose and silhouette for every frame, before rendering | `comp.trackPerson(clip)` → `PersonTrack`, placed with `.place({ width, height, fit })`, cached with `toJSON()` / `PersonTrack.fromJSON()` |+| Pose and silhouette, frame by frame as it renders | `comp.vision(clip).person`, placed with `.place({ width, height, fit })`; results persist in `~/.cache/framefields/vision` |+| Physics that steps with the render | `world.live(comp)`: before each frame draws, its vision is ready and the world advances to it || Arms and head that catch the cards | kinematic circle bodies along shoulder → elbow → wrist (and past the wrist for the hand), `follow`ing `person.point(name, frame)` || The dress pushing particles and cards | `world.field()` with `person.sample(frame, x, y)`: out along the silhouette's gradient, much harder where the silhouette just arrived (the sweep) || Fluid feel | small circles with `linearDamping` 2.2, near-zero gravity, a slow swirling current and a soft pull home || Particles behind her | a second copy of the clip, cut out with `.withVision({ mode: "matte" })`, above the particles |-| Hot particles | `.colorSignal("fillColor", …)` from each body's `speed` |+| Hot particles | `.colorSignal("fillColor", body.speed, { multiplier: 1 / 900, … })` || Screen shake | the whole stage's `y` driven by `world.impact` |-Cards are sized with `FontManager.measure`, so the text always fits.+Cards are sized with `FontManager.measure`, so the text always fits. A live world can't look+ahead, so they drop at fixed offsets from where her head starts and find her hands on the way down.diff --git a/examples/28_physics_dancer/src/film.ts b/examples/28_physics_dancer/src/film.tsindex c32d5e7a..89198b0f 100644--- a/examples/28_physics_dancer/src/film.ts+++ b/examples/28_physics_dancer/src/film.ts@@ -2,11 +2,16 @@* A dancer in a field of physics: word cards drop onto her raised hands and* head, and a cloud of particles flows around her as the dress sweeps through.*+ * Nothing is analysed up front: `comp.vision` analyses each frame of the clip+ * just before it renders, and the live physics world steps to that frame+ * right after, so the preview starts at once and fills in as it plays.+ ** - Her arms and head are kinematic colliders that follow the tracked pose.* - Her silhouette (dress included) is a force field that pushes everything out,* hardest where it is sweeping in.* - A vision cutout of the same clip sits above the particles, so they pass behind her.*/+import fs from "node:fs";import {Composition,FontManager,@@ -17,9 +22,7 @@ import {type PhysicsBody2D,type PhysicsWorld2D,type PoseKeypointName,- Signal,} from "framefields";-import { performer } from "./performer.js";import {CREAM,DANCER,@@ -39,11 +42,12 @@ import {} from "./theme.js";const WORDS = [- { text: "PHYSICS", bg: EMBER, fg: INK, at: 30 },- { text: "MEETS", bg: INK, fg: CREAM, at: 46 },- { text: "VISION", bg: CREAM, fg: INK, at: 62 },- { text: "IN", bg: GOLD, fg: INK, at: 78 },- { text: "FRAMEFIELDS", bg: VIOLET, fg: CREAM, at: 94 },+ // `dx`: where it drops, from her head at the first frame (live worlds can't look ahead).+ { text: "PHYSICS", bg: EMBER, fg: INK, at: 30, dx: -40 },+ { text: "MEETS", bg: INK, fg: CREAM, at: 46, dx: 120 },+ { text: "VISION", bg: CREAM, fg: INK, at: 62, dx: -120 },+ { text: "IN", bg: GOLD, fg: INK, at: 78, dx: 60 },+ { text: "FRAMEFIELDS", bg: VIOLET, fg: CREAM, at: 94, dx: 0 },];const CARD_FONT_SIZE = 64;const PARTICLE_COLORS = [INK, RUST, EMBER, GOLD];@@ -76,7 +80,7 @@ function limb(}}-/** Word cards sized to their text, dropped over whichever hand is highest. */+/** Word cards sized to their text, dropped around where her head starts. */function cards(world: PhysicsWorld2D, person: PersonTrack) {return WORDS.map((w, i) => {const m = FontManager.measure(w.text, {@@ -85,13 +89,11 @@ function cards(world: PhysicsWorld2D, person: PersonTrack) {});const width = Math.ceil(m.width + 72);const height = Math.ceil(m.height + 40);- const hand = (["leftWrist", "rightWrist"] as const)- .map((k) => person.point(k, w.at + 12))- .reduce((p, q) => (p.y < q.y ? p : q));+ const head = person.point("nose", 0);const layer = Layer.box({id: `card-${i}`,position: "absolute",- x: hand.x - width / 2 + (i % 2 ? 30 : -30),+ x: head.x + w.dx - width / 2,y: -height - 200,width,height,@@ -181,7 +183,19 @@ function silhouettePush(person: PersonTrack, strength: number, sweep: number) {};}+export { DURATION as DURATION_FRAMES } from "./theme.js";++export interface Film {+ comp: Composition;+ world: PhysicsWorld2D;+ cards: Array<{ id: string; body: PhysicsBody2D }>;+}+export async function buildFilm(): Promise<Composition> {+ return (await buildScene()).comp;+}++export async function buildScene(): Promise<Film> {await registerFonts();const comp = new Composition({width: W,@@ -191,7 +205,15 @@ export async function buildFilm(): Promise<Composition> {backgroundColor: "#E9D3B4",fonts: [FONT_FILE],});- const person = await performer(comp);+ if (!fs.existsSync(DANCER)) {+ throw new Error(+ `Missing ${DANCER}. Generate it with \`pnpm assets\` in examples/19_framefields_film.`,+ );+ }+ const dancer = comp.vision(DANCER);+ // The particles are laid out around her first pose: analyse just that now.+ await dancer.ready(0);+ const person = dancer.person.place({ width: W, height: H, fit: "cover" });const world = await Physics.world2d({fps: FPS,@@ -237,22 +259,17 @@ export async function buildFilm(): Promise<Composition> {},{ bodies: cloudBodies },);- world.simulate();+ // Steps as the composition renders, after each frame's vision is ready.+ world.live(comp);- // Particles glow as they speed up.+ // Particles glow as they speed up (900 px/s is fully hot).for (const p of cloud) {- const heat = Array.from({ length: DURATION }, (_, f) =>- Math.min(1, p.body.speed.get({ frame: f, fps: FPS } as never) / 900),- );p.layer.animate(- LayerAnimation.create().colorSignal(- "fillColor",- Signal.fromArray(heat, FPS),- {- colorA: p.color,- colorB: HOT,- },- ),+ LayerAnimation.create().colorSignal("fillColor", p.body.speed, {+ colorA: p.color,+ colorB: HOT,+ multiplier: 1 / 900,+ }),);}@@ -292,11 +309,9 @@ export async function buildFilm(): Promise<Composition> {),);- for (const w of words) {- const hit = w.body.collisions[0];- console.log(- `${w.layer.id}: first contact at frame ${hit?.frame ?? "-"} (${hit?.with.type ?? "none"})`,- );- }- return comp;+ return {+ comp,+ world,+ cards: words.map((w) => ({ id: String(w.layer.id), body: w.body })),+ };}diff --git a/examples/28_physics_dancer/src/performer.ts b/examples/28_physics_dancer/src/performer.tsdeleted file mode 100644index fd81fb14..00000000--- a/examples/28_physics_dancer/src/performer.ts+++ /dev/null@@ -1,30 +0,0 @@-import fs from "node:fs";-import path from "node:path";-import { type Composition, PersonTrack } from "framefields";-import { DANCER, H, TRACK_CACHE, W } from "./theme.js";--/**- * The dancer's pose and silhouette for every frame, placed in composition- * pixels. Tracking runs once (about 30 s) and is cached in output/.- */-export async function performer(comp: Composition): Promise<PersonTrack> {- if (!fs.existsSync(DANCER)) {- throw new Error(- `Missing ${DANCER}. Generate it with \`pnpm assets\` in examples/19_framefields_film.`,- );- }- let track: PersonTrack;- if (fs.existsSync(TRACK_CACHE)) {- track = PersonTrack.fromJSON(- JSON.parse(fs.readFileSync(TRACK_CACHE, "utf8")),- );- } else {- console.log("Tracking the dancer (pose + silhouette)…");- track = await comp.trackPerson(DANCER, {- onProgress: (f, n) => f % 40 === 0 && console.log(` ${f}/${n}`),- });- fs.mkdirSync(path.dirname(TRACK_CACHE), { recursive: true });- fs.writeFileSync(TRACK_CACHE, JSON.stringify(track.toJSON()));- }- return track.place({ width: W, height: H, fit: "cover" });-}diff --git a/examples/28_physics_dancer/src/render.ts b/examples/28_physics_dancer/src/render.tsindex 411eafb1..b8fc9fdc 100644--- a/examples/28_physics_dancer/src/render.ts+++ b/examples/28_physics_dancer/src/render.ts@@ -1,20 +1,44 @@/*** pnpm render → output/physics-dancer.mp4- * pnpm render sim → runs tracking and physics only, prints first contacts+ * pnpm render sim → analyses the clip and steps the physics without drawing, prints first contacts+ * pnpm render preview → live player on localhost (add --open); vision and physics fill in as it plays*- * Frames render in order: the cutout runs vision on the previous frame, so- * single frames out of order would show a ghost of another moment.+ * Vision results persist in ~/.cache/framefields/vision, so only the first run analyses the clip.*/import path from "node:path";-import { buildFilm } from "./film.js";+import { startPreview } from "framefields";+import { buildScene, DURATION_FRAMES } from "./film.js";import { OUTPUT } from "./theme.js";-const [mode = "video"] = process.argv.slice(2);-const film = await buildFilm();+const [mode = "video", ...args] = process.argv.slice(2);-if (mode !== "sim") {+if (mode === "preview") {+ const session = await startPreview(+ { entry: new URL("./film.ts", import.meta.url), export: "buildFilm" },+ { title: "Physics dancer", open: args.includes("--open") },+ );+ console.log(`Preview at ${session.url}`);+ await session.closed;+ process.exit(0);+}++const { comp, world, cards } = await buildScene();++if (mode === "sim") {+ const started = Date.now();+ await world.advanceTo(DURATION_FRAMES - 1);+ console.log(+ `Analysed and simulated in ${((Date.now() - started) / 1000).toFixed(1)} s`,+ );+ for (const { id, body } of cards) {+ const hit = body.collisions[0];+ console.log(+ `${id}: first contact at frame ${hit?.frame ?? "-"} (${hit?.with.type ?? "none"})`,+ );+ }+} else {const started = Date.now();- const result = await film.renderVideo({+ const result = await comp.renderVideo({outputPath: path.join(OUTPUT, "physics-dancer.mp4"),quality: "high",});diff --git a/examples/28_physics_dancer/src/theme.ts b/examples/28_physics_dancer/src/theme.tsindex df8fd0c0..8ee95a86 100644--- a/examples/28_physics_dancer/src/theme.ts+++ b/examples/28_physics_dancer/src/theme.ts@@ -11,8 +11,6 @@ export const DANCER = path.resolve(HERE,"../../19_framefields_film/assets/dancer.mp4",);-/** Pose + silhouette analysis of the clip, cached after the first run. */-export const TRACK_CACHE = path.join(OUTPUT, "dancer-track.json");export const W = 1920;export const H = 1080;diff --git a/nodes/node-compositor/src/renderers/webgpu-renderer.ts b/nodes/node-compositor/src/renderers/webgpu-renderer.tsindex 078d024c..06013f21 100644--- a/nodes/node-compositor/src/renderers/webgpu-renderer.ts+++ b/nodes/node-compositor/src/renderers/webgpu-renderer.ts@@ -67,7 +67,7 @@ import {srtLoader,Vector3Math,} from "@framefields/webgpu-renderers";-import { compileTimeline } from "../shared/compiler.js";+import { compileTimeline, seekTimeline } from "../shared/compiler.js";import type { CompositorOperation } from "../shared/config.js";import { shouldPaintContainerBackground } from "./container-bg.js";import { buildLayerMatrix, toDOMMatrix } from "./transform.js";@@ -614,12 +614,12 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {const composeDuration =virtualMedia?.metadata?.durationMs ?? DEFAULT_DURATION_MS;const durationSec = composeDuration / 1000;- const { tl, targetsById } = compileTimeline(- props.renderId,- virtualMedia as any,- { fps: fps ?? 24, durationSec },- );- tl.seek(shiftedFrame / (fps ?? 24));+ const compiled = compileTimeline(props.renderId, virtualMedia as any, {+ fps: fps ?? 24,+ durationSec,+ });+ const { targetsById } = compiled;+ seekTimeline(compiled, shiftedFrame, fps ?? 24);const r3d = getRenderer3D(ctx.device,@@ -2555,13 +2555,11 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {const container3DShutter =target?.motionBlurShutter ?? lop.motionBlurShutter ?? 0;if (container3DShutter > 0 && fps && shiftedFrame !== undefined) {- const currentT = shiftedFrame / fps;- const prevT = (shiftedFrame - 1) / fps;- tl.seek(prevT);+ seekTimeline(compiled, shiftedFrame - 1, fps);const prevTarget = targetId ? targetsById[targetId] : undefined;const prevX = prevTarget?.x ?? baseX;const prevY = prevTarget?.y ?? baseY;- tl.seek(currentT);+ seekTimeline(compiled, shiftedFrame, fps);const vx = sampledX - prevX;const vy = sampledY - prevY;@@ -2842,13 +2840,11 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {const containerShutter =target?.motionBlurShutter ?? lop.motionBlurShutter ?? 0;if (containerShutter > 0 && fps && shiftedFrame !== undefined) {- const currentT = shiftedFrame / fps;- const prevT = (shiftedFrame - 1) / fps;- tl.seek(prevT);+ seekTimeline(compiled, shiftedFrame - 1, fps);const prevTarget = targetId ? targetsById[targetId] : undefined;const prevX = prevTarget?.x ?? baseX;const prevY = prevTarget?.y ?? baseY;- tl.seek(currentT);+ seekTimeline(compiled, shiftedFrame, fps);const vx = sampledX - prevX;const vy = sampledY - prevY;@@ -3683,13 +3679,11 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {let blurPadX = 0;let blurPadY = 0;if (shutterAngle > 0 && fps && shiftedFrame !== undefined) {- const currentT = shiftedFrame / fps;- const prevT = (shiftedFrame - 1) / fps;- tl.seek(prevT);+ seekTimeline(compiled, shiftedFrame - 1, fps);const prevTarget = targetId ? targetsById[targetId] : undefined;const prevX = prevTarget?.x ?? baseX;const prevY = prevTarget?.y ?? baseY;- tl.seek(currentT);+ seekTimeline(compiled, shiftedFrame, fps);const vx = sampledX - prevX;const vy = sampledY - prevY;diff --git a/nodes/node-compositor/src/shared/audio-reactive.test.ts b/nodes/node-compositor/src/shared/audio-reactive.test.tsindex 32346cd6..450bf4c7 100644--- a/nodes/node-compositor/src/shared/audio-reactive.test.ts+++ b/nodes/node-compositor/src/shared/audio-reactive.test.ts@@ -1,5 +1,5 @@import { describe, expect, it } from "vitest";-import { compileTimeline } from "./compiler.js";+import { compileTimeline, seekTimeline } from "./compiler.js";import type { CompositorNodeConfig } from "./config.js";import { processCompositor } from "./processor.js";@@ -166,17 +166,18 @@ describe("Audio-Reactive Motion Graphics Engine Pipeline", () => {expect(signals.audio_beat_handle).toBe(beatSamples);// 2. Compile into timeline- const { tl, targetsById } = compileTimeline("audio-reactive-session", vm, {+ const compiled = compileTimeline("audio-reactive-session", vm, {fps: 24,durationSec: 2,});+ const { targetsById } = compiled;const logoStub = targetsById["bouncing-logo-star"];expect(logoStub).toBeDefined();// Check baseline at frame 0 (t = 0 sec):// Beat signal = 0.0 -> scale = 0.0 * 0.6 + 1.0 = 1.0- tl.seek(0);+ seekTimeline(compiled, 0, 24);expect(logoStub.scale).toBeCloseTo(1.0, 3);// Bass signal = 0.0 -> trimEnd = 0.0 * 0.8 + 0.2 = 0.2expect(logoStub.trimEnd).toBeCloseTo(0.2, 3);@@ -185,16 +186,16 @@ describe("Audio-Reactive Motion Graphics Engine Pipeline", () => {// Check beat hit at frame 12 (t = 12/24 = 0.5 sec):// Beat signal = 1.0 -> scale = 1.0 * 0.6 + 1.0 = 1.6!- tl.seek(12 / 24);+ seekTimeline(compiled, 12, 24);expect(logoStub.scale).toBeCloseTo(1.6, 3);// Check post-beat settling at frame 14:// Beat signal = 0.0 -> scale back to 1.0- tl.seek(14 / 24);+ seekTimeline(compiled, 14, 24);expect(logoStub.scale).toBeCloseTo(1.0, 3);// Check second beat hit at frame 24 (t = 1.0 sec):- tl.seek(24 / 24);+ seekTimeline(compiled, 24, 24);expect(logoStub.scale).toBeCloseTo(1.6, 3);// Check container staggering:@@ -206,17 +207,17 @@ describe("Audio-Reactive Motion Graphics Engine Pipeline", () => {const bar3 = targetsById["bar-3"];// At frame 0:- tl.seek(0);+ seekTimeline(compiled, 0, 24);expect(bar1.opacity).toBe(0);// At frame 4 (bar-1 is halfway faded 0.33, bar-2 is just starting):- tl.seek(4 / 24);+ seekTimeline(compiled, 4, 24);expect(bar1.opacity).toBeGreaterThan(0);expect(bar2.opacity).toBe(0);expect(bar3.opacity).toBe(0);// At frame 8 (bar-3 is just starting):- tl.seek(8 / 24);+ seekTimeline(compiled, 8, 24);expect(bar2.opacity).toBeGreaterThan(0);expect(bar3.opacity).toBe(0);});diff --git a/nodes/node-compositor/src/shared/compiler.test.ts b/nodes/node-compositor/src/shared/compiler.test.tsindex 6ef9b508..37f51408 100644--- a/nodes/node-compositor/src/shared/compiler.test.ts+++ b/nodes/node-compositor/src/shared/compiler.test.ts@@ -1,6 +1,10 @@import gsapModule from "gsap";import { afterEach, describe, expect, it, vi } from "vitest";-import { compileLayerTimeline, compileTimeline } from "./compiler.js";+import {+ compileLayerTimeline,+ compileTimeline,+ seekTimeline,+} from "./compiler.js";const gsap: any = (gsapModule as any).timeline? gsapModule@@ -1114,6 +1118,46 @@ describe("keyframe tail truncation (review M8)", () => {expect(stub.y).toBe(50);});+ it("reads signal tracks when their frame is drawn, not when compiled", () => {+ // A live physics world fills its signals while rendering: values+ // written after compiling must still reach the frame.+ const live = [0, 0, 0, 0];+ const virtualMedia: any = {+ metadata: { width: 1920, height: 1080, durationMs: 2000 },+ operation: { op: "Compositor", signals: { live } },+ children: [+ {+ operation: {+ op: "CompositorLayer",+ id: "live-node",+ x: 0,+ durationFrames: 48,+ animation: {+ tracks: [+ {+ id: "t_live",+ prop: "x" as const,+ source: { type: "signal" as const, inputHandleId: "live" },+ },+ ],+ },+ },+ },+ ],+ };+ const compiled = compileTimeline("render-live-signal-test", virtualMedia, {+ fps: 24,+ durationSec: 2,+ });+ const stub = compiled.targetsById["live-node"];+ live[2] = 120;+ seekTimeline(compiled, 2, 24);+ expect(stub.x).toBe(120);+ live[3] = 160;+ seekTimeline(compiled, 3, 24);+ expect(stub.x).toBe(160);+ });+it("compiles and evaluates procedural wiggle and signal tracks", () => {const virtualMedia: any = {metadata: { width: 1920, height: 1080, durationMs: 2000 },@@ -1160,21 +1204,21 @@ describe("keyframe tail truncation (review M8)", () => {],};- const { tl, targetsById } = compileTimeline(- "render-procedural-test",- virtualMedia,- { fps: 24, durationSec: 2 },- );+ const compiled = compileTimeline("render-procedural-test", virtualMedia, {+ fps: 24,+ durationSec: 2,+ });+ const { targetsById } = compiled;const stub = targetsById["proc-node"];expect(stub).toBeDefined();// Frame 0- tl.seek(0);+ seekTimeline(compiled, 0, 24);expect(stub.scale).toBe(1.0); // 0 * 0.5 + 1.0 = 1.0// Frame 2 (at 24fps = 2/24 sec) -> audio_pulse[2] is 1.0 -> 1.0 * 0.5 + 1.0 = 1.5- tl.seek(2 / 24);+ seekTimeline(compiled, 2, 24);expect(stub.scale).toBeCloseTo(1.5, 4);// Wiggle moves smoothly around baseline 50 +- 40diff --git a/nodes/node-compositor/src/shared/compiler.ts b/nodes/node-compositor/src/shared/compiler.tsindex f844ae79..9e8d0d50 100644--- a/nodes/node-compositor/src/shared/compiler.ts+++ b/nodes/node-compositor/src/shared/compiler.ts@@ -111,6 +111,37 @@ export interface CompileResult {targetsById: Record<string, LayerStub>;/** Node ops the timeline was compiled from (tree order). */nodes: CompilerNodeVM[];+ /**+ * Signal-driven tracks, evaluated at draw time instead of baked into the+ * timeline: a signal may be filled while rendering (a live physics world,+ * on-demand vision), so its value is read when its frame is drawn.+ */+ live: LiveTrack[];+ signals?: Record<string, any>;+}++interface LiveTrack {+ stub: Record<string, any>;+ track: AnimationTrack;+ startFrame: number;+ endFrame: number;+}++/** Seeks the compiled timeline to a frame and applies its live signal tracks. */+export function seekTimeline(+ result: CompileResult,+ frame: number,+ fps: number,+): void {+ result.tl.seek(frame / fps);+ for (const live of result.live) {+ const f = Math.max(live.startFrame, Math.min(live.endFrame, frame));+ live.stub[live.track.prop] = evaluateTrackAtFrame(live.track, f, fps, {+ startFrame: live.startFrame,+ baseValues: live.stub,+ signals: result.signals,+ });+ }}// L2: compiled timelines and their LayerStubs are cached per cacheKey and@@ -341,10 +372,24 @@ function compileTrackToTimeline(fps: number,durationFrames: number,signals?: Record<string, any>,+ live?: LiveTrack[],): { dropped: number; maxAuthoredFrame: number } {const prop = track.prop;const source = track.source ?? { type: "keyframe" };+ if (source.type === "signal" && live && !track.repeat) {+ const endF = Number.isFinite(durationFrames)+ ? durationFrames+ : Math.round(fps * 5);+ stub[prop] = evaluateTrackAtFrame(track, startFrame, fps, {+ startFrame,+ baseValues: stub,+ signals,+ });+ live.push({ stub, track, startFrame, endFrame: startFrame + endF });+ return { dropped: 0, maxAuthoredFrame: 0 };+ }+if (source.type !== "keyframe") {const endF = Number.isFinite(durationFrames)? durationFrames@@ -853,6 +898,7 @@ export function compileTimeline(// them here and warn ONCE per compiled cache key (per-frame seeks hit the// cache, so this is naturally once per render/preview session).const truncations: TruncationInfo[] = [];+ const live: LiveTrack[] = [];nodeOps.forEach((vm) => {const lop = (vm?.operation || {}) as any;@@ -894,6 +940,7 @@ export function compileTimeline(fps,durationFrames,signals,+ live,);if (dropped > 0) {truncations.push({@@ -929,6 +976,8 @@ export function compileTimeline(tl,targetsById,nodes: nodeOps,+ live,+ signals,};setCachedTimeline(cacheKey, compileResult);diff --git a/packages/compositions/src/compositions/canvas-composition.ts b/packages/compositions/src/compositions/canvas-composition.tsindex 7dfc8fb4..e715a1c8 100644--- a/packages/compositions/src/compositions/canvas-composition.ts+++ b/packages/compositions/src/compositions/canvas-composition.ts@@ -247,7 +247,12 @@ export async function drawCompositionTree(const accumulatedClockOffsetSec =inheritedClockOffset + windowStartFrame / fps;if (typeof op.onRequestFrame === "function") {- op.onRequestFrame(frameCtx);+ // May return a promise: per-frame preparation (vision, live physics)+ // finishes before anything in this frame is evaluated.+ const pending = op.onRequestFrame(frameCtx);+ if (pending && typeof (pending as Promise<void>).then === "function") {+ await pending;+ }} else if (op.effect && typeof op.effect.notifyFrame === "function") {op.effect.notifyFrame(frameCtx);}diff --git a/packages/compositions/src/program/animation.ts b/packages/compositions/src/program/animation.tsindex 8583b1d6..0f2bfac6 100644--- a/packages/compositions/src/program/animation.ts+++ b/packages/compositions/src/program/animation.ts@@ -63,6 +63,10 @@ export interface SignalTrackOptions {export interface ColorSignalOptions {colorA: string;colorB: string;+ /** Scales the signal before it picks a color (e.g. `1 / 900` for a speed in px/s). Default 1. */+ multiplier?: number;+ /** Added after `multiplier`. Default 0. */+ offset?: number;colorMode?: "interpolate" | "hueRotate" | "threshold";threshold?: number;smoothing?: number;@@ -834,8 +838,8 @@ export class LayerAnimation implements LayerAnimationSpec {const source: TrackSource = {type: "signal",inputHandleId: handleId,- multiplier: 1.0,- offset: 0.0,+ multiplier: options.multiplier ?? 1.0,+ offset: options.offset ?? 0.0,smoothingWindowFrames: options.smoothing ?? 0,colorMode: options.colorMode ?? "interpolate",colorA: options.colorA,diff --git a/packages/framefields/src/clip-vision.test.ts b/packages/framefields/src/clip-vision.test.tsnew file mode 100644index 00000000..8ba828d0--- /dev/null+++ b/packages/framefields/src/clip-vision.test.ts@@ -0,0 +1,75 @@+import { describe, expect, it } from "vitest";+import { clipVisionFromResults } from "./clip-vision.js";++/** A 160×80 clip: frame f has the left wrist at (20 + 20f, 10) and a 40×40 square silhouette moving right. */+function clip(frames = 4, options = {}) {+ const cell = 4;+ const w = 40;+ const h = 20;+ return clipVisionFromResults(+ 24,+ { width: 160, height: 80, cell },+ Array.from({ length: frames }, (_, f) => {+ const kp = new Array(51).fill(0);+ kp.splice(9 * 3, 3, 20 + 20 * f, 10, 1);+ const mask = new Uint8Array(w * h);+ for (let y = 5; y < 15; y++)+ for (let x = 5 + f * 10; x < 15 + f * 10 && x < w; x++)+ mask[y * w + x] = 255;+ return { kp, mask };+ }),+ { smoothing: 0, ...options },+ );+}++describe("ClipVision", () => {+ it("reads keypoints at composition frames and maps them into a layer box", () => {+ const person = clip().person;+ expect(person.point("leftWrist", 1.5)).toMatchObject({+ x: 50,+ y: 10,+ visibility: 1,+ });+ // 160×80 covering 320×320 at (100, 0): scale 4, centered.+ const placed = person.place({+ x: 100,+ y: 0,+ width: 320,+ height: 320,+ fit: "cover",+ });+ expect(placed.point("leftWrist", 0)).toMatchObject({+ x: 100 - 160 + 80,+ y: 40,+ });+ expect(+ placed.signals("leftWrist").x.get({ frame: 1, fps: 24 } as never),+ ).toBeCloseTo(100 - 160 + 160);+ });++ it("offsets the clip by its start frame", () => {+ const late = clip(4, { startFrame: 10 }).person;+ expect(late.point("leftWrist", 12).x).toBe(60);+ });++ it("smooths over neighbouring frames, weighted by confidence", () => {+ const person = clip(5, { smoothing: 1 }).person;+ // Frames 1..3 at x 40, 60, 80 with weights 1, 2, 1 → 60.+ expect(person.point("leftWrist", 2).x).toBeCloseTo(60);+ });++ it("samples the silhouette with a gradient pointing inward", () => {+ const person = clip().person;+ expect(person.sample(0, 40, 40).alpha).toBeGreaterThan(0.9);+ expect(person.sample(0, 120, 40).alpha).toBe(0);+ const edge = person.sample(0, 62, 40);+ expect(edge.alpha).toBeGreaterThan(0);+ expect(edge.gx).toBeLessThan(0);+ });++ it("explains a read of a frame that was never analysed", () => {+ expect(() => clip(2).person.point("leftWrist", 30)).toThrow(+ /not analysed yet/,+ );+ });+});diff --git a/packages/framefields/src/clip-vision.ts b/packages/framefields/src/clip-vision.tsnew file mode 100644index 00000000..a0df0a9d--- /dev/null+++ b/packages/framefields/src/clip-vision.ts@@ -0,0 +1,665 @@+import { type FrameSignal, Signal } from "@framefields/core";+import { COCO17_KEYPOINT_NAMES, VisionRunner } from "@framefields/vision";+import { mediaDecoderCache } from "@framefields/webgpu-renderers";+import type { VisionProps } from "./effects/index.js";++export type PoseKeypointName = (typeof COCO17_KEYPOINT_NAMES)[number];++export interface ClipVisionOptions {+ /** Composition frame the clip's first frame plays on (default 0). */+ startFrame?: number;+ /** Seconds trimmed off the clip's start, as on its media layer (default 0). */+ trimStartSec?: number;+ /** Analysis frame rate (default: the composition's). */+ fps?: number;+ /** Track 17 COCO keypoints (default true). */+ pose?: boolean;+ /** Track the person's silhouette, dress and held things included (default true). */+ mask?: boolean;+ /** Silhouette cell size in source pixels (default 8: a 1920×1080 clip gives a 240×135 grid). */+ maskCell?: number;+ /** Keypoints are averaged over ±`smoothing` frames, weighted by confidence (default 2; 0 = raw). */+ smoothing?: number;+ variant?: VisionProps["variant"];+ confidence?: number;+ modelsDir?: string;+ baseUrl?: string;+ /**+ * Node: where results persist between runs. Default+ * `$FRAMEFIELDS_VISION_CACHE_DIR` or `~/.cache/framefields/vision`;+ * `false` keeps them in memory only. Browsers always keep them in memory.+ */+ cacheDir?: string | false;+}++/** Where the clip sits in the composition, like a media layer's box and `fit`. */+export interface PersonPlacement {+ x?: number;+ y?: number;+ width: number;+ height: number;+ fit?: "cover" | "contain" | "fill";+}++interface FrameResult {+ /** 17 × [x, y, visibility], source px; null when no person was found. */+ kp: Float32Array | null;+ /** cellW × cellH, 0–255; null when no person was found. */+ mask: Uint8Array | null;+}++interface Transform {+ sx: number;+ sy: number;+ ox: number;+ oy: number;+}++/** Frames looked back over to bridge a gap where the person is lost. */+const HOLD = 24;++const CACHE_VERSION = 1;++/**+ * Vision results for one clip, computed per frame on demand and shared by+ * everything that reads the clip: physics, layout, signals. Unlike the+ * render-time `withVision`, which analyses a layer's previous rendered frame,+ * this analyses the clip's own frames, so a result exists before its frame+ * renders. The composition analyses what each frame needs just before+ * drawing it (and a live physics world before stepping), so preview and+ * export fill the cache as they play; nothing has to be analysed up front.+ *+ * Results depend only on the clip and settings, so they persist on disk in+ * Node: a second run, or a render after a preview, reuses them.+ */+export class ClipVision {+ readonly source: string;+ readonly fps: number;+ readonly startFrame: number;+ readonly trimStartSec: number;+ readonly smoothing: number;+ private readonly options: ClipVisionOptions;+ private readonly results = new Map<number, FrameResult>();+ private sourceWidth = 0;+ private sourceHeight = 0;+ private cellW = 0;+ private cellH = 0;+ private readonly cell: number;+ private queue: Promise<void> = Promise.resolve();+ private runner?: VisionRunner;+ private cacheLoaded?: Promise<void>;+ private unsaved = 0;+ private readonly blurred = new Map<number, Float32Array | null>();+ private readonly smoothed = new Map<string, [number, number, number]>();+ private personView?: PersonTrack;++ /** @internal Use `comp.vision(source, options)`. */+ constructor(source: string, fps: number, options: ClipVisionOptions = {}) {+ this.source = source;+ this.options = options;+ this.fps = options.fps ?? fps;+ this.startFrame = Math.round(options.startFrame ?? 0);+ this.trimStartSec = options.trimStartSec ?? 0;+ this.smoothing = Math.max(0, Math.round(options.smoothing ?? 2));+ this.cell = Math.max(1, Math.round(options.maskCell ?? 8));+ }++ /** The tracked person in source pixels; `.place()` maps it into the composition. */+ get person(): PersonTrack {+ this.personView ??= new PersonTrack(this);+ return this.personView;+ }++ /**+ * Analyses what reading composition frame `frame` needs (it and a few+ * neighbours for smoothing). Pass a second frame to analyse a whole range.+ * Calls queue, so the clip is decoded once, in order.+ */+ ready(frame: number, toFrame: number = frame): Promise<void> {+ const [from, to] = this.window(frame, toFrame);+ this.queue = this.queue.then(() => this.compute(from, to));+ return this.queue;+ }++ /** True when composition frame `frame` can be read synchronously. */+ isReady(frame: number): boolean {+ const [from, to] = this.window(frame, frame);+ for (let c = from; c <= to; c++) if (!this.results.has(c)) return false;+ return true;+ }++ /** Composition frame → clip frame (fractional). */+ clipFrame(frame: number): number {+ return frame - this.startFrame + this.trimStartSec * this.fps;+ }++ private window(frame: number, toFrame: number): [number, number] {+ const a = Math.floor(this.clipFrame(Math.min(frame, toFrame)));+ const b = Math.ceil(this.clipFrame(Math.max(frame, toFrame)));+ return [+ Math.max(0, a - HOLD - this.smoothing),+ Math.max(0, b + this.smoothing),+ ];+ }++ private async compute(from: number, to: number): Promise<void> {+ await (this.cacheLoaded ??= this.loadCache());+ let missing = false;+ for (let c = from; c <= to; c++) if (!this.results.has(c)) missing = true;+ if (!missing) return;++ const wantPose = this.options.pose ?? true;+ const wantMask = this.options.mask ?? true;+ this.runner ??= VisionRunner.create({+ modelsDir: this.options.modelsDir,+ baseUrl: this.options.baseUrl,+ ...(this.options.confidence !== undefined && {+ confidence: this.options.confidence,+ }),+ ...(this.options.variant !== undefined && {+ variant: this.options.variant,+ }),+ });+ const decoder = mediaDecoderCache.getVideo(+ this.source,+ true,+ "clip-vision",+ );+ for (let c = from; c <= to; c++) {+ if (this.results.has(c)) continue;+ const image = await decodedPixels(await decoder.getFrame(c / this.fps));+ if (!image) {+ this.results.set(c, { kp: null, mask: null });+ continue;+ }+ if (!this.sourceWidth) this.setSize(image.width, image.height);+ let kp: Float32Array | null = null;+ let mask: Uint8Array | null = null;+ if (wantPose) {+ const person = (await this.runner.pose(image)).people[0];+ if (person) {+ kp = new Float32Array(51);+ person.keypoints.forEach((p, k) =>+ kp?.set([p.x, p.y, p.visibility], k * 3),+ );+ }+ }+ if (wantMask) {+ const silhouette = personSilhouette(await this.runner.segment(image));+ if (silhouette) {+ mask = new Uint8Array(this.cellW * this.cellH);+ downsampleMean(+ silhouette,+ image.width,+ image.height,+ this.cell,+ mask,+ this.cellW,+ this.cellH,+ );+ }+ }+ this.results.set(c, { kp, mask });+ if (++this.unsaved >= 24) await this.saveCache();+ }+ if (this.unsaved > 0) await this.saveCache();+ }++ private setSize(width: number, height: number): void {+ this.sourceWidth = width;+ this.sourceHeight = height;+ this.cellW = Math.ceil(width / this.cell);+ this.cellH = Math.ceil(height / this.cell);+ }++ /** @internal */+ size(): { width: number; height: number } {+ return { width: this.sourceWidth, height: this.sourceHeight };+ }++ /** @internal Raw result for a clip frame, or throws when it was never analysed. */+ result(c: number): FrameResult | undefined {+ return this.results.get(Math.max(0, c));+ }++ /** @internal Confidence-weighted keypoint over the smoothing window, in source px. */+ keypoint(k: number, c: number): [number, number, number] {+ c = Math.max(0, c);+ const key = `${k}:${c}`;+ const hit = this.smoothed.get(key);+ if (hit) return hit;+ const avg = (center: number): [number, number, number] | null => {+ let sx = 0;+ let sy = 0;+ let sv = 0;+ let sw = 0;+ for (let g = center - this.smoothing; g <= center + this.smoothing; g++) {+ const r = this.mustHave(g < 0 ? 0 : g);+ if (!r.kp) continue;+ const v = r.kp[k * 3 + 2];+ const w = (this.smoothing + 1 - Math.abs(g - center)) * v * v;+ sx += r.kp[k * 3] * w;+ sy += r.kp[k * 3 + 1] * w;+ sv += v * w;+ sw += w;+ }+ return sw > 1e-6 ? [sx / sw, sy / sw, sv / sw] : null;+ };+ let out: [number, number, number] | null = null;+ // Lost? Hold the last place the person was seen (within HOLD frames).+ for (let d = 0; d <= HOLD && !out; d++) if (c - d >= 0) out = avg(c - d);+ out ??= [this.sourceWidth / 2, this.sourceHeight / 2, 0];+ this.smoothed.set(key, out);+ return out;+ }++ /** @internal Softened silhouette grid for a clip frame (null: no person). */+ blurredMask(c: number): Float32Array | null {+ c = Math.max(0, c);+ if (this.blurred.has(c)) return this.blurred.get(c) ?? null;+ const r = this.mustHave(c);+ const grid = r.mask ? boxBlur(r.mask, this.cellW, this.cellH, 2) : null;+ this.blurred.set(c, grid);+ return grid;+ }++ /** @internal */+ grid(): { cell: number; width: number; height: number } {+ return { cell: this.cell, width: this.cellW, height: this.cellH };+ }++ private mustHave(c: number): FrameResult {+ const r = this.results.get(c);+ if (r) return r;+ throw new Error(+ `ClipVision "${this.source}": frame ${c} is not analysed yet. ` ++ "Read it while rendering (the composition prepares each frame), in a live physics world, " ++ `or after \`await clip.ready(frame)\`.`,+ );+ }++ // ── Persistence (Node) ────────────────────────────────────────────────++ private cacheFile?: string | null;++ private async cachePath(): Promise<string | null> {+ if (this.cacheFile !== undefined) return this.cacheFile;+ this.cacheFile = null;+ if (+ this.options.cacheDir === false ||+ typeof process === "undefined" ||+ !process.versions?.node+ ) {+ return null;+ }+ try {+ const [fs, path, os, crypto] = await Promise.all([+ import("node:fs"),+ import("node:path"),+ import("node:os"),+ import("node:crypto"),+ ]);+ const stat = fs.statSync(this.source);+ const dir =+ this.options.cacheDir ??+ process.env.FRAMEFIELDS_VISION_CACHE_DIR ??+ path.join(os.homedir(), ".cache", "framefields", "vision");+ const o = this.options;+ const key = JSON.stringify([+ CACHE_VERSION,+ path.resolve(this.source),+ stat.size,+ stat.mtimeMs,+ this.fps,+ o.pose ?? true,+ o.mask ?? true,+ this.cell,+ o.variant ?? "s",+ o.confidence ?? null,+ ]);+ const hash = crypto+ .createHash("sha256")+ .update(key)+ .digest("hex")+ .slice(0, 24);+ fs.mkdirSync(dir, { recursive: true });+ this.cacheFile = path.join(+ dir,+ `${path.basename(this.source)}.${hash}.json`,+ );+ } catch {+ this.cacheFile = null; // remote URL or unreadable: memory only+ }+ return this.cacheFile;+ }++ private async loadCache(): Promise<void> {+ const file = await this.cachePath();+ if (!file) return;+ try {+ const fs = await import("node:fs");+ if (!fs.existsSync(file)) return;+ const json = JSON.parse(fs.readFileSync(file, "utf8")) as {+ width: number;+ height: number;+ cell: number;+ frames: Record<string, { kp: number[] | null; mask: string | null }>;+ };+ if (json.cell !== this.cell) return;+ this.setSize(json.width, json.height);+ for (const [c, r] of Object.entries(json.frames)) {+ this.results.set(Number(c), {+ kp: r.kp ? Float32Array.from(r.kp) : null,+ mask: r.mask ? new Uint8Array(Buffer.from(r.mask, "base64")) : null,+ });+ }+ } catch {+ // A damaged cache is recomputed.+ }+ }++ private async saveCache(): Promise<void> {+ this.unsaved = 0;+ const file = await this.cachePath();+ if (!file || !this.sourceWidth) return;+ const frames: Record<string, { kp: number[] | null; mask: string | null }> =+ {};+ for (const [c, r] of this.results) {+ frames[c] = {+ kp: r.kp ? Array.from(r.kp, (v) => Math.round(v * 100) / 100) : null,+ mask: r.mask ? Buffer.from(r.mask).toString("base64") : null,+ };+ }+ const fs = await import("node:fs");+ const tmp = `${file}.${process.pid}.tmp`;+ fs.writeFileSync(+ tmp,+ JSON.stringify({+ width: this.sourceWidth,+ height: this.sourceHeight,+ cell: this.cell,+ frames,+ }),+ );+ fs.renameSync(tmp, file);+ }+}++/**+ * The person in a {@link ClipVision}: pose keypoints and silhouette at any+ * composition frame, in source pixels or, after `place()`, composition+ * pixels. Reads are synchronous; the frame must be analysed (it is while+ * rendering, inside a live physics world, or after `await clip.ready(frame)`).+ */+export class PersonTrack {+ /** @internal */+ constructor(+ readonly clip: ClipVision,+ private readonly t?: Transform,+ private readonly placement?: PersonPlacement,+ ) {}++ /** The same person in composition pixels for a clip laid out at `placement`. */+ place(placement: PersonPlacement): PersonTrack {+ return new PersonTrack(this.clip, undefined, placement);+ }++ private transform(): Transform {+ if (this.t) return this.t;+ const p = this.placement;+ const { width: sw, height: sh } = this.clip.size();+ if (!p || !sw) return { sx: 1, sy: 1, ox: 0, oy: 0 };+ let sx = p.width / sw;+ let sy = p.height / sh;+ if (p.fit !== "fill") {+ const s = p.fit === "contain" ? Math.min(sx, sy) : Math.max(sx, sy);+ sx = s;+ sy = s;+ }+ return {+ sx,+ sy,+ ox: (p.x ?? 0) + (p.width - sw * sx) / 2,+ oy: (p.y ?? 0) + (p.height - sh * sy) / 2,+ };+ }++ /** A keypoint at a composition frame (fractional frames interpolate). */+ point(+ name: PoseKeypointName,+ frame: number,+ ): { x: number; y: number; visibility: number } {+ const k = keypointIndex(name);+ const c = this.clip.clipFrame(frame);+ const c0 = Math.floor(c);+ const u = c - c0;+ const a = this.clip.keypoint(k, c0);+ const b = u > 0 ? this.clip.keypoint(k, c0 + 1) : a;+ const t = this.transform();+ return {+ x: (a[0] + (b[0] - a[0]) * u) * t.sx + t.ox,+ y: (a[1] + (b[1] - a[1]) * u) * t.sy + t.oy,+ visibility: a[2] + (b[2] - a[2]) * u,+ };+ }++ /** Signals for a keypoint, for `.signal("x", …)`; read while rendering. */+ signals(name: PoseKeypointName): {+ x: FrameSignal<number>;+ y: FrameSignal<number>;+ visibility: FrameSignal<number>;+ } {+ const read = (axis: "x" | "y" | "visibility") =>+ Signal.builder({+ type: "custom",+ fn: (ctx) => this.point(name, ctx.frame)[axis],+ }) as FrameSignal<number>;+ return { x: read("x"), y: read("y"), visibility: read("visibility") };+ }++ /**+ * The silhouette near a point at a composition frame: `alpha` 0–1 (softened+ * over a few cells; inside > 0.5) and its gradient `gx`, `gy` per px,+ * pointing into the body. Push things out with `-gradient`.+ */+ sample(+ frame: number,+ x: number,+ y: number,+ ): { alpha: number; gx: number; gy: number } {+ const grid = this.clip.blurredMask(Math.round(this.clip.clipFrame(frame)));+ if (!grid) return { alpha: 0, gx: 0, gy: 0 };+ const { cell, width: w, height: h } = this.clip.grid();+ const t = this.transform();+ const cx = (x - t.ox) / t.sx / cell - 0.5;+ const cy = (y - t.oy) / t.sy / cell - 0.5;+ const read = (gx: number, gy: number) =>+ gx < 0 || gy < 0 || gx >= w || gy >= h ? 0 : grid[gy * w + gx];+ const bil = (px: number, py: number) => {+ const x0 = Math.floor(px);+ const y0 = Math.floor(py);+ const ux = px - x0;+ const uy = py - y0;+ const a = read(x0, y0) + (read(x0 + 1, y0) - read(x0, y0)) * ux;+ const b =+ read(x0, y0 + 1) + (read(x0 + 1, y0 + 1) - read(x0, y0 + 1)) * ux;+ return a + (b - a) * uy;+ };+ return {+ alpha: bil(cx, cy),+ gx: (bil(cx + 1, cy) - bil(cx - 1, cy)) / (2 * cell * t.sx),+ gy: (bil(cx, cy + 1) - bil(cx, cy - 1)) / (2 * cell * t.sy),+ };+ }+}++function keypointIndex(name: PoseKeypointName): number {+ const i = COCO17_KEYPOINT_NAMES.indexOf(name);+ if (i < 0) {+ throw new Error(+ `Unknown keypoint "${name}". Use one of ${COCO17_KEYPOINT_NAMES.join(", ")}.`,+ );+ }+ return i;+}++/** RGBA pixels of a decoded frame (headless buffer, or a canvas in the browser). */+async function decodedPixels(+ frame: {+ width: number;+ height: number;+ buffer?: Uint8Array;+ canvas?: unknown;+ } | null,+): Promise<{+ data: Uint8Array | Uint8ClampedArray;+ width: number;+ height: number;+} | null> {+ if (!frame || frame.width <= 0) return null;+ if (frame.buffer)+ return { data: frame.buffer, width: frame.width, height: frame.height };+ const canvas = frame.canvas as+ | {+ getContext(type: "2d"): {+ getImageData(+ x: number,+ y: number,+ w: number,+ h: number,+ ): { data: Uint8ClampedArray };+ } | null;+ }+ | undefined;+ const ctx = canvas?.getContext("2d");+ if (!ctx) return null;+ return {+ data: ctx.getImageData(0, 0, frame.width, frame.height).data,+ width: frame.width,+ height: frame.height,+ };+}++type Segmentation = Awaited<ReturnType<VisionRunner["segment"]>>;++/**+ * The largest person's mask, plus instances that overlap it and are no more+ * than twice its size (a flowing dress often comes back as "kite").+ */+function personSilhouette(seg: Segmentation): Uint8Array | undefined {+ const people = seg.masks.filter((m) => m.category === "person");+ if (people.length === 0) return undefined;+ const person = people.reduce((a, b) => (b.area > a.area ? b : a));+ const box = seg.detections[person.detectionIndex]?.boundingBox;+ const out = new Uint8Array(person.mask);+ if (!box) return out;+ for (const m of seg.masks) {+ if (m === person || m.area > person.area * 2) continue;+ const b = seg.detections[m.detectionIndex]?.boundingBox;+ if (!b) continue;+ const ix = Math.max(+ 0,+ Math.min(box.originX + box.width, b.originX + b.width) -+ Math.max(box.originX, b.originX),+ );+ const iy = Math.max(+ 0,+ Math.min(box.originY + box.height, b.originY + b.height) -+ Math.max(box.originY, b.originY),+ );+ if (ix * iy < 0.3 * b.width * b.height) continue;+ for (let i = 0; i < out.length; i++)+ if (m.mask[i] > out[i]) out[i] = m.mask[i];+ }+ return out;+}++function downsampleMean(+ src: Uint8Array,+ w: number,+ h: number,+ cell: number,+ dst: Uint8Array,+ mw: number,+ mh: number,+): void {+ for (let my = 0; my < mh; my++) {+ for (let mx = 0; mx < mw; mx++) {+ let sum = 0;+ let n = 0;+ for (let y = my * cell; y < Math.min(h, (my + 1) * cell); y++) {+ for (let x = mx * cell; x < Math.min(w, (mx + 1) * cell); x++) {+ sum += src[y * w + x];+ n++;+ }+ }+ dst[my * mw + mx] = n ? Math.round(sum / n) : 0;+ }+ }+}++/** Separable box blur to 0–1, so gradients reach a little past the edge. */+function boxBlur(+ src: Uint8Array,+ w: number,+ h: number,+ r: number,+): Float32Array {+ const tmp = new Float32Array(w * h);+ const dst = new Float32Array(w * h);+ for (let y = 0; y < h; y++) {+ for (let x = 0; x < w; x++) {+ let s = 0;+ let n = 0;+ for (let k = -r; k <= r; k++) {+ const xx = x + k;+ if (xx >= 0 && xx < w) {+ s += src[y * w + xx];+ n++;+ }+ }+ tmp[y * w + x] = s / n / 255;+ }+ }+ for (let y = 0; y < h; y++) {+ for (let x = 0; x < w; x++) {+ let s = 0;+ let n = 0;+ for (let k = -r; k <= r; k++) {+ const yy = y + k;+ if (yy >= 0 && yy < h) {+ s += tmp[yy * w + x];+ n++;+ }+ }+ dst[y * w + x] = s / n;+ }+ }+ return dst;+}++/** @internal Test hook: a clip whose results are given instead of computed. */+export function clipVisionFromResults(+ fps: number,+ size: { width: number; height: number; cell: number },+ frames: Array<{ kp: number[] | null; mask: Uint8Array | null }>,+ options: ClipVisionOptions = {},+): ClipVision {+ const clip = new ClipVision("memory", fps, {+ ...options,+ maskCell: size.cell,+ cacheDir: false,+ });+ const c = clip as unknown as {+ setSize(w: number, h: number): void;+ results: Map<number, FrameResult>;+ };+ c.setSize(size.width, size.height);+ frames.forEach((f, i) => {+ c.results.set(i, {+ kp: f.kp ? Float32Array.from(f.kp) : null,+ mask: f.mask,+ });+ });+ return clip;+}diff --git a/packages/framefields/src/index.ts b/packages/framefields/src/index.tsindex 2b870ab4..20acb7aa 100644--- a/packages/framefields/src/index.ts+++ b/packages/framefields/src/index.ts@@ -188,6 +188,7 @@ import {} from "@framefields/tensor-webgpu";import { inputStore, mediaDecoderCache } from "@framefields/webgpu-renderers";import { buildChart, type ChartOptions } from "./chart.js";+import { ClipVision, type ClipVisionOptions } from "./clip-vision.js";import {ApplyLUT,type ApplyLUTProps,@@ -240,11 +241,6 @@ import {Vision,type VisionProps,} from "./effects/index.js";-import {- type PersonTrack,- type TrackPersonOptions,- trackPerson,-} from "./person-track.js";import {type FrameGridOptions,HeadlessMediaRenderer,@@ -669,6 +665,13 @@ export {type ChartX,DEFAULT_CHART_COLORS,} from "./chart.js";+export {+ ClipVision,+ type ClipVisionOptions,+ type PersonPlacement,+ PersonTrack,+ type PoseKeypointName,+} from "./clip-vision.js";export {ApplyLUT,type ApplyLUTProps,@@ -790,13 +793,6 @@ export {type LightingRig,type LightingRigOptions,} from "./lighting.js";-export {- PersonTrack,- type PersonTrackJSON,- type PersonTrackPlacement,- type PoseKeypointName,- type TrackPersonOptions,-} from "./person-track.js";export {type Body2DOptions,type Body3DOptions,@@ -1107,6 +1103,9 @@ export class Composition {public audioTracks: MediaNode[] = [];public signals: Record<string, unknown> = {};private _frameHooks: Array<(ctx: FrameContext) => void> = [];+ private _prepareHooks: Array<(ctx: FrameContext) => Promise<void> | void> =+ [];+ private readonly _clipVision = new Map<string, ClipVision>();private _visionConfig?: VisionProps;private _visionBundle?: VisionBundle;private _effects: unknown[] = [];@@ -1145,22 +1144,27 @@ export class Composition {}/**- * Tracks one person through a clip before rendering: smoothed pose- * keypoints and a soft silhouette per frame (see {@link PersonTrack}).- * Use it to drive physics with a performer (limbs as kinematic bodies,- * the silhouette as a force field) or to keyframe things to a body.- * Analysis takes a few hundred ms per frame; cache it with `toJSON()` /- * `PersonTrack.fromJSON()`.+ * Vision for a clip, analysed per frame on demand and shared by everything+ * that reads it (see {@link ClipVision}): `comp.vision("dancer.mp4").person`+ * gives pose keypoints and a silhouette at any frame. Each frame is+ * analysed just before it renders, so nothing runs up front. Results+ * persist on disk in Node. The same clip and options return one instance.*/- public async trackPerson(- source: string,- options: TrackPersonOptions = {},- ): Promise<PersonTrack> {- const fps = options.fps ?? this.fps;- const durationMs = this.durationMs ?? (await this.computeDuration());- const totalFrames =- options.totalFrames ?? Math.max(1, Math.round((durationMs / 1000) * fps));- return trackPerson(source, { ...options, fps, totalFrames });+ public vision(source: string, options: ClipVisionOptions = {}): ClipVision {+ const key = `${source}\u0000${JSON.stringify(options)}`;+ let clip = this._clipVision.get(key);+ if (!clip) {+ clip = new ClipVision(source, this.fps, options);+ this._clipVision.set(key, clip);+ }+ return clip;+ }++ /** @internal Analyses every `comp.vision` clip for a composition frame. */+ public async prepareVision(frame: number): Promise<void> {+ await Promise.all(+ [...this._clipVision.values()].map((clip) => clip.ready(frame)),+ );}/**@@ -1253,10 +1257,37 @@ export class Composition {return this;}- public notifyFrame(ctx: FrameContext): void {+ /**+ * Async work a frame needs before it is evaluated (vision, a live physics+ * world). Every render path awaits these, in order, before drawing the+ * frame; they may run more than once per frame, so keep them idempotent.+ */+ public onPrepareFrame(+ callback: (ctx: FrameContext) => Promise<void> | void,+ ): this {+ this._prepareHooks.push(callback);+ return this;+ }++ public notifyFrame(ctx: FrameContext): Promise<void> | undefined {for (const hook of this._frameHooks) {hook(ctx);}+ if (this._clipVision.size === 0 && this._prepareHooks.length === 0) {+ return undefined;+ }+ return (async () => {+ await this.prepareVision(ctx.frame);+ for (const hook of this._prepareHooks) await hook(ctx);+ })();+ }++ private hasFrameHooks(): boolean {+ return (+ this._frameHooks.length > 0 ||+ this._prepareHooks.length > 0 ||+ this._clipVision.size > 0+ );}constructor(options: CompositionOptions) {@@ -1789,7 +1820,7 @@ export class Composition {signals: this.signals,...(this._effects.length > 0 && { effects: this._effects }),...(this._visionConfig && { vision: this._visionConfig }),- ...(this._frameHooks.length > 0 && {+ ...(this.hasFrameHooks() && {onRequestFrame: (ctx: FrameContext) => this.notifyFrame(ctx),}),} as unknown as CompositorProgramConfig;@@ -1843,13 +1874,13 @@ export class Composition {renderer?: HeadlessMediaRenderer;} = {},): Promise<Buffer> {- if (this._frameHooks.length > 0) {+ if (this.hasFrameHooks()) {const targetFrame =options.frame ??(options.atMs !== undefined? Math.round((options.atMs / 1000) * this.fps): 0);- this.notifyFrame({+ await this.notifyFrame({frame: targetFrame,fps: this.fps,time: this.fps > 0 ? targetFrame / this.fps : 0,diff --git a/packages/framefields/src/person-track.test.ts b/packages/framefields/src/person-track.test.tsdeleted file mode 100644index 51abd185..00000000--- a/packages/framefields/src/person-track.test.ts+++ /dev/null@@ -1,81 +0,0 @@-import { describe, expect, it } from "vitest";-import { PersonTrack, type PersonTrackJSON } from "./index.js";--/** Two frames of a 160×80 clip: a wrist moving right, a 40×40 square silhouette moving right. */-function synthetic(): PersonTrack {- const frames = 2;- const keypoints = new Array(frames * 17 * 3).fill(0);- const set = (f: number, k: number, x: number, y: number) =>- keypoints.splice((f * 17 + k) * 3, 3, x, y, 1);- set(0, 9, 20, 10); // leftWrist- set(1, 9, 60, 30);- const cell = 4;- const w = 40;- const h = 20;- const mask = new Uint8Array(frames * w * h);- for (let f = 0; f < frames; f++) {- for (let y = 5; y < 15; y++)- for (let x = 5 + f * 10; x < 15 + f * 10; x++)- mask[f * w * h + y * w + x] = 255;- }- const json: PersonTrackJSON = {- version: 1,- fps: 24,- frames,- sourceWidth: 160,- sourceHeight: 80,- keypoints,- mask: {- cell,- width: w,- height: h,- data: Buffer.from(mask).toString("base64"),- },- };- return PersonTrack.fromJSON(JSON.parse(JSON.stringify(json)));-}--describe("PersonTrack", () => {- it("interpolates keypoints and maps them into a layer box", () => {- const t = synthetic();- expect(t.point("leftWrist", 0.5)).toMatchObject({- x: 40,- y: 20,- visibility: 1,- });- // 160×80 covering 320×320 at (100, 0): scale 4, centered → x offset 100 − 160.- const placed = t.place({- x: 100,- y: 0,- width: 320,- height: 320,- fit: "cover",- });- expect(placed.point("leftWrist", 0)).toMatchObject({- x: 100 - 160 + 80,- y: 40,- });- const sig = placed.signals("leftWrist");- expect(sig.x.get({ frame: 1, fps: 24 } as never)).toBeCloseTo(- 100 - 160 + 240,- );- });-- it("samples the silhouette with a gradient pointing inward", () => {- const t = synthetic();- // Frame 0 square spans source px 20–60 × 20–60.- expect(t.sample(0, 40, 40).alpha).toBeGreaterThan(0.9);- expect(t.sample(0, 120, 40).alpha).toBe(0);- const edge = t.sample(0, 62, 40);- expect(edge.alpha).toBeGreaterThan(0);- expect(edge.gx).toBeLessThan(0); // inside is to the left- expect(t.sample(1, 40, 40).alpha).toBeLessThan(t.sample(1, 80, 40).alpha);- });-- it("round-trips through JSON", () => {- const t = synthetic();- const again = PersonTrack.fromJSON(t.toJSON());- expect(again.point("leftWrist", 1)).toEqual(t.point("leftWrist", 1));- expect(again.sample(0, 40, 40)).toEqual(t.sample(0, 40, 40));- });-});diff --git a/packages/framefields/src/person-track.ts b/packages/framefields/src/person-track.tsdeleted file mode 100644index 8001a196..00000000--- a/packages/framefields/src/person-track.ts+++ /dev/null@@ -1,511 +0,0 @@-import { type FrameSignal, Signal } from "@framefields/core";-import { COCO17_KEYPOINT_NAMES, VisionRunner } from "@framefields/vision";-import { mediaDecoderCache } from "@framefields/webgpu-renderers";-import type { VisionProps } from "./effects/index.js";--export type PoseKeypointName = (typeof COCO17_KEYPOINT_NAMES)[number];--export interface TrackPersonOptions {- /** Frames per second to sample (default: the composition's). */- fps?: number;- /** How many frames (default: the composition's length). */- totalFrames?: number;- /** Track 17 COCO keypoints (default true). */- pose?: boolean;- /** Track the person's silhouette, dress and held things included (default true). */- mask?: boolean;- /** Silhouette cell size in source pixels (default 8: a 1920×1080 clip gives a 240×135 grid). */- maskCell?: number;- /** Frames of zero-lag smoothing on keypoints (default 2; 0 for raw). */- smoothing?: number;- /** Keypoints below this visibility count as missing and are filled from neighbours (default 0.3). */- minVisibility?: number;- variant?: VisionProps["variant"];- confidence?: number;- modelsDir?: string;- baseUrl?: string;- onProgress?: (frame: number, total: number) => void;-}--/** Where the clip sits in the composition, like a media layer's box and `fit`. */-export interface PersonTrackPlacement {- x?: number;- y?: number;- width: number;- height: number;- fit?: "cover" | "contain" | "fill";-}--/** Plain-JSON form, for caching an analysis between runs. */-export interface PersonTrackJSON {- version: 1;- fps: number;- frames: number;- sourceWidth: number;- sourceHeight: number;- /** frames × 17 × [x, y, visibility], source px, smoothed. */- keypoints?: number[];- mask?: { cell: number; width: number; height: number; data: string };-}--interface Transform {- sx: number;- sy: number;- ox: number;- oy: number;-}--const IDENTITY: Transform = { sx: 1, sy: 1, ox: 0, oy: 0 };--/**- * One person tracked through a clip ahead of rendering: smoothed pose- * keypoints and a soft silhouette, frame by frame. Unlike the render-time- * vision signals it is complete before the first frame renders, so it can- * drive physics (limbs as kinematic bodies, the silhouette as a force field),- * be keyframed, or be inspected. `place()` maps it into composition pixels.- */-export class PersonTrack {- readonly fps: number;- readonly frames: number;- readonly sourceWidth: number;- readonly sourceHeight: number;- private readonly kp?: Float32Array;- private readonly maskData?: {- cell: number;- width: number;- height: number;- data: Uint8Array;- };- private blurred?: Float32Array[];-- private constructor(- json: Omit<PersonTrackJSON, "keypoints" | "mask" | "version">,- kp: Float32Array | undefined,- mask: PersonTrack["maskData"],- private readonly t: Transform = IDENTITY,- shared?: PersonTrack,- ) {- this.fps = json.fps;- this.frames = json.frames;- this.sourceWidth = json.sourceWidth;- this.sourceHeight = json.sourceHeight;- this.kp = kp;- this.maskData = mask;- if (shared) this.blurred = shared.blurredMasks();- }-- static fromJSON(json: PersonTrackJSON): PersonTrack {- const mask = json.mask- ? {- cell: json.mask.cell,- width: json.mask.width,- height: json.mask.height,- data: new Uint8Array(Buffer.from(json.mask.data, "base64")),- }- : undefined;- return new PersonTrack(- json,- json.keypoints ? Float32Array.from(json.keypoints) : undefined,- mask,- );- }-- toJSON(): PersonTrackJSON {- const m = this.maskData;- return {- version: 1,- fps: this.fps,- frames: this.frames,- sourceWidth: this.sourceWidth,- sourceHeight: this.sourceHeight,- ...(this.kp && {- keypoints: Array.from(this.kp, (v) => Math.round(v * 100) / 100),- }),- ...(m && {- mask: {- cell: m.cell,- width: m.width,- height: m.height,- data: Buffer.from(m.data).toString("base64"),- },- }),- };- }-- /** The same track in composition pixels for a clip laid out at `placement`. */- place(placement: PersonTrackPlacement): PersonTrack {- const { width: W, height: H } = placement;- const sw = this.sourceWidth;- const sh = this.sourceHeight;- let sx = W / sw;- let sy = H / sh;- if (placement.fit !== "fill") {- const s =- placement.fit === "contain" ? Math.min(sx, sy) : Math.max(sx, sy);- sx = s;- sy = s;- }- const t: Transform = {- sx,- sy,- ox: (placement.x ?? 0) + (W - sw * sx) / 2,- oy: (placement.y ?? 0) + (H - sh * sy) / 2,- };- return new PersonTrack(this, this.kp, this.maskData, t, this);- }-- /** Whether the clip had pose / silhouette tracking. */- get hasPose(): boolean {- return this.kp !== undefined;- }- get hasMask(): boolean {- return this.maskData !== undefined;- }-- /** A keypoint at a frame (fractional frames interpolate), placed px. */- point(- name: PoseKeypointName,- frame: number,- ): { x: number; y: number; visibility: number } {- const kp = this.requirePose();- const k = keypointIndex(name);- const f = Math.max(0, Math.min(this.frames - 1, frame));- const f0 = Math.floor(f);- const f1 = Math.min(this.frames - 1, f0 + 1);- const u = f - f0;- const at = (fi: number, c: number) => kp[(fi * 17 + k) * 3 + c];- const lerp = (c: number) => at(f0, c) + (at(f1, c) - at(f0, c)) * u;- return {- x: lerp(0) * this.t.sx + this.t.ox,- y: lerp(1) * this.t.sy + this.t.oy,- visibility: lerp(2),- };- }-- /** Per-frame signals for a keypoint (placed px), for `.signal("x", …)` or physics `follow`. */- signals(- name: PoseKeypointName,- options: { startFrame?: number } = {},- ): {- x: FrameSignal<number>;- y: FrameSignal<number>;- visibility: FrameSignal<number>;- } {- const start = options.startFrame ?? 0;- const x = new Float32Array(start + this.frames);- const y = new Float32Array(start + this.frames);- const v = new Float32Array(start + this.frames);- for (let f = 0; f < start + this.frames; f++) {- const p = this.point(name, f - start);- x[f] = p.x;- y[f] = p.y;- v[f] = p.visibility;- }- return {- x: Signal.fromArray(x, this.fps),- y: Signal.fromArray(y, this.fps),- visibility: Signal.fromArray(v, this.fps),- };- }-- /**- * The silhouette near a point (placed px) at a frame: `alpha` 0–1 (softened- * over a few cells), and its gradient `gx`, `gy` per px, pointing into the- * body. Push things out with `-gradient`; `alpha > 0.5` is inside.- */- sample(- frame: number,- x: number,- y: number,- ): { alpha: number; gx: number; gy: number } {- const m = this.requireMask();- const grids = this.blurredMasks();- const fi = Math.max(0, Math.min(this.frames - 1, Math.round(frame)));- const grid = grids[fi];- // Placed px → mask cells.- const cx = (x - this.t.ox) / this.t.sx / m.cell - 0.5;- const cy = (y - this.t.oy) / this.t.sy / m.cell - 0.5;- const read = (gx: number, gy: number) => {- if (gx < 0 || gy < 0 || gx >= m.width || gy >= m.height) return 0;- return grid[gy * m.width + gx];- };- const bil = (px: number, py: number) => {- const x0 = Math.floor(px);- const y0 = Math.floor(py);- const ux = px - x0;- const uy = py - y0;- const a = read(x0, y0) + (read(x0 + 1, y0) - read(x0, y0)) * ux;- const b =- read(x0, y0 + 1) + (read(x0 + 1, y0 + 1) - read(x0, y0 + 1)) * ux;- return a + (b - a) * uy;- };- const alpha = bil(cx, cy);- const pxPerCellX = m.cell * this.t.sx;- const pxPerCellY = m.cell * this.t.sy;- return {- alpha,- gx: (bil(cx + 1, cy) - bil(cx - 1, cy)) / (2 * pxPerCellX),- gy: (bil(cx, cy + 1) - bil(cx, cy - 1)) / (2 * pxPerCellY),- };- }-- private requirePose(): Float32Array {- if (!this.kp) throw new Error("PersonTrack: tracked without `pose`.");- return this.kp;- }-- private requireMask(): NonNullable<PersonTrack["maskData"]> {- if (!this.maskData) throw new Error("PersonTrack: tracked without `mask`.");- return this.maskData;- }-- /** Silhouette per frame, softened by a small box blur so gradients reach past the edge. */- private blurredMasks(): Float32Array[] {- if (this.blurred) return this.blurred;- const m = this.requireMask();- const { width: w, height: h } = m;- const r = 2;- const out: Float32Array[] = [];- const tmp = new Float32Array(w * h);- for (let f = 0; f < this.frames; f++) {- const src = m.data.subarray(f * w * h, (f + 1) * w * h);- const dst = new Float32Array(w * h);- for (let y = 0; y < h; y++) {- for (let x = 0; x < w; x++) {- let s = 0;- let n = 0;- for (let k = -r; k <= r; k++) {- const xx = x + k;- if (xx >= 0 && xx < w) {- s += src[y * w + xx];- n++;- }- }- tmp[y * w + x] = s / n / 255;- }- }- for (let y = 0; y < h; y++) {- for (let x = 0; x < w; x++) {- let s = 0;- let n = 0;- for (let k = -r; k <= r; k++) {- const yy = y + k;- if (yy >= 0 && yy < h) {- s += tmp[yy * w + x];- n++;- }- }- dst[y * w + x] = s / n;- }- }- out.push(dst);- }- this.blurred = out;- return out;- }-- /** @internal */- static fromAnalysis(- meta: {- fps: number;- frames: number;- sourceWidth: number;- sourceHeight: number;- },- kp: Float32Array | undefined,- mask: PersonTrack["maskData"],- ): PersonTrack {- return new PersonTrack(meta, kp, mask);- }-}--function keypointIndex(name: PoseKeypointName): number {- const i = COCO17_KEYPOINT_NAMES.indexOf(name);- if (i < 0) {- throw new Error(- `PersonTrack: unknown keypoint "${name}". Use one of ${COCO17_KEYPOINT_NAMES.join(", ")}.`,- );- }- return i;-}--/** Runs pose and segmentation over a clip and returns a {@link PersonTrack}. */-export async function trackPerson(- source: string,- options: TrackPersonOptions & { fps: number; totalFrames: number },-): Promise<PersonTrack> {- const { fps, totalFrames } = options;- const wantPose = options.pose ?? true;- const wantMask = options.mask ?? true;- const cell = Math.max(1, Math.round(options.maskCell ?? 8));- const minVis = options.minVisibility ?? 0.3;- const runner = VisionRunner.create({- modelsDir: options.modelsDir,- baseUrl: options.baseUrl,- ...(options.confidence !== undefined && { confidence: options.confidence }),- ...(options.variant !== undefined && { variant: options.variant }),- });- const decoder = mediaDecoderCache.getVideo(source, true);-- let sourceWidth = 0;- let sourceHeight = 0;- const kp = wantPose ? new Float32Array(totalFrames * 17 * 3) : undefined;- let maskW = 0;- let maskH = 0;- let maskData: Uint8Array | undefined;-- try {- for (let i = 0; i < totalFrames; i++) {- const decoded = await decoder.getFrame(i / fps);- if (!decoded?.buffer || decoded.width <= 0) continue;- const frame = {- data: decoded.buffer,- width: decoded.width,- height: decoded.height,- };- if (!sourceWidth) {- sourceWidth = frame.width;- sourceHeight = frame.height;- maskW = Math.ceil(sourceWidth / cell);- maskH = Math.ceil(sourceHeight / cell);- if (wantMask) maskData = new Uint8Array(totalFrames * maskW * maskH);- }- if (kp) {- const { people } = await runner.pose(frame);- const person = people[0];- if (person) {- person.keypoints.forEach((p, k) => {- kp.set([p.x, p.y, p.visibility], (i * 17 + k) * 3);- });- }- }- if (maskData) {- const seg = await runner.segment(frame);- const union = personSilhouette(seg);- if (union)- downsampleMean(- union,- frame.width,- frame.height,- cell,- maskData,- i * maskW * maskH,- maskW,- maskH,- );- }- options.onProgress?.(i + 1, totalFrames);- }- } finally {- runner.close();- decoder.destroy();- }- if (!sourceWidth)- throw new Error(`trackPerson: could not decode any frame of "${source}".`);- if (kp) smoothKeypoints(kp, totalFrames, minVis, options.smoothing ?? 2);-- return PersonTrack.fromAnalysis(- { fps, frames: totalFrames, sourceWidth, sourceHeight },- kp,- maskData && { cell, width: maskW, height: maskH, data: maskData },- );-}--type Segmentation = Awaited<ReturnType<VisionRunner["segment"]>>;--/**- * The largest person's mask, plus instances that overlap it and are no more- * than twice its size (a flowing dress often comes back as "kite").- */-function personSilhouette(seg: Segmentation): Uint8Array | undefined {- const people = seg.masks.filter((m) => m.category === "person");- if (people.length === 0) return undefined;- const person = people.reduce((a, b) => (b.area > a.area ? b : a));- const box = seg.detections[person.detectionIndex]?.boundingBox;- const out = new Uint8Array(person.mask);- if (!box) return out;- for (const m of seg.masks) {- if (m === person || m.area > person.area * 2) continue;- const b = seg.detections[m.detectionIndex]?.boundingBox;- if (!b) continue;- const ix = Math.max(- 0,- Math.min(box.originX + box.width, b.originX + b.width) -- Math.max(box.originX, b.originX),- );- const iy = Math.max(- 0,- Math.min(box.originY + box.height, b.originY + b.height) -- Math.max(box.originY, b.originY),- );- if (ix * iy < 0.3 * b.width * b.height) continue;- for (let i = 0; i < out.length; i++)- if (m.mask[i] > out[i]) out[i] = m.mask[i];- }- return out;-}--function downsampleMean(- src: Uint8Array,- w: number,- h: number,- cell: number,- dst: Uint8Array,- offset: number,- mw: number,- mh: number,-): void {- for (let my = 0; my < mh; my++) {- for (let mx = 0; mx < mw; mx++) {- let sum = 0;- let n = 0;- for (let y = my * cell; y < Math.min(h, (my + 1) * cell); y++) {- for (let x = mx * cell; x < Math.min(w, (mx + 1) * cell); x++) {- sum += src[y * w + x];- n++;- }- }- dst[offset + my * mw + mx] = n ? Math.round(sum / n) : 0;- }- }-}--/**- * Fills missing keypoints from their nearest seen frames, then runs a- * forward and a backward exponential filter (no lag, no overshoot).- */-function smoothKeypoints(- kp: Float32Array,- frames: number,- minVis: number,- smoothing: number,-): void {- for (let k = 0; k < 17; k++) {- const idx = (f: number, c: number) => (f * 17 + k) * 3 + c;- const seen: number[] = [];- for (let f = 0; f < frames; f++) if (kp[idx(f, 2)] >= minVis) seen.push(f);- if (seen.length === 0) continue;- let s = 0;- for (let f = 0; f < frames; f++) {- if (kp[idx(f, 2)] >= minVis) continue;- while (s + 1 < seen.length && seen[s + 1] < f) s++;- const a = seen[s];- const b = seen.find((v) => v > f);- for (const c of [0, 1]) {- if (f < a || b === undefined) kp[idx(f, c)] = kp[idx(a, c)];- else {- const u = (f - a) / (b - a);- kp[idx(f, c)] = kp[idx(a, c)] + (kp[idx(b, c)] - kp[idx(a, c)]) * u;- }- }- }- if (smoothing <= 0) continue;- const alpha = 1 / (1 + smoothing);- for (const c of [0, 1]) {- for (let f = 1; f < frames; f++)- kp[idx(f, c)] += (kp[idx(f - 1, c)] - kp[idx(f, c)]) * (1 - alpha);- for (let f = frames - 2; f >= 0; f--)- kp[idx(f, c)] += (kp[idx(f + 1, c)] - kp[idx(f, c)]) * (1 - alpha);- }- }-}diff --git a/packages/framefields/src/physics/backend.ts b/packages/framefields/src/physics/backend.tsindex 4b99c277..6f684e00 100644--- a/packages/framefields/src/physics/backend.ts+++ b/packages/framefields/src/physics/backend.ts@@ -116,14 +116,18 @@ export interface PhysicsBackend {type Rapier2D = typeof RAPIER2D;type Rapier3D = typeof RAPIER3D;-// Variable specifiers keep bundlers from inlining the optional WASM packages.+// Literal specifiers: the preview's browser bundle splits each engine into a+// chunk loaded on first use; Node keeps them as external optional packages.const RAPIER_2D = "@dimforge/rapier2d-deterministic-compat";const RAPIER_3D = "@dimforge/rapier3d-deterministic-compat";-async function loadRapier<T>(specifier: string): Promise<T> {+async function loadRapier<T>(+ specifier: string,+ load: () => Promise<unknown>,+): Promise<T> {let mod: { default?: T } & T;try {- mod = (await import(/* @vite-ignore */ specifier)) as { default?: T } & T;+ mod = (await load()) as { default?: T } & T;} catch (error) {throw new Error(`Physics needs the optional dependency "${specifier}". Install it with \`npm i ${specifier}\`.`,@@ -141,7 +145,10 @@ let rapier3d: Promise<Rapier3D> | undefined;/** Loads and initializes Rapier once per process; 2D and 3D are separate modules. */export async function rapierBackend(dims: 2 | 3): Promise<PhysicsBackend> {if (dims === 2) {- rapier2d ??= loadRapier<Rapier2D>(RAPIER_2D);+ rapier2d ??= loadRapier<Rapier2D>(+ RAPIER_2D,+ () => import("@dimforge/rapier2d-deterministic-compat"),+ );const R = await rapier2d;return {createWorld2D: (gravity) => new RapierWorld2D(R, gravity),@@ -150,7 +157,10 @@ export async function rapierBackend(dims: 2 | 3): Promise<PhysicsBackend> {},};}- rapier3d ??= loadRapier<Rapier3D>(RAPIER_3D);+ rapier3d ??= loadRapier<Rapier3D>(+ RAPIER_3D,+ () => import("@dimforge/rapier3d-deterministic-compat"),+ );const R = await rapier3d;return {createWorld2D: () => {diff --git a/packages/framefields/src/physics/core.ts b/packages/framefields/src/physics/core.tsindex 10a478bd..08622852 100644--- a/packages/framefields/src/physics/core.ts+++ b/packages/framefields/src/physics/core.ts@@ -67,6 +67,24 @@ export interface BodyCommonOptions {output?: "signals" | "keyframes" | "none";}+/** What `world.live()` needs from a composition. */+export interface LiveHost {+ onPrepareFrame(+ callback: (ctx: { frame: number }) => Promise<void> | void,+ ): unknown;+ prepareVision?(frame: number): Promise<void>;+}++interface Stepper<B, V, R> {+ /** Last simulated frame, relative to the world's start. */+ current: number;+ freed: boolean;+ bySim: Map<SimBody<V, R>, B>;+ kinematicFrom: Map<B, { position: V; rotation: R }>;+ /** Impact envelope level per body after `current`. */+ level: Map<B, number>;+}+export interface PhysicsCollision<B> {/** Composition frame of first contact. */frame: number;@@ -248,7 +266,11 @@ export abstract class WorldCore<}> = [];private readonly worldRecording: Recording;private readonly allCollisions: Array<PhysicsCollision<B> & { body: B }> = [];- private simulated = false;+ /** No more bodies, joints or forces: the world has started (or finished). */+ private locked = false;+ private liveMode = false;+ private stepper?: Stepper<B, V, R>;+ private chain: Promise<void> = Promise.resolve();protected constructor(protected readonly sim: SimWorld<V, SimBody<V, R>, unknown>,@@ -269,7 +291,7 @@ export abstract class WorldCore<this.worldRecording = this.newRecording();}- /** Every first contact in the simulation, in frame order. */+ /** First contacts so far (all of them once simulated), in frame order. */get collisions(): ReadonlyArray<PhysicsCollision<B> & { body: B }> {this.ensureSimulated();return this.allCollisions;@@ -280,17 +302,21 @@ export abstract class WorldCore<return this.worldRecording.signal("impact");}- /** Runs the simulation (once) and binds every body's layer per its `output`. */+ /** Runs the whole simulation now (once) and binds every body's layer per its `output`. */simulate(): this {+ if (this.liveMode)+ throw new Error(+ "Physics: this world is live; it steps as frames render.",+ );this.ensureSimulated();return this;}/** Like {@link simulate}, but writes keyframes for every body whose `output` was left at its default. */bakeKeyframes(options: PhysicsBakeOptions = {}): this {- if (this.simulated) {+ if (this.locked) {throw new Error(- "Physics: bakeKeyframes() must come before simulate() or reading any body signal.",+ "Physics: bakeKeyframes() must come before simulate(), live() or reading any body signal.",);}this.bakeDefault = options;@@ -298,6 +324,67 @@ export abstract class WorldCore<return this;}+ /**+ * Steps the world as the composition renders instead of all at once: before+ * each frame draws, the world advances to it, after the composition's+ * vision clips (`comp.vision`) and any `inputs` have that frame ready. Use+ * it when bodies follow a performer, so nothing is analysed up front and a+ * preview starts at once. Frames already simulated are kept, so scrubbing+ * back costs nothing; jumping ahead simulates the frames in between.+ * Signals only; `bakeKeyframes()` needs the whole run up front.+ */+ live(+ comp: LiveHost,+ options: {+ inputs?: ReadonlyArray<{ ready(frame: number): Promise<void> }>;+ } = {},+ ): this {+ if (this.locked) {+ throw new Error(+ "Physics: live() must come before simulate() or reading any body signal.",+ );+ }+ this.locked = true;+ this.liveMode = true;+ const prepare = async (frame: number) => {+ await comp.prepareVision?.(frame);+ for (const input of options.inputs ?? []) await input.ready(frame);+ };+ this.prepareInputs = prepare;+ for (const body of this.bodies) this.bind(body);+ comp.onPrepareFrame((ctx) => this.advanceTo(ctx.frame));+ return this;+ }++ /**+ * Live worlds: simulates up to composition frame `frame` (no-op when it+ * already has). Calls queue, so frames are stepped once, in order.+ */+ advanceTo(frame: number): Promise<void> {+ this.chain = this.chain.then(async () => {+ const target = Math.min(+ this.frames - 1,+ Math.round(frame) - this.startFrame,+ );+ if (!this.stepper) {+ await this.prepareInputs?.(this.startFrame);+ this.stepper = this.begin();+ }+ const st = this.stepper;+ while (st.current < target) {+ const next = st.current + 1;+ await this.prepareInputs?.(this.startFrame + next);+ this.stepFrame(st, next);+ }+ if (st.current === this.frames - 1 && !st.freed) {+ st.freed = true;+ this.sim.free();+ }+ });+ return this.chain;+ }++ private prepareInputs?: (frame: number) => Promise<void>;private bakeDefault?: PhysicsBakeOptions;protected newRecording(): Recording {@@ -307,9 +394,9 @@ export abstract class WorldCore<}protected assertOpen(): void {- if (this.simulated) {+ if (this.locked) {throw new Error(- "Physics: the world has already simulated; add every body, joint and force before simulate() or reading signals.",+ "Physics: the world has already started; add every body, joint and force before simulate(), live() or reading signals.",);}}@@ -346,171 +433,176 @@ export abstract class WorldCore<}protected ensureSimulated(): void {- if (this.simulated) return;- this.simulated = true;- this.run();+ if (this.locked) return;+ this.locked = true;+ const st = this.begin();+ for (let f = 1; f < this.frames; f++) this.stepFrame(st, f);+ st.freed = true;this.sim.free();for (const body of this.bodies) this.bind(body);}- private run(): void {- const { fps, frames, startFrame, substeps } = this;- const dt = 1 / (fps * substeps);+ /** Records frame 0 and sets every channel to it, so frames not simulated yet hold the start pose. */+ private begin(): Stepper<B, V, R> {+ const { startFrame } = this;const bodies = this.bodies;- const n = startFrame + frames;- const rawImpact = new Map<B, Float32Array>();- for (const b of bodies) rawImpact.set(b, new Float32Array(n));const bySim = new Map<SimBody<V, R>, B>();for (const b of bodies) bySim.set(b.sim, b);-for (const b of bodies) if (b.startFrame > 0) b.sim.setEnabled(false);- for (const b of bodies) b.record(startFrame, -1);+ const kinematicFrom = new Map<B, { position: V; rotation: R }>();+ for (const b of bodies) {+ b.record(startFrame, -1);+ b.recording.channel("impact");+ const t = b.kinematicTarget?.(startFrame);+ if (t) kinematicFrom.set(b, t);+ }+ this.worldRecording.channel("impact");+ for (const rec of [+ ...bodies.map((b) => b.recording),+ this.worldRecording,+ ]) {+ for (const arr of rec.channels.values()) arr.fill(arr[startFrame]);+ }+ return {+ current: 0,+ freed: false,+ bySim,+ kinematicFrom,+ level: new Map(bodies.map((b) => [b, 0])),+ };+ }+ private stepFrame(st: Stepper<B, V, R>, f: number): void {+ const { fps, startFrame, substeps } = this;+ const dt = 1 / (fps * substeps);+ const bodies = this.bodies;+ const frame = startFrame + f;const dims = this.gravity.length;const prevVel = new Map<B, number[]>();- let kinematicFrom = new Map<B, { position: V; rotation: R }>();for (const b of bodies) {- const t = b.kinematicTarget?.(startFrame);- if (t) kinematicFrom.set(b, t);+ // A body sits at its start pose on its start frame and moves after.+ if (b.startFrame > 0 && f === b.startFrame + 1) b.sim.setEnabled(true);+ prevVel.set(b, b.sim.velocity() as number[]);+ }+ const isActive = (b: B) => b.type === "dynamic" && f > b.startFrame;+ // Velocity each body is expected to gain this frame from gravity,+ // forces, fields and impulses: anything beyond it came from a contact.+ const expected = new Map<B, number[]>();+ for (const b of bodies) {+ expected.set(+ b,+ this.gravity.map((g) => (isActive(b) ? (g * b.gravityScale) / fps : 0)),+ );+ }+ for (const imp of this.impulses) {+ if (imp.frame !== f) continue;+ const body = imp.body as unknown as B;+ if (!isActive(body)) continue;+ const m = body.sim.mass();+ body.sim.applyImpulse(imp.dv.map((v) => v * m) as V);+ const e = expected.get(body) as number[];+ for (let i = 0; i < dims; i++) e[i] += imp.dv[i];+ }+ const frameForces = this.forces+ .filter((fe) => isActive(fe.body as unknown as B))+ .map((fe) => {+ const body = fe.body as unknown as B;+ const a = fe.accel(frame);+ const e = expected.get(body) as number[];+ for (let i = 0; i < dims; i++) e[i] += a[i] / fps;+ return { body, a };+ });+ const kinematicTo = new Map<B, { position: V; rotation: R }>();+ for (const b of bodies) {+ const t = b.kinematicTarget?.(frame);+ if (t) kinematicTo.set(b, t);}- for (let f = 1; f < frames; f++) {- const frame = startFrame + f;- for (const b of bodies) {- // A body sits at its start pose on its start frame and moves after.- if (b.startFrame > 0 && f === b.startFrame + 1) b.sim.setEnabled(true);- prevVel.set(b, b.sim.velocity() as number[]);+ const started = new Set<string>();+ const pairs: Array<[B, B]> = [];+ for (let s = 1; s <= substeps; s++) {+ for (const { body, a } of frameForces) {+ const m = body.sim.mass();+ body.sim.applyImpulse(a.map((v) => v * m * dt) as V);}- // Velocity each body is expected to gain this frame from gravity,- // forces and impulses: anything beyond it came from a contact.- const expected = new Map<B, number[]>();- for (const b of bodies) {- const active = b.type === "dynamic" && f > b.startFrame;- expected.set(- b,- this.gravity.map((g) => (active ? (g * b.gravityScale) / fps : 0)),- );+ for (const field of this.fields) {+ for (const b of bodies) {+ if (!isActive(b) || (field.targets && !field.targets.has(b)))+ continue;+ const a = field.accel(b, frame);+ if (!a) continue;+ const m = b.sim.mass();+ b.sim.applyImpulse(a.map((v) => v * m * dt) as V);+ const e = expected.get(b) as number[];+ for (let i = 0; i < dims; i++) e[i] += a[i] * dt;+ }}- const isActive = (b: B) => b.type === "dynamic" && f > b.startFrame;- for (const imp of this.impulses) {- if (imp.frame !== f) continue;- const body = imp.body as unknown as B;- if (!isActive(body)) continue;- const m = body.sim.mass();- body.sim.applyImpulse(imp.dv.map((v) => v * m) as V);- const e = expected.get(body) as number[];- for (let i = 0; i < dims; i++) e[i] += imp.dv[i];+ for (const [b, to] of kinematicTo) {+ const from = st.kinematicFrom.get(b) ?? to;+ const k = s / substeps;+ b.sim.setKinematicTarget(+ lerpVec(from.position, to.position, k),+ this.lerpRotation(from.rotation, to.rotation, k),+ );}- const frameForces = this.forces- .filter((fe) => isActive(fe.body as unknown as B))- .map((fe) => {- const body = fe.body as unknown as B;- const a = fe.accel(frame);- const e = expected.get(body) as number[];- for (let i = 0; i < dims; i++) e[i] += a[i] / fps;- return { body, a };- });- const kinematicTo = new Map<B, { position: V; rotation: R }>();- for (const b of bodies) {- const t = b.kinematicTarget?.(frame);- if (t) kinematicTo.set(b, t);+ this.sim.step(dt);+ for (const [sa, sb] of this.sim.drainCollisionStarts()) {+ const a = st.bySim.get(sa);+ const b = st.bySim.get(sb);+ if (!a || !b) continue;+ const key = `${bodies.indexOf(a)}:${bodies.indexOf(b)}`;+ if (started.has(key)) continue;+ started.add(key);+ pairs.push([a, b]);}-- const started = new Set<string>();- const pairs: Array<[B, B]> = [];- for (let s = 1; s <= substeps; s++) {- for (const { body, a } of frameForces) {- const m = body.sim.mass();- body.sim.applyImpulse(a.map((v) => v * m * dt) as V);- }- for (const field of this.fields) {- for (const b of bodies) {- if (!isActive(b) || (field.targets && !field.targets.has(b)))- continue;- const a = field.accel(b, frame);- if (!a) continue;- const m = b.sim.mass();- b.sim.applyImpulse(a.map((v) => v * m * dt) as V);- const e = expected.get(b) as number[];- for (let i = 0; i < dims; i++) e[i] += a[i] * dt;- }- }- for (const [b, to] of kinematicTo) {- const from = kinematicFrom.get(b) ?? to;- const k = s / substeps;- b.sim.setKinematicTarget(- lerpVec(from.position, to.position, k),- this.lerpRotation(from.rotation, to.rotation, k),- );+ }+ st.kinematicFrom = kinematicTo;++ for (const b of bodies) b.record(frame, frame - 1);++ const raw = new Map<B, number>();+ if (pairs.length > 0) {+ const ppm = this.pixelsPerMeter;+ const hit = (b: B): number => {+ if (b.type !== "dynamic") return 0;+ const v = b.sim.velocity() as number[];+ const v0 = prevVel.get(b) as number[];+ const e = expected.get(b) as number[];+ let sq = 0;+ for (let i = 0; i < dims; i++) {+ const d = v[i] - (v0[i] + e[i]);+ sq += d * d;}- this.sim.step(dt);- for (const [sa, sb] of this.sim.drainCollisionStarts()) {- const a = bySim.get(sa);- const b = bySim.get(sb);- if (!a || !b) continue;- const key = `${bodies.indexOf(a)}:${bodies.indexOf(b)}`;- if (started.has(key)) continue;- started.add(key);- pairs.push([a, b]);- }- }- kinematicFrom = kinematicTo;-- for (const b of bodies) b.record(frame, frame - 1);-- if (pairs.length > 0) {- const ppm = this.pixelsPerMeter;- const hit = (b: B): number => {- if (b.type !== "dynamic") return 0;- const v = b.sim.velocity() as number[];- const v0 = prevVel.get(b) as number[];- const e = expected.get(b) as number[];- let sq = 0;- for (let i = 0; i < dims; i++) {- const d = v[i] - (v0[i] + e[i]);- sq += d * d;- }- return Math.sqrt(sq) * ppm;- };- for (const [a, b] of pairs) {- const speed = Math.max(hit(a), hit(b));- for (const [self, other] of [- [a, b],- [b, a],- ] as const) {- const raw = rawImpact.get(self) as Float32Array;- raw[frame] = Math.max(raw[frame], speed);- const c = { frame, with: other, speed };- self.collisions.push(c as never);- if (self === a) this.allCollisions.push({ ...c, body: a });- }+ return Math.sqrt(sq) * ppm;+ };+ for (const [a, b] of pairs) {+ const speed = Math.max(hit(a), hit(b));+ for (const [self, other] of [+ [a, b],+ [b, a],+ ] as const) {+ raw.set(self, Math.max(raw.get(self) ?? 0, speed));+ const c = { frame, with: other, speed };+ self.collisions.push(c as never);+ if (self === a) this.allCollisions.push({ ...c, body: a });}}}// Impact envelopes: instant attack, exponential release.const decay = Math.exp(-1 / this.impactDecay);- const worldImpact = this.worldRecording.channel("impact");+ let loudest = 0;for (const b of bodies) {- const raw = rawImpact.get(b) as Float32Array;- const env = b.recording.channel("impact");- let level = 0;- for (let i = startFrame; i < n; i++) {- level = Math.max(raw[i] / this.impactReference, level * decay);- env[i] = level;- if (level > worldImpact[i]) worldImpact[i] = level;- }- }-- // Before the world starts every channel holds its frame-0 value.- for (const rec of [- ...bodies.map((b) => b.recording),- this.worldRecording,- ]) {- for (const arr of rec.channels.values()) {- arr.fill(arr[startFrame], 0, startFrame);- }+ const level = Math.max(+ (raw.get(b) ?? 0) / this.impactReference,+ (st.level.get(b) ?? 0) * decay,+ );+ st.level.set(b, level);+ b.recording.channel("impact")[frame] = level;+ if (level > loudest) loudest = level;}+ this.worldRecording.channel("impact")[frame] = loudest;+ st.current = f;}protected abstract lerpRotation(a: R, b: R, t: number): R;@@ -523,6 +615,11 @@ export abstract class WorldCore<if (typeof layer?.animate !== "function" || output === "none") return;if (body.type === "fixed") return;const channels = body.boundChannels();+ if (output === "keyframes" && this.liveMode) {+ throw new Error(+ 'Physics: live worlds bind signals; drop `output: "keyframes"` or use simulate().',+ );+ }if (output === "signals") {const anim = LayerAnimation.create();for (const [prop, channel] of channels) {diff --git a/packages/framefields/src/physics/physics.test.ts b/packages/framefields/src/physics/physics.test.tsindex 6e007f24..6d5b1a8d 100644--- a/packages/framefields/src/physics/physics.test.ts+++ b/packages/framefields/src/physics/physics.test.ts@@ -263,7 +263,90 @@ describe("Physics.world2d", () => {);const b = world.body(card("a", 0, 0));void b.y;- expect(() => world.body(card("b", 0, 0))).toThrow(/already simulated/);+ expect(() => world.body(card("b", 0, 0))).toThrow(/already started/);+ });+});++describe("live worlds", () => {+ /** A stand-in composition: collects prepare hooks, records which frames were prepared. */+ function host() {+ const hooks: Array<(ctx: { frame: number }) => Promise<void> | void> = [];+ const prepared: number[] = [];+ return {+ prepared,+ onPrepareFrame(cb: (ctx: { frame: number }) => Promise<void> | void) {+ hooks.push(cb);+ },+ async prepareVision(frame: number) {+ prepared.push(frame);+ },+ async render(frame: number) {+ for (const h of hooks) await h({ frame });+ },+ };+ }++ async function scene() {+ const world = await Physics.world2d({ fps: FPS, frames: 60 });+ world.walls({ width: 1920, height: 1080 });+ const layers = Array.from({ length: 6 }, (_, i) =>+ card(`c${i}`, 300 + i * 200, -100 - i * 150, { rotation: i * 11 }),+ );+ const bodies = layers.map((l, i) => world.body(l, { startFrame: i * 3 }));+ world.force(bodies[0], { x: (f) => (f > 20 ? 900 : 0) });+ return { world, layers, bodies };+ }++ it("steps on demand and matches simulate() exactly, in any request order", async () => {+ const batch = await scene();+ batch.world.simulate();++ const live = await scene();+ const comp = host();+ live.world.live(comp);+ // Layers are bound before anything is simulated.+ expect(tracksOf(live.layers[0]).map((t) => t.source?.type)).toEqual([+ "signal",+ "signal",+ "signal",+ ]);+ await comp.render(40);+ await comp.render(10); // scrubbing back: already simulated+ await comp.render(59);+ expect(comp.prepared).toEqual(Array.from({ length: 60 }, (_, f) => f));++ for (let i = 0; i < batch.bodies.length; i++) {+ for (const f of [0, 10, 33, 59]) {+ expect(at(live.bodies[i].x, f)).toBe(at(batch.bodies[i].x, f));+ expect(at(live.bodies[i].rotation, f)).toBe(+ at(batch.bodies[i].rotation, f),+ );+ expect(at(live.bodies[i].impact, f)).toBe(+ at(batch.bodies[i].impact, f),+ );+ }+ }+ expect(live.world.collisions.length).toBe(batch.world.collisions.length);+ });++ it("holds the start pose for frames not simulated yet", async () => {+ const { world, bodies } = await scene();+ const comp = host();+ world.live(comp);+ await comp.render(5);+ expect(at(bodies[0].y, 30)).toBe(-100);+ expect(at(bodies[0].y, 5)).toBeGreaterThan(-100);+ });++ it("waits for extra inputs before each frame and locks the world", async () => {+ const { world } = await scene();+ const comp = host();+ const seen: number[] = [];+ world.live(comp, { inputs: [{ ready: async (f) => void seen.push(f) }] });+ await comp.render(3);+ expect(seen).toEqual([0, 1, 2, 3]);+ expect(() => world.simulate()).toThrow(/live/);+ expect(() => world.body(card("late", 0, 0))).toThrow(/already started/);});});diff --git a/plugins/framefields/skills/framefields-compose/SKILL.md b/plugins/framefields/skills/framefields-compose/SKILL.mdindex be775a71..a7d58639 100644--- a/plugins/framefields/skills/framefields-compose/SKILL.md+++ b/plugins/framefields/skills/framefields-compose/SKILL.md@@ -400,11 +400,13 @@ stage.animate(LayerAnimation.create().signal("y", world.impact, { multiplier: 14- **Joints:** `world.pin(body, [x, y])` (swing around a point), `world.hinge(a, b, [x, y])`, `world.weld(a, b)`, `world.spring(a, bOrPoint, { frequency: 2, dampingRatio: 0.2, length })`, `world.rope(a, bOrPoint, { length })`. Points are world px.- **Driving the simulation:** `world.force(body, { x, y })` (px/s², mass-independent), `world.impulse(body, frame, [vx, vy])` (a kick), kinematic `follow: { x, y, rotation }`. Each of these takes a number, a signal (audio, `Signal.fromArray`) or `(frame) => value`: e.g. a floor that bounces on the kick.- **Force fields:** `world.field((s) => [ax, ay], { bodies })` is called every solver step with a body's center `s.x`, `s.y` (px), velocity `s.vx`, `s.vy` and `s.frame`; return px/s² or nothing. Wind, vortices, attractors, flow fields — and performers (below).-- **Performers in the simulation:** `const person = (await comp.trackPerson("dancer.mp4")).place({ width: W, height: H, fit: "cover" })` analyses the clip *before* rendering (pose + silhouette, a few hundred ms/frame; cache with `toJSON()` / `PersonTrack.fromJSON()`). Then:+- **Performers in the simulation:** `const dancer = comp.vision("dancer.mp4")` gives the clip's pose and silhouette per frame, analysed on demand (see `framefields-effects`). Make the world **live** so it steps as frames render instead of all at once: `world.live(comp)` in place of `simulate()`. Before each frame draws, the clip's vision for that frame is computed (or read from the cache) and the world advances to it — nothing is analysed up front, the preview plays at once and fills in, and export gives the identical result (same simulation, same order). Then, with `const person = dancer.person.place({ width: W, height: H, fit: "cover" })`:- limbs as kinematic colliders that catch and bat things: `world.body({ x, y, width: 2 * r, height: 2 * r }, { type: "kinematic", shape: "circle", follow: { x: (f) => person.point("leftWrist", f).x - r, y: (f) => person.point("leftWrist", f).y - r } })`, a few balls along each shoulder → elbow → wrist;- the silhouette (dress included) as a force field: `person.sample(frame, x, y)` returns `alpha` (inside > 0.5) and a gradient `gx`, `gy` pointing inward, so `world.field((s) => { const m = person.sample(s.frame, s.x, s.y); … return [-m.gx * k, -m.gy * k]; })` makes particles and props flow around a moving body.+ - Reads are synchronous and need the frame analysed: inside a live world, while rendering, or after `await dancer.ready(frame)` (e.g. `ready(0)` to lay things out around the first pose). A live world can't look ahead, so place things from early frames.- Layer a vision cutout of the same clip (`Layer.video(src).withVision({ mode: "matte" })`) above the particles so they pass behind the performer.-- **Particles today:** up to ~600 small `Layer.shape("circle")` bodies with `linearDamping` ≈ 1–1.5, low `friction`, `gravityScale` < 1 read as a fluid; tint them by speed with `.colorSignal("fillColor", heat, { colorA, colorB })` built from `body.speed`.+ - Live worlds bind signals only (no `bakeKeyframes()`); `await world.advanceTo(frame)` steps one without rendering (e.g. to read `collisions`).+- **Particles today:** up to ~600 small `Layer.shape("circle")` bodies with `linearDamping` ≈ 1–1.5, low `friction`, `gravityScale` < 1 read as a fluid; tint them by speed with `.colorSignal("fillColor", body.speed, { colorA, colorB, multiplier: 1 / 900 })` (`multiplier`/`offset` scale the signal into 0–1).- **Keyframes instead of signals:** `world.bakeKeyframes({ tolerance: 0.25 })` in place of `simulate()` writes plain, simplified linear keyframes (editable, plain JSON).- **3D:** `Physics.world3d` takes `Layer3D.roundedBox / sphere / cylinder` and models (convex hull of the mesh); same options with 3-vectors, `world.hinge(a, b, at, axis)`, `world.walls({ width, height, depth })`.- **Timing:** signals are on the composition clock. In a scene that starts at frame `S`, pass `startFrame: S` to the world. Before it and after `frames` the signals hold their first/last value. Add every body, joint and force before `simulate()` (or before reading any signal): the world then locks.diff --git a/plugins/framefields/skills/framefields-effects/SKILL.md b/plugins/framefields/skills/framefields-effects/SKILL.mdindex 1f3b0da1..d4b52ebc 100644--- a/plugins/framefields/skills/framefields-effects/SKILL.md+++ b/plugins/framefields/skills/framefields-effects/SKILL.md@@ -236,16 +236,20 @@ const vision = comp.withVision({ classes: ["person"] });layer.blurRegion(vision.objects.byCategory("person"), { strength: 30 });```-### 5. Track a Performer Before Rendering-Render-time vision signals lag a frame and only exist while rendering. For anything that must know the motion up front (physics, keyframes, layout), track the person first:+### 5. Clip Vision: Results Before the Frame Renders+`withVision` analyses a layer's *previous* rendered frame, so its results trail by a frame and exist only while rendering. When something must know the motion *at* the frame — physics, layout, keyframes — use `comp.vision`, which analyses the clip's own frames on demand:```typescript-const person = (await comp.trackPerson("assets/dancer.mp4")).place({ width: W, height: H, fit: "cover" });-person.point("rightWrist", 48); // { x, y, visibility } in composition px, smoothed, gaps filled-person.signals("nose").x; // per-frame signal for .signal("x", …)-person.sample(48, x, y); // silhouette alpha + inward gradient (dress and held things included)-writeFileSync("dancer-track.json", JSON.stringify(person.toJSON())); // reuse with PersonTrack.fromJSON+const dancer = comp.vision("assets/dancer.mp4", { startFrame: 0 }); // pose + silhouette (person)+const person = dancer.person.place({ width: W, height: H, fit: "cover" });+person.point("rightWrist", 48); // { x, y, visibility } in composition px, smoothed over ±2 frames+person.signals("nose").x; // a signal for .signal("x", …)+person.sample(48, x, y); // silhouette alpha + inward gradient (dress and held things included)+await dancer.ready(0); // analyse a frame (or a range: ready(0, 120)) to read it at build time```-Options: `pose`, `mask` (both default true), `maskCell` (silhouette grid, default 8 px), `smoothing` (frames, default 2), `variant`. See *Physics* in `framefields-compose` for limbs as colliders and the silhouette as a force field.+- The composition analyses each frame's clips just before drawing it, in every render path and the preview, and a live physics world (`world.live(comp)`) before stepping. Nothing runs up front; a preview starts at once.+- Results persist in Node (`~/.cache/framefields/vision`, or `$FRAMEFIELDS_VISION_CACHE_DIR`; `cacheDir: false` to opt out), keyed by the file and settings, so later runs and renders after a preview are instant.+- Options: `startFrame` / `trimStartSec` (where the clip plays), `pose`, `mask`, `maskCell` (silhouette grid, default 8 px), `smoothing` (frames, default 2), `variant`. When the person is lost the last pose is held for up to a second.+- Analysis costs about 200 ms per frame on CPU (segmentation dominates); on WebGPU it is faster. In a live preview the playhead waits for frames not analysed yet.### 6. Analyze Before AuthoringOne-shot, ffmpeg-free report of what is in a clip (tracks, classes, mask coverage):