Skip to content

Code-first video & image, rendered natively on WebGPU.

feat(physics): live worlds and on-demand clip vision, no up-front tracking

okanasl119b397db38b
diff --git a/.changeset/physics-rapier.md b/.changeset/physics-rapier.md
index 3672d828..f570bbbc 100644
--- a/.changeset/physics-rapier.md
+++ b/.changeset/physics-rapier.md
@@ -4,4 +4,6 @@
Add rigid-body physics: `Physics.world2d` / `Physics.world3d` (Rapier, deterministic) simulate once while the composition is built and replay as per-frame signals. Bodies come from layers (boxes, circles, convex hulls of shapes, measured text, Layer3D primitives and models); `simulate()` binds each layer, and every body exposes `impact`, `speed`, velocity and pose signals for driving any prop or effect. Includes ground/walls, pins, hinges, welds, springs, ropes, signal-driven forces and kinematic bodies, collision lists, and `bakeKeyframes()` for plain keyframes.
-`world.field()` adds position-dependent forces (wind, vortices, flow fields). `comp.trackPerson(source)` tracks a performer before rendering — smoothed pose keypoints and a soft silhouette per frame, placeable into composition pixels and cacheable as JSON — so limbs can be kinematic colliders and the silhouette a force field.
+`world.field()` adds position-dependent forces (wind, vortices, flow fields). `world.live(comp)` steps a world as the composition renders instead of all at once: before each frame draws it advances to it, after that frame's inputs are ready, giving the same result as `simulate()`.
+
+`comp.vision(clip)` analyses a clip's own frames on demand (pose keypoints and a soft silhouette) and shares the results with everything that reads them; every render path and the preview await it before drawing a frame, and results persist on disk in Node. `comp.onPrepareFrame()` registers async per-frame work, and the renderer awaits an `onRequestFrame` that returns a promise. `colorSignal` takes `multiplier` and `offset`.
diff --git a/examples/28_physics_dancer/README.md b/examples/28_physics_dancer/README.md
index 86e9e6d6..ba7ca65a 100644
--- a/examples/28_physics_dancer/README.md
+++ b/examples/28_physics_dancer/README.md
@@ -6,8 +6,9 @@ her as the silk dress sweeps through it.
```bash
pnpm fonts # Syne from google/fonts → assets/fonts/
-pnpm render sim # tracks the dancer (once, ~30 s, cached in output/) and runs the physics
+pnpm render sim # analyses the clip and steps the physics without drawing (~35 s cold, ~6 s cached)
pnpm render # output/physics-dancer.mp4
+pnpm render preview # live player; vision and physics fill in as it plays
pnpm render:src # same, on the engine's source (no package builds)
```
@@ -17,12 +18,14 @@ The clip is example 19's `assets/dancer.mp4`; generate it there with `pnpm asset
| Piece | API |
|---|---|
-| Pose and silhouette for every frame, before rendering | `comp.trackPerson(clip)` → `PersonTrack`, placed with `.place({ width, height, fit })`, cached with `toJSON()` / `PersonTrack.fromJSON()` |
+| Pose and silhouette, frame by frame as it renders | `comp.vision(clip).person`, placed with `.place({ width, height, fit })`; results persist in `~/.cache/framefields/vision` |
+| Physics that steps with the render | `world.live(comp)`: before each frame draws, its vision is ready and the world advances to it |
| Arms and head that catch the cards | kinematic circle bodies along shoulder → elbow → wrist (and past the wrist for the hand), `follow`ing `person.point(name, frame)` |
| The dress pushing particles and cards | `world.field()` with `person.sample(frame, x, y)`: out along the silhouette's gradient, much harder where the silhouette just arrived (the sweep) |
| Fluid feel | small circles with `linearDamping` 2.2, near-zero gravity, a slow swirling current and a soft pull home |
| Particles behind her | a second copy of the clip, cut out with `.withVision({ mode: "matte" })`, above the particles |
-| Hot particles | `.colorSignal("fillColor", …)` from each body's `speed` |
+| Hot particles | `.colorSignal("fillColor", body.speed, { multiplier: 1 / 900, … })` |
| Screen shake | the whole stage's `y` driven by `world.impact` |
-Cards are sized with `FontManager.measure`, so the text always fits.
+Cards are sized with `FontManager.measure`, so the text always fits. A live world can't look
+ahead, so they drop at fixed offsets from where her head starts and find her hands on the way down.
diff --git a/examples/28_physics_dancer/src/film.ts b/examples/28_physics_dancer/src/film.ts
index c32d5e7a..89198b0f 100644
--- a/examples/28_physics_dancer/src/film.ts
+++ b/examples/28_physics_dancer/src/film.ts
@@ -2,11 +2,16 @@
* A dancer in a field of physics: word cards drop onto her raised hands and
* head, and a cloud of particles flows around her as the dress sweeps through.
*
+ * Nothing is analysed up front: `comp.vision` analyses each frame of the clip
+ * just before it renders, and the live physics world steps to that frame
+ * right after, so the preview starts at once and fills in as it plays.
+ *
* - Her arms and head are kinematic colliders that follow the tracked pose.
* - Her silhouette (dress included) is a force field that pushes everything out,
* hardest where it is sweeping in.
* - A vision cutout of the same clip sits above the particles, so they pass behind her.
*/
+import fs from "node:fs";
import {
Composition,
FontManager,
@@ -17,9 +22,7 @@ import {
type PhysicsBody2D,
type PhysicsWorld2D,
type PoseKeypointName,
- Signal,
} from "framefields";
-import { performer } from "./performer.js";
import {
CREAM,
DANCER,
@@ -39,11 +42,12 @@ import {
} from "./theme.js";
const WORDS = [
- { text: "PHYSICS", bg: EMBER, fg: INK, at: 30 },
- { text: "MEETS", bg: INK, fg: CREAM, at: 46 },
- { text: "VISION", bg: CREAM, fg: INK, at: 62 },
- { text: "IN", bg: GOLD, fg: INK, at: 78 },
- { text: "FRAMEFIELDS", bg: VIOLET, fg: CREAM, at: 94 },
+ // `dx`: where it drops, from her head at the first frame (live worlds can't look ahead).
+ { text: "PHYSICS", bg: EMBER, fg: INK, at: 30, dx: -40 },
+ { text: "MEETS", bg: INK, fg: CREAM, at: 46, dx: 120 },
+ { text: "VISION", bg: CREAM, fg: INK, at: 62, dx: -120 },
+ { text: "IN", bg: GOLD, fg: INK, at: 78, dx: 60 },
+ { text: "FRAMEFIELDS", bg: VIOLET, fg: CREAM, at: 94, dx: 0 },
];
const CARD_FONT_SIZE = 64;
const PARTICLE_COLORS = [INK, RUST, EMBER, GOLD];
@@ -76,7 +80,7 @@ function limb(
}
}
-/** Word cards sized to their text, dropped over whichever hand is highest. */
+/** Word cards sized to their text, dropped around where her head starts. */
function cards(world: PhysicsWorld2D, person: PersonTrack) {
return WORDS.map((w, i) => {
const m = FontManager.measure(w.text, {
@@ -85,13 +89,11 @@ function cards(world: PhysicsWorld2D, person: PersonTrack) {
});
const width = Math.ceil(m.width + 72);
const height = Math.ceil(m.height + 40);
- const hand = (["leftWrist", "rightWrist"] as const)
- .map((k) => person.point(k, w.at + 12))
- .reduce((p, q) => (p.y < q.y ? p : q));
+ const head = person.point("nose", 0);
const layer = Layer.box({
id: `card-${i}`,
position: "absolute",
- x: hand.x - width / 2 + (i % 2 ? 30 : -30),
+ x: head.x + w.dx - width / 2,
y: -height - 200,
width,
height,
@@ -181,7 +183,19 @@ function silhouettePush(person: PersonTrack, strength: number, sweep: number) {
};
}
+export { DURATION as DURATION_FRAMES } from "./theme.js";
+
+export interface Film {
+ comp: Composition;
+ world: PhysicsWorld2D;
+ cards: Array<{ id: string; body: PhysicsBody2D }>;
+}
+
export async function buildFilm(): Promise<Composition> {
+ return (await buildScene()).comp;
+}
+
+export async function buildScene(): Promise<Film> {
await registerFonts();
const comp = new Composition({
width: W,
@@ -191,7 +205,15 @@ export async function buildFilm(): Promise<Composition> {
backgroundColor: "#E9D3B4",
fonts: [FONT_FILE],
});
- const person = await performer(comp);
+ if (!fs.existsSync(DANCER)) {
+ throw new Error(
+ `Missing ${DANCER}. Generate it with \`pnpm assets\` in examples/19_framefields_film.`,
+ );
+ }
+ const dancer = comp.vision(DANCER);
+ // The particles are laid out around her first pose: analyse just that now.
+ await dancer.ready(0);
+ const person = dancer.person.place({ width: W, height: H, fit: "cover" });
const world = await Physics.world2d({
fps: FPS,
@@ -237,22 +259,17 @@ export async function buildFilm(): Promise<Composition> {
},
{ bodies: cloudBodies },
);
- world.simulate();
+ // Steps as the composition renders, after each frame's vision is ready.
+ world.live(comp);
- // Particles glow as they speed up.
+ // Particles glow as they speed up (900 px/s is fully hot).
for (const p of cloud) {
- const heat = Array.from({ length: DURATION }, (_, f) =>
- Math.min(1, p.body.speed.get({ frame: f, fps: FPS } as never) / 900),
- );
p.layer.animate(
- LayerAnimation.create().colorSignal(
- "fillColor",
- Signal.fromArray(heat, FPS),
- {
- colorA: p.color,
- colorB: HOT,
- },
- ),
+ LayerAnimation.create().colorSignal("fillColor", p.body.speed, {
+ colorA: p.color,
+ colorB: HOT,
+ multiplier: 1 / 900,
+ }),
);
}
@@ -292,11 +309,9 @@ export async function buildFilm(): Promise<Composition> {
),
);
- for (const w of words) {
- const hit = w.body.collisions[0];
- console.log(
- `${w.layer.id}: first contact at frame ${hit?.frame ?? "-"} (${hit?.with.type ?? "none"})`,
- );
- }
- return comp;
+ return {
+ comp,
+ world,
+ cards: words.map((w) => ({ id: String(w.layer.id), body: w.body })),
+ };
}
diff --git a/examples/28_physics_dancer/src/performer.ts b/examples/28_physics_dancer/src/performer.ts
deleted file mode 100644
index fd81fb14..00000000
--- a/examples/28_physics_dancer/src/performer.ts
+++ /dev/null
@@ -1,30 +0,0 @@
-import fs from "node:fs";
-import path from "node:path";
-import { type Composition, PersonTrack } from "framefields";
-import { DANCER, H, TRACK_CACHE, W } from "./theme.js";
-
-/**
- * The dancer's pose and silhouette for every frame, placed in composition
- * pixels. Tracking runs once (about 30 s) and is cached in output/.
- */
-export async function performer(comp: Composition): Promise<PersonTrack> {
- if (!fs.existsSync(DANCER)) {
- throw new Error(
- `Missing ${DANCER}. Generate it with \`pnpm assets\` in examples/19_framefields_film.`,
- );
- }
- let track: PersonTrack;
- if (fs.existsSync(TRACK_CACHE)) {
- track = PersonTrack.fromJSON(
- JSON.parse(fs.readFileSync(TRACK_CACHE, "utf8")),
- );
- } else {
- console.log("Tracking the dancer (pose + silhouette)…");
- track = await comp.trackPerson(DANCER, {
- onProgress: (f, n) => f % 40 === 0 && console.log(` ${f}/${n}`),
- });
- fs.mkdirSync(path.dirname(TRACK_CACHE), { recursive: true });
- fs.writeFileSync(TRACK_CACHE, JSON.stringify(track.toJSON()));
- }
- return track.place({ width: W, height: H, fit: "cover" });
-}
diff --git a/examples/28_physics_dancer/src/render.ts b/examples/28_physics_dancer/src/render.ts
index 411eafb1..b8fc9fdc 100644
--- a/examples/28_physics_dancer/src/render.ts
+++ b/examples/28_physics_dancer/src/render.ts
@@ -1,20 +1,44 @@
/**
* pnpm render → output/physics-dancer.mp4
- * pnpm render sim → runs tracking and physics only, prints first contacts
+ * pnpm render sim → analyses the clip and steps the physics without drawing, prints first contacts
+ * pnpm render preview → live player on localhost (add --open); vision and physics fill in as it plays
*
- * Frames render in order: the cutout runs vision on the previous frame, so
- * single frames out of order would show a ghost of another moment.
+ * Vision results persist in ~/.cache/framefields/vision, so only the first run analyses the clip.
*/
import path from "node:path";
-import { buildFilm } from "./film.js";
+import { startPreview } from "framefields";
+import { buildScene, DURATION_FRAMES } from "./film.js";
import { OUTPUT } from "./theme.js";
-const [mode = "video"] = process.argv.slice(2);
-const film = await buildFilm();
+const [mode = "video", ...args] = process.argv.slice(2);
-if (mode !== "sim") {
+if (mode === "preview") {
+ const session = await startPreview(
+ { entry: new URL("./film.ts", import.meta.url), export: "buildFilm" },
+ { title: "Physics dancer", open: args.includes("--open") },
+ );
+ console.log(`Preview at ${session.url}`);
+ await session.closed;
+ process.exit(0);
+}
+
+const { comp, world, cards } = await buildScene();
+
+if (mode === "sim") {
+ const started = Date.now();
+ await world.advanceTo(DURATION_FRAMES - 1);
+ console.log(
+ `Analysed and simulated in ${((Date.now() - started) / 1000).toFixed(1)} s`,
+ );
+ for (const { id, body } of cards) {
+ const hit = body.collisions[0];
+ console.log(
+ `${id}: first contact at frame ${hit?.frame ?? "-"} (${hit?.with.type ?? "none"})`,
+ );
+ }
+} else {
const started = Date.now();
- const result = await film.renderVideo({
+ const result = await comp.renderVideo({
outputPath: path.join(OUTPUT, "physics-dancer.mp4"),
quality: "high",
});
diff --git a/examples/28_physics_dancer/src/theme.ts b/examples/28_physics_dancer/src/theme.ts
index df8fd0c0..8ee95a86 100644
--- a/examples/28_physics_dancer/src/theme.ts
+++ b/examples/28_physics_dancer/src/theme.ts
@@ -11,8 +11,6 @@ export const DANCER = path.resolve(
HERE,
"../../19_framefields_film/assets/dancer.mp4",
);
-/** Pose + silhouette analysis of the clip, cached after the first run. */
-export const TRACK_CACHE = path.join(OUTPUT, "dancer-track.json");
export const W = 1920;
export const H = 1080;
diff --git a/nodes/node-compositor/src/renderers/webgpu-renderer.ts b/nodes/node-compositor/src/renderers/webgpu-renderer.ts
index 078d024c..06013f21 100644
--- a/nodes/node-compositor/src/renderers/webgpu-renderer.ts
+++ b/nodes/node-compositor/src/renderers/webgpu-renderer.ts
@@ -67,7 +67,7 @@ import {
srtLoader,
Vector3Math,
} from "@framefields/webgpu-renderers";
-import { compileTimeline } from "../shared/compiler.js";
+import { compileTimeline, seekTimeline } from "../shared/compiler.js";
import type { CompositorOperation } from "../shared/config.js";
import { shouldPaintContainerBackground } from "./container-bg.js";
import { buildLayerMatrix, toDOMMatrix } from "./transform.js";
@@ -614,12 +614,12 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
const composeDuration =
virtualMedia?.metadata?.durationMs ?? DEFAULT_DURATION_MS;
const durationSec = composeDuration / 1000;
- const { tl, targetsById } = compileTimeline(
- props.renderId,
- virtualMedia as any,
- { fps: fps ?? 24, durationSec },
- );
- tl.seek(shiftedFrame / (fps ?? 24));
+ const compiled = compileTimeline(props.renderId, virtualMedia as any, {
+ fps: fps ?? 24,
+ durationSec,
+ });
+ const { targetsById } = compiled;
+ seekTimeline(compiled, shiftedFrame, fps ?? 24);
const r3d = getRenderer3D(
ctx.device,
@@ -2555,13 +2555,11 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
const container3DShutter =
target?.motionBlurShutter ?? lop.motionBlurShutter ?? 0;
if (container3DShutter > 0 && fps && shiftedFrame !== undefined) {
- const currentT = shiftedFrame / fps;
- const prevT = (shiftedFrame - 1) / fps;
- tl.seek(prevT);
+ seekTimeline(compiled, shiftedFrame - 1, fps);
const prevTarget = targetId ? targetsById[targetId] : undefined;
const prevX = prevTarget?.x ?? baseX;
const prevY = prevTarget?.y ?? baseY;
- tl.seek(currentT);
+ seekTimeline(compiled, shiftedFrame, fps);
const vx = sampledX - prevX;
const vy = sampledY - prevY;
@@ -2842,13 +2840,11 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
const containerShutter =
target?.motionBlurShutter ?? lop.motionBlurShutter ?? 0;
if (containerShutter > 0 && fps && shiftedFrame !== undefined) {
- const currentT = shiftedFrame / fps;
- const prevT = (shiftedFrame - 1) / fps;
- tl.seek(prevT);
+ seekTimeline(compiled, shiftedFrame - 1, fps);
const prevTarget = targetId ? targetsById[targetId] : undefined;
const prevX = prevTarget?.x ?? baseX;
const prevY = prevTarget?.y ?? baseY;
- tl.seek(currentT);
+ seekTimeline(compiled, shiftedFrame, fps);
const vx = sampledX - prevX;
const vy = sampledY - prevY;
@@ -3683,13 +3679,11 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
let blurPadX = 0;
let blurPadY = 0;
if (shutterAngle > 0 && fps && shiftedFrame !== undefined) {
- const currentT = shiftedFrame / fps;
- const prevT = (shiftedFrame - 1) / fps;
- tl.seek(prevT);
+ seekTimeline(compiled, shiftedFrame - 1, fps);
const prevTarget = targetId ? targetsById[targetId] : undefined;
const prevX = prevTarget?.x ?? baseX;
const prevY = prevTarget?.y ?? baseY;
- tl.seek(currentT);
+ seekTimeline(compiled, shiftedFrame, fps);
const vx = sampledX - prevX;
const vy = sampledY - prevY;
diff --git a/nodes/node-compositor/src/shared/audio-reactive.test.ts b/nodes/node-compositor/src/shared/audio-reactive.test.ts
index 32346cd6..450bf4c7 100644
--- a/nodes/node-compositor/src/shared/audio-reactive.test.ts
+++ b/nodes/node-compositor/src/shared/audio-reactive.test.ts
@@ -1,5 +1,5 @@
import { describe, expect, it } from "vitest";
-import { compileTimeline } from "./compiler.js";
+import { compileTimeline, seekTimeline } from "./compiler.js";
import type { CompositorNodeConfig } from "./config.js";
import { processCompositor } from "./processor.js";
@@ -166,17 +166,18 @@ describe("Audio-Reactive Motion Graphics Engine Pipeline", () => {
expect(signals.audio_beat_handle).toBe(beatSamples);
// 2. Compile into timeline
- const { tl, targetsById } = compileTimeline("audio-reactive-session", vm, {
+ const compiled = compileTimeline("audio-reactive-session", vm, {
fps: 24,
durationSec: 2,
});
+ const { targetsById } = compiled;
const logoStub = targetsById["bouncing-logo-star"];
expect(logoStub).toBeDefined();
// Check baseline at frame 0 (t = 0 sec):
// Beat signal = 0.0 -> scale = 0.0 * 0.6 + 1.0 = 1.0
- tl.seek(0);
+ seekTimeline(compiled, 0, 24);
expect(logoStub.scale).toBeCloseTo(1.0, 3);
// Bass signal = 0.0 -> trimEnd = 0.0 * 0.8 + 0.2 = 0.2
expect(logoStub.trimEnd).toBeCloseTo(0.2, 3);
@@ -185,16 +186,16 @@ describe("Audio-Reactive Motion Graphics Engine Pipeline", () => {
// Check beat hit at frame 12 (t = 12/24 = 0.5 sec):
// Beat signal = 1.0 -> scale = 1.0 * 0.6 + 1.0 = 1.6!
- tl.seek(12 / 24);
+ seekTimeline(compiled, 12, 24);
expect(logoStub.scale).toBeCloseTo(1.6, 3);
// Check post-beat settling at frame 14:
// Beat signal = 0.0 -> scale back to 1.0
- tl.seek(14 / 24);
+ seekTimeline(compiled, 14, 24);
expect(logoStub.scale).toBeCloseTo(1.0, 3);
// Check second beat hit at frame 24 (t = 1.0 sec):
- tl.seek(24 / 24);
+ seekTimeline(compiled, 24, 24);
expect(logoStub.scale).toBeCloseTo(1.6, 3);
// Check container staggering:
@@ -206,17 +207,17 @@ describe("Audio-Reactive Motion Graphics Engine Pipeline", () => {
const bar3 = targetsById["bar-3"];
// At frame 0:
- tl.seek(0);
+ seekTimeline(compiled, 0, 24);
expect(bar1.opacity).toBe(0);
// At frame 4 (bar-1 is halfway faded 0.33, bar-2 is just starting):
- tl.seek(4 / 24);
+ seekTimeline(compiled, 4, 24);
expect(bar1.opacity).toBeGreaterThan(0);
expect(bar2.opacity).toBe(0);
expect(bar3.opacity).toBe(0);
// At frame 8 (bar-3 is just starting):
- tl.seek(8 / 24);
+ seekTimeline(compiled, 8, 24);
expect(bar2.opacity).toBeGreaterThan(0);
expect(bar3.opacity).toBe(0);
});
diff --git a/nodes/node-compositor/src/shared/compiler.test.ts b/nodes/node-compositor/src/shared/compiler.test.ts
index 6ef9b508..37f51408 100644
--- a/nodes/node-compositor/src/shared/compiler.test.ts
+++ b/nodes/node-compositor/src/shared/compiler.test.ts
@@ -1,6 +1,10 @@
import gsapModule from "gsap";
import { afterEach, describe, expect, it, vi } from "vitest";
-import { compileLayerTimeline, compileTimeline } from "./compiler.js";
+import {
+ compileLayerTimeline,
+ compileTimeline,
+ seekTimeline,
+} from "./compiler.js";
const gsap: any = (gsapModule as any).timeline
? gsapModule
@@ -1114,6 +1118,46 @@ describe("keyframe tail truncation (review M8)", () => {
expect(stub.y).toBe(50);
});
+ it("reads signal tracks when their frame is drawn, not when compiled", () => {
+ // A live physics world fills its signals while rendering: values
+ // written after compiling must still reach the frame.
+ const live = [0, 0, 0, 0];
+ const virtualMedia: any = {
+ metadata: { width: 1920, height: 1080, durationMs: 2000 },
+ operation: { op: "Compositor", signals: { live } },
+ children: [
+ {
+ operation: {
+ op: "CompositorLayer",
+ id: "live-node",
+ x: 0,
+ durationFrames: 48,
+ animation: {
+ tracks: [
+ {
+ id: "t_live",
+ prop: "x" as const,
+ source: { type: "signal" as const, inputHandleId: "live" },
+ },
+ ],
+ },
+ },
+ },
+ ],
+ };
+ const compiled = compileTimeline("render-live-signal-test", virtualMedia, {
+ fps: 24,
+ durationSec: 2,
+ });
+ const stub = compiled.targetsById["live-node"];
+ live[2] = 120;
+ seekTimeline(compiled, 2, 24);
+ expect(stub.x).toBe(120);
+ live[3] = 160;
+ seekTimeline(compiled, 3, 24);
+ expect(stub.x).toBe(160);
+ });
+
it("compiles and evaluates procedural wiggle and signal tracks", () => {
const virtualMedia: any = {
metadata: { width: 1920, height: 1080, durationMs: 2000 },
@@ -1160,21 +1204,21 @@ describe("keyframe tail truncation (review M8)", () => {
],
};
- const { tl, targetsById } = compileTimeline(
- "render-procedural-test",
- virtualMedia,
- { fps: 24, durationSec: 2 },
- );
+ const compiled = compileTimeline("render-procedural-test", virtualMedia, {
+ fps: 24,
+ durationSec: 2,
+ });
+ const { targetsById } = compiled;
const stub = targetsById["proc-node"];
expect(stub).toBeDefined();
// Frame 0
- tl.seek(0);
+ seekTimeline(compiled, 0, 24);
expect(stub.scale).toBe(1.0); // 0 * 0.5 + 1.0 = 1.0
// Frame 2 (at 24fps = 2/24 sec) -> audio_pulse[2] is 1.0 -> 1.0 * 0.5 + 1.0 = 1.5
- tl.seek(2 / 24);
+ seekTimeline(compiled, 2, 24);
expect(stub.scale).toBeCloseTo(1.5, 4);
// Wiggle moves smoothly around baseline 50 +- 40
diff --git a/nodes/node-compositor/src/shared/compiler.ts b/nodes/node-compositor/src/shared/compiler.ts
index f844ae79..9e8d0d50 100644
--- a/nodes/node-compositor/src/shared/compiler.ts
+++ b/nodes/node-compositor/src/shared/compiler.ts
@@ -111,6 +111,37 @@ export interface CompileResult {
targetsById: Record<string, LayerStub>;
/** Node ops the timeline was compiled from (tree order). */
nodes: CompilerNodeVM[];
+ /**
+ * Signal-driven tracks, evaluated at draw time instead of baked into the
+ * timeline: a signal may be filled while rendering (a live physics world,
+ * on-demand vision), so its value is read when its frame is drawn.
+ */
+ live: LiveTrack[];
+ signals?: Record<string, any>;
+}
+
+interface LiveTrack {
+ stub: Record<string, any>;
+ track: AnimationTrack;
+ startFrame: number;
+ endFrame: number;
+}
+
+/** Seeks the compiled timeline to a frame and applies its live signal tracks. */
+export function seekTimeline(
+ result: CompileResult,
+ frame: number,
+ fps: number,
+): void {
+ result.tl.seek(frame / fps);
+ for (const live of result.live) {
+ const f = Math.max(live.startFrame, Math.min(live.endFrame, frame));
+ live.stub[live.track.prop] = evaluateTrackAtFrame(live.track, f, fps, {
+ startFrame: live.startFrame,
+ baseValues: live.stub,
+ signals: result.signals,
+ });
+ }
}
// L2: compiled timelines and their LayerStubs are cached per cacheKey and
@@ -341,10 +372,24 @@ function compileTrackToTimeline(
fps: number,
durationFrames: number,
signals?: Record<string, any>,
+ live?: LiveTrack[],
): { dropped: number; maxAuthoredFrame: number } {
const prop = track.prop;
const source = track.source ?? { type: "keyframe" };
+ if (source.type === "signal" && live && !track.repeat) {
+ const endF = Number.isFinite(durationFrames)
+ ? durationFrames
+ : Math.round(fps * 5);
+ stub[prop] = evaluateTrackAtFrame(track, startFrame, fps, {
+ startFrame,
+ baseValues: stub,
+ signals,
+ });
+ live.push({ stub, track, startFrame, endFrame: startFrame + endF });
+ return { dropped: 0, maxAuthoredFrame: 0 };
+ }
+
if (source.type !== "keyframe") {
const endF = Number.isFinite(durationFrames)
? durationFrames
@@ -853,6 +898,7 @@ export function compileTimeline(
// them here and warn ONCE per compiled cache key (per-frame seeks hit the
// cache, so this is naturally once per render/preview session).
const truncations: TruncationInfo[] = [];
+ const live: LiveTrack[] = [];
nodeOps.forEach((vm) => {
const lop = (vm?.operation || {}) as any;
@@ -894,6 +940,7 @@ export function compileTimeline(
fps,
durationFrames,
signals,
+ live,
);
if (dropped > 0) {
truncations.push({
@@ -929,6 +976,8 @@ export function compileTimeline(
tl,
targetsById,
nodes: nodeOps,
+ live,
+ signals,
};
setCachedTimeline(cacheKey, compileResult);
diff --git a/packages/compositions/src/compositions/canvas-composition.ts b/packages/compositions/src/compositions/canvas-composition.ts
index 7dfc8fb4..e715a1c8 100644
--- a/packages/compositions/src/compositions/canvas-composition.ts
+++ b/packages/compositions/src/compositions/canvas-composition.ts
@@ -247,7 +247,12 @@ export async function drawCompositionTree(
const accumulatedClockOffsetSec =
inheritedClockOffset + windowStartFrame / fps;
if (typeof op.onRequestFrame === "function") {
- op.onRequestFrame(frameCtx);
+ // May return a promise: per-frame preparation (vision, live physics)
+ // finishes before anything in this frame is evaluated.
+ const pending = op.onRequestFrame(frameCtx);
+ if (pending && typeof (pending as Promise<void>).then === "function") {
+ await pending;
+ }
} else if (op.effect && typeof op.effect.notifyFrame === "function") {
op.effect.notifyFrame(frameCtx);
}
diff --git a/packages/compositions/src/program/animation.ts b/packages/compositions/src/program/animation.ts
index 8583b1d6..0f2bfac6 100644
--- a/packages/compositions/src/program/animation.ts
+++ b/packages/compositions/src/program/animation.ts
@@ -63,6 +63,10 @@ export interface SignalTrackOptions {
export interface ColorSignalOptions {
colorA: string;
colorB: string;
+ /** Scales the signal before it picks a color (e.g. `1 / 900` for a speed in px/s). Default 1. */
+ multiplier?: number;
+ /** Added after `multiplier`. Default 0. */
+ offset?: number;
colorMode?: "interpolate" | "hueRotate" | "threshold";
threshold?: number;
smoothing?: number;
@@ -834,8 +838,8 @@ export class LayerAnimation implements LayerAnimationSpec {
const source: TrackSource = {
type: "signal",
inputHandleId: handleId,
- multiplier: 1.0,
- offset: 0.0,
+ multiplier: options.multiplier ?? 1.0,
+ offset: options.offset ?? 0.0,
smoothingWindowFrames: options.smoothing ?? 0,
colorMode: options.colorMode ?? "interpolate",
colorA: options.colorA,
diff --git a/packages/framefields/src/clip-vision.test.ts b/packages/framefields/src/clip-vision.test.ts
new file mode 100644
index 00000000..8ba828d0
--- /dev/null
+++ b/packages/framefields/src/clip-vision.test.ts
@@ -0,0 +1,75 @@
+import { describe, expect, it } from "vitest";
+import { clipVisionFromResults } from "./clip-vision.js";
+
+/** A 160×80 clip: frame f has the left wrist at (20 + 20f, 10) and a 40×40 square silhouette moving right. */
+function clip(frames = 4, options = {}) {
+ const cell = 4;
+ const w = 40;
+ const h = 20;
+ return clipVisionFromResults(
+ 24,
+ { width: 160, height: 80, cell },
+ Array.from({ length: frames }, (_, f) => {
+ const kp = new Array(51).fill(0);
+ kp.splice(9 * 3, 3, 20 + 20 * f, 10, 1);
+ const mask = new Uint8Array(w * h);
+ for (let y = 5; y < 15; y++)
+ for (let x = 5 + f * 10; x < 15 + f * 10 && x < w; x++)
+ mask[y * w + x] = 255;
+ return { kp, mask };
+ }),
+ { smoothing: 0, ...options },
+ );
+}
+
+describe("ClipVision", () => {
+ it("reads keypoints at composition frames and maps them into a layer box", () => {
+ const person = clip().person;
+ expect(person.point("leftWrist", 1.5)).toMatchObject({
+ x: 50,
+ y: 10,
+ visibility: 1,
+ });
+ // 160×80 covering 320×320 at (100, 0): scale 4, centered.
+ const placed = person.place({
+ x: 100,
+ y: 0,
+ width: 320,
+ height: 320,
+ fit: "cover",
+ });
+ expect(placed.point("leftWrist", 0)).toMatchObject({
+ x: 100 - 160 + 80,
+ y: 40,
+ });
+ expect(
+ placed.signals("leftWrist").x.get({ frame: 1, fps: 24 } as never),
+ ).toBeCloseTo(100 - 160 + 160);
+ });
+
+ it("offsets the clip by its start frame", () => {
+ const late = clip(4, { startFrame: 10 }).person;
+ expect(late.point("leftWrist", 12).x).toBe(60);
+ });
+
+ it("smooths over neighbouring frames, weighted by confidence", () => {
+ const person = clip(5, { smoothing: 1 }).person;
+ // Frames 1..3 at x 40, 60, 80 with weights 1, 2, 1 → 60.
+ expect(person.point("leftWrist", 2).x).toBeCloseTo(60);
+ });
+
+ it("samples the silhouette with a gradient pointing inward", () => {
+ const person = clip().person;
+ expect(person.sample(0, 40, 40).alpha).toBeGreaterThan(0.9);
+ expect(person.sample(0, 120, 40).alpha).toBe(0);
+ const edge = person.sample(0, 62, 40);
+ expect(edge.alpha).toBeGreaterThan(0);
+ expect(edge.gx).toBeLessThan(0);
+ });
+
+ it("explains a read of a frame that was never analysed", () => {
+ expect(() => clip(2).person.point("leftWrist", 30)).toThrow(
+ /not analysed yet/,
+ );
+ });
+});
diff --git a/packages/framefields/src/clip-vision.ts b/packages/framefields/src/clip-vision.ts
new file mode 100644
index 00000000..a0df0a9d
--- /dev/null
+++ b/packages/framefields/src/clip-vision.ts
@@ -0,0 +1,665 @@
+import { type FrameSignal, Signal } from "@framefields/core";
+import { COCO17_KEYPOINT_NAMES, VisionRunner } from "@framefields/vision";
+import { mediaDecoderCache } from "@framefields/webgpu-renderers";
+import type { VisionProps } from "./effects/index.js";
+
+export type PoseKeypointName = (typeof COCO17_KEYPOINT_NAMES)[number];
+
+export interface ClipVisionOptions {
+ /** Composition frame the clip's first frame plays on (default 0). */
+ startFrame?: number;
+ /** Seconds trimmed off the clip's start, as on its media layer (default 0). */
+ trimStartSec?: number;
+ /** Analysis frame rate (default: the composition's). */
+ fps?: number;
+ /** Track 17 COCO keypoints (default true). */
+ pose?: boolean;
+ /** Track the person's silhouette, dress and held things included (default true). */
+ mask?: boolean;
+ /** Silhouette cell size in source pixels (default 8: a 1920×1080 clip gives a 240×135 grid). */
+ maskCell?: number;
+ /** Keypoints are averaged over ±`smoothing` frames, weighted by confidence (default 2; 0 = raw). */
+ smoothing?: number;
+ variant?: VisionProps["variant"];
+ confidence?: number;
+ modelsDir?: string;
+ baseUrl?: string;
+ /**
+ * Node: where results persist between runs. Default
+ * `$FRAMEFIELDS_VISION_CACHE_DIR` or `~/.cache/framefields/vision`;
+ * `false` keeps them in memory only. Browsers always keep them in memory.
+ */
+ cacheDir?: string | false;
+}
+
+/** Where the clip sits in the composition, like a media layer's box and `fit`. */
+export interface PersonPlacement {
+ x?: number;
+ y?: number;
+ width: number;
+ height: number;
+ fit?: "cover" | "contain" | "fill";
+}
+
+interface FrameResult {
+ /** 17 × [x, y, visibility], source px; null when no person was found. */
+ kp: Float32Array | null;
+ /** cellW × cellH, 0–255; null when no person was found. */
+ mask: Uint8Array | null;
+}
+
+interface Transform {
+ sx: number;
+ sy: number;
+ ox: number;
+ oy: number;
+}
+
+/** Frames looked back over to bridge a gap where the person is lost. */
+const HOLD = 24;
+
+const CACHE_VERSION = 1;
+
+/**
+ * Vision results for one clip, computed per frame on demand and shared by
+ * everything that reads the clip: physics, layout, signals. Unlike the
+ * render-time `withVision`, which analyses a layer's previous rendered frame,
+ * this analyses the clip's own frames, so a result exists before its frame
+ * renders. The composition analyses what each frame needs just before
+ * drawing it (and a live physics world before stepping), so preview and
+ * export fill the cache as they play; nothing has to be analysed up front.
+ *
+ * Results depend only on the clip and settings, so they persist on disk in
+ * Node: a second run, or a render after a preview, reuses them.
+ */
+export class ClipVision {
+ readonly source: string;
+ readonly fps: number;
+ readonly startFrame: number;
+ readonly trimStartSec: number;
+ readonly smoothing: number;
+ private readonly options: ClipVisionOptions;
+ private readonly results = new Map<number, FrameResult>();
+ private sourceWidth = 0;
+ private sourceHeight = 0;
+ private cellW = 0;
+ private cellH = 0;
+ private readonly cell: number;
+ private queue: Promise<void> = Promise.resolve();
+ private runner?: VisionRunner;
+ private cacheLoaded?: Promise<void>;
+ private unsaved = 0;
+ private readonly blurred = new Map<number, Float32Array | null>();
+ private readonly smoothed = new Map<string, [number, number, number]>();
+ private personView?: PersonTrack;
+
+ /** @internal Use `comp.vision(source, options)`. */
+ constructor(source: string, fps: number, options: ClipVisionOptions = {}) {
+ this.source = source;
+ this.options = options;
+ this.fps = options.fps ?? fps;
+ this.startFrame = Math.round(options.startFrame ?? 0);
+ this.trimStartSec = options.trimStartSec ?? 0;
+ this.smoothing = Math.max(0, Math.round(options.smoothing ?? 2));
+ this.cell = Math.max(1, Math.round(options.maskCell ?? 8));
+ }
+
+ /** The tracked person in source pixels; `.place()` maps it into the composition. */
+ get person(): PersonTrack {
+ this.personView ??= new PersonTrack(this);
+ return this.personView;
+ }
+
+ /**
+ * Analyses what reading composition frame `frame` needs (it and a few
+ * neighbours for smoothing). Pass a second frame to analyse a whole range.
+ * Calls queue, so the clip is decoded once, in order.
+ */
+ ready(frame: number, toFrame: number = frame): Promise<void> {
+ const [from, to] = this.window(frame, toFrame);
+ this.queue = this.queue.then(() => this.compute(from, to));
+ return this.queue;
+ }
+
+ /** True when composition frame `frame` can be read synchronously. */
+ isReady(frame: number): boolean {
+ const [from, to] = this.window(frame, frame);
+ for (let c = from; c <= to; c++) if (!this.results.has(c)) return false;
+ return true;
+ }
+
+ /** Composition frame → clip frame (fractional). */
+ clipFrame(frame: number): number {
+ return frame - this.startFrame + this.trimStartSec * this.fps;
+ }
+
+ private window(frame: number, toFrame: number): [number, number] {
+ const a = Math.floor(this.clipFrame(Math.min(frame, toFrame)));
+ const b = Math.ceil(this.clipFrame(Math.max(frame, toFrame)));
+ return [
+ Math.max(0, a - HOLD - this.smoothing),
+ Math.max(0, b + this.smoothing),
+ ];
+ }
+
+ private async compute(from: number, to: number): Promise<void> {
+ await (this.cacheLoaded ??= this.loadCache());
+ let missing = false;
+ for (let c = from; c <= to; c++) if (!this.results.has(c)) missing = true;
+ if (!missing) return;
+
+ const wantPose = this.options.pose ?? true;
+ const wantMask = this.options.mask ?? true;
+ this.runner ??= VisionRunner.create({
+ modelsDir: this.options.modelsDir,
+ baseUrl: this.options.baseUrl,
+ ...(this.options.confidence !== undefined && {
+ confidence: this.options.confidence,
+ }),
+ ...(this.options.variant !== undefined && {
+ variant: this.options.variant,
+ }),
+ });
+ const decoder = mediaDecoderCache.getVideo(
+ this.source,
+ true,
+ "clip-vision",
+ );
+ for (let c = from; c <= to; c++) {
+ if (this.results.has(c)) continue;
+ const image = await decodedPixels(await decoder.getFrame(c / this.fps));
+ if (!image) {
+ this.results.set(c, { kp: null, mask: null });
+ continue;
+ }
+ if (!this.sourceWidth) this.setSize(image.width, image.height);
+ let kp: Float32Array | null = null;
+ let mask: Uint8Array | null = null;
+ if (wantPose) {
+ const person = (await this.runner.pose(image)).people[0];
+ if (person) {
+ kp = new Float32Array(51);
+ person.keypoints.forEach((p, k) =>
+ kp?.set([p.x, p.y, p.visibility], k * 3),
+ );
+ }
+ }
+ if (wantMask) {
+ const silhouette = personSilhouette(await this.runner.segment(image));
+ if (silhouette) {
+ mask = new Uint8Array(this.cellW * this.cellH);
+ downsampleMean(
+ silhouette,
+ image.width,
+ image.height,
+ this.cell,
+ mask,
+ this.cellW,
+ this.cellH,
+ );
+ }
+ }
+ this.results.set(c, { kp, mask });
+ if (++this.unsaved >= 24) await this.saveCache();
+ }
+ if (this.unsaved > 0) await this.saveCache();
+ }
+
+ private setSize(width: number, height: number): void {
+ this.sourceWidth = width;
+ this.sourceHeight = height;
+ this.cellW = Math.ceil(width / this.cell);
+ this.cellH = Math.ceil(height / this.cell);
+ }
+
+ /** @internal */
+ size(): { width: number; height: number } {
+ return { width: this.sourceWidth, height: this.sourceHeight };
+ }
+
+ /** @internal Raw result for a clip frame, or throws when it was never analysed. */
+ result(c: number): FrameResult | undefined {
+ return this.results.get(Math.max(0, c));
+ }
+
+ /** @internal Confidence-weighted keypoint over the smoothing window, in source px. */
+ keypoint(k: number, c: number): [number, number, number] {
+ c = Math.max(0, c);
+ const key = `${k}:${c}`;
+ const hit = this.smoothed.get(key);
+ if (hit) return hit;
+ const avg = (center: number): [number, number, number] | null => {
+ let sx = 0;
+ let sy = 0;
+ let sv = 0;
+ let sw = 0;
+ for (let g = center - this.smoothing; g <= center + this.smoothing; g++) {
+ const r = this.mustHave(g < 0 ? 0 : g);
+ if (!r.kp) continue;
+ const v = r.kp[k * 3 + 2];
+ const w = (this.smoothing + 1 - Math.abs(g - center)) * v * v;
+ sx += r.kp[k * 3] * w;
+ sy += r.kp[k * 3 + 1] * w;
+ sv += v * w;
+ sw += w;
+ }
+ return sw > 1e-6 ? [sx / sw, sy / sw, sv / sw] : null;
+ };
+ let out: [number, number, number] | null = null;
+ // Lost? Hold the last place the person was seen (within HOLD frames).
+ for (let d = 0; d <= HOLD && !out; d++) if (c - d >= 0) out = avg(c - d);
+ out ??= [this.sourceWidth / 2, this.sourceHeight / 2, 0];
+ this.smoothed.set(key, out);
+ return out;
+ }
+
+ /** @internal Softened silhouette grid for a clip frame (null: no person). */
+ blurredMask(c: number): Float32Array | null {
+ c = Math.max(0, c);
+ if (this.blurred.has(c)) return this.blurred.get(c) ?? null;
+ const r = this.mustHave(c);
+ const grid = r.mask ? boxBlur(r.mask, this.cellW, this.cellH, 2) : null;
+ this.blurred.set(c, grid);
+ return grid;
+ }
+
+ /** @internal */
+ grid(): { cell: number; width: number; height: number } {
+ return { cell: this.cell, width: this.cellW, height: this.cellH };
+ }
+
+ private mustHave(c: number): FrameResult {
+ const r = this.results.get(c);
+ if (r) return r;
+ throw new Error(
+ `ClipVision "${this.source}": frame ${c} is not analysed yet. ` +
+ "Read it while rendering (the composition prepares each frame), in a live physics world, " +
+ `or after \`await clip.ready(frame)\`.`,
+ );
+ }
+
+ // ── Persistence (Node) ────────────────────────────────────────────────
+
+ private cacheFile?: string | null;
+
+ private async cachePath(): Promise<string | null> {
+ if (this.cacheFile !== undefined) return this.cacheFile;
+ this.cacheFile = null;
+ if (
+ this.options.cacheDir === false ||
+ typeof process === "undefined" ||
+ !process.versions?.node
+ ) {
+ return null;
+ }
+ try {
+ const [fs, path, os, crypto] = await Promise.all([
+ import("node:fs"),
+ import("node:path"),
+ import("node:os"),
+ import("node:crypto"),
+ ]);
+ const stat = fs.statSync(this.source);
+ const dir =
+ this.options.cacheDir ??
+ process.env.FRAMEFIELDS_VISION_CACHE_DIR ??
+ path.join(os.homedir(), ".cache", "framefields", "vision");
+ const o = this.options;
+ const key = JSON.stringify([
+ CACHE_VERSION,
+ path.resolve(this.source),
+ stat.size,
+ stat.mtimeMs,
+ this.fps,
+ o.pose ?? true,
+ o.mask ?? true,
+ this.cell,
+ o.variant ?? "s",
+ o.confidence ?? null,
+ ]);
+ const hash = crypto
+ .createHash("sha256")
+ .update(key)
+ .digest("hex")
+ .slice(0, 24);
+ fs.mkdirSync(dir, { recursive: true });
+ this.cacheFile = path.join(
+ dir,
+ `${path.basename(this.source)}.${hash}.json`,
+ );
+ } catch {
+ this.cacheFile = null; // remote URL or unreadable: memory only
+ }
+ return this.cacheFile;
+ }
+
+ private async loadCache(): Promise<void> {
+ const file = await this.cachePath();
+ if (!file) return;
+ try {
+ const fs = await import("node:fs");
+ if (!fs.existsSync(file)) return;
+ const json = JSON.parse(fs.readFileSync(file, "utf8")) as {
+ width: number;
+ height: number;
+ cell: number;
+ frames: Record<string, { kp: number[] | null; mask: string | null }>;
+ };
+ if (json.cell !== this.cell) return;
+ this.setSize(json.width, json.height);
+ for (const [c, r] of Object.entries(json.frames)) {
+ this.results.set(Number(c), {
+ kp: r.kp ? Float32Array.from(r.kp) : null,
+ mask: r.mask ? new Uint8Array(Buffer.from(r.mask, "base64")) : null,
+ });
+ }
+ } catch {
+ // A damaged cache is recomputed.
+ }
+ }
+
+ private async saveCache(): Promise<void> {
+ this.unsaved = 0;
+ const file = await this.cachePath();
+ if (!file || !this.sourceWidth) return;
+ const frames: Record<string, { kp: number[] | null; mask: string | null }> =
+ {};
+ for (const [c, r] of this.results) {
+ frames[c] = {
+ kp: r.kp ? Array.from(r.kp, (v) => Math.round(v * 100) / 100) : null,
+ mask: r.mask ? Buffer.from(r.mask).toString("base64") : null,
+ };
+ }
+ const fs = await import("node:fs");
+ const tmp = `${file}.${process.pid}.tmp`;
+ fs.writeFileSync(
+ tmp,
+ JSON.stringify({
+ width: this.sourceWidth,
+ height: this.sourceHeight,
+ cell: this.cell,
+ frames,
+ }),
+ );
+ fs.renameSync(tmp, file);
+ }
+}
+
+/**
+ * The person in a {@link ClipVision}: pose keypoints and silhouette at any
+ * composition frame, in source pixels or, after `place()`, composition
+ * pixels. Reads are synchronous; the frame must be analysed (it is while
+ * rendering, inside a live physics world, or after `await clip.ready(frame)`).
+ */
+export class PersonTrack {
+ /** @internal */
+ constructor(
+ readonly clip: ClipVision,
+ private readonly t?: Transform,
+ private readonly placement?: PersonPlacement,
+ ) {}
+
+ /** The same person in composition pixels for a clip laid out at `placement`. */
+ place(placement: PersonPlacement): PersonTrack {
+ return new PersonTrack(this.clip, undefined, placement);
+ }
+
+ private transform(): Transform {
+ if (this.t) return this.t;
+ const p = this.placement;
+ const { width: sw, height: sh } = this.clip.size();
+ if (!p || !sw) return { sx: 1, sy: 1, ox: 0, oy: 0 };
+ let sx = p.width / sw;
+ let sy = p.height / sh;
+ if (p.fit !== "fill") {
+ const s = p.fit === "contain" ? Math.min(sx, sy) : Math.max(sx, sy);
+ sx = s;
+ sy = s;
+ }
+ return {
+ sx,
+ sy,
+ ox: (p.x ?? 0) + (p.width - sw * sx) / 2,
+ oy: (p.y ?? 0) + (p.height - sh * sy) / 2,
+ };
+ }
+
+ /** A keypoint at a composition frame (fractional frames interpolate). */
+ point(
+ name: PoseKeypointName,
+ frame: number,
+ ): { x: number; y: number; visibility: number } {
+ const k = keypointIndex(name);
+ const c = this.clip.clipFrame(frame);
+ const c0 = Math.floor(c);
+ const u = c - c0;
+ const a = this.clip.keypoint(k, c0);
+ const b = u > 0 ? this.clip.keypoint(k, c0 + 1) : a;
+ const t = this.transform();
+ return {
+ x: (a[0] + (b[0] - a[0]) * u) * t.sx + t.ox,
+ y: (a[1] + (b[1] - a[1]) * u) * t.sy + t.oy,
+ visibility: a[2] + (b[2] - a[2]) * u,
+ };
+ }
+
+ /** Signals for a keypoint, for `.signal("x", …)`; read while rendering. */
+ signals(name: PoseKeypointName): {
+ x: FrameSignal<number>;
+ y: FrameSignal<number>;
+ visibility: FrameSignal<number>;
+ } {
+ const read = (axis: "x" | "y" | "visibility") =>
+ Signal.builder({
+ type: "custom",
+ fn: (ctx) => this.point(name, ctx.frame)[axis],
+ }) as FrameSignal<number>;
+ return { x: read("x"), y: read("y"), visibility: read("visibility") };
+ }
+
+ /**
+ * The silhouette near a point at a composition frame: `alpha` 0–1 (softened
+ * over a few cells; inside > 0.5) and its gradient `gx`, `gy` per px,
+ * pointing into the body. Push things out with `-gradient`.
+ */
+ sample(
+ frame: number,
+ x: number,
+ y: number,
+ ): { alpha: number; gx: number; gy: number } {
+ const grid = this.clip.blurredMask(Math.round(this.clip.clipFrame(frame)));
+ if (!grid) return { alpha: 0, gx: 0, gy: 0 };
+ const { cell, width: w, height: h } = this.clip.grid();
+ const t = this.transform();
+ const cx = (x - t.ox) / t.sx / cell - 0.5;
+ const cy = (y - t.oy) / t.sy / cell - 0.5;
+ const read = (gx: number, gy: number) =>
+ gx < 0 || gy < 0 || gx >= w || gy >= h ? 0 : grid[gy * w + gx];
+ const bil = (px: number, py: number) => {
+ const x0 = Math.floor(px);
+ const y0 = Math.floor(py);
+ const ux = px - x0;
+ const uy = py - y0;
+ const a = read(x0, y0) + (read(x0 + 1, y0) - read(x0, y0)) * ux;
+ const b =
+ read(x0, y0 + 1) + (read(x0 + 1, y0 + 1) - read(x0, y0 + 1)) * ux;
+ return a + (b - a) * uy;
+ };
+ return {
+ alpha: bil(cx, cy),
+ gx: (bil(cx + 1, cy) - bil(cx - 1, cy)) / (2 * cell * t.sx),
+ gy: (bil(cx, cy + 1) - bil(cx, cy - 1)) / (2 * cell * t.sy),
+ };
+ }
+}
+
+function keypointIndex(name: PoseKeypointName): number {
+ const i = COCO17_KEYPOINT_NAMES.indexOf(name);
+ if (i < 0) {
+ throw new Error(
+ `Unknown keypoint "${name}". Use one of ${COCO17_KEYPOINT_NAMES.join(", ")}.`,
+ );
+ }
+ return i;
+}
+
+/** RGBA pixels of a decoded frame (headless buffer, or a canvas in the browser). */
+async function decodedPixels(
+ frame: {
+ width: number;
+ height: number;
+ buffer?: Uint8Array;
+ canvas?: unknown;
+ } | null,
+): Promise<{
+ data: Uint8Array | Uint8ClampedArray;
+ width: number;
+ height: number;
+} | null> {
+ if (!frame || frame.width <= 0) return null;
+ if (frame.buffer)
+ return { data: frame.buffer, width: frame.width, height: frame.height };
+ const canvas = frame.canvas as
+ | {
+ getContext(type: "2d"): {
+ getImageData(
+ x: number,
+ y: number,
+ w: number,
+ h: number,
+ ): { data: Uint8ClampedArray };
+ } | null;
+ }
+ | undefined;
+ const ctx = canvas?.getContext("2d");
+ if (!ctx) return null;
+ return {
+ data: ctx.getImageData(0, 0, frame.width, frame.height).data,
+ width: frame.width,
+ height: frame.height,
+ };
+}
+
+type Segmentation = Awaited<ReturnType<VisionRunner["segment"]>>;
+
+/**
+ * The largest person's mask, plus instances that overlap it and are no more
+ * than twice its size (a flowing dress often comes back as "kite").
+ */
+function personSilhouette(seg: Segmentation): Uint8Array | undefined {
+ const people = seg.masks.filter((m) => m.category === "person");
+ if (people.length === 0) return undefined;
+ const person = people.reduce((a, b) => (b.area > a.area ? b : a));
+ const box = seg.detections[person.detectionIndex]?.boundingBox;
+ const out = new Uint8Array(person.mask);
+ if (!box) return out;
+ for (const m of seg.masks) {
+ if (m === person || m.area > person.area * 2) continue;
+ const b = seg.detections[m.detectionIndex]?.boundingBox;
+ if (!b) continue;
+ const ix = Math.max(
+ 0,
+ Math.min(box.originX + box.width, b.originX + b.width) -
+ Math.max(box.originX, b.originX),
+ );
+ const iy = Math.max(
+ 0,
+ Math.min(box.originY + box.height, b.originY + b.height) -
+ Math.max(box.originY, b.originY),
+ );
+ if (ix * iy < 0.3 * b.width * b.height) continue;
+ for (let i = 0; i < out.length; i++)
+ if (m.mask[i] > out[i]) out[i] = m.mask[i];
+ }
+ return out;
+}
+
+function downsampleMean(
+ src: Uint8Array,
+ w: number,
+ h: number,
+ cell: number,
+ dst: Uint8Array,
+ mw: number,
+ mh: number,
+): void {
+ for (let my = 0; my < mh; my++) {
+ for (let mx = 0; mx < mw; mx++) {
+ let sum = 0;
+ let n = 0;
+ for (let y = my * cell; y < Math.min(h, (my + 1) * cell); y++) {
+ for (let x = mx * cell; x < Math.min(w, (mx + 1) * cell); x++) {
+ sum += src[y * w + x];
+ n++;
+ }
+ }
+ dst[my * mw + mx] = n ? Math.round(sum / n) : 0;
+ }
+ }
+}
+
+/** Separable box blur to 0–1, so gradients reach a little past the edge. */
+function boxBlur(
+ src: Uint8Array,
+ w: number,
+ h: number,
+ r: number,
+): Float32Array {
+ const tmp = new Float32Array(w * h);
+ const dst = new Float32Array(w * h);
+ for (let y = 0; y < h; y++) {
+ for (let x = 0; x < w; x++) {
+ let s = 0;
+ let n = 0;
+ for (let k = -r; k <= r; k++) {
+ const xx = x + k;
+ if (xx >= 0 && xx < w) {
+ s += src[y * w + xx];
+ n++;
+ }
+ }
+ tmp[y * w + x] = s / n / 255;
+ }
+ }
+ for (let y = 0; y < h; y++) {
+ for (let x = 0; x < w; x++) {
+ let s = 0;
+ let n = 0;
+ for (let k = -r; k <= r; k++) {
+ const yy = y + k;
+ if (yy >= 0 && yy < h) {
+ s += tmp[yy * w + x];
+ n++;
+ }
+ }
+ dst[y * w + x] = s / n;
+ }
+ }
+ return dst;
+}
+
+/** @internal Test hook: a clip whose results are given instead of computed. */
+export function clipVisionFromResults(
+ fps: number,
+ size: { width: number; height: number; cell: number },
+ frames: Array<{ kp: number[] | null; mask: Uint8Array | null }>,
+ options: ClipVisionOptions = {},
+): ClipVision {
+ const clip = new ClipVision("memory", fps, {
+ ...options,
+ maskCell: size.cell,
+ cacheDir: false,
+ });
+ const c = clip as unknown as {
+ setSize(w: number, h: number): void;
+ results: Map<number, FrameResult>;
+ };
+ c.setSize(size.width, size.height);
+ frames.forEach((f, i) => {
+ c.results.set(i, {
+ kp: f.kp ? Float32Array.from(f.kp) : null,
+ mask: f.mask,
+ });
+ });
+ return clip;
+}
diff --git a/packages/framefields/src/index.ts b/packages/framefields/src/index.ts
index 2b870ab4..20acb7aa 100644
--- a/packages/framefields/src/index.ts
+++ b/packages/framefields/src/index.ts
@@ -188,6 +188,7 @@ import {
} from "@framefields/tensor-webgpu";
import { inputStore, mediaDecoderCache } from "@framefields/webgpu-renderers";
import { buildChart, type ChartOptions } from "./chart.js";
+import { ClipVision, type ClipVisionOptions } from "./clip-vision.js";
import {
ApplyLUT,
type ApplyLUTProps,
@@ -240,11 +241,6 @@ import {
Vision,
type VisionProps,
} from "./effects/index.js";
-import {
- type PersonTrack,
- type TrackPersonOptions,
- trackPerson,
-} from "./person-track.js";
import {
type FrameGridOptions,
HeadlessMediaRenderer,
@@ -669,6 +665,13 @@ export {
type ChartX,
DEFAULT_CHART_COLORS,
} from "./chart.js";
+export {
+ ClipVision,
+ type ClipVisionOptions,
+ type PersonPlacement,
+ PersonTrack,
+ type PoseKeypointName,
+} from "./clip-vision.js";
export {
ApplyLUT,
type ApplyLUTProps,
@@ -790,13 +793,6 @@ export {
type LightingRig,
type LightingRigOptions,
} from "./lighting.js";
-export {
- PersonTrack,
- type PersonTrackJSON,
- type PersonTrackPlacement,
- type PoseKeypointName,
- type TrackPersonOptions,
-} from "./person-track.js";
export {
type Body2DOptions,
type Body3DOptions,
@@ -1107,6 +1103,9 @@ export class Composition {
public audioTracks: MediaNode[] = [];
public signals: Record<string, unknown> = {};
private _frameHooks: Array<(ctx: FrameContext) => void> = [];
+ private _prepareHooks: Array<(ctx: FrameContext) => Promise<void> | void> =
+ [];
+ private readonly _clipVision = new Map<string, ClipVision>();
private _visionConfig?: VisionProps;
private _visionBundle?: VisionBundle;
private _effects: unknown[] = [];
@@ -1145,22 +1144,27 @@ export class Composition {
}
/**
- * Tracks one person through a clip before rendering: smoothed pose
- * keypoints and a soft silhouette per frame (see {@link PersonTrack}).
- * Use it to drive physics with a performer (limbs as kinematic bodies,
- * the silhouette as a force field) or to keyframe things to a body.
- * Analysis takes a few hundred ms per frame; cache it with `toJSON()` /
- * `PersonTrack.fromJSON()`.
+ * Vision for a clip, analysed per frame on demand and shared by everything
+ * that reads it (see {@link ClipVision}): `comp.vision("dancer.mp4").person`
+ * gives pose keypoints and a silhouette at any frame. Each frame is
+ * analysed just before it renders, so nothing runs up front. Results
+ * persist on disk in Node. The same clip and options return one instance.
*/
- public async trackPerson(
- source: string,
- options: TrackPersonOptions = {},
- ): Promise<PersonTrack> {
- const fps = options.fps ?? this.fps;
- const durationMs = this.durationMs ?? (await this.computeDuration());
- const totalFrames =
- options.totalFrames ?? Math.max(1, Math.round((durationMs / 1000) * fps));
- return trackPerson(source, { ...options, fps, totalFrames });
+ public vision(source: string, options: ClipVisionOptions = {}): ClipVision {
+ const key = `${source}\u0000${JSON.stringify(options)}`;
+ let clip = this._clipVision.get(key);
+ if (!clip) {
+ clip = new ClipVision(source, this.fps, options);
+ this._clipVision.set(key, clip);
+ }
+ return clip;
+ }
+
+ /** @internal Analyses every `comp.vision` clip for a composition frame. */
+ public async prepareVision(frame: number): Promise<void> {
+ await Promise.all(
+ [...this._clipVision.values()].map((clip) => clip.ready(frame)),
+ );
}
/**
@@ -1253,10 +1257,37 @@ export class Composition {
return this;
}
- public notifyFrame(ctx: FrameContext): void {
+ /**
+ * Async work a frame needs before it is evaluated (vision, a live physics
+ * world). Every render path awaits these, in order, before drawing the
+ * frame; they may run more than once per frame, so keep them idempotent.
+ */
+ public onPrepareFrame(
+ callback: (ctx: FrameContext) => Promise<void> | void,
+ ): this {
+ this._prepareHooks.push(callback);
+ return this;
+ }
+
+ public notifyFrame(ctx: FrameContext): Promise<void> | undefined {
for (const hook of this._frameHooks) {
hook(ctx);
}
+ if (this._clipVision.size === 0 && this._prepareHooks.length === 0) {
+ return undefined;
+ }
+ return (async () => {
+ await this.prepareVision(ctx.frame);
+ for (const hook of this._prepareHooks) await hook(ctx);
+ })();
+ }
+
+ private hasFrameHooks(): boolean {
+ return (
+ this._frameHooks.length > 0 ||
+ this._prepareHooks.length > 0 ||
+ this._clipVision.size > 0
+ );
}
constructor(options: CompositionOptions) {
@@ -1789,7 +1820,7 @@ export class Composition {
signals: this.signals,
...(this._effects.length > 0 && { effects: this._effects }),
...(this._visionConfig && { vision: this._visionConfig }),
- ...(this._frameHooks.length > 0 && {
+ ...(this.hasFrameHooks() && {
onRequestFrame: (ctx: FrameContext) => this.notifyFrame(ctx),
}),
} as unknown as CompositorProgramConfig;
@@ -1843,13 +1874,13 @@ export class Composition {
renderer?: HeadlessMediaRenderer;
} = {},
): Promise<Buffer> {
- if (this._frameHooks.length > 0) {
+ if (this.hasFrameHooks()) {
const targetFrame =
options.frame ??
(options.atMs !== undefined
? Math.round((options.atMs / 1000) * this.fps)
: 0);
- this.notifyFrame({
+ await this.notifyFrame({
frame: targetFrame,
fps: this.fps,
time: this.fps > 0 ? targetFrame / this.fps : 0,
diff --git a/packages/framefields/src/person-track.test.ts b/packages/framefields/src/person-track.test.ts
deleted file mode 100644
index 51abd185..00000000
--- a/packages/framefields/src/person-track.test.ts
+++ /dev/null
@@ -1,81 +0,0 @@
-import { describe, expect, it } from "vitest";
-import { PersonTrack, type PersonTrackJSON } from "./index.js";
-
-/** Two frames of a 160×80 clip: a wrist moving right, a 40×40 square silhouette moving right. */
-function synthetic(): PersonTrack {
- const frames = 2;
- const keypoints = new Array(frames * 17 * 3).fill(0);
- const set = (f: number, k: number, x: number, y: number) =>
- keypoints.splice((f * 17 + k) * 3, 3, x, y, 1);
- set(0, 9, 20, 10); // leftWrist
- set(1, 9, 60, 30);
- const cell = 4;
- const w = 40;
- const h = 20;
- const mask = new Uint8Array(frames * w * h);
- for (let f = 0; f < frames; f++) {
- for (let y = 5; y < 15; y++)
- for (let x = 5 + f * 10; x < 15 + f * 10; x++)
- mask[f * w * h + y * w + x] = 255;
- }
- const json: PersonTrackJSON = {
- version: 1,
- fps: 24,
- frames,
- sourceWidth: 160,
- sourceHeight: 80,
- keypoints,
- mask: {
- cell,
- width: w,
- height: h,
- data: Buffer.from(mask).toString("base64"),
- },
- };
- return PersonTrack.fromJSON(JSON.parse(JSON.stringify(json)));
-}
-
-describe("PersonTrack", () => {
- it("interpolates keypoints and maps them into a layer box", () => {
- const t = synthetic();
- expect(t.point("leftWrist", 0.5)).toMatchObject({
- x: 40,
- y: 20,
- visibility: 1,
- });
- // 160×80 covering 320×320 at (100, 0): scale 4, centered → x offset 100 − 160.
- const placed = t.place({
- x: 100,
- y: 0,
- width: 320,
- height: 320,
- fit: "cover",
- });
- expect(placed.point("leftWrist", 0)).toMatchObject({
- x: 100 - 160 + 80,
- y: 40,
- });
- const sig = placed.signals("leftWrist");
- expect(sig.x.get({ frame: 1, fps: 24 } as never)).toBeCloseTo(
- 100 - 160 + 240,
- );
- });
-
- it("samples the silhouette with a gradient pointing inward", () => {
- const t = synthetic();
- // Frame 0 square spans source px 20–60 × 20–60.
- expect(t.sample(0, 40, 40).alpha).toBeGreaterThan(0.9);
- expect(t.sample(0, 120, 40).alpha).toBe(0);
- const edge = t.sample(0, 62, 40);
- expect(edge.alpha).toBeGreaterThan(0);
- expect(edge.gx).toBeLessThan(0); // inside is to the left
- expect(t.sample(1, 40, 40).alpha).toBeLessThan(t.sample(1, 80, 40).alpha);
- });
-
- it("round-trips through JSON", () => {
- const t = synthetic();
- const again = PersonTrack.fromJSON(t.toJSON());
- expect(again.point("leftWrist", 1)).toEqual(t.point("leftWrist", 1));
- expect(again.sample(0, 40, 40)).toEqual(t.sample(0, 40, 40));
- });
-});
diff --git a/packages/framefields/src/person-track.ts b/packages/framefields/src/person-track.ts
deleted file mode 100644
index 8001a196..00000000
--- a/packages/framefields/src/person-track.ts
+++ /dev/null
@@ -1,511 +0,0 @@
-import { type FrameSignal, Signal } from "@framefields/core";
-import { COCO17_KEYPOINT_NAMES, VisionRunner } from "@framefields/vision";
-import { mediaDecoderCache } from "@framefields/webgpu-renderers";
-import type { VisionProps } from "./effects/index.js";
-
-export type PoseKeypointName = (typeof COCO17_KEYPOINT_NAMES)[number];
-
-export interface TrackPersonOptions {
- /** Frames per second to sample (default: the composition's). */
- fps?: number;
- /** How many frames (default: the composition's length). */
- totalFrames?: number;
- /** Track 17 COCO keypoints (default true). */
- pose?: boolean;
- /** Track the person's silhouette, dress and held things included (default true). */
- mask?: boolean;
- /** Silhouette cell size in source pixels (default 8: a 1920×1080 clip gives a 240×135 grid). */
- maskCell?: number;
- /** Frames of zero-lag smoothing on keypoints (default 2; 0 for raw). */
- smoothing?: number;
- /** Keypoints below this visibility count as missing and are filled from neighbours (default 0.3). */
- minVisibility?: number;
- variant?: VisionProps["variant"];
- confidence?: number;
- modelsDir?: string;
- baseUrl?: string;
- onProgress?: (frame: number, total: number) => void;
-}
-
-/** Where the clip sits in the composition, like a media layer's box and `fit`. */
-export interface PersonTrackPlacement {
- x?: number;
- y?: number;
- width: number;
- height: number;
- fit?: "cover" | "contain" | "fill";
-}
-
-/** Plain-JSON form, for caching an analysis between runs. */
-export interface PersonTrackJSON {
- version: 1;
- fps: number;
- frames: number;
- sourceWidth: number;
- sourceHeight: number;
- /** frames × 17 × [x, y, visibility], source px, smoothed. */
- keypoints?: number[];
- mask?: { cell: number; width: number; height: number; data: string };
-}
-
-interface Transform {
- sx: number;
- sy: number;
- ox: number;
- oy: number;
-}
-
-const IDENTITY: Transform = { sx: 1, sy: 1, ox: 0, oy: 0 };
-
-/**
- * One person tracked through a clip ahead of rendering: smoothed pose
- * keypoints and a soft silhouette, frame by frame. Unlike the render-time
- * vision signals it is complete before the first frame renders, so it can
- * drive physics (limbs as kinematic bodies, the silhouette as a force field),
- * be keyframed, or be inspected. `place()` maps it into composition pixels.
- */
-export class PersonTrack {
- readonly fps: number;
- readonly frames: number;
- readonly sourceWidth: number;
- readonly sourceHeight: number;
- private readonly kp?: Float32Array;
- private readonly maskData?: {
- cell: number;
- width: number;
- height: number;
- data: Uint8Array;
- };
- private blurred?: Float32Array[];
-
- private constructor(
- json: Omit<PersonTrackJSON, "keypoints" | "mask" | "version">,
- kp: Float32Array | undefined,
- mask: PersonTrack["maskData"],
- private readonly t: Transform = IDENTITY,
- shared?: PersonTrack,
- ) {
- this.fps = json.fps;
- this.frames = json.frames;
- this.sourceWidth = json.sourceWidth;
- this.sourceHeight = json.sourceHeight;
- this.kp = kp;
- this.maskData = mask;
- if (shared) this.blurred = shared.blurredMasks();
- }
-
- static fromJSON(json: PersonTrackJSON): PersonTrack {
- const mask = json.mask
- ? {
- cell: json.mask.cell,
- width: json.mask.width,
- height: json.mask.height,
- data: new Uint8Array(Buffer.from(json.mask.data, "base64")),
- }
- : undefined;
- return new PersonTrack(
- json,
- json.keypoints ? Float32Array.from(json.keypoints) : undefined,
- mask,
- );
- }
-
- toJSON(): PersonTrackJSON {
- const m = this.maskData;
- return {
- version: 1,
- fps: this.fps,
- frames: this.frames,
- sourceWidth: this.sourceWidth,
- sourceHeight: this.sourceHeight,
- ...(this.kp && {
- keypoints: Array.from(this.kp, (v) => Math.round(v * 100) / 100),
- }),
- ...(m && {
- mask: {
- cell: m.cell,
- width: m.width,
- height: m.height,
- data: Buffer.from(m.data).toString("base64"),
- },
- }),
- };
- }
-
- /** The same track in composition pixels for a clip laid out at `placement`. */
- place(placement: PersonTrackPlacement): PersonTrack {
- const { width: W, height: H } = placement;
- const sw = this.sourceWidth;
- const sh = this.sourceHeight;
- let sx = W / sw;
- let sy = H / sh;
- if (placement.fit !== "fill") {
- const s =
- placement.fit === "contain" ? Math.min(sx, sy) : Math.max(sx, sy);
- sx = s;
- sy = s;
- }
- const t: Transform = {
- sx,
- sy,
- ox: (placement.x ?? 0) + (W - sw * sx) / 2,
- oy: (placement.y ?? 0) + (H - sh * sy) / 2,
- };
- return new PersonTrack(this, this.kp, this.maskData, t, this);
- }
-
- /** Whether the clip had pose / silhouette tracking. */
- get hasPose(): boolean {
- return this.kp !== undefined;
- }
- get hasMask(): boolean {
- return this.maskData !== undefined;
- }
-
- /** A keypoint at a frame (fractional frames interpolate), placed px. */
- point(
- name: PoseKeypointName,
- frame: number,
- ): { x: number; y: number; visibility: number } {
- const kp = this.requirePose();
- const k = keypointIndex(name);
- const f = Math.max(0, Math.min(this.frames - 1, frame));
- const f0 = Math.floor(f);
- const f1 = Math.min(this.frames - 1, f0 + 1);
- const u = f - f0;
- const at = (fi: number, c: number) => kp[(fi * 17 + k) * 3 + c];
- const lerp = (c: number) => at(f0, c) + (at(f1, c) - at(f0, c)) * u;
- return {
- x: lerp(0) * this.t.sx + this.t.ox,
- y: lerp(1) * this.t.sy + this.t.oy,
- visibility: lerp(2),
- };
- }
-
- /** Per-frame signals for a keypoint (placed px), for `.signal("x", …)` or physics `follow`. */
- signals(
- name: PoseKeypointName,
- options: { startFrame?: number } = {},
- ): {
- x: FrameSignal<number>;
- y: FrameSignal<number>;
- visibility: FrameSignal<number>;
- } {
- const start = options.startFrame ?? 0;
- const x = new Float32Array(start + this.frames);
- const y = new Float32Array(start + this.frames);
- const v = new Float32Array(start + this.frames);
- for (let f = 0; f < start + this.frames; f++) {
- const p = this.point(name, f - start);
- x[f] = p.x;
- y[f] = p.y;
- v[f] = p.visibility;
- }
- return {
- x: Signal.fromArray(x, this.fps),
- y: Signal.fromArray(y, this.fps),
- visibility: Signal.fromArray(v, this.fps),
- };
- }
-
- /**
- * The silhouette near a point (placed px) at a frame: `alpha` 0–1 (softened
- * over a few cells), and its gradient `gx`, `gy` per px, pointing into the
- * body. Push things out with `-gradient`; `alpha > 0.5` is inside.
- */
- sample(
- frame: number,
- x: number,
- y: number,
- ): { alpha: number; gx: number; gy: number } {
- const m = this.requireMask();
- const grids = this.blurredMasks();
- const fi = Math.max(0, Math.min(this.frames - 1, Math.round(frame)));
- const grid = grids[fi];
- // Placed px → mask cells.
- const cx = (x - this.t.ox) / this.t.sx / m.cell - 0.5;
- const cy = (y - this.t.oy) / this.t.sy / m.cell - 0.5;
- const read = (gx: number, gy: number) => {
- if (gx < 0 || gy < 0 || gx >= m.width || gy >= m.height) return 0;
- return grid[gy * m.width + gx];
- };
- const bil = (px: number, py: number) => {
- const x0 = Math.floor(px);
- const y0 = Math.floor(py);
- const ux = px - x0;
- const uy = py - y0;
- const a = read(x0, y0) + (read(x0 + 1, y0) - read(x0, y0)) * ux;
- const b =
- read(x0, y0 + 1) + (read(x0 + 1, y0 + 1) - read(x0, y0 + 1)) * ux;
- return a + (b - a) * uy;
- };
- const alpha = bil(cx, cy);
- const pxPerCellX = m.cell * this.t.sx;
- const pxPerCellY = m.cell * this.t.sy;
- return {
- alpha,
- gx: (bil(cx + 1, cy) - bil(cx - 1, cy)) / (2 * pxPerCellX),
- gy: (bil(cx, cy + 1) - bil(cx, cy - 1)) / (2 * pxPerCellY),
- };
- }
-
- private requirePose(): Float32Array {
- if (!this.kp) throw new Error("PersonTrack: tracked without `pose`.");
- return this.kp;
- }
-
- private requireMask(): NonNullable<PersonTrack["maskData"]> {
- if (!this.maskData) throw new Error("PersonTrack: tracked without `mask`.");
- return this.maskData;
- }
-
- /** Silhouette per frame, softened by a small box blur so gradients reach past the edge. */
- private blurredMasks(): Float32Array[] {
- if (this.blurred) return this.blurred;
- const m = this.requireMask();
- const { width: w, height: h } = m;
- const r = 2;
- const out: Float32Array[] = [];
- const tmp = new Float32Array(w * h);
- for (let f = 0; f < this.frames; f++) {
- const src = m.data.subarray(f * w * h, (f + 1) * w * h);
- const dst = new Float32Array(w * h);
- for (let y = 0; y < h; y++) {
- for (let x = 0; x < w; x++) {
- let s = 0;
- let n = 0;
- for (let k = -r; k <= r; k++) {
- const xx = x + k;
- if (xx >= 0 && xx < w) {
- s += src[y * w + xx];
- n++;
- }
- }
- tmp[y * w + x] = s / n / 255;
- }
- }
- for (let y = 0; y < h; y++) {
- for (let x = 0; x < w; x++) {
- let s = 0;
- let n = 0;
- for (let k = -r; k <= r; k++) {
- const yy = y + k;
- if (yy >= 0 && yy < h) {
- s += tmp[yy * w + x];
- n++;
- }
- }
- dst[y * w + x] = s / n;
- }
- }
- out.push(dst);
- }
- this.blurred = out;
- return out;
- }
-
- /** @internal */
- static fromAnalysis(
- meta: {
- fps: number;
- frames: number;
- sourceWidth: number;
- sourceHeight: number;
- },
- kp: Float32Array | undefined,
- mask: PersonTrack["maskData"],
- ): PersonTrack {
- return new PersonTrack(meta, kp, mask);
- }
-}
-
-function keypointIndex(name: PoseKeypointName): number {
- const i = COCO17_KEYPOINT_NAMES.indexOf(name);
- if (i < 0) {
- throw new Error(
- `PersonTrack: unknown keypoint "${name}". Use one of ${COCO17_KEYPOINT_NAMES.join(", ")}.`,
- );
- }
- return i;
-}
-
-/** Runs pose and segmentation over a clip and returns a {@link PersonTrack}. */
-export async function trackPerson(
- source: string,
- options: TrackPersonOptions & { fps: number; totalFrames: number },
-): Promise<PersonTrack> {
- const { fps, totalFrames } = options;
- const wantPose = options.pose ?? true;
- const wantMask = options.mask ?? true;
- const cell = Math.max(1, Math.round(options.maskCell ?? 8));
- const minVis = options.minVisibility ?? 0.3;
- const runner = VisionRunner.create({
- modelsDir: options.modelsDir,
- baseUrl: options.baseUrl,
- ...(options.confidence !== undefined && { confidence: options.confidence }),
- ...(options.variant !== undefined && { variant: options.variant }),
- });
- const decoder = mediaDecoderCache.getVideo(source, true);
-
- let sourceWidth = 0;
- let sourceHeight = 0;
- const kp = wantPose ? new Float32Array(totalFrames * 17 * 3) : undefined;
- let maskW = 0;
- let maskH = 0;
- let maskData: Uint8Array | undefined;
-
- try {
- for (let i = 0; i < totalFrames; i++) {
- const decoded = await decoder.getFrame(i / fps);
- if (!decoded?.buffer || decoded.width <= 0) continue;
- const frame = {
- data: decoded.buffer,
- width: decoded.width,
- height: decoded.height,
- };
- if (!sourceWidth) {
- sourceWidth = frame.width;
- sourceHeight = frame.height;
- maskW = Math.ceil(sourceWidth / cell);
- maskH = Math.ceil(sourceHeight / cell);
- if (wantMask) maskData = new Uint8Array(totalFrames * maskW * maskH);
- }
- if (kp) {
- const { people } = await runner.pose(frame);
- const person = people[0];
- if (person) {
- person.keypoints.forEach((p, k) => {
- kp.set([p.x, p.y, p.visibility], (i * 17 + k) * 3);
- });
- }
- }
- if (maskData) {
- const seg = await runner.segment(frame);
- const union = personSilhouette(seg);
- if (union)
- downsampleMean(
- union,
- frame.width,
- frame.height,
- cell,
- maskData,
- i * maskW * maskH,
- maskW,
- maskH,
- );
- }
- options.onProgress?.(i + 1, totalFrames);
- }
- } finally {
- runner.close();
- decoder.destroy();
- }
- if (!sourceWidth)
- throw new Error(`trackPerson: could not decode any frame of "${source}".`);
- if (kp) smoothKeypoints(kp, totalFrames, minVis, options.smoothing ?? 2);
-
- return PersonTrack.fromAnalysis(
- { fps, frames: totalFrames, sourceWidth, sourceHeight },
- kp,
- maskData && { cell, width: maskW, height: maskH, data: maskData },
- );
-}
-
-type Segmentation = Awaited<ReturnType<VisionRunner["segment"]>>;
-
-/**
- * The largest person's mask, plus instances that overlap it and are no more
- * than twice its size (a flowing dress often comes back as "kite").
- */
-function personSilhouette(seg: Segmentation): Uint8Array | undefined {
- const people = seg.masks.filter((m) => m.category === "person");
- if (people.length === 0) return undefined;
- const person = people.reduce((a, b) => (b.area > a.area ? b : a));
- const box = seg.detections[person.detectionIndex]?.boundingBox;
- const out = new Uint8Array(person.mask);
- if (!box) return out;
- for (const m of seg.masks) {
- if (m === person || m.area > person.area * 2) continue;
- const b = seg.detections[m.detectionIndex]?.boundingBox;
- if (!b) continue;
- const ix = Math.max(
- 0,
- Math.min(box.originX + box.width, b.originX + b.width) -
- Math.max(box.originX, b.originX),
- );
- const iy = Math.max(
- 0,
- Math.min(box.originY + box.height, b.originY + b.height) -
- Math.max(box.originY, b.originY),
- );
- if (ix * iy < 0.3 * b.width * b.height) continue;
- for (let i = 0; i < out.length; i++)
- if (m.mask[i] > out[i]) out[i] = m.mask[i];
- }
- return out;
-}
-
-function downsampleMean(
- src: Uint8Array,
- w: number,
- h: number,
- cell: number,
- dst: Uint8Array,
- offset: number,
- mw: number,
- mh: number,
-): void {
- for (let my = 0; my < mh; my++) {
- for (let mx = 0; mx < mw; mx++) {
- let sum = 0;
- let n = 0;
- for (let y = my * cell; y < Math.min(h, (my + 1) * cell); y++) {
- for (let x = mx * cell; x < Math.min(w, (mx + 1) * cell); x++) {
- sum += src[y * w + x];
- n++;
- }
- }
- dst[offset + my * mw + mx] = n ? Math.round(sum / n) : 0;
- }
- }
-}
-
-/**
- * Fills missing keypoints from their nearest seen frames, then runs a
- * forward and a backward exponential filter (no lag, no overshoot).
- */
-function smoothKeypoints(
- kp: Float32Array,
- frames: number,
- minVis: number,
- smoothing: number,
-): void {
- for (let k = 0; k < 17; k++) {
- const idx = (f: number, c: number) => (f * 17 + k) * 3 + c;
- const seen: number[] = [];
- for (let f = 0; f < frames; f++) if (kp[idx(f, 2)] >= minVis) seen.push(f);
- if (seen.length === 0) continue;
- let s = 0;
- for (let f = 0; f < frames; f++) {
- if (kp[idx(f, 2)] >= minVis) continue;
- while (s + 1 < seen.length && seen[s + 1] < f) s++;
- const a = seen[s];
- const b = seen.find((v) => v > f);
- for (const c of [0, 1]) {
- if (f < a || b === undefined) kp[idx(f, c)] = kp[idx(a, c)];
- else {
- const u = (f - a) / (b - a);
- kp[idx(f, c)] = kp[idx(a, c)] + (kp[idx(b, c)] - kp[idx(a, c)]) * u;
- }
- }
- }
- if (smoothing <= 0) continue;
- const alpha = 1 / (1 + smoothing);
- for (const c of [0, 1]) {
- for (let f = 1; f < frames; f++)
- kp[idx(f, c)] += (kp[idx(f - 1, c)] - kp[idx(f, c)]) * (1 - alpha);
- for (let f = frames - 2; f >= 0; f--)
- kp[idx(f, c)] += (kp[idx(f + 1, c)] - kp[idx(f, c)]) * (1 - alpha);
- }
- }
-}
diff --git a/packages/framefields/src/physics/backend.ts b/packages/framefields/src/physics/backend.ts
index 4b99c277..6f684e00 100644
--- a/packages/framefields/src/physics/backend.ts
+++ b/packages/framefields/src/physics/backend.ts
@@ -116,14 +116,18 @@ export interface PhysicsBackend {
type Rapier2D = typeof RAPIER2D;
type Rapier3D = typeof RAPIER3D;
-// Variable specifiers keep bundlers from inlining the optional WASM packages.
+// Literal specifiers: the preview's browser bundle splits each engine into a
+// chunk loaded on first use; Node keeps them as external optional packages.
const RAPIER_2D = "@dimforge/rapier2d-deterministic-compat";
const RAPIER_3D = "@dimforge/rapier3d-deterministic-compat";
-async function loadRapier<T>(specifier: string): Promise<T> {
+async function loadRapier<T>(
+ specifier: string,
+ load: () => Promise<unknown>,
+): Promise<T> {
let mod: { default?: T } & T;
try {
- mod = (await import(/* @vite-ignore */ specifier)) as { default?: T } & T;
+ mod = (await load()) as { default?: T } & T;
} catch (error) {
throw new Error(
`Physics needs the optional dependency "${specifier}". Install it with \`npm i ${specifier}\`.`,
@@ -141,7 +145,10 @@ let rapier3d: Promise<Rapier3D> | undefined;
/** Loads and initializes Rapier once per process; 2D and 3D are separate modules. */
export async function rapierBackend(dims: 2 | 3): Promise<PhysicsBackend> {
if (dims === 2) {
- rapier2d ??= loadRapier<Rapier2D>(RAPIER_2D);
+ rapier2d ??= loadRapier<Rapier2D>(
+ RAPIER_2D,
+ () => import("@dimforge/rapier2d-deterministic-compat"),
+ );
const R = await rapier2d;
return {
createWorld2D: (gravity) => new RapierWorld2D(R, gravity),
@@ -150,7 +157,10 @@ export async function rapierBackend(dims: 2 | 3): Promise<PhysicsBackend> {
},
};
}
- rapier3d ??= loadRapier<Rapier3D>(RAPIER_3D);
+ rapier3d ??= loadRapier<Rapier3D>(
+ RAPIER_3D,
+ () => import("@dimforge/rapier3d-deterministic-compat"),
+ );
const R = await rapier3d;
return {
createWorld2D: () => {
diff --git a/packages/framefields/src/physics/core.ts b/packages/framefields/src/physics/core.ts
index 10a478bd..08622852 100644
--- a/packages/framefields/src/physics/core.ts
+++ b/packages/framefields/src/physics/core.ts
@@ -67,6 +67,24 @@ export interface BodyCommonOptions {
output?: "signals" | "keyframes" | "none";
}
+/** What `world.live()` needs from a composition. */
+export interface LiveHost {
+ onPrepareFrame(
+ callback: (ctx: { frame: number }) => Promise<void> | void,
+ ): unknown;
+ prepareVision?(frame: number): Promise<void>;
+}
+
+interface Stepper<B, V, R> {
+ /** Last simulated frame, relative to the world's start. */
+ current: number;
+ freed: boolean;
+ bySim: Map<SimBody<V, R>, B>;
+ kinematicFrom: Map<B, { position: V; rotation: R }>;
+ /** Impact envelope level per body after `current`. */
+ level: Map<B, number>;
+}
+
export interface PhysicsCollision<B> {
/** Composition frame of first contact. */
frame: number;
@@ -248,7 +266,11 @@ export abstract class WorldCore<
}> = [];
private readonly worldRecording: Recording;
private readonly allCollisions: Array<PhysicsCollision<B> & { body: B }> = [];
- private simulated = false;
+ /** No more bodies, joints or forces: the world has started (or finished). */
+ private locked = false;
+ private liveMode = false;
+ private stepper?: Stepper<B, V, R>;
+ private chain: Promise<void> = Promise.resolve();
protected constructor(
protected readonly sim: SimWorld<V, SimBody<V, R>, unknown>,
@@ -269,7 +291,7 @@ export abstract class WorldCore<
this.worldRecording = this.newRecording();
}
- /** Every first contact in the simulation, in frame order. */
+ /** First contacts so far (all of them once simulated), in frame order. */
get collisions(): ReadonlyArray<PhysicsCollision<B> & { body: B }> {
this.ensureSimulated();
return this.allCollisions;
@@ -280,17 +302,21 @@ export abstract class WorldCore<
return this.worldRecording.signal("impact");
}
- /** Runs the simulation (once) and binds every body's layer per its `output`. */
+ /** Runs the whole simulation now (once) and binds every body's layer per its `output`. */
simulate(): this {
+ if (this.liveMode)
+ throw new Error(
+ "Physics: this world is live; it steps as frames render.",
+ );
this.ensureSimulated();
return this;
}
/** Like {@link simulate}, but writes keyframes for every body whose `output` was left at its default. */
bakeKeyframes(options: PhysicsBakeOptions = {}): this {
- if (this.simulated) {
+ if (this.locked) {
throw new Error(
- "Physics: bakeKeyframes() must come before simulate() or reading any body signal.",
+ "Physics: bakeKeyframes() must come before simulate(), live() or reading any body signal.",
);
}
this.bakeDefault = options;
@@ -298,6 +324,67 @@ export abstract class WorldCore<
return this;
}
+ /**
+ * Steps the world as the composition renders instead of all at once: before
+ * each frame draws, the world advances to it, after the composition's
+ * vision clips (`comp.vision`) and any `inputs` have that frame ready. Use
+ * it when bodies follow a performer, so nothing is analysed up front and a
+ * preview starts at once. Frames already simulated are kept, so scrubbing
+ * back costs nothing; jumping ahead simulates the frames in between.
+ * Signals only; `bakeKeyframes()` needs the whole run up front.
+ */
+ live(
+ comp: LiveHost,
+ options: {
+ inputs?: ReadonlyArray<{ ready(frame: number): Promise<void> }>;
+ } = {},
+ ): this {
+ if (this.locked) {
+ throw new Error(
+ "Physics: live() must come before simulate() or reading any body signal.",
+ );
+ }
+ this.locked = true;
+ this.liveMode = true;
+ const prepare = async (frame: number) => {
+ await comp.prepareVision?.(frame);
+ for (const input of options.inputs ?? []) await input.ready(frame);
+ };
+ this.prepareInputs = prepare;
+ for (const body of this.bodies) this.bind(body);
+ comp.onPrepareFrame((ctx) => this.advanceTo(ctx.frame));
+ return this;
+ }
+
+ /**
+ * Live worlds: simulates up to composition frame `frame` (no-op when it
+ * already has). Calls queue, so frames are stepped once, in order.
+ */
+ advanceTo(frame: number): Promise<void> {
+ this.chain = this.chain.then(async () => {
+ const target = Math.min(
+ this.frames - 1,
+ Math.round(frame) - this.startFrame,
+ );
+ if (!this.stepper) {
+ await this.prepareInputs?.(this.startFrame);
+ this.stepper = this.begin();
+ }
+ const st = this.stepper;
+ while (st.current < target) {
+ const next = st.current + 1;
+ await this.prepareInputs?.(this.startFrame + next);
+ this.stepFrame(st, next);
+ }
+ if (st.current === this.frames - 1 && !st.freed) {
+ st.freed = true;
+ this.sim.free();
+ }
+ });
+ return this.chain;
+ }
+
+ private prepareInputs?: (frame: number) => Promise<void>;
private bakeDefault?: PhysicsBakeOptions;
protected newRecording(): Recording {
@@ -307,9 +394,9 @@ export abstract class WorldCore<
}
protected assertOpen(): void {
- if (this.simulated) {
+ if (this.locked) {
throw new Error(
- "Physics: the world has already simulated; add every body, joint and force before simulate() or reading signals.",
+ "Physics: the world has already started; add every body, joint and force before simulate(), live() or reading signals.",
);
}
}
@@ -346,171 +433,176 @@ export abstract class WorldCore<
}
protected ensureSimulated(): void {
- if (this.simulated) return;
- this.simulated = true;
- this.run();
+ if (this.locked) return;
+ this.locked = true;
+ const st = this.begin();
+ for (let f = 1; f < this.frames; f++) this.stepFrame(st, f);
+ st.freed = true;
this.sim.free();
for (const body of this.bodies) this.bind(body);
}
- private run(): void {
- const { fps, frames, startFrame, substeps } = this;
- const dt = 1 / (fps * substeps);
+ /** Records frame 0 and sets every channel to it, so frames not simulated yet hold the start pose. */
+ private begin(): Stepper<B, V, R> {
+ const { startFrame } = this;
const bodies = this.bodies;
- const n = startFrame + frames;
- const rawImpact = new Map<B, Float32Array>();
- for (const b of bodies) rawImpact.set(b, new Float32Array(n));
const bySim = new Map<SimBody<V, R>, B>();
for (const b of bodies) bySim.set(b.sim, b);
-
for (const b of bodies) if (b.startFrame > 0) b.sim.setEnabled(false);
- for (const b of bodies) b.record(startFrame, -1);
+ const kinematicFrom = new Map<B, { position: V; rotation: R }>();
+ for (const b of bodies) {
+ b.record(startFrame, -1);
+ b.recording.channel("impact");
+ const t = b.kinematicTarget?.(startFrame);
+ if (t) kinematicFrom.set(b, t);
+ }
+ this.worldRecording.channel("impact");
+ for (const rec of [
+ ...bodies.map((b) => b.recording),
+ this.worldRecording,
+ ]) {
+ for (const arr of rec.channels.values()) arr.fill(arr[startFrame]);
+ }
+ return {
+ current: 0,
+ freed: false,
+ bySim,
+ kinematicFrom,
+ level: new Map(bodies.map((b) => [b, 0])),
+ };
+ }
+ private stepFrame(st: Stepper<B, V, R>, f: number): void {
+ const { fps, startFrame, substeps } = this;
+ const dt = 1 / (fps * substeps);
+ const bodies = this.bodies;
+ const frame = startFrame + f;
const dims = this.gravity.length;
const prevVel = new Map<B, number[]>();
- let kinematicFrom = new Map<B, { position: V; rotation: R }>();
for (const b of bodies) {
- const t = b.kinematicTarget?.(startFrame);
- if (t) kinematicFrom.set(b, t);
+ // A body sits at its start pose on its start frame and moves after.
+ if (b.startFrame > 0 && f === b.startFrame + 1) b.sim.setEnabled(true);
+ prevVel.set(b, b.sim.velocity() as number[]);
+ }
+ const isActive = (b: B) => b.type === "dynamic" && f > b.startFrame;
+ // Velocity each body is expected to gain this frame from gravity,
+ // forces, fields and impulses: anything beyond it came from a contact.
+ const expected = new Map<B, number[]>();
+ for (const b of bodies) {
+ expected.set(
+ b,
+ this.gravity.map((g) => (isActive(b) ? (g * b.gravityScale) / fps : 0)),
+ );
+ }
+ for (const imp of this.impulses) {
+ if (imp.frame !== f) continue;
+ const body = imp.body as unknown as B;
+ if (!isActive(body)) continue;
+ const m = body.sim.mass();
+ body.sim.applyImpulse(imp.dv.map((v) => v * m) as V);
+ const e = expected.get(body) as number[];
+ for (let i = 0; i < dims; i++) e[i] += imp.dv[i];
+ }
+ const frameForces = this.forces
+ .filter((fe) => isActive(fe.body as unknown as B))
+ .map((fe) => {
+ const body = fe.body as unknown as B;
+ const a = fe.accel(frame);
+ const e = expected.get(body) as number[];
+ for (let i = 0; i < dims; i++) e[i] += a[i] / fps;
+ return { body, a };
+ });
+ const kinematicTo = new Map<B, { position: V; rotation: R }>();
+ for (const b of bodies) {
+ const t = b.kinematicTarget?.(frame);
+ if (t) kinematicTo.set(b, t);
}
- for (let f = 1; f < frames; f++) {
- const frame = startFrame + f;
- for (const b of bodies) {
- // A body sits at its start pose on its start frame and moves after.
- if (b.startFrame > 0 && f === b.startFrame + 1) b.sim.setEnabled(true);
- prevVel.set(b, b.sim.velocity() as number[]);
+ const started = new Set<string>();
+ const pairs: Array<[B, B]> = [];
+ for (let s = 1; s <= substeps; s++) {
+ for (const { body, a } of frameForces) {
+ const m = body.sim.mass();
+ body.sim.applyImpulse(a.map((v) => v * m * dt) as V);
}
- // Velocity each body is expected to gain this frame from gravity,
- // forces and impulses: anything beyond it came from a contact.
- const expected = new Map<B, number[]>();
- for (const b of bodies) {
- const active = b.type === "dynamic" && f > b.startFrame;
- expected.set(
- b,
- this.gravity.map((g) => (active ? (g * b.gravityScale) / fps : 0)),
- );
+ for (const field of this.fields) {
+ for (const b of bodies) {
+ if (!isActive(b) || (field.targets && !field.targets.has(b)))
+ continue;
+ const a = field.accel(b, frame);
+ if (!a) continue;
+ const m = b.sim.mass();
+ b.sim.applyImpulse(a.map((v) => v * m * dt) as V);
+ const e = expected.get(b) as number[];
+ for (let i = 0; i < dims; i++) e[i] += a[i] * dt;
+ }
}
- const isActive = (b: B) => b.type === "dynamic" && f > b.startFrame;
- for (const imp of this.impulses) {
- if (imp.frame !== f) continue;
- const body = imp.body as unknown as B;
- if (!isActive(body)) continue;
- const m = body.sim.mass();
- body.sim.applyImpulse(imp.dv.map((v) => v * m) as V);
- const e = expected.get(body) as number[];
- for (let i = 0; i < dims; i++) e[i] += imp.dv[i];
+ for (const [b, to] of kinematicTo) {
+ const from = st.kinematicFrom.get(b) ?? to;
+ const k = s / substeps;
+ b.sim.setKinematicTarget(
+ lerpVec(from.position, to.position, k),
+ this.lerpRotation(from.rotation, to.rotation, k),
+ );
}
- const frameForces = this.forces
- .filter((fe) => isActive(fe.body as unknown as B))
- .map((fe) => {
- const body = fe.body as unknown as B;
- const a = fe.accel(frame);
- const e = expected.get(body) as number[];
- for (let i = 0; i < dims; i++) e[i] += a[i] / fps;
- return { body, a };
- });
- const kinematicTo = new Map<B, { position: V; rotation: R }>();
- for (const b of bodies) {
- const t = b.kinematicTarget?.(frame);
- if (t) kinematicTo.set(b, t);
+ this.sim.step(dt);
+ for (const [sa, sb] of this.sim.drainCollisionStarts()) {
+ const a = st.bySim.get(sa);
+ const b = st.bySim.get(sb);
+ if (!a || !b) continue;
+ const key = `${bodies.indexOf(a)}:${bodies.indexOf(b)}`;
+ if (started.has(key)) continue;
+ started.add(key);
+ pairs.push([a, b]);
}
-
- const started = new Set<string>();
- const pairs: Array<[B, B]> = [];
- for (let s = 1; s <= substeps; s++) {
- for (const { body, a } of frameForces) {
- const m = body.sim.mass();
- body.sim.applyImpulse(a.map((v) => v * m * dt) as V);
- }
- for (const field of this.fields) {
- for (const b of bodies) {
- if (!isActive(b) || (field.targets && !field.targets.has(b)))
- continue;
- const a = field.accel(b, frame);
- if (!a) continue;
- const m = b.sim.mass();
- b.sim.applyImpulse(a.map((v) => v * m * dt) as V);
- const e = expected.get(b) as number[];
- for (let i = 0; i < dims; i++) e[i] += a[i] * dt;
- }
- }
- for (const [b, to] of kinematicTo) {
- const from = kinematicFrom.get(b) ?? to;
- const k = s / substeps;
- b.sim.setKinematicTarget(
- lerpVec(from.position, to.position, k),
- this.lerpRotation(from.rotation, to.rotation, k),
- );
+ }
+ st.kinematicFrom = kinematicTo;
+
+ for (const b of bodies) b.record(frame, frame - 1);
+
+ const raw = new Map<B, number>();
+ if (pairs.length > 0) {
+ const ppm = this.pixelsPerMeter;
+ const hit = (b: B): number => {
+ if (b.type !== "dynamic") return 0;
+ const v = b.sim.velocity() as number[];
+ const v0 = prevVel.get(b) as number[];
+ const e = expected.get(b) as number[];
+ let sq = 0;
+ for (let i = 0; i < dims; i++) {
+ const d = v[i] - (v0[i] + e[i]);
+ sq += d * d;
}
- this.sim.step(dt);
- for (const [sa, sb] of this.sim.drainCollisionStarts()) {
- const a = bySim.get(sa);
- const b = bySim.get(sb);
- if (!a || !b) continue;
- const key = `${bodies.indexOf(a)}:${bodies.indexOf(b)}`;
- if (started.has(key)) continue;
- started.add(key);
- pairs.push([a, b]);
- }
- }
- kinematicFrom = kinematicTo;
-
- for (const b of bodies) b.record(frame, frame - 1);
-
- if (pairs.length > 0) {
- const ppm = this.pixelsPerMeter;
- const hit = (b: B): number => {
- if (b.type !== "dynamic") return 0;
- const v = b.sim.velocity() as number[];
- const v0 = prevVel.get(b) as number[];
- const e = expected.get(b) as number[];
- let sq = 0;
- for (let i = 0; i < dims; i++) {
- const d = v[i] - (v0[i] + e[i]);
- sq += d * d;
- }
- return Math.sqrt(sq) * ppm;
- };
- for (const [a, b] of pairs) {
- const speed = Math.max(hit(a), hit(b));
- for (const [self, other] of [
- [a, b],
- [b, a],
- ] as const) {
- const raw = rawImpact.get(self) as Float32Array;
- raw[frame] = Math.max(raw[frame], speed);
- const c = { frame, with: other, speed };
- self.collisions.push(c as never);
- if (self === a) this.allCollisions.push({ ...c, body: a });
- }
+ return Math.sqrt(sq) * ppm;
+ };
+ for (const [a, b] of pairs) {
+ const speed = Math.max(hit(a), hit(b));
+ for (const [self, other] of [
+ [a, b],
+ [b, a],
+ ] as const) {
+ raw.set(self, Math.max(raw.get(self) ?? 0, speed));
+ const c = { frame, with: other, speed };
+ self.collisions.push(c as never);
+ if (self === a) this.allCollisions.push({ ...c, body: a });
}
}
}
// Impact envelopes: instant attack, exponential release.
const decay = Math.exp(-1 / this.impactDecay);
- const worldImpact = this.worldRecording.channel("impact");
+ let loudest = 0;
for (const b of bodies) {
- const raw = rawImpact.get(b) as Float32Array;
- const env = b.recording.channel("impact");
- let level = 0;
- for (let i = startFrame; i < n; i++) {
- level = Math.max(raw[i] / this.impactReference, level * decay);
- env[i] = level;
- if (level > worldImpact[i]) worldImpact[i] = level;
- }
- }
-
- // Before the world starts every channel holds its frame-0 value.
- for (const rec of [
- ...bodies.map((b) => b.recording),
- this.worldRecording,
- ]) {
- for (const arr of rec.channels.values()) {
- arr.fill(arr[startFrame], 0, startFrame);
- }
+ const level = Math.max(
+ (raw.get(b) ?? 0) / this.impactReference,
+ (st.level.get(b) ?? 0) * decay,
+ );
+ st.level.set(b, level);
+ b.recording.channel("impact")[frame] = level;
+ if (level > loudest) loudest = level;
}
+ this.worldRecording.channel("impact")[frame] = loudest;
+ st.current = f;
}
protected abstract lerpRotation(a: R, b: R, t: number): R;
@@ -523,6 +615,11 @@ export abstract class WorldCore<
if (typeof layer?.animate !== "function" || output === "none") return;
if (body.type === "fixed") return;
const channels = body.boundChannels();
+ if (output === "keyframes" && this.liveMode) {
+ throw new Error(
+ 'Physics: live worlds bind signals; drop `output: "keyframes"` or use simulate().',
+ );
+ }
if (output === "signals") {
const anim = LayerAnimation.create();
for (const [prop, channel] of channels) {
diff --git a/packages/framefields/src/physics/physics.test.ts b/packages/framefields/src/physics/physics.test.ts
index 6e007f24..6d5b1a8d 100644
--- a/packages/framefields/src/physics/physics.test.ts
+++ b/packages/framefields/src/physics/physics.test.ts
@@ -263,7 +263,90 @@ describe("Physics.world2d", () => {
);
const b = world.body(card("a", 0, 0));
void b.y;
- expect(() => world.body(card("b", 0, 0))).toThrow(/already simulated/);
+ expect(() => world.body(card("b", 0, 0))).toThrow(/already started/);
+ });
+});
+
+describe("live worlds", () => {
+ /** A stand-in composition: collects prepare hooks, records which frames were prepared. */
+ function host() {
+ const hooks: Array<(ctx: { frame: number }) => Promise<void> | void> = [];
+ const prepared: number[] = [];
+ return {
+ prepared,
+ onPrepareFrame(cb: (ctx: { frame: number }) => Promise<void> | void) {
+ hooks.push(cb);
+ },
+ async prepareVision(frame: number) {
+ prepared.push(frame);
+ },
+ async render(frame: number) {
+ for (const h of hooks) await h({ frame });
+ },
+ };
+ }
+
+ async function scene() {
+ const world = await Physics.world2d({ fps: FPS, frames: 60 });
+ world.walls({ width: 1920, height: 1080 });
+ const layers = Array.from({ length: 6 }, (_, i) =>
+ card(`c${i}`, 300 + i * 200, -100 - i * 150, { rotation: i * 11 }),
+ );
+ const bodies = layers.map((l, i) => world.body(l, { startFrame: i * 3 }));
+ world.force(bodies[0], { x: (f) => (f > 20 ? 900 : 0) });
+ return { world, layers, bodies };
+ }
+
+ it("steps on demand and matches simulate() exactly, in any request order", async () => {
+ const batch = await scene();
+ batch.world.simulate();
+
+ const live = await scene();
+ const comp = host();
+ live.world.live(comp);
+ // Layers are bound before anything is simulated.
+ expect(tracksOf(live.layers[0]).map((t) => t.source?.type)).toEqual([
+ "signal",
+ "signal",
+ "signal",
+ ]);
+ await comp.render(40);
+ await comp.render(10); // scrubbing back: already simulated
+ await comp.render(59);
+ expect(comp.prepared).toEqual(Array.from({ length: 60 }, (_, f) => f));
+
+ for (let i = 0; i < batch.bodies.length; i++) {
+ for (const f of [0, 10, 33, 59]) {
+ expect(at(live.bodies[i].x, f)).toBe(at(batch.bodies[i].x, f));
+ expect(at(live.bodies[i].rotation, f)).toBe(
+ at(batch.bodies[i].rotation, f),
+ );
+ expect(at(live.bodies[i].impact, f)).toBe(
+ at(batch.bodies[i].impact, f),
+ );
+ }
+ }
+ expect(live.world.collisions.length).toBe(batch.world.collisions.length);
+ });
+
+ it("holds the start pose for frames not simulated yet", async () => {
+ const { world, bodies } = await scene();
+ const comp = host();
+ world.live(comp);
+ await comp.render(5);
+ expect(at(bodies[0].y, 30)).toBe(-100);
+ expect(at(bodies[0].y, 5)).toBeGreaterThan(-100);
+ });
+
+ it("waits for extra inputs before each frame and locks the world", async () => {
+ const { world } = await scene();
+ const comp = host();
+ const seen: number[] = [];
+ world.live(comp, { inputs: [{ ready: async (f) => void seen.push(f) }] });
+ await comp.render(3);
+ expect(seen).toEqual([0, 1, 2, 3]);
+ expect(() => world.simulate()).toThrow(/live/);
+ expect(() => world.body(card("late", 0, 0))).toThrow(/already started/);
});
});
diff --git a/plugins/framefields/skills/framefields-compose/SKILL.md b/plugins/framefields/skills/framefields-compose/SKILL.md
index be775a71..a7d58639 100644
--- a/plugins/framefields/skills/framefields-compose/SKILL.md
+++ b/plugins/framefields/skills/framefields-compose/SKILL.md
@@ -400,11 +400,13 @@ stage.animate(LayerAnimation.create().signal("y", world.impact, { multiplier: 14
- **Joints:** `world.pin(body, [x, y])` (swing around a point), `world.hinge(a, b, [x, y])`, `world.weld(a, b)`, `world.spring(a, bOrPoint, { frequency: 2, dampingRatio: 0.2, length })`, `world.rope(a, bOrPoint, { length })`. Points are world px.
- **Driving the simulation:** `world.force(body, { x, y })` (px/s², mass-independent), `world.impulse(body, frame, [vx, vy])` (a kick), kinematic `follow: { x, y, rotation }`. Each of these takes a number, a signal (audio, `Signal.fromArray`) or `(frame) => value`: e.g. a floor that bounces on the kick.
- **Force fields:** `world.field((s) => [ax, ay], { bodies })` is called every solver step with a body's center `s.x`, `s.y` (px), velocity `s.vx`, `s.vy` and `s.frame`; return px/s² or nothing. Wind, vortices, attractors, flow fields — and performers (below).
-- **Performers in the simulation:** `const person = (await comp.trackPerson("dancer.mp4")).place({ width: W, height: H, fit: "cover" })` analyses the clip *before* rendering (pose + silhouette, a few hundred ms/frame; cache with `toJSON()` / `PersonTrack.fromJSON()`). Then:
+- **Performers in the simulation:** `const dancer = comp.vision("dancer.mp4")` gives the clip's pose and silhouette per frame, analysed on demand (see `framefields-effects`). Make the world **live** so it steps as frames render instead of all at once: `world.live(comp)` in place of `simulate()`. Before each frame draws, the clip's vision for that frame is computed (or read from the cache) and the world advances to it — nothing is analysed up front, the preview plays at once and fills in, and export gives the identical result (same simulation, same order). Then, with `const person = dancer.person.place({ width: W, height: H, fit: "cover" })`:
- limbs as kinematic colliders that catch and bat things: `world.body({ x, y, width: 2 * r, height: 2 * r }, { type: "kinematic", shape: "circle", follow: { x: (f) => person.point("leftWrist", f).x - r, y: (f) => person.point("leftWrist", f).y - r } })`, a few balls along each shoulder → elbow → wrist;
- the silhouette (dress included) as a force field: `person.sample(frame, x, y)` returns `alpha` (inside > 0.5) and a gradient `gx`, `gy` pointing inward, so `world.field((s) => { const m = person.sample(s.frame, s.x, s.y); … return [-m.gx * k, -m.gy * k]; })` makes particles and props flow around a moving body.
+ - Reads are synchronous and need the frame analysed: inside a live world, while rendering, or after `await dancer.ready(frame)` (e.g. `ready(0)` to lay things out around the first pose). A live world can't look ahead, so place things from early frames.
- Layer a vision cutout of the same clip (`Layer.video(src).withVision({ mode: "matte" })`) above the particles so they pass behind the performer.
-- **Particles today:** up to ~600 small `Layer.shape("circle")` bodies with `linearDamping` ≈ 1–1.5, low `friction`, `gravityScale` < 1 read as a fluid; tint them by speed with `.colorSignal("fillColor", heat, { colorA, colorB })` built from `body.speed`.
+ - Live worlds bind signals only (no `bakeKeyframes()`); `await world.advanceTo(frame)` steps one without rendering (e.g. to read `collisions`).
+- **Particles today:** up to ~600 small `Layer.shape("circle")` bodies with `linearDamping` ≈ 1–1.5, low `friction`, `gravityScale` < 1 read as a fluid; tint them by speed with `.colorSignal("fillColor", body.speed, { colorA, colorB, multiplier: 1 / 900 })` (`multiplier`/`offset` scale the signal into 0–1).
- **Keyframes instead of signals:** `world.bakeKeyframes({ tolerance: 0.25 })` in place of `simulate()` writes plain, simplified linear keyframes (editable, plain JSON).
- **3D:** `Physics.world3d` takes `Layer3D.roundedBox / sphere / cylinder` and models (convex hull of the mesh); same options with 3-vectors, `world.hinge(a, b, at, axis)`, `world.walls({ width, height, depth })`.
- **Timing:** signals are on the composition clock. In a scene that starts at frame `S`, pass `startFrame: S` to the world. Before it and after `frames` the signals hold their first/last value. Add every body, joint and force before `simulate()` (or before reading any signal): the world then locks.
diff --git a/plugins/framefields/skills/framefields-effects/SKILL.md b/plugins/framefields/skills/framefields-effects/SKILL.md
index 1f3b0da1..d4b52ebc 100644
--- a/plugins/framefields/skills/framefields-effects/SKILL.md
+++ b/plugins/framefields/skills/framefields-effects/SKILL.md
@@ -236,16 +236,20 @@ const vision = comp.withVision({ classes: ["person"] });
layer.blurRegion(vision.objects.byCategory("person"), { strength: 30 });
```
-### 5. Track a Performer Before Rendering
-Render-time vision signals lag a frame and only exist while rendering. For anything that must know the motion up front (physics, keyframes, layout), track the person first:
+### 5. Clip Vision: Results Before the Frame Renders
+`withVision` analyses a layer's *previous* rendered frame, so its results trail by a frame and exist only while rendering. When something must know the motion *at* the frame — physics, layout, keyframes — use `comp.vision`, which analyses the clip's own frames on demand:
```typescript
-const person = (await comp.trackPerson("assets/dancer.mp4")).place({ width: W, height: H, fit: "cover" });
-person.point("rightWrist", 48); // { x, y, visibility } in composition px, smoothed, gaps filled
-person.signals("nose").x; // per-frame signal for .signal("x", …)
-person.sample(48, x, y); // silhouette alpha + inward gradient (dress and held things included)
-writeFileSync("dancer-track.json", JSON.stringify(person.toJSON())); // reuse with PersonTrack.fromJSON
+const dancer = comp.vision("assets/dancer.mp4", { startFrame: 0 }); // pose + silhouette (person)
+const person = dancer.person.place({ width: W, height: H, fit: "cover" });
+person.point("rightWrist", 48); // { x, y, visibility } in composition px, smoothed over ±2 frames
+person.signals("nose").x; // a signal for .signal("x", …)
+person.sample(48, x, y); // silhouette alpha + inward gradient (dress and held things included)
+await dancer.ready(0); // analyse a frame (or a range: ready(0, 120)) to read it at build time
```
-Options: `pose`, `mask` (both default true), `maskCell` (silhouette grid, default 8 px), `smoothing` (frames, default 2), `variant`. See *Physics* in `framefields-compose` for limbs as colliders and the silhouette as a force field.
+- The composition analyses each frame's clips just before drawing it, in every render path and the preview, and a live physics world (`world.live(comp)`) before stepping. Nothing runs up front; a preview starts at once.
+- Results persist in Node (`~/.cache/framefields/vision`, or `$FRAMEFIELDS_VISION_CACHE_DIR`; `cacheDir: false` to opt out), keyed by the file and settings, so later runs and renders after a preview are instant.
+- Options: `startFrame` / `trimStartSec` (where the clip plays), `pose`, `mask`, `maskCell` (silhouette grid, default 8 px), `smoothing` (frames, default 2), `variant`. When the person is lost the last pose is held for up to a second.
+- Analysis costs about 200 ms per frame on CPU (segmentation dominates); on WebGPU it is faster. In a live preview the playhead waits for frames not analysed yet.
### 6. Analyze Before Authoring
One-shot, ffmpeg-free report of what is in a clip (tracks, classes, mask coverage):