Skip to content

Code-first video & image, rendered natively on WebGPU.

fix(3d): render each 3D scene in isolation; stop per-frame GPU leaks

okanasl01954c0c2ad2
diff --git a/.changeset/scene-isolation.md b/.changeset/scene-isolation.md
new file mode 100644
index 00000000..a14d2061
--- /dev/null
+++ b/.changeset/scene-isolation.md
@@ -0,0 +1,8 @@
+---
+"@framefields/compositions": minor
+"@framefields/node-compositor": patch
+"@framefields/webgpu-renderers": patch
+"framefields": minor
+---
+
+3D scenes render in isolation. A box with a fixed size that holds a camera is its own scene: its camera, lights and environment see only its layers, it runs on its own clock, and it is placed like a picture, so `Layer.sequence` transitions between 3D scenes blend two finished pictures instead of drawing both through one camera. Signal tracks take `{ clock: "layer" }` to sample at the layer's own frame. `comp.lint()` reports `duplicate_id`. Per-pass 3D buffers are recycled once per frame (per command encoder), so a frame holding several 3D scenes never reuses a buffer an earlier pass still reads, and bloom mip chains live until the frame is done. Size-dependent 3D render targets (depth, HDR, MSAA, reflections) are cached per size and kept for the frame, so scenes of different sizes can share a frame.
diff --git a/nodes/node-compositor/SKILL.md b/nodes/node-compositor/SKILL.md
index 54a2d4d6..2094abb8 100644
--- a/nodes/node-compositor/SKILL.md
+++ b/nodes/node-compositor/SKILL.md
@@ -210,6 +210,7 @@ Each track represents animatable property modifications:
- **Watermarking a Video:** a `flex` row (`align: "end"`, `justify: "end"`, full canvas) containing a `media` node bound to the PNG input; low `opacity`.
- **Picture-in-Picture:** a `flex` row with two `media` nodes (`fit: "cover"`, each `grow: 1`).
- **Audio-Reactive 3D Models:** a `kind: "model3d"` node loading an OBJ/FBX/GLTF/STL/PLY mesh with `audioDeform: { audioTrackId, mode: "normal_extrusion" | "radial_pulse" | "harmonic_wave" | "twist" | "ripple", amplitudeMultiplier, damping }`, deformed directly in VRAM via WebGPU compute shaders driven by audio frequency spectra and transient energies. `twist` turns each slice about Y by its height (harder on the bass); `ripple` runs concentric waves across XY, struck by the drums. `audioTrackId` may be an audio layer's id or an audio file path.
+- **Several 3D scenes:** a `box`/`flex`/`block` with a numeric `width` and `height` that holds a `camera` and the layers it films renders as its own scene (one holding only a camera stays a shot in a camera cut list): a nested composition with only its own camera, lights and environment, on its own clock (children start from the box's frame 0), placed in its parent like a media layer. Transitions, opacity and mattes on the box then blend finished pictures. Without that, every 3D layer shares one world and the active camera (the latest to start).
- **3D scene capacity:** direct lights and skin joints use dynamically sized GPU storage; directional and spot lights can each cast shadows, up to the device texture-array layer limit.
- **Transparent meshes:** glTF `BLEND` materials and meshes with reduced layer, material, tint, or vertex alpha draw after opaque items, back to front by the camera-space center of each mesh’s transformed bounds, without writing depth. This is object-level sorting: intersecting meshes, self-overlap, skinned/deformed bounds, translucent flat layers, glass, and transparent shadows have separate limitations.
- **Mesh wireframe:** `wireframe: true` on a `model3d` draws triangle edges as one-pixel lines, including triangulation and back edges; it also applies to skinned and audio-deformed meshes and their shadows.
diff --git a/nodes/node-compositor/src/renderers/webgpu-renderer.ts b/nodes/node-compositor/src/renderers/webgpu-renderer.ts
index 13f21780..a6b0ab7b 100644
--- a/nodes/node-compositor/src/renderers/webgpu-renderer.ts
+++ b/nodes/node-compositor/src/renderers/webgpu-renderer.ts
@@ -225,6 +225,9 @@ const swarmStateMap = new WeakMap<object, Swarm3DState>();
const deviceRenderer3DCache = new WeakMap<GPUDevice, Map<string, Renderer3D>>();
/** One Renderer3D per device, target format and MSAA sample count. */
+/** The command encoder the 3D pools were last recycled for, per device. */
+const poolEncoder = new WeakMap<GPUDevice, GPUCommandEncoder>();
+
function getRenderer3D(
device: GPUDevice,
format: GPUTextureFormat,
@@ -990,13 +993,18 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
ctx.renderer.format ?? "rgba8unorm",
sampleCount3d,
);
- r3d.resetPools();
-
const relightPipe = getScreenSpaceRelightPipeline(
ctx.device,
ctx.renderer.format ?? "rgba8unorm",
);
- relightPipe.resetPools();
+ // The pools recycle per-pass uniforms once per frame, and a frame is one
+ // command encoder: a nested scene drawn into the same encoder must not
+ // recycle buffers the passes encoded before it still read.
+ if (poolEncoder.get(ctx.device) !== encoder) {
+ poolEncoder.set(ctx.device, encoder);
+ r3d.resetPools();
+ relightPipe.resetPools();
+ }
// 2. Initial clear
const clearColor = parseColor(vop.backgroundColor);
diff --git a/packages/compositions/src/program/animation.ts b/packages/compositions/src/program/animation.ts
index 72e0ad3d..1165b203 100644
--- a/packages/compositions/src/program/animation.ts
+++ b/packages/compositions/src/program/animation.ts
@@ -4,6 +4,7 @@ import type {
AnimationTrack,
EaseRef,
LayerAnimationSpec,
+ SignalClock,
TrackSource,
} from "./schema.js";
@@ -66,6 +67,13 @@ export interface SignalTrackOptions {
mode?: "continuous" | "accumulate";
threshold?: number;
debounceFrames?: number;
+ /**
+ * `"composition"` (default) samples the signal at the composition frame,
+ * as audio analysis and shared signals need. `"layer"` samples it at the
+ * layer's own frame, counted from its start like keyframes: for motion
+ * authored per scene, which then plays right inside `Layer.sequence`.
+ */
+ clock?: SignalClock;
}
export interface ColorSignalOptions {
@@ -963,6 +971,7 @@ export class LayerAnimation<TProp extends AnimatableProp = AnimatableProp>
accumulateThreshold: options.threshold ?? 0.15,
debounceFrames: options.debounceFrames ?? 2,
channel: options.channel ?? "primary",
+ ...(options.clock && { clock: options.clock }),
};
if (inline) attachInlineSignal(source, inline);
diff --git a/packages/compositions/src/program/evaluator.test.ts b/packages/compositions/src/program/evaluator.test.ts
index 3bd807ef..69212095 100644
--- a/packages/compositions/src/program/evaluator.test.ts
+++ b/packages/compositions/src/program/evaluator.test.ts
@@ -502,6 +502,32 @@ describe("Container Staggering", () => {
});
describe("Evaluator — Signal to Color Modulation", () => {
+ it("reads a signal on the composition clock unless the track asks for its layer's", () => {
+ const ramp = [0, 10, 20, 30, 40, 50];
+ const track = (clock?: "composition" | "layer"): AnimationTrack => ({
+ id: "sig-clock",
+ prop: "x",
+ source: {
+ type: "signal",
+ inputHandleId: "ramp",
+ multiplier: 1,
+ offset: 0,
+ smoothingWindowFrames: 0,
+ ...(clock && { clock }),
+ },
+ keyframes: [],
+ });
+ const at = (clock?: "composition" | "layer") =>
+ evaluateTrackAtFrame(track(clock), 4, 30, {
+ startFrame: 3,
+ signals: { ramp },
+ });
+ // A layer starting on frame 3, seen on composition frame 4.
+ expect(at()).toBe(40);
+ expect(at("composition")).toBe(40);
+ expect(at("layer")).toBe(10);
+ });
+
it("interpolates colors based on signal amplitude", () => {
const track: AnimationTrack = {
id: "sig-color-1",
diff --git a/packages/compositions/src/program/evaluator.ts b/packages/compositions/src/program/evaluator.ts
index c2bde79a..cc3ba07a 100644
--- a/packages/compositions/src/program/evaluator.ts
+++ b/packages/compositions/src/program/evaluator.ts
@@ -787,7 +787,8 @@ export function evaluateTrackAtFrame(
}
case "signal": {
- const sigVal = evaluateSignal(source, frame, fps, context.signals);
+ const at = source.clock === "layer" ? localFrame : frame;
+ const sigVal = evaluateSignal(source, at, fps, context.signals);
if (COLOR_PROPS.has(track.prop)) {
const src = source as {
colorMode?: "interpolate" | "hueRotate" | "threshold";
diff --git a/packages/compositions/src/program/lint.test.ts b/packages/compositions/src/program/lint.test.ts
index 2f9ce14a..5fdea5b5 100644
--- a/packages/compositions/src/program/lint.test.ts
+++ b/packages/compositions/src/program/lint.test.ts
@@ -8,6 +8,14 @@ const codes = (layout: unknown[], extra = {}) =>
);
describe("lintComposition", () => {
+ it("flags two layers with one id, wherever they sit in the tree", () => {
+ expect(
+ codes([
+ { id: "scene-a", kind: "box", children: [{ id: "hero", kind: "box" }] },
+ { id: "scene-b", kind: "box", children: [{ id: "hero", kind: "box" }] },
+ ]),
+ ).toEqual(["duplicate_id"]);
+ });
it("flags unreadable colors and gradients", () => {
expect(
codes([
diff --git a/packages/compositions/src/program/lint.ts b/packages/compositions/src/program/lint.ts
index 053b382d..957b4040 100644
--- a/packages/compositions/src/program/lint.ts
+++ b/packages/compositions/src/program/lint.ts
@@ -120,9 +120,20 @@ export function lintComposition(
let has3D = false;
const pbrNodes: string[] = [];
const emojiText: string[] = [];
+ const seenIds = new Set<string>();
const visit = (node: AnyNode, backdrop: string | null, atRoot: boolean) => {
const id = String(node.id ?? "?");
+ if (typeof node.id === "string") {
+ if (seenIds.has(node.id))
+ push(
+ "error",
+ "duplicate_id",
+ `Two layers share the id '${id}': the renderer keeps one and drops the other. Give every layer its own id (scenes reused in one composition need their own prefix).`,
+ id,
+ );
+ seenIds.add(node.id);
+ }
if (
options.emojiFont === false &&
node.kind === "text" &&
diff --git a/packages/compositions/src/program/schema.ts b/packages/compositions/src/program/schema.ts
index abf5f343..0f9d1fc2 100644
--- a/packages/compositions/src/program/schema.ts
+++ b/packages/compositions/src/program/schema.ts
@@ -469,6 +469,13 @@ export type Keyframe = z.infer<typeof KeyframeSchema>;
export const SignalModeSchema = z.enum(["continuous", "accumulate"]);
export type SignalMode = z.infer<typeof SignalModeSchema>;
+/**
+ * Which clock a signal track reads: the composition's (default), as audio
+ * analysis and shared signals need, or its layer's own, from the layer's
+ * start frame, as keyframes do: motion authored for one scene of a sequence.
+ */
+export const SignalClockSchema = z.enum(["composition", "layer"]);
+export type SignalClock = z.infer<typeof SignalClockSchema>;
const RawTrackSourceSchema = z.discriminatedUnion("type", [
z.object({
@@ -489,6 +496,7 @@ const RawTrackSourceSchema = z.discriminatedUnion("type", [
accumulateThreshold: z.number().optional(),
debounceFrames: z.number().int().min(0).optional(),
channel: z.string().optional(),
+ clock: SignalClockSchema.optional(),
}),
z.object({
type: z.literal("wiggle"),
@@ -530,6 +538,7 @@ export const TrackSourceSchema = z.preprocess((val: unknown) => {
accumulateThreshold: thresh,
debounceFrames: obj.debounceFrames ?? 2,
channel: typeof obj.channel === "string" ? obj.channel : "primary",
+ ...(obj.clock !== undefined && { clock: obj.clock }),
};
}
return val;
diff --git a/packages/compositions/src/program/to-virtual-media.test.ts b/packages/compositions/src/program/to-virtual-media.test.ts
index cb80967b..73bb77d4 100644
--- a/packages/compositions/src/program/to-virtual-media.test.ts
+++ b/packages/compositions/src/program/to-virtual-media.test.ts
@@ -6,7 +6,7 @@ import {
} from "@framefields/core";
import { describe, expect, it } from "vitest";
import type { CompositorProgramConfig, LayoutNode } from "./schema.js";
-import { compositorToProgram } from "./to-virtual-media.js";
+import { compositorToProgram, isScene3D } from "./to-virtual-media.js";
const doc = {
width: 1920,
@@ -441,3 +441,75 @@ describe("compositorToProgram", () => {
expect(textOp?.effects).toEqual([{ op: "FilmGrain", strength: 0.1 }]);
});
});
+
+describe("compositorToProgram: 3D scenes", () => {
+ const scene = (id: string, startFrame: number): LayoutNode =>
+ ({
+ id,
+ kind: "box",
+ position: "absolute",
+ width: 1080,
+ height: 1920,
+ background: "#202020",
+ startFrame,
+ durationFrames: 60,
+ children: [
+ { id: `${id}-cam`, kind: "camera", z: -800 },
+ {
+ id: `${id}-cube`,
+ kind: "box",
+ is3D: true,
+ width: 100,
+ height: 100,
+ startFrame: 10,
+ },
+ ],
+ }) as unknown as LayoutNode;
+ const ramp = [0, 1, 2];
+ const program = compositorToProgram(
+ {
+ width: 1080,
+ height: 1920,
+ fps: 30,
+ mode: "Video",
+ layout: [scene("a", 0), scene("b", 50)],
+ signals: { ramp },
+ } as unknown as CompositorProgramConfig,
+ { isVideoMode: true, fps: 30, durationMs: 4000 },
+ );
+ const [a, b] = program.children as VirtualMediaData[];
+
+ it("places a container holding a camera as media whose content is its own composition", () => {
+ const op = b.operation as Record<string, unknown>;
+ expect(op.kind).toBe("media");
+ expect(op.startFrame).toBe(50);
+ expect(op.background).toBeUndefined();
+ const nested = b.children?.[0];
+ expect(nested?.operation.op).toBe("Compositor");
+ expect((nested?.operation as Record<string, unknown>).backgroundColor).toBe(
+ "#202020",
+ );
+ expect((nested?.operation as Record<string, unknown>).signals).toEqual({
+ ramp,
+ });
+ });
+ it("leaves a box holding only a camera as a shot over the shared world", () => {
+ expect(
+ isScene3D({
+ id: "shot",
+ kind: "box",
+ width: 1080,
+ height: 1920,
+ children: [{ id: "shot-cam", kind: "camera" }],
+ } as unknown as LayoutNode),
+ ).toBe(false);
+ });
+ it("keeps the scene's layers on the scene's own clock", () => {
+ const layers = b.children?.[0].children ?? [];
+ const cube = layers.find(
+ (l) => (l.operation as { id?: string }).id === "b-cube",
+ );
+ expect(cube?.operation.startFrame).toBe(10);
+ expect(a.children?.[0].children?.length).toBe(2);
+ });
+});
diff --git a/packages/compositions/src/program/to-virtual-media.ts b/packages/compositions/src/program/to-virtual-media.ts
index 9d32fd80..9540c9cb 100644
--- a/packages/compositions/src/program/to-virtual-media.ts
+++ b/packages/compositions/src/program/to-virtual-media.ts
@@ -64,6 +64,27 @@ export type MediaSignalResolver = (
layer: MediaSignalLayer,
) => ResolvedMediaTimeRemap;
+/**
+ * A container holding a camera and the layers it films is a 3D scene of its
+ * own: its camera, lights and environment see only its layers. It needs a
+ * fixed size to render at. A container holding only a camera (and lights) is
+ * a shot in a camera cut list over the shared world, and stays one.
+ */
+export function isScene3D(node: LayoutNode): boolean {
+ if (
+ (node.kind !== "box" && node.kind !== "flex" && node.kind !== "block") ||
+ typeof node.width !== "number" ||
+ typeof node.height !== "number" ||
+ !Array.isArray(node.children)
+ )
+ return false;
+ const kids = node.children;
+ return (
+ kids.some((c) => c.kind === "camera") &&
+ kids.some((c) => c.kind !== "camera" && c.kind !== "light")
+ );
+}
+
function nodeToVirtualMedia(
node: LayoutNode,
resolveMedia: CompositorToProgramOptions["resolveMedia"],
@@ -224,6 +245,10 @@ function nodeToVirtualMedia(
? Math.round(node.durationFrames)
: inheritedDurationFrames;
const nodeChildren = node.children;
+ // A 3D scene renders as a composition of its own on its own clock, which
+ // this layer then places like a picture: transitions, opacity and mattes
+ // on the layer blend whole scenes, each through its own camera.
+ const scene = isScene3D(node);
children = nodeChildren.map((c, idx) => {
const delay = computeStaggerDelay(
idx,
@@ -231,7 +256,7 @@ function nodeToVirtualMedia(
staggerFrames,
staggerDirection,
);
- const childOffset = inheritedOffset + nodeStart + delay;
+ const childOffset = scene ? delay : inheritedOffset + nodeStart + delay;
return nodeToVirtualMedia(
c,
resolveMedia,
@@ -242,6 +267,33 @@ function nodeToVirtualMedia(
resolveSignal,
);
});
+ if (scene) {
+ const frames =
+ nodeDuration ?? Math.round(((compDurationMs ?? 10000) / 1000) * fps);
+ const durationMs = (frames / fps) * 1000;
+ media = {
+ metadata: {
+ width: node.width as number,
+ height: node.height as number,
+ fps,
+ durationMs,
+ },
+ operation: {
+ op: "Compositor",
+ width: node.width as number,
+ height: node.height as number,
+ fps,
+ mode: "Video",
+ dataType: "Video",
+ volume: 1,
+ ...(typeof node.background === "string" && {
+ backgroundColor: node.background,
+ }),
+ },
+ children,
+ } as unknown as VirtualMediaData;
+ children = [media];
+ }
}
let dataType: DataType = "Image";
@@ -346,6 +398,12 @@ function nodeToVirtualMedia(
operation: {
op: "CompositorLayer",
...node,
+ // A scene is placed as media: its nested composition paints its background.
+ ...(isScene3D(node) && {
+ kind: "media",
+ background: undefined,
+ children: undefined,
+ }),
startFrame,
durationFrames: dur,
dataType,
@@ -428,6 +486,8 @@ export function compositorToProgram(
),
} as unknown as VirtualMediaData;
+ shareSignals(rootMedia.children ?? [], programSignals);
+
const effectsList: ProgramEffect[] = Array.isArray(options.effects)
? options.effects
: Array.isArray(config.effects)
@@ -471,10 +531,26 @@ export function compositorToProgram(
return rootMedia;
}
-/** Recursively gather every node op from a render tree (in order). */
+/** Hands every nested scene composition the program's signals, which its tracks read. */
+function shareSignals(
+ nodes: readonly VirtualMediaData[],
+ signals: Record<string, unknown>,
+) {
+ for (const vm of nodes) {
+ if (vm?.operation?.op === "Compositor")
+ (vm.operation as { signals?: unknown }).signals = signals;
+ shareSignals(vm?.children ?? [], signals);
+ }
+}
+
+/**
+ * Recursively gather every node op from a render tree (in order), stopping at
+ * nested scene compositions: their layers belong to their own render.
+ */
export function collectNodeOps(nodes: VirtualMediaData[]): VirtualMediaData[] {
const out: VirtualMediaData[] = [];
const walk = (vm: VirtualMediaData) => {
+ if (vm?.operation?.op === "Compositor") return;
if (vm?.operation?.op === "CompositorLayer") {
out.push(vm);
}
diff --git a/packages/framefields/src/render-memory.test.ts b/packages/framefields/src/render-memory.test.ts
new file mode 100644
index 00000000..71861389
--- /dev/null
+++ b/packages/framefields/src/render-memory.test.ts
@@ -0,0 +1,60 @@
+import { describe, expect, it } from "vitest";
+import {
+ Composition,
+ HeadlessMediaRenderer,
+ Layer,
+ Layer3D,
+ LayerAnimation,
+} from "./index.js";
+
+type Proto = Record<string, (...args: unknown[]) => unknown>;
+
+describe("render memory", () => {
+ it("recycles 3D buffers from frame to frame instead of allocating new ones", async () => {
+ const comp = new Composition({
+ width: 200,
+ height: 200,
+ fps: 30,
+ durationFrames: 60,
+ backgroundColor: "#000000",
+ });
+ comp.add([
+ Layer.camera({ x: 100, y: 100, z: -400, targetX: 100, targetY: 100 }),
+ Layer.ambientLight("#ffffff", 1),
+ ...[0, 1, 2, 3].map((i) =>
+ Layer3D.roundedBox({
+ x: 40 + i * 40,
+ y: 100,
+ z: 0,
+ width: 30,
+ height: 30,
+ depth: 30,
+ color: "#ff8800",
+ }).animate(
+ LayerAnimation.create().keys("rotateY", [
+ [0, 0],
+ [60, 360],
+ ]),
+ ),
+ ),
+ ]);
+ const renderer = new HeadlessMediaRenderer();
+ for (let f = 0; f < 4; f++) await comp.renderFrame({ frame: f, renderer });
+ const proto = (globalThis as unknown as { GPUDevice: { prototype: Proto } })
+ .GPUDevice.prototype;
+ const createBuffer = proto.createBuffer;
+ let created = 0;
+ proto.createBuffer = function (...args: unknown[]) {
+ created++;
+ return createBuffer.apply(this, args);
+ };
+ try {
+ for (let f = 4; f < 34; f++)
+ await comp.renderFrame({ frame: f, renderer });
+ } finally {
+ proto.createBuffer = createBuffer;
+ }
+ // Thirty more frames of the same scene reuse the pools: next to nothing new.
+ expect(created).toBeLessThan(30);
+ });
+});
diff --git a/packages/framefields/src/scene-isolation-render.test.ts b/packages/framefields/src/scene-isolation-render.test.ts
new file mode 100644
index 00000000..935c4a15
--- /dev/null
+++ b/packages/framefields/src/scene-isolation-render.test.ts
@@ -0,0 +1,90 @@
+import { Canvas, loadImage } from "skia-canvas";
+import { describe, expect, it } from "vitest";
+import {
+ Composition,
+ HeadlessMediaRenderer,
+ Layer,
+ Layer3D,
+ Transition,
+} from "./index.js";
+
+async function centre(png: Buffer) {
+ const img = await loadImage(png);
+ const canvas = new Canvas(img.width, img.height);
+ const ctx = canvas.getContext("2d");
+ ctx.drawImage(img, 0, 0);
+ const [r, g, b] = ctx.getImageData(img.width / 2, img.height / 2, 1, 1).data;
+ return { r, g, b };
+}
+
+/** A scene that films one glowing cube at `x` through its own camera. */
+function scene(x: number, color: string) {
+ return Layer.box({
+ width: 200,
+ height: 200,
+ background: "#000000",
+ children: [
+ Layer.camera({
+ x,
+ y: 100,
+ z: -400,
+ targetX: x,
+ targetY: 100,
+ targetZ: 0,
+ }),
+ Layer.ambientLight("#ffffff", 1),
+ Layer3D.roundedBox({
+ x,
+ y: 100,
+ z: 0,
+ width: 160,
+ height: 160,
+ depth: 160,
+ radius: 4,
+ color,
+ emissive: color,
+ emissiveIntensity: 1,
+ }),
+ ],
+ });
+}
+
+describe("3D scenes in a sequence", () => {
+ it("renders each scene through its own camera, and blends them in a transition", async () => {
+ const comp = new Composition({
+ width: 200,
+ height: 200,
+ fps: 30,
+ durationFrames: 50,
+ backgroundColor: "#000000",
+ });
+ // The cubes stand far apart: one shared world filmed by either camera
+ // would show only one of them.
+ comp.add(
+ Layer.sequence(
+ [
+ { scene: scene(100, "#ff0000"), duration: 30 },
+ {
+ scene: scene(5000, "#0000ff"),
+ duration: 30,
+ transition: Transition.crossfade(10),
+ },
+ ],
+ { width: 200, height: 200 },
+ ),
+ );
+ const renderer = new HeadlessMediaRenderer();
+ const at = async (frame: number) =>
+ centre(await comp.renderFrame({ frame, renderer }));
+ const first = await at(10);
+ const both = await at(25);
+ const second = await at(40);
+ expect(first.r).toBeGreaterThan(150);
+ expect(first.b).toBeLessThan(60);
+ expect(second.b).toBeGreaterThan(150);
+ expect(second.r).toBeLessThan(60);
+ // Mid-crossfade both cubes show, each through its own camera.
+ expect(both.r).toBeGreaterThan(50);
+ expect(both.b).toBeGreaterThan(50);
+ });
+});
diff --git a/packages/webgpu-renderers/src/renderer3d/bloom-pipeline.ts b/packages/webgpu-renderers/src/renderer3d/bloom-pipeline.ts
index 6ddcaa9f..c1e2b621 100644
--- a/packages/webgpu-renderers/src/renderer3d/bloom-pipeline.ts
+++ b/packages/webgpu-renderers/src/renderer3d/bloom-pipeline.ts
@@ -159,8 +159,13 @@ export class BloomPipeline {
private module: GPUShaderModule;
private pipelineLayout: GPUPipelineLayout;
private sampler: GPUSampler;
- private mips: GPUTexture[] = [];
- private mipKey = "";
+ /**
+ * Mip chains by size and depth. Scenes with different bloom share a frame
+ * (one command buffer), so a chain stays alive until the frame is done:
+ * chains not used in a frame are destroyed at the next `resetPools`.
+ */
+ private chains = new Map<string, GPUTexture[]>();
+ private usedThisFrame = new Set<string>();
private placeholder: GPUTexture;
private uniformPool = new BufferPool(
48,
@@ -267,17 +272,22 @@ export class BloomPipeline {
return pipe;
}
- private ensureMips(width: number, height: number, levels: number): void {
+ private mipChain(
+ width: number,
+ height: number,
+ levels: number,
+ ): GPUTexture[] {
const key = `${width}x${height}x${levels}`;
- if (key === this.mipKey) return;
- for (const t of this.mips) t.destroy();
- this.mips = [];
+ this.usedThisFrame.add(key);
+ const cached = this.chains.get(key);
+ if (cached) return cached;
+ const chain: GPUTexture[] = [];
let w = width;
let h = height;
for (let i = 0; i < levels; i++) {
w = Math.max(1, Math.floor(w / 2));
h = Math.max(1, Math.floor(h / 2));
- this.mips.push(
+ chain.push(
this.device.createTexture({
label: `BloomMip${i}`,
size: { width: w, height: h },
@@ -288,7 +298,8 @@ export class BloomPipeline {
);
if (w === 1 && h === 1) break;
}
- this.mipKey = key;
+ this.chains.set(key, chain);
+ return chain;
}
private uniforms(
@@ -371,8 +382,7 @@ export class BloomPipeline {
mode: display ? 1 : 0,
};
const levels = Math.max(1, Math.min(8, Math.round(options.radius ?? 6)));
- this.ensureMips(source.width, source.height, levels);
- const mips = this.mips;
+ const mips = this.mipChain(source.width, source.height, levels);
const u = (w: number, h: number) => this.uniforms(1 / w, 1 / h, opts);
const glowSource = hdrSource ?? source;
@@ -420,13 +430,24 @@ export class BloomPipeline {
);
}
- /** Call once per frame before `apply`: recycles the per-pass uniforms. */
+ /**
+ * Call once per frame before `apply`: recycles the per-pass uniforms and
+ * frees the mip chains the last frame did not use.
+ */
resetPools(): void {
this.uniformPool.reset();
+ for (const [key, chain] of this.chains) {
+ if (this.usedThisFrame.has(key)) continue;
+ for (const t of chain) t.destroy();
+ this.chains.delete(key);
+ }
+ this.usedThisFrame.clear();
}
destroy(): void {
- for (const t of this.mips) t.destroy();
+ for (const chain of this.chains.values())
+ for (const t of chain) t.destroy();
+ this.chains.clear();
this.placeholder.destroy();
this.uniformPool.destroy();
}
diff --git a/packages/webgpu-renderers/src/renderer3d/renderer3d.ts b/packages/webgpu-renderers/src/renderer3d/renderer3d.ts
index 120a2548..166d7cf2 100644
--- a/packages/webgpu-renderers/src/renderer3d/renderer3d.ts
+++ b/packages/webgpu-renderers/src/renderer3d/renderer3d.ts
@@ -418,7 +418,41 @@ export class Renderer3D {
});
}
+ /**
+ * Size-dependent render targets, by label, size, samples and usage. One
+ * frame (one command buffer) can render 3D scenes of different sizes, so
+ * a target is never destroyed while the frame may use it: targets the last
+ * frame did not use are destroyed at the next `resetPools`.
+ */
+ private attachments = new Map<string, GPUTexture>();
+ private attachmentsUsed = new Set<string>();
+ private attachment(
+ label: string,
+ width: number,
+ height: number,
+ desc: { format: GPUTextureFormat; usage: number; sampleCount?: number },
+ ): GPUTexture {
+ const key = `${label}:${width}x${height}:${desc.sampleCount ?? 1}:${desc.usage}`;
+ this.attachmentsUsed.add(key);
+ let texture = this.attachments.get(key);
+ if (!texture) {
+ texture = this.device.createTexture({
+ label,
+ size: { width, height },
+ ...desc,
+ });
+ this.attachments.set(key, texture);
+ }
+ return texture;
+ }
+
resetPools(): void {
+ for (const [key, texture] of this.attachments) {
+ if (this.attachmentsUsed.has(key)) continue;
+ texture.destroy();
+ this.attachments.delete(key);
+ }
+ this.attachmentsUsed.clear();
this.bloomPipeline?.resetPools();
this.cameraPool.reset();
// The cached camera buffer goes back to the pool and may be handed
@@ -437,37 +471,27 @@ export class Renderer3D {
}
getOrCreateDepthTexture(width: number, height: number): GPUTexture {
- if (
- !this.depthTexture ||
- this.depthTexture.width !== width ||
- this.depthTexture.height !== height
- ) {
- this.depthTexture?.destroy();
- this.depthTexture = this.device.createTexture({
- label: "Renderer3DDepthTexture",
- size: { width, height },
+ this.depthTexture = this.attachment(
+ "Renderer3DDepthTexture",
+ width,
+ height,
+ {
format: "depth24plus",
sampleCount: this.sampleCount,
usage: GPUTextureUsage.RENDER_ATTACHMENT,
- });
- }
+ },
+ );
return this.depthTexture;
}
private getOrCreateMsaaTexture(
- current: GPUTexture | null,
+ _current: GPUTexture | null,
label: string,
format: GPUTextureFormat,
width: number,
height: number,
): GPUTexture {
- if (current && current.width === width && current.height === height) {
- return current;
- }
- current?.destroy();
- return this.device.createTexture({
- label,
- size: { width, height },
+ return this.attachment(label, width, height, {
format,
sampleCount: this.sampleCount,
usage: GPUTextureUsage.RENDER_ATTACHMENT,
@@ -480,110 +504,85 @@ export class Renderer3D {
}
private getOrCreateHdrTexture(width: number, height: number): GPUTexture {
- if (
- !this.hdrTexture ||
- this.hdrTexture.width !== width ||
- this.hdrTexture.height !== height
- ) {
- this.hdrTexture?.destroy();
- this.hdrTexture = this.device.createTexture({
- label: "Renderer3DHdrRadiance",
- size: { width, height },
- format: "rgba16float",
- usage:
- GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING,
- });
- }
+ this.hdrTexture = this.attachment("Renderer3DHdrRadiance", width, height, {
+ format: "rgba16float",
+
+ usage:
+ GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING,
+ });
return this.hdrTexture;
}
getOrCreateLinearDepthTexture(width: number, height: number): GPUTexture {
- if (
- !this.linearDepthTexture ||
- this.linearDepthTexture.width !== width ||
- this.linearDepthTexture.height !== height
- ) {
- this.linearDepthTexture?.destroy();
- this.linearDepthTexture = this.device.createTexture({
- label: "Renderer3DLinearDepthTexture",
- size: { width, height },
+ this.linearDepthTexture = this.attachment(
+ "Renderer3DLinearDepthTexture",
+ width,
+ height,
+ {
format: "rgba16float",
+
usage:
GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING,
- });
- }
+ },
+ );
return this.linearDepthTexture;
}
getOrCreateReflectionTexture(width: number, height: number): GPUTexture {
- if (
- !this.reflectionColorTexture ||
- this.reflectionColorTexture.width !== width ||
- this.reflectionColorTexture.height !== height
- ) {
- this.reflectionColorTexture?.destroy();
- this.reflectionColorTexture = this.device.createTexture({
- label: "Renderer3DReflectionColor",
- size: { width, height },
+ this.reflectionColorTexture = this.attachment(
+ "Renderer3DReflectionColor",
+ width,
+ height,
+ {
format: this.format,
+
usage:
GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING,
- });
- }
+ },
+ );
return this.reflectionColorTexture;
}
getOrCreateReflectionMsaaTexture(width: number, height: number): GPUTexture {
- if (
- !this.reflectionMsaaColorTexture ||
- this.reflectionMsaaColorTexture.width !== width ||
- this.reflectionMsaaColorTexture.height !== height
- ) {
- this.reflectionMsaaColorTexture?.destroy();
- this.reflectionMsaaColorTexture = this.device.createTexture({
- label: "Renderer3DReflectionMsaaColor",
- size: { width, height },
+ this.reflectionMsaaColorTexture = this.attachment(
+ "Renderer3DReflectionMsaaColor",
+ width,
+ height,
+ {
format: this.format,
sampleCount: this.sampleCount,
usage: GPUTextureUsage.RENDER_ATTACHMENT,
- });
- }
+ },
+ );
return this.reflectionMsaaColorTexture;
}
getOrCreateReflectionDepthTexture(width: number, height: number): GPUTexture {
- if (
- !this.reflectionDepthTexture ||
- this.reflectionDepthTexture.width !== width ||
- this.reflectionDepthTexture.height !== height
- ) {
- this.reflectionDepthTexture?.destroy();
- this.reflectionDepthTexture = this.device.createTexture({
- label: "Renderer3DReflectionDepth",
- size: { width, height },
+ this.reflectionDepthTexture = this.attachment(
+ "Renderer3DReflectionDepth",
+ width,
+ height,
+ {
format: "depth24plus",
sampleCount: this.sampleCount,
usage: GPUTextureUsage.RENDER_ATTACHMENT,
- });
- }
+ },
+ );
return this.reflectionDepthTexture;
}
getOrCreateReflectionTempTexture(width: number, height: number): GPUTexture {
- if (
- !this.reflectionTempTexture ||
- this.reflectionTempTexture.width !== width ||
- this.reflectionTempTexture.height !== height
- ) {
- this.reflectionTempTexture?.destroy();
- this.reflectionTempTexture = this.device.createTexture({
- label: "Renderer3DReflectionTemp",
- size: { width, height },
+ this.reflectionTempTexture = this.attachment(
+ "Renderer3DReflectionTemp",
+ width,
+ height,
+ {
format: this.format,
+
usage:
GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING,
- });
- }
+ },
+ );
return this.reflectionTempTexture;
}
@@ -591,20 +590,17 @@ export class Renderer3D {
width: number,
height: number,
): GPUTexture {
- if (
- !this.reflectionBlurredTexture ||
- this.reflectionBlurredTexture.width !== width ||
- this.reflectionBlurredTexture.height !== height
- ) {
- this.reflectionBlurredTexture?.destroy();
- this.reflectionBlurredTexture = this.device.createTexture({
- label: "Renderer3DReflectionBlurred",
- size: { width, height },
+ this.reflectionBlurredTexture = this.attachment(
+ "Renderer3DReflectionBlurred",
+ width,
+ height,
+ {
format: this.format,
+
usage:
GPUTextureUsage.RENDER_ATTACHMENT | GPUTextureUsage.TEXTURE_BINDING,
- });
- }
+ },
+ );
return this.reflectionBlurredTexture;
}
@@ -2021,6 +2017,8 @@ export class Renderer3D {
}
destroy(): void {
+ for (const texture of this.attachments.values()) texture.destroy();
+ this.attachments.clear();
this.bloomPipeline?.destroy();
this.depthTexture?.destroy();
for (const texture of this.shadowTextures.values()) texture.destroy();
diff --git a/plugins/framefields/skills/framefields-compose/SKILL.md b/plugins/framefields/skills/framefields-compose/SKILL.md
index 324bc747..a181dac1 100644
--- a/plugins/framefields/skills/framefields-compose/SKILL.md
+++ b/plugins/framefields/skills/framefields-compose/SKILL.md
@@ -104,7 +104,7 @@ All `Layer.*` builders return `AnimatableNode<T>` instances with chainable `.ani
| `Layer.environment(preset, options)` | `"light"` | The 3D scene's world: studio image-based light, exposure, tone mapping, fog, bloom. **Adding one switches the scene to physically based shading.** See *3D Look* below. |
| `Layer.sequence(entries, options)` | `"box"` | Scenes back to back with `Transition`s between them; returns a box whose `totalFrames` is the length. See *Scenes & Transitions*. |
| `Layer.model(options)` | `"model3d"` | 3D mesh loader (`.gltf`, `.glb`, `.obj`, `.fbx`, `.stl`, `.ply`). Supports skinning and audio deformation compute shaders. |
-| `Layer3D.roundedBox / sphere / cylinder / capsule / torus / ring / prism / pyramid / polyhedron / icosahedron / icosphere / shard(options)` | `"model3d"` | Procedural solid meshes, centered on `x`/`y`/`z` (y is down, so "top" is -y). `roundedBox`: `width`, `height`, `depth`, `radius`, `segments`, `taper`. `sphere`: `radius`. `cylinder`: `radius` or `radiusTop`/`radiusBottom`, `height`, `openEnded`, `flatShaded`. `capsule`: `radius`, `height` (cylindrical shaft) or `totalHeight`. `torus`: `radius`, `tubeRadius`, `arc`. `ring`: flat annulus (`innerRadius`, `outerRadius`). `prism` / `pyramid`: solid capped flat-shaded N-gon prism or pyramid (`sides`, `radius`, `height`). `polyhedron` (`type: "tetrahedron" \| "octahedron" \| "dodecahedron" \| "icosahedron"`), `icosahedron`, `icosphere` (`radius`, `detail`, `flatShaded`). `shard`: procedural flat-faceted polyhedron rock/crystal (`radius`, `cuts`, `irregularity`, `seed`). Surface: `color`, `tint`, `roughness`, `metallic`, `emissive`, `emissiveIntensity`, `material` (`"lit" \| "glass" \| "acrylic" \| "frosted"`), `castShadow`, `wireframe`. Shared GPU buffers cache identical shapes. |
+| `Layer3D.roundedBox / sphere / cylinder / capsule / torus / ring / prism / pyramid / polyhedron / icosahedron / icosphere / shard(options)` | `"model3d"` | Procedural solid meshes, centered on `x`/`y`/`z` (y is down, so "top" is -y). `roundedBox`: `width`, `height`, `depth`, `radius`, `segments`, `taper`. `sphere`: `radius`. `cylinder`: `radius` or `radiusTop`/`radiusBottom`, `height`, `openEnded`, `flatShaded`. `capsule`: `radius`, `height` (cylindrical shaft) or `totalHeight`. `torus`: `radius`, `tube` (tube radius), `arc`. `ring`: flat annulus (`innerRadius`, `outerRadius`). `prism` / `pyramid`: solid capped flat-shaded N-gon prism or pyramid (`sides`, `radius`, `height`). `polyhedron` (`type: "tetrahedron" \| "octahedron" \| "dodecahedron" \| "icosahedron"`), `icosahedron`, `icosphere` (`radius`, `detail`, `flatShaded`). `shard`: procedural flat-faceted polyhedron rock/crystal (`radius`, `cuts`, `irregularity`, `seed`). Surface: `color`, `tint`, `roughness`, `metallic`, `emissive`, `emissiveIntensity`, `material` (`"lit" \| "glass" \| "acrylic" \| "frosted"`), `castShadow`, `wireframe`. Shared GPU buffers cache identical shapes. |
| `Layer3D.extrude / lathe / tube / line / points(options)` | `"model3d"` | Custom & path-driven 3D geometry. `extrude`: extrude a 2D SVG path `d` (with holes) or `contour`/`holes` (`depth`, `bevelThickness`, `bevelSize`, `bevelSegments`, `bevelColor`). `lathe`: revolve a 2D `points` profile `[r, y][]` around the Y axis (`segments`, `phiStart`, `phiLength`, `closed`, `flatShaded`). `tube`: sweep a solid or thick-walled hollow pipe (`radius`, `innerRadius`, `radiusTaper`, `closed`, `capped`) along a `Path3D` or 3D control `points` using rotation-minimizing frames. `line`: 3D polyline (`points` or `Path3D`, `dashSize`, `gapSize`, per-vertex `colors`). `points`: 3D point cloud (`points`, `pointSize` for lit sphere dots or `0` for `point-list`, per-point `colors`). |
| `Layer3D.instances(options)` | `"model3d"` | Hardware-instanced single or multi-variant mesh rendering in 1 GPU draw call. Options: `count`, `mesh` / `meshes` (array of Layer3D primitives or meshes), `instanceAt(index, frame): { position, rotation, scale, color, emissive }`, or bind directly to Rapier `Physics.world3d` bodies via `bodies: Body3D[]` with `emissiveBySpeed` and `colorRamp`. |
| `Layer3D.swarm(options)` | `"model3d"` | Continuous 3D GPU particle swarm with compute-driven physics, 3D curl turbulence, vortex/cyclone field (`swirl`, `pull`, `axial`), ground plane collisions with Coulomb sliding friction and rotational roll torque, kinetic heat accumulation/cooling, and zero-copy packing directly into GPU instance buffers. |
@@ -271,7 +271,9 @@ const beatAnim2 = LayerAnimation.create()
object is carried on its track and registered with the program automatically (under a generated
`inline_signal_N` id, so the spec stays plain JSON); a string is looked up in `comp.addSignal` names.
Anything else (a number, `undefined`) throws. Signals are sampled at the **composition** frame
-(`ctx.frame`), not the layer's local frame.
+(`ctx.frame`) by default, which shared and audio signals need. Motion authored for one scene
+(`Signal.fromArray` of per-frame poses) passes `{ clock: "layer" }` to sample at the layer's own
+frame, counted from its start like keyframes, so it plays right inside `Layer.sequence`.
To pull signals out of audio, use `AudioSignal`, which runs on the GPU and accepts audio or video files.
`AudioSignal.extract(src, { fps })` gives `primary`, `beat`, `bass` and `energy` envelopes.
@@ -405,7 +407,9 @@ Where depth jumps (an object's outline) the surface stretches across with the ed
## Scenes & Transitions
-`Layer.sequence` places full-frame scenes end to end, overlapping each pair by its transition, and writes the scenes' `startFrame` / `durationFrames` for you (each scene keys from its own frame 0). Transitions are ordinary keyframes and mattes merged into the scenes:
+`Layer.sequence` places full-frame scenes end to end, overlapping each pair by its transition, and writes the scenes' `startFrame` / `durationFrames` for you (each scene keys from its own frame 0). Transitions are ordinary keyframes and mattes merged into the scenes.
+
+**3D scenes:** a box with a fixed `width`/`height` that holds a `Layer.camera` and the layers it films is a 3D scene of its own (a box holding only a camera stays a shot in a camera cut list over the shared world): its camera, lights and environment see only its layers, it renders on its own clock, and the box is placed like a picture. So a crossfade, whip or iris between two 3D scenes blends two finished pictures, each through its own camera. Give layer ids a per-scene prefix when the same scene code builds several scenes (`comp.lint()` reports `duplicate_id`).
```typescript
import { Layer, Transition } from "framefields";
diff --git a/plugins/framefields/skills/framefields-render/SKILL.md b/plugins/framefields/skills/framefields-render/SKILL.md
index 5c5927ee..940a6735 100644
--- a/plugins/framefields/skills/framefields-render/SKILL.md
+++ b/plugins/framefields/skills/framefields-render/SKILL.md
@@ -423,6 +423,7 @@ if (!result.qa!.passed) process.exitCode = 1; // fail CI / the agent loop
| `unknown_color` | warning | A color string that is not a color: it draws transparent |
| `low_contrast_text` | warning | Text below 3:1 contrast against the solid color behind it (suggests a readable one) |
| `offscreen_layer` | warning | A static absolute layer entirely outside the canvas |
+| `duplicate_id` | error | Two layers share an id: the renderer keeps one and drops the other |
| `shadow_unsupported_light` | warning | `castShadow` on a point or ambient light |
| `pbr_without_environment` | info | `roughness` / `metallic` set but no `Layer.environment()` (no physical shading) |
| `static_track` | info | Every keyframe on a track holds the same value |