Skip to content

Code-first video & image, rendered natively on WebGPU.

Work in progress across packages: renderer video QA, render diagnostics, loader/renderer3d fixes, launch example updates

okanasl2ceb93e17b2a
diff --git a/README.md b/README.md
index 011c115e..b3cda2f2 100644
--- a/README.md
+++ b/README.md
@@ -101,22 +101,6 @@ Developers generating video programmatically commonly weigh **Remotion** (React/
| **Headless Verification** | **FrameGrid contact sheets**, single-frame snapshots, Skia MSE pixel-invariant assertions | Playwright/Puppeteer visual snapshots | Manual frame inspection / canvas diffing |
| **Docker / Cloud Portability** | **Compact** (~500 MB slim image with native GPU/Vulkan drivers) | **Heavy** (~2–3 GB with Chromium, fonts, X11/Mesa) | Moderate container size |
-### Why Gitframes? Deep Architectural Analysis
-
-#### 1. Why not Remotion for high-performance pipelines?
-Remotion pioneered the "video in React" paradigm and is a great fit for marketing templates built from existing web UI. But because it is coupled to **headless Chromium**:
-- **Scalability ceiling** — Chromium is an interactive document browser, not a real-time compositor. Every frame pays DOM style recalculation, layout reflow, paint traversal, and cross-process buffer copies over CDP.
-- **GPU inefficiency** — WebGL inside headless Chromium is sandboxed and cannot use compute shaders, storage buffers, or native WebGPU multi-render targets (MRT) efficiently.
-- **Infrastructure cost** — Scaling in production needs high-memory CPU instances to survive Chromium OOM crashes.
-
-**Gitframes eliminates the browser entirely.** Running directly on the GPU via native WebGPU, it behaves like a game engine or DAW and delivers order-of-magnitude throughput gains.
-
-#### 2. Why not Hyperframes or Canvas2D?
-Canvas engines avoid DOM overhead but:
-- **CPU-bound rasterization** — `ctx.arc` / `ctx.bezierCurveTo` execute largely on the CPU; hundreds of animated type layers stall quickly.
-- **Shallow post-processing** — no multi-pass fragment pipelines, so pro grading (3D LUTs, selective color, bilateral SSAO, bokeh DoF) is effectively impossible.
-- **Disconnected audio** — Canvas frameworks treat audio as an external soundtrack; Gitframes unifies synthesis and animation through frame-accurate signals.
-
---
## Key Features & Capabilities
diff --git a/examples/22_gitframes_launch/package.json b/examples/22_gitframes_launch/package.json
index a647b528..381f826a 100644
--- a/examples/22_gitframes_launch/package.json
+++ b/examples/22_gitframes_launch/package.json
@@ -9,6 +9,7 @@
"images": "tsx src/generate-images.ts",
"precomp": "tsx src/precomp.ts",
"render": "tsx src/render.ts",
+ "render:src": "tsx --tsconfig tsconfig.dev.json --conditions=development src/render.ts",
"test": "vitest run",
"typecheck": "tsc --noEmit -p ."
},
diff --git a/examples/22_gitframes_launch/src/render.ts b/examples/22_gitframes_launch/src/render.ts
index d07cae14..16cfbb20 100644
--- a/examples/22_gitframes_launch/src/render.ts
+++ b/examples/22_gitframes_launch/src/render.ts
@@ -1,11 +1,12 @@
/**
- * pnpm render → output/gitframes-launch.mp4 (with the score)
+ * pnpm render → output/gitframes-launch.mp4 (with the score) + QA report; exits 1 if QA fails
* pnpm render frames 0 80 320 → output/frames/f0000.png … for visual checks
* pnpm render sheet 0 320 20 → every 20th frame from 0 to 320, plus a contact sheet
+ * pnpm render:src … → any of the above, run on the engine's source (no package builds)
*/
import fs from "node:fs/promises";
import path from "node:path";
-import { HeadlessMediaRenderer } from "gitframes";
+import { formatQaReport, HeadlessMediaRenderer } from "gitframes";
import { buildFilm } from "./film.js";
import { OUTPUT } from "./paths.js";
@@ -39,10 +40,15 @@ if (mode === "frames" || mode === "sheet") {
const result = await film.renderVideo({
outputPath: path.join(OUTPUT, "gitframes-launch.mp4"),
quality: "high",
+ qa: true,
});
await result.cleanup?.().catch(() => {});
console.log(
`${path.join(OUTPUT, "gitframes-launch.mp4")} (${((Date.now() - started) / 1000).toFixed(1)} s)`,
);
+ if (result.qa) {
+ console.log(formatQaReport(result.qa));
+ if (!result.qa.passed) process.exitCode = 1;
+ }
}
-process.exit(0);
+process.exit(process.exitCode ?? 0);
diff --git a/examples/22_gitframes_launch/src/scenes/logo.ts b/examples/22_gitframes_launch/src/scenes/logo.ts
index 2ea01a32..67219805 100644
--- a/examples/22_gitframes_launch/src/scenes/logo.ts
+++ b/examples/22_gitframes_launch/src/scenes/logo.ts
@@ -85,10 +85,8 @@ export function logoScene() {
fontFamily: DISPLAY,
fontWeight: 900,
fontSize: 210,
- fill: FG,
- bevelColor: ACCENT,
+ fill: ACCENT,
depth: 90,
- slices: 24,
x: CX,
y: CY,
z: 0,
diff --git a/examples/22_gitframes_launch/src/scenes/not.ts b/examples/22_gitframes_launch/src/scenes/not.ts
index 1672586a..fa33f847 100644
--- a/examples/22_gitframes_launch/src/scenes/not.ts
+++ b/examples/22_gitframes_launch/src/scenes/not.ts
@@ -53,7 +53,9 @@ function denials(): Denial[] {
function denial(d: Denial, i: number, from: number) {
const at = Math.min(d.no.at, d.thing.at) - from;
const until = d.until - from;
- const strikeAt = d.thing.at - from + BEAT;
+ // The bar lands a beat after the word, but never so late that its 8-frame
+ // sweep would run past the clip (the last word is followed quickly).
+ const strikeAt = Math.min(d.thing.at - from + BEAT, until - 9);
const step = until - at;
return [
word({
diff --git a/examples/22_gitframes_launch/src/scenes/outro.ts b/examples/22_gitframes_launch/src/scenes/outro.ts
index a324f56d..21906b04 100644
--- a/examples/22_gitframes_launch/src/scenes/outro.ts
+++ b/examples/22_gitframes_launch/src/scenes/outro.ts
@@ -103,10 +103,8 @@ function lockup() {
fontFamily: DISPLAY,
fontWeight: 900,
fontSize: 200,
- fill: FG,
- bevelColor: ACCENT,
+ fill: ACCENT,
depth: 80,
- slices: 24,
x: CX,
y: CY - 40,
z: 0,
diff --git a/examples/22_gitframes_launch/src/scenes/vfx.ts b/examples/22_gitframes_launch/src/scenes/vfx.ts
index f75048c9..c74576b7 100644
--- a/examples/22_gitframes_launch/src/scenes/vfx.ts
+++ b/examples/22_gitframes_launch/src/scenes/vfx.ts
@@ -388,6 +388,7 @@ function passShot(p: Pass, sceneFrom: number) {
id: `${p.id}-words`,
words: p.words,
from: p.verb.at,
+ until: len,
y: (H - 210) / 2,
size: 170,
color: p.id === "vfx-loud" ? ACCENT : FG,
diff --git a/examples/22_gitframes_launch/src/scenes/world.ts b/examples/22_gitframes_launch/src/scenes/world.ts
index d40b4455..67784eef 100644
--- a/examples/22_gitframes_launch/src/scenes/world.ts
+++ b/examples/22_gitframes_launch/src/scenes/world.ts
@@ -232,10 +232,8 @@ function depth(probeAt: number, lightAt: number, depthAt: number, len: number) {
fontFamily: DISPLAY,
fontWeight: 900,
fontSize: 300,
- fill: ACCENT,
- bevelColor: ACCENT_DEEP,
+ fill: ACCENT_DEEP,
depth: 140,
- slices: 28,
x: CX,
y: CY + 40,
z: DEPTH_Z,
diff --git a/examples/22_gitframes_launch/src/theme.ts b/examples/22_gitframes_launch/src/theme.ts
index 13911e64..1221838e 100644
--- a/examples/22_gitframes_launch/src/theme.ts
+++ b/examples/22_gitframes_launch/src/theme.ts
@@ -82,9 +82,7 @@ export function keys(
list: Key[],
anim: Anim = LayerAnimation.create(),
): Anim {
- for (const [frame, value, ease] of list)
- anim.keyframe(prop, Math.round(frame), value, ease);
- return anim;
+ return anim.keys(prop, list);
}
export interface SceneOptions {
@@ -280,6 +278,8 @@ export interface SungLineOptions {
justify?: "start" | "center" | "end";
/** Frame (scene-local) the whole line leaves. */
outAt?: number;
+ /** Frame (relative to `from`) the line's clip ends: entrances are shortened to finish by then. */
+ until?: number;
enter?: "pop" | "rise";
/** Stand the line in the 3D scene at this depth. */
z?: number;
@@ -302,18 +302,27 @@ export function sungLine(o: SungLineOptions): Node {
const at = Math.max(0, w.at - o.from);
const text = o.upper === false ? bare(w) : bare(w).toUpperCase();
const anim = LayerAnimation.create();
+ // A word sung just before the cut still lands at rest on its last frame.
+ const fit = (frames: number) =>
+ o.until === undefined
+ ? frames
+ : Math.max(1, Math.min(frames, o.until - 1 - at));
if ((o.enter ?? "pop") === "pop") {
anim
- .fromTo("scale", 1.35, 1, { start: at, end: at + 8, ease: EASE_OUT })
- .fromTo("opacity", 0, 1, { start: at, end: at + 2, ease: "none" });
+ .fromTo("scale", 1.35, 1, {
+ start: at,
+ end: at + fit(8),
+ ease: EASE_OUT,
+ })
+ .fromTo("opacity", 0, 1, { start: at, end: at + fit(2), ease: "none" });
} else {
anim
.fromTo("y", size * 0.45, 0, {
start: at,
- end: at + 10,
+ end: at + fit(10),
ease: EASE_OUT,
})
- .fromTo("opacity", 0, 1, { start: at, end: at + 3, ease: "none" });
+ .fromTo("opacity", 0, 1, { start: at, end: at + fit(3), ease: "none" });
}
return Layer.text(text, {
id: `${o.id}-${i}`,
diff --git a/examples/22_gitframes_launch/tsconfig.dev.json b/examples/22_gitframes_launch/tsconfig.dev.json
new file mode 100644
index 00000000..8c05246f
--- /dev/null
+++ b/examples/22_gitframes_launch/tsconfig.dev.json
@@ -0,0 +1,9 @@
+{
+ "extends": "./tsconfig.json",
+ "include": [
+ "src/**/*",
+ "../../nodes/**/*",
+ "../../packages/**/*",
+ "../../node_modules/@gitframes/**/*"
+ ]
+}
diff --git a/nodes/node-compositor/src/renderers/webgpu-renderer.ts b/nodes/node-compositor/src/renderers/webgpu-renderer.ts
index 8b472047..bca37a21 100644
--- a/nodes/node-compositor/src/renderers/webgpu-renderer.ts
+++ b/nodes/node-compositor/src/renderers/webgpu-renderer.ts
@@ -18,7 +18,6 @@ import type { WebGPUNodeRenderer } from "@gitframes/node-sdk";
import { measureText } from "@gitframes/renderers";
import { TemporalDeflickerPipeline } from "@gitframes/tensor-webgpu";
import {
- AudioLatentEngine,
AudioLatentTrackCache,
type BlendMode,
Camera3D,
@@ -204,16 +203,24 @@ function getHomographyResources(
}
const audioTrackLatentCache = new Map<string, AudioLatentTrackCache>();
-const deviceRenderer3DCache = new WeakMap<GPUDevice, Renderer3D>();
+const deviceRenderer3DCache = new WeakMap<GPUDevice, Map<string, Renderer3D>>();
+/** One Renderer3D per device, target format and MSAA sample count. */
function getRenderer3D(
device: GPUDevice,
format: GPUTextureFormat,
+ sampleCount: number,
): Renderer3D {
- let r3d = deviceRenderer3DCache.get(device);
- if (!r3d || r3d.format !== format) {
- r3d = new Renderer3D(device, format);
- deviceRenderer3DCache.set(device, r3d);
+ let byConfig = deviceRenderer3DCache.get(device);
+ if (!byConfig) {
+ byConfig = new Map();
+ deviceRenderer3DCache.set(device, byConfig);
+ }
+ const key = `${format}|${sampleCount}`;
+ let r3d = byConfig.get(key);
+ if (!r3d) {
+ r3d = new Renderer3D(device, format, sampleCount);
+ byConfig.set(key, r3d);
}
return r3d;
}
@@ -582,6 +589,8 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
if (op.op === "Compositor") {
const vop = op as CompositorOperation;
+ // 4x MSAA in the 3D pass unless the document opts out (antialias3d: false).
+ const sampleCount3d = vop.antialias3d === false ? 1 : 4;
const containerWidth = props.containerWidth ?? ctx.surface.width;
const containerHeight = props.containerHeight ?? ctx.surface.height;
@@ -622,7 +631,11 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
);
tl.seek(shiftedFrame / (fps ?? 24));
- const r3d = getRenderer3D(ctx.device, ctx.renderer.format ?? "rgba8unorm");
+ const r3d = getRenderer3D(
+ ctx.device,
+ ctx.renderer.format ?? "rgba8unorm",
+ sampleCount3d,
+ );
r3d.resetPools();
const relightPipe = getScreenSpaceRelightPipeline(
@@ -990,6 +1003,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
const r3d = getRenderer3D(
ctx.device,
ctx.renderer.format ?? "rgba8unorm",
+ sampleCount3d,
);
// Execute audio mesh deformation compute passes before beginning render pass
@@ -1253,11 +1267,16 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
const r3d = getRenderer3D(
ctx.device,
ctx.renderer.format ?? "rgba8unorm",
+ sampleCount3d,
);
let modelSrc = lop.modelData ?? lop.src;
if (!modelSrc && lop.text3dOptions) {
- const tOpts = lop.text3dOptions as ExtrudedTextGeometryOptions;
+ // The node's Layer3D.extrudedText options: geometry options plus a family name.
+ const tOpts = lop.text3dOptions as Omit<
+ ExtrudedTextGeometryOptions,
+ "font"
+ > & { fontFamily?: string };
const font =
(tOpts.fontFamily
? SlugFontCache.getParsed(tOpts.fontFamily)
@@ -1268,7 +1287,8 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
if (font) {
const res = generateExtrudedTextGeometry({
...tOpts,
- font,
+ // SlugFontCache holds fontkit fonts under a narrower type.
+ font: font as unknown as ExtrudedTextGeometryOptions["font"],
});
modelSrc = res.modelData;
lop.modelData = modelSrc;
@@ -2628,6 +2648,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
const r3dInstance = getRenderer3D(
ctx.device,
ctx.renderer.format ?? "rgba8unorm",
+ sampleCount3d,
);
const activeLights = synthesizeRelightLights(
@@ -2635,7 +2656,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
sceneLights,
rect,
shiftedFrame,
- fps,
+ fps ?? 24,
);
const lightsBuffer =
@@ -2668,7 +2689,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
relightOpts.roughness,
0.35,
shiftedFrame,
- fps,
+ fps ?? 24,
),
specularStrength: resolveVal(
target?.specularStrength ??
@@ -2676,7 +2697,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
relightOpts.specularStrength,
0.7,
shiftedFrame,
- fps,
+ fps ?? 24,
),
metallic: resolveVal(
target?.metallic ??
@@ -2684,7 +2705,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
relightOpts.metallic,
0.0,
shiftedFrame,
- fps,
+ fps ?? 24,
),
ambientIntensity: resolveVal(
target?.ambientIntensity ??
@@ -2692,7 +2713,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
relightOpts.ambientIntensity,
0.15,
shiftedFrame,
- fps,
+ fps ?? 24,
),
depthScale: resolveVal(
target?.depthScale ??
@@ -2700,7 +2721,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
relightOpts.depthScale,
1.0,
shiftedFrame,
- fps,
+ fps ?? 24,
),
opacity: 1.0,
},
@@ -3311,13 +3332,14 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
const r3dInstance = getRenderer3D(
ctx.device,
ctx.renderer.format ?? "rgba8unorm",
+ sampleCount3d,
);
const activeLights = synthesizeRelightLights(
relightOpts as Record<string, unknown>,
sceneLights,
{ x: sampledX, y: sampledY, width: texW, height: texH },
shiftedFrame,
- fps,
+ fps ?? 24,
);
const lightsBuffer = r3dInstance.getOrCreateLightsBuffer(activeLights);
@@ -3349,7 +3371,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
relightOpts.roughness,
0.35,
shiftedFrame,
- fps,
+ fps ?? 24,
),
specularStrength: resolveVal(
target?.specularStrength ??
@@ -3357,7 +3379,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
relightOpts.specularStrength,
0.7,
shiftedFrame,
- fps,
+ fps ?? 24,
),
metallic: resolveVal(
target?.metallic ??
@@ -3365,7 +3387,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
relightOpts.metallic,
0.0,
shiftedFrame,
- fps,
+ fps ?? 24,
),
ambientIntensity: resolveVal(
target?.ambientIntensity ??
@@ -3373,7 +3395,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
relightOpts.ambientIntensity,
0.15,
shiftedFrame,
- fps,
+ fps ?? 24,
),
depthScale: resolveVal(
target?.depthScale ??
@@ -3381,7 +3403,7 @@ export const CompositorWebGPURenderer: WebGPUNodeRenderer = async (args) => {
relightOpts.depthScale,
1.0,
shiftedFrame,
- fps,
+ fps ?? 24,
),
opacity: 1.0,
},
diff --git a/nodes/node-compositor/src/shared/compiler.test.ts b/nodes/node-compositor/src/shared/compiler.test.ts
index d37f584c..c011fda8 100644
--- a/nodes/node-compositor/src/shared/compiler.test.ts
+++ b/nodes/node-compositor/src/shared/compiler.test.ts
@@ -913,6 +913,21 @@ describe("keyframe tail truncation (review M8)", () => {
expect(warn).toHaveBeenCalledTimes(2);
});
+ it("warns once across single-frame renders of the same doc (frame-by-frame probes)", () => {
+ const warn = spyWarn();
+ const input = vmWithTrack(
+ [
+ { frame: 0, value: 0 },
+ { frame: 31, value: 100 },
+ ],
+ 24,
+ );
+ for (const id of ["img-probe-1", "img-probe-2", "img-probe-3"]) {
+ compileTimeline(id, input, { fps: 24, durationSec: 5 });
+ }
+ expect(warn).toHaveBeenCalledTimes(1);
+ });
+
it("compiles and evaluates text reveal animation tracks accurately", () => {
const virtualMedia = {
operation: { op: "Compositor", width: 1920, height: 1080 },
diff --git a/nodes/node-compositor/src/shared/compiler.ts b/nodes/node-compositor/src/shared/compiler.ts
index c5871896..d692699f 100644
--- a/nodes/node-compositor/src/shared/compiler.ts
+++ b/nodes/node-compositor/src/shared/compiler.ts
@@ -4,11 +4,12 @@ import {
collectNodeOps,
type EaseRef,
evaluateTrackAtFrame,
- type LayerAnimation,
+ type LayerAnimationSpec,
} from "@gitframes/compositions/program";
import {
DEFAULT_DURATION_MS,
getActiveMediaMetadata,
+ reportRenderDiagnostic,
type VirtualMediaData,
} from "@gitframes/core";
import gsapModule from "gsap";
@@ -462,7 +463,12 @@ export interface TruncationInfo {
maxAuthoredFrame: number;
}
-/** Dedupe per compiled cache key — a fresh render of the same broken doc SHOULD re-warn. */
+/**
+ * Dedupe per compiled cache key — a fresh render of the same broken doc SHOULD
+ * re-warn. Single-frame renders are the exception: each `renderFrame` call has
+ * its own `img-` render id, so probing a doc frame by frame would repeat the
+ * warning on every frame; those dedupe per document instead.
+ */
const truncationWarnedKeys = new Set<string>();
const TRUNCATION_WARNED_CAP = 100;
@@ -789,7 +795,20 @@ export function compileTimeline(
}
});
- if (truncations.length > 0) warnTruncationOnce(cacheKey, truncations);
+ if (truncations.length > 0) {
+ warnTruncationOnce(
+ renderId.startsWith("img-") ? `frame_${irHash}` : cacheKey,
+ truncations,
+ );
+ for (const t of truncations) {
+ reportRenderDiagnostic(renderId, {
+ severity: "warning",
+ code: "animation_truncated",
+ layerId: t.layerId,
+ message: `'${t.prop}' on '${t.layerId}': ${t.dropped} keyframe(s) past the clip window (last authored frame ${t.maxAuthoredFrame}) never play`,
+ });
+ }
+ }
tl.seek(1e-6);
tl.seek(0);
@@ -818,7 +837,7 @@ const layerTimelineCache = new Map<
export function compileLayerTimeline(
layerId: string,
- animation: LayerAnimation | undefined,
+ animation: LayerAnimationSpec | undefined,
baseVolume: number,
baseMuted: boolean,
fps: number,
diff --git a/packages/compositions/package.json b/packages/compositions/package.json
index be4f4ed8..0426f3d2 100644
--- a/packages/compositions/package.json
+++ b/packages/compositions/package.json
@@ -42,6 +42,7 @@
"@gitframes/server-utils": "workspace:*",
"@gitframes/tsconfig": "workspace:*",
"@types/react": "^19.2.4",
+ "@webgpu/types": "^0.1.70",
"dotenv": "^16.4.5",
"tsdown": "^0.19.0",
"tsx": "^4.20.3",
diff --git a/packages/compositions/src/program/animation.test.ts b/packages/compositions/src/program/animation.test.ts
index eea72839..de421029 100644
--- a/packages/compositions/src/program/animation.test.ts
+++ b/packages/compositions/src/program/animation.test.ts
@@ -127,3 +127,32 @@ describe("LayerAnimation builder & keyframe anchoring", () => {
expect(cf25.perspective).toBe(1200);
});
});
+
+describe("LayerAnimation.keys", () => {
+ it("adds a keyframe per tuple, rounding frames and parsing eases", () => {
+ const anim = LayerAnimation.keys("scale", [
+ [0, 0],
+ [11.6, 1.1, "back.out"],
+ [20, 1],
+ ]);
+ expect(anim.tracks).toHaveLength(1);
+ const [a, b, c] = anim.tracks[0]!.keyframes;
+ expect([a!.frame, b!.frame, c!.frame]).toEqual([0, 12, 20]);
+ expect(b!.value).toBe(1.1);
+ expect(b!.ease).toEqual({ name: "back", dir: "out" });
+ expect(a!.ease).toBeUndefined();
+ expect(evaluateAnimationAtFrame(anim, 20).scale).toBe(1);
+ });
+
+ it("chains onto an existing animation and its track for the same prop", () => {
+ const anim = LayerAnimation.create()
+ .keyframe("opacity", 0, 0)
+ .keys("opacity", [[10, 1]])
+ .keys("y", [
+ [0, 40],
+ [10, 0],
+ ]);
+ expect(anim.tracks.map((t) => t.prop)).toEqual(["opacity", "y"]);
+ expect(anim.tracks[0]!.keyframes.map((k) => k.frame)).toEqual([0, 10]);
+ });
+});
diff --git a/packages/compositions/src/program/animation.ts b/packages/compositions/src/program/animation.ts
index 06d0d0ad..d263b0df 100644
--- a/packages/compositions/src/program/animation.ts
+++ b/packages/compositions/src/program/animation.ts
@@ -6,6 +6,16 @@ import type {
TrackSource,
} from "./schema.js";
+/**
+ * One keyframe as a tuple: `[frame, value]` or `[frame, value, ease]`. The
+ * ease shapes the motion arriving at this keyframe from the one before.
+ */
+export type KeyframeTuple = readonly [
+ frame: number,
+ value: number | string | boolean,
+ ease?: string | EaseRef,
+];
+
export interface FromToOptions {
from?: number;
to?: number;
@@ -271,6 +281,14 @@ export class LayerAnimation implements LayerAnimationSpec {
return new LayerAnimation().fromTo(prop, fromValue, toValue, options);
}
+ /** A track from a list of `[frame, value, ease?]` keyframes; see {@link LayerAnimation.keys}. */
+ public static keys(
+ prop: AnimatableProp,
+ keyframes: readonly KeyframeTuple[],
+ ): LayerAnimation {
+ return new LayerAnimation().keys(prop, keyframes);
+ }
+
public static signal(
prop: AnimatableProp,
signalOrHandleId: unknown,
@@ -499,6 +517,18 @@ export class LayerAnimation implements LayerAnimationSpec {
return this;
}
+ /**
+ * Adds several keyframes to prop's track at once:
+ * `.keys("scale", [[0, 0], [12, 1.1, "back.out"], [20, 1]])`.
+ * Frames are rounded to whole frames, so beat math can be passed straight in.
+ */
+ public keys(prop: AnimatableProp, keyframes: readonly KeyframeTuple[]): this {
+ for (const [frame, value, ease] of keyframes) {
+ this.keyframe(prop, Math.round(frame), value, ease);
+ }
+ return this;
+ }
+
/**
* Animates opacity from 0 to 1 between startFrame and endFrame.
*/
diff --git a/packages/compositions/src/program/evaluator.test.ts b/packages/compositions/src/program/evaluator.test.ts
index 3f38432a..df04fac5 100644
--- a/packages/compositions/src/program/evaluator.test.ts
+++ b/packages/compositions/src/program/evaluator.test.ts
@@ -137,6 +137,7 @@ describe("Evaluator — Signal Binding", () => {
inputHandleId: "sine_osc",
multiplier: 1.0,
offset: 1.0,
+ smoothingWindowFrames: 0,
},
keyframes: [],
},
@@ -396,7 +397,9 @@ describe("Evaluator — Master evaluateTrackAtFrame & evaluateAnimationAtFrame",
inputHandleId: "beat_pulse",
multiplier: 0.5,
offset: 1.0,
+ smoothingWindowFrames: 0,
},
+ keyframes: [],
},
],
};
diff --git a/packages/compositions/src/program/schema.test.ts b/packages/compositions/src/program/schema.test.ts
index f6e78cac..7c14629f 100644
--- a/packages/compositions/src/program/schema.test.ts
+++ b/packages/compositions/src/program/schema.test.ts
@@ -2,6 +2,7 @@ import { describe, expect, it } from "vitest";
import {
CompositorProgramSchema,
EaseRefSchema,
+ type FlexNode,
type LayoutNode,
} from "./schema.js";
import {
@@ -87,7 +88,7 @@ describe("CompositorProgramSchema (document v2)", () => {
const doc = parsed.data;
expect(doc.width).toBe(1920);
expect(doc.layout).toHaveLength(1);
- const hero = doc.layout[0];
+ const hero = doc.layout[0] as FlexNode;
expect(hero.kind).toBe("flex");
expect(hero.position).toBe("relative"); // default
expect(hero.zIndex).toBe(0);
diff --git a/packages/compositions/src/program/schema.ts b/packages/compositions/src/program/schema.ts
index 482b53cf..b62861f8 100644
--- a/packages/compositions/src/program/schema.ts
+++ b/packages/compositions/src/program/schema.ts
@@ -981,7 +981,8 @@ export const CameraDoFSchema = z
})
.strict();
-export type CameraDoF = z.infer<typeof CameraDoFSchema>;
+/** Depth of field as authored: fields with schema defaults may be left out. */
+export type CameraDoF = z.input<typeof CameraDoFSchema>;
export const CameraShakeSchema = z
.object({
@@ -1555,7 +1556,8 @@ export const ChartGPUOptionsSchema = z
samples: z.union([z.literal(1), z.literal(4)]).default(4),
})
.strict();
-export type ChartGPUOptions = z.infer<typeof ChartGPUOptionsSchema>;
+/** Chart options as authored: fields with schema defaults may be left out. */
+export type ChartGPUOptions = z.input<typeof ChartGPUOptionsSchema>;
export const ChartNodeSchema: z.ZodType<ChartNode> = z
.object({
@@ -1631,6 +1633,11 @@ export const CompositorProgramSchema = z
volume: z.number().min(0).max(1).default(1),
fps: z.number().int().min(1).max(120).default(24),
mode: z.enum(["Video", "Image"]).default("Video"),
+ /**
+ * Multisample the 3D pass (4 samples per pixel) so mesh edges are smooth.
+ * Default true; false saves the extra target memory (~60-125 MB at 1080p).
+ */
+ antialias3d: z.boolean().optional(),
layout: z.array(LayoutNodeSchema).default([]),
fonts: z.array(z.string()).optional(),
signals: z.record(z.string(), z.unknown()).optional(),
diff --git a/packages/compositions/src/program/sequence.ts b/packages/compositions/src/program/sequence.ts
index 24b98fff..86404381 100644
--- a/packages/compositions/src/program/sequence.ts
+++ b/packages/compositions/src/program/sequence.ts
@@ -88,7 +88,7 @@ export const SceneDefinitionSchema = z
carrier: CarrierHandoffConfigSchema.optional(),
whipDirection: z.enum(["left", "right", "up", "down"]).default("left"),
})
- .default({ type: "cut", durationFrames: 0 }),
+ .default({ type: "cut", durationFrames: 0, whipDirection: "left" }),
})
.passthrough();
diff --git a/packages/compositions/src/program/text-animations/advanced-types.ts b/packages/compositions/src/program/text-animations/advanced-types.ts
index e75850cf..aeee67c2 100644
--- a/packages/compositions/src/program/text-animations/advanced-types.ts
+++ b/packages/compositions/src/program/text-animations/advanced-types.ts
@@ -340,6 +340,8 @@ export const ExtendedGlyphPropertiesSchema = z.object({
rotationX: z.number().optional(),
rotationY: z.number().optional(),
rotationZ: z.number().optional(),
+ /** Volumetric 3D arrangement (set by `volumetricFormation()`) */
+ formation: VolumetricFormationConfigSchema.optional(),
skew: z.number().optional(),
skewAxis: z.number().optional(),
diff --git a/packages/compositions/src/program/to-virtual-media.ts b/packages/compositions/src/program/to-virtual-media.ts
index 166caa15..de358868 100644
--- a/packages/compositions/src/program/to-virtual-media.ts
+++ b/packages/compositions/src/program/to-virtual-media.ts
@@ -202,7 +202,8 @@ function nodeToVirtualMedia(
// A container without its own timing keeps its parent's clock — its keyframes
// are authored parent-relative and its background shows before its first child
// arrives — and stays active until its last child ends.
- if (node.durationFrames === undefined) {
+ // Also when its own durationFrames is unusable (0, NaN): `dur` is unset then.
+ if (dur === undefined) {
const childEnds = children.map(
(c) =>
((c.operation as any)?.startFrame ?? 0) +
@@ -275,6 +276,9 @@ export function compositorToProgram(
fps,
mode: config.mode ?? "Video",
backgroundColor: config.backgroundColor,
+ ...(config.antialias3d !== undefined && {
+ antialias3d: config.antialias3d,
+ }),
// Master gain — consumed by the compositor's audio processor when
// the extractor dispatches the root op (audio-processor.ts).
volume: config.volume ?? 1,
diff --git a/packages/compositions/src/utils/audio-extractor.ts b/packages/compositions/src/utils/audio-extractor.ts
index aeb0e4c5..696d6a2d 100644
--- a/packages/compositions/src/utils/audio-extractor.ts
+++ b/packages/compositions/src/utils/audio-extractor.ts
@@ -9,9 +9,15 @@ import { AudioSampleSink } from "mediabunny";
import { computeRenderParams } from "./apply-operations.js";
import { normalizeTimeline } from "./normalization.js";
+// Progress chatter (every decode step) is debug output: shown only with LOG_LEVEL=debug or trace.
+const verbose =
+ typeof process !== "undefined" &&
+ /^(debug|trace)$/i.test(process.env?.LOG_LEVEL ?? "");
+
// Fallback logger for browser environment. In Node.js environment, we dynamically load @gitframes/server-utils to avoid bundling it on the frontend.
let mediaLogger = {
info: (msg: unknown, ...args: unknown[]) => {
+ if (!verbose) return;
if (typeof msg === "object" && msg !== null) {
console.info("[AudioExtractor]", msg, ...args);
} else {
diff --git a/packages/compositions/src/utils/layout/resolve-layout.ts b/packages/compositions/src/utils/layout/resolve-layout.ts
index 98a02948..54158186 100644
--- a/packages/compositions/src/utils/layout/resolve-layout.ts
+++ b/packages/compositions/src/utils/layout/resolve-layout.ts
@@ -25,7 +25,18 @@ import type { Yoga, Node as YogaNode } from "yoga-layout/load";
export type { Yoga };
import { loadYoga } from "yoga-layout/load";
-import type { LayoutNode, SizeSpec } from "../../program/schema.js";
+import type {
+ CompositionNodeBase,
+ LayoutNode,
+ SizeSpec,
+} from "../../program/schema.js";
+
+/**
+ * A node as layout reads it. Camera, light and model nodes carry only some
+ * of the base layout fields; the rest read as undefined, which layout handles.
+ */
+type LaidOutNode = LayoutNode & Partial<CompositionNodeBase>;
+type YogaPosition = number | `${number}%`;
export interface Rect {
x: number;
@@ -222,7 +233,8 @@ function containerStyle(node: LayoutNode): {
};
}
-function isAbsolutePositioned(node: LayoutNode): boolean {
+function isAbsolutePositioned(layoutNode: LayoutNode): boolean {
+ const node = layoutNode as LaidOutNode;
return (
node.position === "absolute" ||
(node.position === undefined &&
@@ -259,11 +271,12 @@ interface NodeMapping {
function buildYogaNode(
yoga: Yoga,
- node: LayoutNode,
+ layoutNode: LayoutNode,
parentExtent: { width?: number; height?: number } | undefined,
measure: MeasureFn | undefined,
parentDir: "row" | "column" | null,
): NodeMapping {
+ const node = layoutNode as LaidOutNode;
const yn = yoga.Node.create();
const dir = dirOf(node);
const isContainer = dir !== null;
@@ -276,8 +289,9 @@ function buildYogaNode(
yn.setPositionType(yoga.POSITION_TYPE_ABSOLUTE);
const posX = toYogaValue(node.x);
const posY = toYogaValue(node.y);
- if (posX !== undefined) yn.setPosition(yoga.EDGE_LEFT, posX);
- if (posY !== undefined) yn.setPosition(yoga.EDGE_TOP, posY);
+ if (posX !== undefined)
+ yn.setPosition(yoga.EDGE_LEFT, posX as YogaPosition);
+ if (posY !== undefined) yn.setPosition(yoga.EDGE_TOP, posY as YogaPosition);
} else {
yn.setPositionType(yoga.POSITION_TYPE_RELATIVE);
// Do not set EDGE_LEFT/EDGE_TOP offsets on the Yoga node itself;
@@ -572,8 +586,8 @@ export function computeLayoutSync(
const rootLayout = mapping.yogaNode.getComputedLayout();
const isRootAbsolute =
- root.position === "absolute" ||
- (root.position === undefined &&
+ (root as LaidOutNode).position === "absolute" ||
+ ((root as LaidOutNode).position === undefined &&
(root.x !== undefined || root.y !== undefined));
const rootX = isRootAbsolute ? rootLayout.left : Number(root.x ?? 0);
const rootY = isRootAbsolute ? rootLayout.top : Number(root.y ?? 0);
diff --git a/packages/core/package.json b/packages/core/package.json
index b325e2aa..e15b3184 100644
--- a/packages/core/package.json
+++ b/packages/core/package.json
@@ -29,8 +29,10 @@
},
"devDependencies": {
"@gitframes/tsconfig": "workspace:*",
+ "@webgpu/types": "^0.1.70",
"tsdown": "^0.19.0",
- "typescript": "^5.8.3"
+ "typescript": "^5.8.3",
+ "vitest": "^4.1.5"
},
"dependencies": {
"nanoid": "^5.1.6",
diff --git a/packages/core/src/index.ts b/packages/core/src/index.ts
index c052f8ea..6ca21cc8 100644
--- a/packages/core/src/index.ts
+++ b/packages/core/src/index.ts
@@ -2,6 +2,7 @@ export * from "./effects/index.js";
export * from "./errors.js";
export * from "./id-gen.js";
export * from "./node-metadata.js";
+export * from "./render-diagnostics.js";
export * from "./signals/index.js";
export * from "./types/index.js";
export * from "./utils/is-equal.js";
diff --git a/packages/core/src/render-diagnostics.ts b/packages/core/src/render-diagnostics.ts
new file mode 100644
index 00000000..9f54d49b
--- /dev/null
+++ b/packages/core/src/render-diagnostics.ts
@@ -0,0 +1,56 @@
+/**
+ * Problems a render noticed about its document (as opposed to its output),
+ * collected per render id so the renderer can put them in its QA report.
+ * Producers report; the renderer takes them when the render ends.
+ */
+export interface RenderDiagnostic {
+ readonly severity: "error" | "warning" | "info";
+ /** Stable machine-readable kind, e.g. "animation_truncated" */
+ readonly code: string;
+ readonly message: string;
+ readonly layerId?: string;
+}
+
+// Kept on globalThis: bundles can each carry their own copy of this module
+// (the compositor reports through one, the renderer collects through another),
+// and they must all see one store.
+const STORE_KEY = Symbol.for("gitframes.renderDiagnostics");
+const diagnosticsByRender: Map<string, Map<string, RenderDiagnostic>> =
+ ((globalThis as Record<symbol, unknown>)[STORE_KEY] as
+ | Map<string, Map<string, RenderDiagnostic>>
+ | undefined) ??
+ ((globalThis as Record<symbol, unknown>)[STORE_KEY] = new Map());
+// A render that never collects (a crashed or frame-only render) must not grow this forever.
+const MAX_TRACKED_RENDERS = 256;
+
+/** Record a diagnostic for a render; repeats (same code, layer and message) are kept once. */
+export function reportRenderDiagnostic(
+ renderId: string,
+ diagnostic: RenderDiagnostic,
+): void {
+ let forRender = diagnosticsByRender.get(renderId);
+ if (!forRender) {
+ if (diagnosticsByRender.size >= MAX_TRACKED_RENDERS) {
+ const oldest = diagnosticsByRender.keys().next().value;
+ if (oldest !== undefined) diagnosticsByRender.delete(oldest);
+ }
+ forRender = new Map();
+ diagnosticsByRender.set(renderId, forRender);
+ }
+ const key = `${diagnostic.code}|${diagnostic.layerId ?? ""}|${diagnostic.message}`;
+ if (!forRender.has(key)) forRender.set(key, diagnostic);
+}
+
+/**
+ * Everything reported for a render, including its nested compositions (they
+ * render under derived ids, `${renderId}-…`), removing it from the store.
+ */
+export function takeRenderDiagnostics(renderId: string): RenderDiagnostic[] {
+ const collected = new Map<string, RenderDiagnostic>();
+ for (const [id, forRender] of diagnosticsByRender) {
+ if (id !== renderId && !id.startsWith(`${renderId}-`)) continue;
+ for (const [key, diagnostic] of forRender) collected.set(key, diagnostic);
+ diagnosticsByRender.delete(id);
+ }
+ return [...collected.values()];
+}
diff --git a/packages/core/src/signals/programmatic.ts b/packages/core/src/signals/programmatic.ts
index 19565faf..3eb6f4ad 100644
--- a/packages/core/src/signals/programmatic.ts
+++ b/packages/core/src/signals/programmatic.ts
@@ -35,7 +35,9 @@ export class ProgrammaticSignal implements FrameSignal<number> {
private _cachedFrame = -1;
private _cachedValue = 0;
private _seekedFrame = -1;
- private _subscribers = new Set<{ markDirty(): void }>();
+ // Holds the computeds that read this signal (registerSignalDependency adds them).
+ private _subscribers: Parameters<typeof registerSignalDependency>[0] =
+ new Set();
private _children = new Set<ProgrammaticSignal>();
public get fps(): number {
diff --git a/packages/core/src/signals/signals.test.ts b/packages/core/src/signals/signals.test.ts
index f458aff6..15d796d4 100644
--- a/packages/core/src/signals/signals.test.ts
+++ b/packages/core/src/signals/signals.test.ts
@@ -110,6 +110,8 @@ describe("Signals Reactive Core", () => {
});
it("should support primitive valueOf and arithmetic coercion without NaN", () => {
+ // Frame-driven signals read the global clock; earlier tests leave it advanced.
+ updateClockSignals(0, 24);
const stateSig = signal(15);
const compSig = computed(() => stateSig.value * 2);
const arraySig = frameArraySignal([10, 20, 30], 24);
diff --git a/packages/core/src/types/tensor.ts b/packages/core/src/types/tensor.ts
index e69b7a20..23b7b77c 100644
--- a/packages/core/src/types/tensor.ts
+++ b/packages/core/src/types/tensor.ts
@@ -1,3 +1,4 @@
+/// <reference types="@webgpu/types" />
import type { Signal } from "../signals/index.js";
import type { NormalizedLandmarkList } from "./vision.js";
diff --git a/packages/gitframes/src/chartgpu-conformance.test.ts b/packages/gitframes/src/chartgpu-conformance.test.ts
index bdde8145..9aeccbbd 100644
--- a/packages/gitframes/src/chartgpu-conformance.test.ts
+++ b/packages/gitframes/src/chartgpu-conformance.test.ts
@@ -61,7 +61,8 @@ describe("ChartGPU Conformance Test Suite (Headless WebGPU & Compositor Integrat
height: 200,
});
- const testOptions: ChartGPUOptions = {
+ // The bridge takes chartgpu's own option type, not the document schema's.
+ const testOptions: Parameters<ChartGPUEngineBridge["initialize"]>[0] = {
theme: "dark",
series: [
{
diff --git a/packages/gitframes/src/font-manager.test.ts b/packages/gitframes/src/font-manager.test.ts
index cc30ac39..c95b9b60 100644
--- a/packages/gitframes/src/font-manager.test.ts
+++ b/packages/gitframes/src/font-manager.test.ts
@@ -1,39 +1,78 @@
+import { existsSync } from "node:fs";
import fs from "node:fs/promises";
+import os from "node:os";
import path from "node:path";
-import { beforeEach, describe, expect, it } from "vitest";
+import { afterAll, beforeAll, beforeEach, describe, expect, it } from "vitest";
import { Composition, FontManager, Layer } from "./index.js";
+// Fonts are kept on disk, not in git (.gitignore: *.ttf). Every test that
+// needs a real font uses the repo's Inter and is skipped where it is absent.
+const INTER = "assets/fonts/Inter.ttf";
+const hasInter = existsSync(
+ path.resolve(import.meta.dirname, "../../..", INTER),
+);
+if (!hasInter) {
+ console.warn(
+ `[font-manager.test] ${INTER} not found at the repo root; skipping tests that need a font file.`,
+ );
+}
+
describe("FontManager Singleton & Typography Binding", () => {
- beforeEach(() => {
- FontManager.clear();
+ // A second, distinct font file: a copy of Inter registered under its own family.
+ let secondFont = "";
+
+ beforeAll(async () => {
+ if (!hasInter) return;
+ const dir = await fs.mkdtemp(path.join(os.tmpdir(), "font-manager-"));
+ secondFont = path.join(dir, "SecondFace.ttf");
+ await fs.copyFile(
+ path.resolve(import.meta.dirname, "../../..", INTER),
+ secondFont,
+ );
});
- it("1. registers a valid TTF font and parses metadata", async () => {
- const font = await FontManager.register("assets/fonts/Inter.ttf");
- expect(font).toBeDefined();
- expect(font.family).toBe("Inter");
- expect(font.format).toBe("truetype");
- expect(font.unitsPerEm).toBe(2048);
- expect(font.filePath).toBeDefined();
-
- expect(FontManager.has("Inter")).toBe(true);
- expect(FontManager.get("Inter")?.family).toBe("Inter");
+ afterAll(async () => {
+ if (secondFont) {
+ await fs.rm(path.dirname(secondFont), { recursive: true, force: true });
+ }
});
- it("2. registers a font with custom family name and options object", async () => {
- const font = await FontManager.register({
- family: "CinzelDisplay",
- source: "assets/fonts/Cinzel.ttf",
- weight: 700,
- style: "normal",
- });
-
- expect(font.family).toBe("CinzelDisplay");
- expect(font.format).toBe("truetype");
- expect(font.weight).toBe(700);
- expect(FontManager.has("CinzelDisplay")).toBe(true);
+ beforeEach(() => {
+ FontManager.clear();
});
+ it.skipIf(!hasInter)(
+ "1. registers a valid TTF font and parses metadata",
+ async () => {
+ const font = await FontManager.register(INTER);
+ expect(font).toBeDefined();
+ expect(font.family).toBe("Inter");
+ expect(font.format).toBe("truetype");
+ expect(font.unitsPerEm).toBe(2048);
+ expect(font.filePath).toBeDefined();
+
+ expect(FontManager.has("Inter")).toBe(true);
+ expect(FontManager.get("Inter")?.family).toBe("Inter");
+ },
+ );
+
+ it.skipIf(!hasInter)(
+ "2. registers a font with custom family name and options object",
+ async () => {
+ const font = await FontManager.register({
+ family: "InterDisplay",
+ source: INTER,
+ weight: 700,
+ style: "normal",
+ });
+
+ expect(font.family).toBe("InterDisplay");
+ expect(font.format).toBe("truetype");
+ expect(font.weight).toBe(700);
+ expect(FontManager.has("InterDisplay")).toBe(true);
+ },
+ );
+
it("3. validates font binary and rejects non-font files", async () => {
const tmpBadFile = path.resolve(process.cwd(), "scratch_not_a_font.txt");
await fs.writeFile(
@@ -56,51 +95,65 @@ describe("FontManager Singleton & Typography Binding", () => {
);
});
- it("5. automatically provides registered font paths to Composition specs", async () => {
- await FontManager.register("assets/fonts/Inter.ttf");
- await FontManager.register("assets/fonts/SpaceGrotesk.ttf");
-
- const comp = new Composition({
- width: 640,
- height: 360,
- fps: 30,
- });
-
- comp.add(Layer.text("Hello World", { fontFamily: "Inter", fontSize: 32 }));
- comp.add(
- Layer.caption("captions.vtt", {
- fontFamily: "SpaceGrotesk",
- fontSize: 24,
- }),
- );
+ it.skipIf(!hasInter)(
+ "5. automatically provides registered font paths to Composition specs",
+ async () => {
+ await FontManager.register(INTER);
+ await FontManager.register({ family: "SecondFace", source: secondFont });
- const spec = comp.toSpec();
- expect(spec.fonts).toBeDefined();
- expect(spec.fonts?.length).toBeGreaterThanOrEqual(2);
- expect(spec.fonts?.some((p) => p.includes("Inter.ttf"))).toBe(true);
- expect(spec.fonts?.some((p) => p.includes("SpaceGrotesk.ttf"))).toBe(true);
- });
-
- it("6. comp.registerFont delegates directly to FontManager", async () => {
- const comp = new Composition({
- width: 640,
- height: 360,
- });
-
- await comp.registerFont("MontserratCustom", "assets/fonts/Montserrat.ttf");
- expect(FontManager.has("MontserratCustom")).toBe(true);
-
- const spec = comp.toSpec();
- expect(spec.fonts?.some((p) => p.includes("Montserrat.ttf"))).toBe(true);
- });
+ const comp = new Composition({
+ width: 640,
+ height: 360,
+ fps: 30,
+ });
- it("7. Layer.text and Layer.caption automatically use registered font family when omitted", async () => {
- const font = await FontManager.register("assets/fonts/SpaceGrotesk.ttf");
-
- const textLayer = Layer.text("Auto Font Text");
- const captionLayer = Layer.caption("subtitles.vtt");
+ comp.add(
+ Layer.text("Hello World", { fontFamily: "Inter", fontSize: 32 }),
+ );
+ comp.add(
+ Layer.caption("captions.vtt", {
+ fontFamily: "SecondFace",
+ fontSize: 24,
+ }),
+ );
- expect(textLayer.fontFamily).toBe(font.family);
- expect(captionLayer.fontFamily).toBe(font.family);
- });
+ const spec = comp.toSpec();
+ expect(spec.fonts).toBeDefined();
+ expect(spec.fonts?.length).toBeGreaterThanOrEqual(2);
+ expect(spec.fonts?.some((p) => p.includes("Inter.ttf"))).toBe(true);
+ expect(spec.fonts?.some((p) => p.includes("SecondFace.ttf"))).toBe(true);
+ },
+ );
+
+ it.skipIf(!hasInter)(
+ "6. comp.registerFont delegates directly to FontManager",
+ async () => {
+ const comp = new Composition({
+ width: 640,
+ height: 360,
+ });
+
+ await comp.registerFont("InterCustom", INTER);
+ expect(FontManager.has("InterCustom")).toBe(true);
+
+ const spec = comp.toSpec();
+ expect(spec.fonts?.some((p) => p.includes("Inter.ttf"))).toBe(true);
+ },
+ );
+
+ it.skipIf(!hasInter)(
+ "7. Layer.text and Layer.caption automatically use registered font family when omitted",
+ async () => {
+ const font = await FontManager.register({
+ family: "SecondFace",
+ source: secondFont,
+ });
+
+ const textLayer = Layer.text("Auto Font Text");
+ const captionLayer = Layer.caption("subtitles.vtt");
+
+ expect(textLayer.fontFamily).toBe(font.family);
+ expect(captionLayer.fontFamily).toBe(font.family);
+ },
+ );
});
diff --git a/packages/gitframes/src/fonts/manager.ts b/packages/gitframes/src/fonts/manager.ts
index f6316ad4..13eb8341 100644
--- a/packages/gitframes/src/fonts/manager.ts
+++ b/packages/gitframes/src/fonts/manager.ts
@@ -302,7 +302,7 @@ export class FontManager {
if (filePath) {
await registerHeadlessFont(resolvedFamily, filePath);
}
- SlugFontCache.parsedFontCache.set(resolvedFamily, parsedFontkit);
+ SlugFontCache.setParsed(resolvedFamily, parsedFontkit);
return registered;
}
diff --git a/packages/gitframes/src/http-assets.test.ts b/packages/gitframes/src/http-assets.test.ts
index 4af43895..cc9454a3 100644
--- a/packages/gitframes/src/http-assets.test.ts
+++ b/packages/gitframes/src/http-assets.test.ts
@@ -214,7 +214,7 @@ describe("Web HTTP Asset Loading Pipeline", () => {
Layer.text("Cloud Typography", {
fontFamily: "CloudInter",
fontSize: 48,
- color: "#ffffff",
+ fill: "#ffffff",
}),
);
diff --git a/packages/gitframes/src/ids.ts b/packages/gitframes/src/ids.ts
new file mode 100644
index 00000000..b1949938
--- /dev/null
+++ b/packages/gitframes/src/ids.ts
@@ -0,0 +1,12 @@
+/**
+ * Default ids for nodes created without one. A counter per prefix rather than
+ * a random suffix: the same composition script builds the same ids on every
+ * run and in every render worker, so specs, caches and diffs stay stable.
+ */
+const counters = new Map<string, number>();
+
+export function autoId(prefix: string): string {
+ const n = (counters.get(prefix) ?? 0) + 1;
+ counters.set(prefix, n);
+ return `${prefix}-auto-${n}`;
+}
diff --git a/packages/gitframes/src/index.ts b/packages/gitframes/src/index.ts
index ac19c6a6..499d5e3e 100644
--- a/packages/gitframes/src/index.ts
+++ b/packages/gitframes/src/index.ts
@@ -20,6 +20,7 @@ import {
type RegisteredFont,
type RegisterFontOptions,
} from "./fonts/index.js";
+import { autoId } from "./ids.js";
import { normalizeTextSpans, type TextSpan } from "./rich-text.js";
export {
@@ -149,6 +150,7 @@ import {
import {
analyzeSequence as analyzeVisionFrames,
type LandmarkCoordinateSignals,
+ type MaskTrackSignals,
type ObjectTrackSignals,
type PinToLandmarkOptions,
type PinToObjectOptions,
@@ -227,6 +229,8 @@ import {
type FrameGridOptions,
HeadlessMediaRenderer,
renderFrameGrid,
+ type VideoQaOptions,
+ type VideoQaReport,
} from "./renderer/index.js";
import {
type Carousel3DOptions,
@@ -260,14 +264,31 @@ export interface SectionOptions {
export type SectionNode = AnimatableNode<BoxNode> & {
readonly innerMedia: AnimatableNode<MediaNode>;
- readonly effects: Effect[];
- addEffect(effect: Effect): SectionNode;
- withEffect(effect: Effect): SectionNode;
- withEffects(effects: Effect[]): SectionNode;
+ // Any effect, whatever its config shape (Blur, FilmGrain, …).
+ readonly effects: Effect<object>[];
+ addEffect(effect: Effect<object>): SectionNode;
+ withEffect(effect: Effect<object>): SectionNode;
+ withEffects(effects: Effect<object>[]): SectionNode;
};
export type TrackedRegionOptions = SectionOptions;
+/** Editing helpers attached to every tracked object (see decorateTrackSignals). */
+export interface TrackEditing {
+ /** A section of the source that follows this track */
+ section(
+ sourceOrOptions?: string | MediaNode | SectionOptions,
+ options?: SectionOptions,
+ ): SectionNode;
+ /** A blur confined to this track's region */
+ blurEffect(options?: Partial<BlurProps>): Blur;
+ /** The tracked region of `source`, cut out as a section */
+ isolate(source: string | MediaNode, options?: TrackedRegionOptions): SectionNode;
+}
+
+/** A tracked object with its editing helpers, as vision hands it out. */
+export type EditableObjectTrack = ObjectTrackSignals & TrackEditing;
+
function decorateTrackSignals<T extends ObjectTrackSignals>(
track: T,
source?: string | MediaNode,
@@ -471,6 +492,7 @@ export {
getScramblePool,
getShuffledOrder,
type Keyframe,
+ type KeyframeTuple,
type KineticMarqueeConfig,
KineticMarqueeConfigSchema,
LayerAnimation,
@@ -703,6 +725,11 @@ export interface CompositionOptions {
backgroundColor?: string;
fonts?: string[];
signals?: Record<string, unknown>;
+ /**
+ * Smooth the edges of 3D meshes (extruded text, models) with 4x MSAA.
+ * Default true; set false to save ~60-125 MB of GPU memory at 1080p.
+ */
+ antialias3d?: boolean;
}
/**
@@ -819,25 +846,9 @@ export class Media {
* Renders this media pipeline into an MP4/WebM video file with audio.
*/
public async renderVideo(
- outputPathOrOptions:
- | string
- | {
- codec?: string;
- audioCodec?: string;
- quality?: string;
- concurrency?: number;
- outputPath?: string;
- renderer?: HeadlessMediaRenderer;
- } = {},
- options: {
- codec?: string;
- audioCodec?: string;
- quality?: string;
- concurrency?: number;
- outputPath?: string;
- renderer?: HeadlessMediaRenderer;
- } = {},
- ): Promise<{ filePath: string; cleanup: () => Promise<void> }> {
+ outputPathOrOptions: string | RenderVideoOptions = {},
+ options: RenderVideoOptions = {},
+ ): Promise<RenderVideoResult> {
const opts =
typeof outputPathOrOptions === "string"
? { ...options, outputPath: outputPathOrOptions }
@@ -852,6 +863,29 @@ export class Media {
}
}
+export interface RenderVideoOptions {
+ codec?: string;
+ audioCodec?: string;
+ quality?: string;
+ concurrency?: number;
+ outputPath?: string;
+ renderer?: HeadlessMediaRenderer;
+ /**
+ * Check the output while it renders and return the findings as `qa`:
+ * dropped, black or frozen frames, loudness (LUFS), clipping, silence and
+ * document warnings such as truncated animations. `true` uses defaults;
+ * pass thresholds (e.g. `{ targetLufs: -14 }`) to tighten them.
+ */
+ qa?: boolean | VideoQaOptions;
+}
+
+export interface RenderVideoResult {
+ filePath: string;
+ cleanup: () => Promise<void>;
+ /** Present when rendered with `qa`; print it with `formatQaReport` */
+ qa?: VideoQaReport;
+}
+
export interface SubjectSandwichOptions {
readonly source: string | MediaNode;
readonly subject?: MaskTrackSignals | ObjectTrackSignals;
@@ -898,6 +932,7 @@ export class Composition {
public fps: number;
public durationMs?: number;
public backgroundColor: string;
+ public antialias3d?: boolean;
public fonts: string[];
public children: LayoutNode[] = [];
public audioTracks: MediaNode[] = [];
@@ -1046,6 +1081,7 @@ export class Composition {
? (options.durationFrames / this.fps) * 1000
: undefined);
this.backgroundColor = options.backgroundColor ?? "#000000";
+ this.antialias3d = options.antialias3d;
this.fonts = options.fonts ?? [];
this.signals = options.signals ?? {};
}
@@ -1148,7 +1184,7 @@ export class Composition {
const src =
typeof options.source === "string"
? options.source
- : (options.source.src ?? "");
+ : (options.source.inputHandleId ?? "");
const fit = options.fit ?? "cover";
const feather = options.feather ?? 4;
const featherVal = feather > 1 ? feather / 100 : feather;
@@ -1205,7 +1241,7 @@ export class Composition {
const src =
typeof options.source === "string"
? options.source
- : (options.source.src ?? "");
+ : (options.source.inputHandleId ?? "");
const damping = Math.max(0, Math.min(0.99, options.damping ?? 0.8));
const headroom = options.leadHeadroom ?? 0.18;
const compW = this.width;
@@ -1325,7 +1361,7 @@ export class Composition {
volume: options.muted ? 0 : (options.volume ?? 1),
});
- container.children.push(videoPlate);
+ (container.children ??= []).push(videoPlate);
this.add(container);
return this;
}
@@ -1353,7 +1389,7 @@ export class Composition {
const src =
typeof options.source === "string"
? options.source
- : (options.source?.src ?? "");
+ : (options.source?.inputHandleId ?? "");
const outlineLayer = Layer.box({
width: compW,
@@ -1378,7 +1414,7 @@ export class Composition {
mode: "mask",
enableSegmentation: true,
});
- outlineLayer.children.push(maskVideo);
+ (outlineLayer.children ??= []).push(maskVideo);
}
outlineLayer.apply(
@@ -1547,6 +1583,7 @@ export class Composition {
fps: this.fps,
mode: "Video",
backgroundColor: this.backgroundColor,
+ ...(this.antialias3d !== undefined && { antialias3d: this.antialias3d }),
layout: layoutItems,
fonts: mergedFonts,
signals: this.signals,
@@ -1652,25 +1689,9 @@ export class Composition {
* Renders this composition into an MP4/WebM video file with audio.
*/
public async renderVideo(
- outputPathOrOptions:
- | string
- | {
- codec?: string;
- audioCodec?: string;
- quality?: string;
- concurrency?: number;
- outputPath?: string;
- renderer?: HeadlessMediaRenderer;
- } = {},
- options: {
- codec?: string;
- audioCodec?: string;
- quality?: string;
- concurrency?: number;
- outputPath?: string;
- renderer?: HeadlessMediaRenderer;
- } = {},
- ): Promise<{ filePath: string; cleanup: () => Promise<void> }> {
+ outputPathOrOptions: string | RenderVideoOptions = {},
+ options: RenderVideoOptions = {},
+ ): Promise<RenderVideoResult> {
const opts =
typeof outputPathOrOptions === "string"
? { ...options, outputPath: outputPathOrOptions }
@@ -2039,7 +2060,7 @@ export const Layer = {
options: Partial<MediaNode> = {},
): AnimatableNode<MediaNode> =>
withAnimation({
- id: options.id ?? "video-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("video"),
kind: "media",
inputHandleId: src,
dataType: "Video",
@@ -2051,7 +2072,7 @@ export const Layer = {
options: Partial<MediaNode> = {},
): AnimatableNode<MediaNode> =>
withAnimation({
- id: options.id ?? "img-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("img"),
kind: "media",
inputHandleId: src,
dataType: "Image",
@@ -2063,7 +2084,7 @@ export const Layer = {
options: Partial<MediaNode> = {},
): AnimatableNode<MediaNode> =>
withAnimation({
- id: options.id ?? "svg-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("svg"),
kind: "media",
inputHandleId: src,
dataType: "SVG",
@@ -2075,7 +2096,7 @@ export const Layer = {
options: Partial<MediaNode> = {},
): AnimatableNode<MediaNode> =>
withAnimation({
- id: options.id ?? "lottie-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("lottie"),
kind: "media",
inputHandleId: src,
dataType: "Lottie",
@@ -2087,7 +2108,7 @@ export const Layer = {
options: Partial<MediaNode> = {},
): AnimatableNode<MediaNode> =>
withAnimation({
- id: options.id ?? "audio-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("audio"),
kind: "media",
inputHandleId: src,
dataType: "Audio",
@@ -2135,7 +2156,7 @@ export const Layer = {
}
return withAnimation({
- id: options.id ?? "text-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("text"),
kind: "text",
fontFamily:
options.fontFamily ?? FontManager.getAll()[0]?.family ?? "Inter",
@@ -2151,7 +2172,7 @@ export const Layer = {
options: Partial<MediaNode> = {},
): AnimatableNode<MediaNode> =>
withAnimation({
- id: options.id ?? "caption-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("caption"),
kind: "media",
inputHandleId: src,
dataType: options.dataType ?? "Caption",
@@ -2173,7 +2194,7 @@ export const Layer = {
options: Partial<ShapeNode> = {},
): AnimatableNode<ShapeNode> =>
withAnimation({
- id: options.id ?? "shape-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("shape"),
kind: "shape",
shapeType: options.shapeType ?? shapeType,
fillColor: options.fillColor ?? "#3b82f6",
@@ -2181,7 +2202,7 @@ export const Layer = {
}),
flex: (options: Partial<FlexNode> = {}): AnimatableNode<FlexNode> =>
withAnimation({
- id: options.id ?? "flex-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("flex"),
kind: "flex",
dir: options.dir ?? "column",
children: options.children ?? [],
@@ -2189,7 +2210,7 @@ export const Layer = {
}),
box: (options: Partial<BoxNode> = {}): AnimatableNode<BoxNode> =>
withAnimation({
- id: options.id ?? "box-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("box"),
kind: "box",
background: options.background ?? "transparent",
children: options.children ?? [],
@@ -2312,9 +2333,7 @@ export const Layer = {
const boxId =
opts.id ??
- (target
- ? `section-track-${target.trackId}-${Math.random().toString(36).slice(2, 8)}`
- : `section-${Math.random().toString(36).slice(2, 8)}`);
+ (target ? autoId(`section-track-${target.trackId}`) : autoId("section"));
let box = Layer.box({
id: boxId,
@@ -2417,14 +2436,14 @@ export const Layer = {
props: Partial<ChartNode> = {},
): AnimatableNode<ChartNode> =>
withAnimation({
- id: props.id ?? "chart-" + Math.random().toString(36).slice(2, 9),
+ id: props.id ?? autoId("chart"),
kind: "chart",
chartOptions: options,
...props,
}),
camera: (options: Partial<CameraNode> = {}): AnimatableNode<CameraNode> =>
withAnimation<CameraNode>({
- id: options.id ?? "camera-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("camera"),
kind: "camera",
mode: options.mode ?? "lookAt",
pitch: options.pitch ?? 0,
@@ -2434,7 +2453,7 @@ export const Layer = {
}),
light: (options: Partial<LightNode> = {}): AnimatableNode<LightNode> =>
withAnimation<LightNode>({
- id: options.id ?? "light-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("light"),
kind: "light",
lightType: options.lightType ?? "point",
color: options.color ?? "#ffffff",
@@ -2463,7 +2482,7 @@ export const Layer = {
Layer3D.prism(options),
model: (options: Partial<Model3DNode> = {}): AnimatableNode<Model3DNode> =>
withAnimation({
- id: options.id ?? "model3d-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("model3d"),
kind: "model3d",
is3D: options.is3D ?? true,
modelFormat: options.modelFormat ?? "auto",
@@ -2567,8 +2586,7 @@ export const Light = {
options: Partial<LightNode> = {},
): AnimatableNode<LightNode> =>
withAnimation({
- id:
- options.id ?? "light-ambient-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("light-ambient"),
kind: "light",
lightType: "ambient",
color,
@@ -2577,9 +2595,7 @@ export const Light = {
}),
directional: (options: Partial<LightNode> = {}): AnimatableNode<LightNode> =>
withAnimation({
- id:
- options.id ??
- "light-directional-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("light-directional"),
kind: "light",
lightType: "directional",
color: options.color ?? "#ffffff",
@@ -2594,7 +2610,7 @@ export const Light = {
}),
point: (options: Partial<LightNode> = {}): AnimatableNode<LightNode> =>
withAnimation({
- id: options.id ?? "light-point-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("light-point"),
kind: "light",
lightType: "point",
color: options.color ?? "#ffffff",
@@ -2608,7 +2624,7 @@ export const Light = {
}),
spot: (options: Partial<LightNode> = {}): AnimatableNode<LightNode> =>
withAnimation({
- id: options.id ?? "light-spot-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("light-spot"),
kind: "light",
lightType: "spot",
color: options.color ?? "#ffffff",
@@ -2632,11 +2648,18 @@ export {
type FrameGridOptions,
type FrameGridRenderable,
type FrameSamplePoint,
+ formatQaReport,
HeadlessMediaRenderer,
HeadlessWebGPURenderer,
+ type QaIssue,
+ type QaSegment,
renderFrameGrid,
renderSemaphore,
resolveSamplePoints,
+ type AudioQaStats,
+ type VideoQaOptions,
+ type VideoQaReport,
+ type VideoQaStats,
} from "./renderer/index.js";
export function section(
diff --git a/packages/gitframes/src/lights-3d-render.test.ts b/packages/gitframes/src/lights-3d-render.test.ts
index 506bee14..c4213b54 100644
--- a/packages/gitframes/src/lights-3d-render.test.ts
+++ b/packages/gitframes/src/lights-3d-render.test.ts
@@ -1,6 +1,6 @@
import { Canvas, loadImage } from "skia-canvas";
import { describe, expect, it } from "vitest";
-import { Composition, Layer, LayerAnimation, Light } from "./index.js";
+import { Composition, Layer, LayerAnimation } from "./index.js";
interface DecodedImage {
width: number;
diff --git a/packages/gitframes/src/model3d-render.test.ts b/packages/gitframes/src/model3d-render.test.ts
index dddd6ecc..a6556b55 100644
--- a/packages/gitframes/src/model3d-render.test.ts
+++ b/packages/gitframes/src/model3d-render.test.ts
@@ -1,7 +1,7 @@
import fs from "node:fs/promises";
import os from "node:os";
import path from "node:path";
-import { Signal, signal } from "@gitframes/core";
+import { Signal } from "@gitframes/core";
import { Canvas, loadImage } from "skia-canvas";
import { describe, expect, it } from "vitest";
import { Composition, Layer, Layer3D, LayerAnimation, Light } from "./index.js";
diff --git a/packages/gitframes/src/render-conformance.test.ts b/packages/gitframes/src/render-conformance.test.ts
index 1df57c3a..af09714f 100644
--- a/packages/gitframes/src/render-conformance.test.ts
+++ b/packages/gitframes/src/render-conformance.test.ts
@@ -1,4 +1,3 @@
-import fs from "node:fs/promises";
import type { VirtualMediaData } from "@gitframes/core";
import { Canvas, loadImage } from "skia-canvas";
import { describe, expect, it } from "vitest";
diff --git a/packages/gitframes/src/renderer/index.ts b/packages/gitframes/src/renderer/index.ts
index fbd89382..fa53f35d 100644
--- a/packages/gitframes/src/renderer/index.ts
+++ b/packages/gitframes/src/renderer/index.ts
@@ -1,7 +1,14 @@
export {
+ type AudioQaStats,
+ formatQaReport,
HeadlessMediaRenderer,
HeadlessWebGPURenderer,
+ type QaIssue,
+ type QaSegment,
renderSemaphore,
+ type VideoQaOptions,
+ type VideoQaReport,
+ type VideoQaStats,
} from "@gitframes/renderer";
export { initHeadlessWebGPU } from "@gitframes/webgpu-renderers";
diff --git a/packages/gitframes/src/scene-3d-regressions.test.ts b/packages/gitframes/src/scene-3d-regressions.test.ts
index 5196db8a..db41deeb 100644
--- a/packages/gitframes/src/scene-3d-regressions.test.ts
+++ b/packages/gitframes/src/scene-3d-regressions.test.ts
@@ -3,7 +3,13 @@ import os from "node:os";
import path from "node:path";
import { Canvas, loadImage } from "skia-canvas";
import { describe, expect, it } from "vitest";
-import { Composition, GradientMap, Layer, Layer3D } from "./index.js";
+import {
+ type BoxNode,
+ Composition,
+ GradientMap,
+ Layer,
+ Layer3D,
+} from "./index.js";
const W = 800;
const H = 600;
@@ -34,7 +40,7 @@ describe("3D scene regressions", () => {
divisions: 10,
lineWidth: 4,
});
- const lines = grid.children ?? [];
+ const lines = (grid.children ?? []) as BoxNode[];
// 11 lines each way; long lines are split into segments of at most 4096 px.
expect(lines.length).toBe(
11 * Math.ceil(14000 / 4096) + 11 * Math.ceil(9000 / 4096),
@@ -52,7 +58,7 @@ describe("3D scene regressions", () => {
Layer.box({ id: `c${i}`, width: 100, height: 60 }),
);
const ring = Layer3D.carousel({ radius: 500, items });
- const [front, right, back, left] = ring.items ?? [];
+ const [front, right, back, left] = (ring.items ?? []) as BoxNode[];
// The default camera looks down +z from -z: the front item sits at -radius, unrotated.
expect(front.z).toBeCloseTo(-500);
expect(front.rotateY).toBeCloseTo(0);
diff --git a/packages/gitframes/src/screen-space-relight.test.ts b/packages/gitframes/src/screen-space-relight.test.ts
index cc6fdc16..de96b7fa 100644
--- a/packages/gitframes/src/screen-space-relight.test.ts
+++ b/packages/gitframes/src/screen-space-relight.test.ts
@@ -165,8 +165,10 @@ describe("Screen-Space 3D Normal Relighting Conformance", () => {
z: 50,
radius: 200,
decay: 1.0,
+ // Sweep across over the first second (30 fps).
animation: LayerAnimation.create().fromTo("x", 80, 320, {
- duration: 1.0,
+ start: 0,
+ end: 30,
}),
}),
);
diff --git a/packages/gitframes/src/section-effects.test.ts b/packages/gitframes/src/section-effects.test.ts
index ed7902a6..c0401153 100644
--- a/packages/gitframes/src/section-effects.test.ts
+++ b/packages/gitframes/src/section-effects.test.ts
@@ -72,7 +72,7 @@ describe("Unified Composition & Layer Section Architecture Suite", () => {
borderRadius: 16,
borderColor: "#38bdf8",
borderWidth: 2,
- effects: [new Blur({ strength: 24 }), new FilmGrain({ intensity: 0.35 })],
+ effects: [new Blur({ strength: 24 }), new FilmGrain({ strength: 35 })],
});
expect(section.kind).toBe("box");
@@ -148,8 +148,8 @@ describe("Unified Composition & Layer Section Architecture Suite", () => {
height: 400,
borderRadius: 20,
effects: [
- new Vignette({ radius: 0.7, amount: 0.8 }),
- new FilmGrain({ intensity: 0.4 }),
+ new Vignette({ radius: 0.7, strength: 80 }),
+ new FilmGrain({ strength: 40 }),
],
});
@@ -226,7 +226,7 @@ describe("Unified Composition & Layer Section Architecture Suite", () => {
// Chain multiple effects
section.withEffects([
- new FilmGrain({ intensity: 0.2 }),
+ new FilmGrain({ strength: 20 }),
new Vignette({ radius: 0.9 }),
]);
expect(section.effects).toHaveLength(3);
@@ -246,7 +246,7 @@ describe("Unified Composition & Layer Section Architecture Suite", () => {
comp.add(video);
const sec = comp.addSection(targetPerson, {
- effects: [new Blur({ strength: 20 }), new FilmGrain({ intensity: 0.3 })],
+ effects: [new Blur({ strength: 20 }), new FilmGrain({ strength: 30 })],
});
const vm = comp.toVirtualMedia();
@@ -299,7 +299,7 @@ describe("Unified Composition & Layer Section Architecture Suite", () => {
});
expect(sec).toBeDefined();
- const png = await comp.renderFrame(0);
+ const png = await comp.renderFrame({ frame: 0 });
expect(Buffer.isBuffer(png)).toBe(true);
expect(png.length).toBeGreaterThan(100);
// Validate PNG magic bytes
diff --git a/packages/gitframes/src/shapes-3d-render.test.ts b/packages/gitframes/src/shapes-3d-render.test.ts
index dd2aaad9..0652990a 100644
--- a/packages/gitframes/src/shapes-3d-render.test.ts
+++ b/packages/gitframes/src/shapes-3d-render.test.ts
@@ -1,7 +1,6 @@
import { Canvas, loadImage } from "skia-canvas";
import { describe, expect, it } from "vitest";
import {
- CameraAnimation,
Composition,
Layer,
Layer3D,
@@ -289,33 +288,23 @@ describe("WebGPU 3D Shapes & Path Follower Headless Conformance", () => {
const img15 = await decodePngPixels(f15Buf);
const img30 = await decodePngPixels(f30Buf);
- // Find where orange pixels are in img0
- let found0: { x: number; y: number } | null = null;
- for (let y = 0; y < img0.height; y++) {
- for (let x = 0; x < img0.width; x++) {
- const pix = getPixel(img0, x, y);
- if (pix.r > 200 && pix.g > 100 && pix.b < 50) {
- found0 = { x, y };
- break;
+ // First orange (follower) pixel in a frame, scanning row by row.
+ const findOrange = (img: typeof img0) => {
+ for (let y = 0; y < img.height; y++) {
+ for (let x = 0; x < img.width; x++) {
+ const pix = getPixel(img, x, y);
+ if (pix.r > 200 && pix.g > 100 && pix.b < 50) return { x, y };
}
}
- if (found0) break;
- }
-
- // Find where orange pixels are in img30
- let found30: { x: number; y: number } | null = null;
- for (let y = 0; y < img30.height; y++) {
- for (let x = 0; x < img30.width; x++) {
- const pix = getPixel(img30, x, y);
- if (pix.r > 200 && pix.g > 100 && pix.b < 50) {
- found30 = { x, y };
- break;
- }
- }
- if (found30) break;
- }
+ return null;
+ };
+ const found0 = findOrange(img0);
+ const found15 = findOrange(img15);
+ const found30 = findOrange(img30);
expect(found0).not.toBeNull();
+ // Halfway along the path the follower is still on screen.
+ expect(found15).not.toBeNull();
expect(found30).not.toBeNull();
const p0 = getPixel(img0, found0!.x, found0!.y);
diff --git a/packages/gitframes/src/shapes3d.ts b/packages/gitframes/src/shapes3d.ts
index 4869348a..049af630 100644
--- a/packages/gitframes/src/shapes3d.ts
+++ b/packages/gitframes/src/shapes3d.ts
@@ -3,13 +3,13 @@ import type {
LayoutNode,
MaterialType,
Model3DNode,
- TextNode,
} from "@gitframes/compositions/program";
import {
generateExtrudedTextGeometry,
type Model3DData,
SlugFontCache,
} from "@gitframes/webgpu-renderers";
+import { autoId } from "./ids.js";
import { type AnimatableNode, Layer } from "./index.js";
export interface CubeFaceConfig {
@@ -169,6 +169,10 @@ export interface Text3DOptions {
rotateX?: number;
rotateY?: number;
rotateZ?: number;
+ scale?: number;
+ scaleX?: number;
+ scaleY?: number;
+ scaleZ?: number;
align?: "start" | "center" | "end" | "left" | "right";
verticalAlign?: "top" | "middle" | "bottom";
letterSpacing?: number;
@@ -193,14 +197,23 @@ export interface Text3DOptions {
stroke?: string;
strokeWidth?: number;
extrusionDepth?: number;
+ /** @deprecated No effect: extruded text is one solid mesh, not a stack of slices. */
extrusionSteps?: number;
+ /**
+ * Extruded text only: color of the side walls. `fill` colors the letter
+ * faces (front and back). Defaults to `fill`.
+ */
bevelColor?: string;
}
export interface ExtrudedText3DOptions extends Text3DOptions {
depth: number;
+ /** @deprecated No effect: extruded text is one solid mesh, not a stack of slices. */
slices?: number;
+ /** Fixed straight segments per curve; by default curves split as finely as their on-screen size needs. */
curveSegments?: number;
+ /** Most a flattened curve strays from the outline, in on-screen px at scale 1 (default 0.1). */
+ curveTolerance?: number;
}
export type Shape3DContainer = AnimatableNode<BoxNode> & {
@@ -209,6 +222,16 @@ export type Shape3DContainer = AnimatableNode<BoxNode> & {
[Symbol.iterator](): Iterator<LayoutNode>;
};
+/** Largest initial scale of an extruded text node along its outline's axes. */
+function extrudedTextScale(options: ExtrudedText3DOptions): number {
+ const scale = options.scale ?? 1;
+ const s = Math.max(
+ Math.abs(scale * (options.scaleX ?? 1)),
+ Math.abs(scale * (options.scaleY ?? 1)),
+ );
+ return s > 0 ? s : 1;
+}
+
export const Layer3D = {
/**
* Creates a true 3D volumetric cube with 6 textured or colored faces.
@@ -345,7 +368,7 @@ export const Layer3D = {
];
const container = Layer.box({
- id: options.id ?? "cube-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("cube"),
position: "absolute",
x: options.x ?? 0,
y: options.y ?? 0,
@@ -401,7 +424,7 @@ export const Layer3D = {
});
const container = Layer.box({
- id: options.id ?? "carousel-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("carousel"),
position: "absolute",
x: options.x ?? 0,
y: options.y ?? 0,
@@ -445,7 +468,8 @@ export const Layer3D = {
const color =
typeof cfg === "string"
? cfg
- : (cfg?.background ?? `hsl(${(i * 360) / n}, 70%, 50%)`);
+ : ((cfg as { background?: string } | undefined)?.background ??
+ `hsl(${(i * 360) / n}, 70%, 50%)`);
panels.push(
Layer.box({
@@ -465,7 +489,7 @@ export const Layer3D = {
}
const container = Layer.box({
- id: options.id ?? "prism-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("prism"),
position: "absolute",
x: options.x ?? 0,
y: options.y ?? 0,
@@ -495,7 +519,7 @@ export const Layer3D = {
*/
plane(options: Plane3DOptions): AnimatableNode<BoxNode> {
return Layer.box({
- id: options.id ?? "plane-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("plane"),
position: "absolute",
width: options.width,
height: options.height,
@@ -520,7 +544,7 @@ export const Layer3D = {
*/
grid(options: Grid3DOptions): AnimatableNode<BoxNode> {
const { width, height } = options;
- const id = options.id ?? "grid-" + Math.random().toString(36).slice(2, 9);
+ const id = options.id ?? autoId("grid");
const divisions = Math.max(1, Math.round(options.divisions ?? 10));
const lineWidth = options.lineWidth ?? 2;
const color = options.color ?? "rgba(59, 130, 246, 0.5)";
@@ -583,19 +607,18 @@ export const Layer3D = {
},
/**
- * Creates a standalone planar 3D text node or multi-slice extruded 3D text.
+ * Creates a standalone planar 3D text node, or solid extruded 3D text when `extrusionDepth` is set.
*/
text(options: Text3DOptions): AnimatableNode<LayoutNode> {
if (options.extrusionDepth && options.extrusionDepth > 0) {
return Layer3D.extrudedText({
...options,
depth: options.extrusionDepth,
- slices: options.extrusionSteps,
});
}
return Layer.text(options.text, {
- id: options.id ?? "text3d-" + Math.random().toString(36).slice(2, 9),
+ id: options.id ?? autoId("text3d"),
position: "absolute",
x: options.x ?? 0,
y: options.y ?? 0,
@@ -630,12 +653,16 @@ export const Layer3D = {
},
/**
- * Creates multi-slice volumetric extruded 3D typography with depth shading.
+ * Creates solid extruded 3D typography: `fill` colors the letter faces,
+ * `bevelColor` the side walls; curves are tessellated to `curveTolerance`.
*/
extrudedText(options: ExtrudedText3DOptions): Shape3DContainer {
const { depth, text, fill = "#ffffff", bevelColor } = options;
let modelData: Model3DData | undefined;
+ // Tolerance is in on-screen px: a node scaled up 3x needs 3x finer geometry.
+ const curveTolerance =
+ (options.curveTolerance ?? 0.1) / extrudedTextScale(options);
const parsedFont =
(options.fontFamily
? SlugFontCache.getParsed(options.fontFamily)
@@ -647,28 +674,36 @@ export const Layer3D = {
if (parsedFont) {
const res = generateExtrudedTextGeometry({
text,
- font: parsedFont,
+ // SlugFontCache holds fontkit fonts under a narrower type.
+ font: parsedFont as unknown as Parameters<
+ typeof generateExtrudedTextGeometry
+ >[0]["font"],
fontSize: options.fontSize ?? 48,
depth,
fill,
bevelColor,
- curveSegments: options.curveSegments ?? 5,
+ curveSegments: options.curveSegments,
+ curveTolerance,
align: options.align,
verticalAlign: options.verticalAlign,
letterSpacing: options.letterSpacing,
- material: options.material,
+ // The mesh's own shading only knows these; glass and the like are
+ // layer materials, applied by the model node below.
+ material:
+ options.material === "lit" ||
+ options.material === "unlit" ||
+ options.material === "toon"
+ ? options.material
+ : undefined,
shininess: options.shininess,
roughness: options.roughness,
- specularIntensity: options.specularIntensity,
- ambientIntensity: options.ambientIntensity,
metallic: options.metallic,
});
modelData = res.modelData;
}
const modelNode = Layer.model({
- id:
- options.id ?? `extruded-text-${Math.random().toString(36).slice(2, 9)}`,
+ id: options.id ?? autoId("extruded-text"),
modelData,
is3D: true,
x: options.x ?? 0,
@@ -689,11 +724,13 @@ export const Layer3D = {
ambientIntensity: options.ambientIntensity ?? 1.0,
metallic: options.metallic,
twoSided: options.twoSided ?? true,
- text3dOptions: options,
+ // What the compositor rebuilds the mesh from when modelData did not
+ // survive (a spec sent as JSON): the same tolerance as here.
+ text3dOptions: { ...options, curveTolerance },
});
return Object.assign(modelNode, {
- faces: [modelNode] as AnimatableNode<BoxNode>[],
+ faces: [modelNode] as unknown as AnimatableNode<BoxNode>[],
items: [modelNode],
[Symbol.iterator]: function* () {
yield modelNode;
diff --git a/packages/gitframes/src/tracked-region-blur.test.ts b/packages/gitframes/src/tracked-region-blur.test.ts
index 72df46b0..f36449a4 100644
--- a/packages/gitframes/src/tracked-region-blur.test.ts
+++ b/packages/gitframes/src/tracked-region-blur.test.ts
@@ -3,15 +3,15 @@ import {
Blur,
Composition,
Effect,
+ type EditableObjectTrack,
Layer,
- type ObjectTrackSignals,
VisionNode,
} from "./index.js";
describe("Tracked Section Editing & Blur Engine Suite", () => {
function setupVisionWithSyntheticTrack(): {
vision: ReturnType<typeof VisionNode.attach>;
- targetPerson: ObjectTrackSignals;
+ targetPerson: EditableObjectTrack;
} {
const video = Layer.video("test.mp4");
const vision = VisionNode.attach(video, {
@@ -48,7 +48,10 @@ describe("Tracked Section Editing & Blur Engine Suite", () => {
},
]);
- const targetPerson = vision.objects.byCategory("person");
+ // Vision decorates every track with its editing helpers.
+ const targetPerson = vision.objects.byCategory(
+ "person",
+ ) as EditableObjectTrack;
return { vision, targetPerson };
}
@@ -137,7 +140,7 @@ describe("Tracked Section Editing & Blur Engine Suite", () => {
// Inside the box, a nested video plate counter-offsets by -screenX + padding
expect(trackedBox.children).toHaveLength(1);
- const innerVideo = trackedBox.children[0] as unknown as {
+ const innerVideo = trackedBox.children?.[0] as unknown as {
kind: string;
x: { get: (ctx: unknown) => number };
y: { get: (ctx: unknown) => number };
@@ -254,7 +257,7 @@ describe("Tracked Section Editing & Blur Engine Suite", () => {
});
comp.add(bgCard);
- const png = await comp.renderFrame(0);
+ const png = await comp.renderFrame({ frame: 0 });
expect(Buffer.isBuffer(png)).toBe(true);
expect(png.length).toBeGreaterThan(100);
// Check PNG signature: 0x89 'P' 'N' 'G'
diff --git a/packages/node-sdk/package.json b/packages/node-sdk/package.json
index 8c6aa21e..dae58f8f 100644
--- a/packages/node-sdk/package.json
+++ b/packages/node-sdk/package.json
@@ -36,6 +36,7 @@
"devDependencies": {
"@gitframes/tsconfig": "workspace:*",
"@types/node": "^24.1.0",
+ "@webgpu/types": "^0.1.70",
"tsdown": "^0.19.0",
"typescript": "^5.8.3",
"webgpu": "^0.4.0"
diff --git a/packages/node-sdk/renderer/index.ts b/packages/node-sdk/renderer/index.ts
index 38a363f9..d3421ee8 100644
--- a/packages/node-sdk/renderer/index.ts
+++ b/packages/node-sdk/renderer/index.ts
@@ -1,13 +1,10 @@
+/// <reference types="@webgpu/types" />
import type { VirtualMediaData } from "@gitframes/core";
import type { RenderContextValue } from "@gitframes/webgpu-renderers";
-// @ts-expect-error
export type GPUCommandEncoder = globalThis.GPUCommandEncoder;
-// @ts-expect-error
export type GPURenderPassEncoder = globalThis.GPURenderPassEncoder;
-// @ts-expect-error
export type GPUTextureView = globalThis.GPUTextureView;
-// @ts-expect-error
export type GPUTexture = globalThis.GPUTexture;
export interface NodeRenderProps {
@@ -33,6 +30,8 @@ export interface NodeRenderProps {
isVideoMode?: boolean;
isPlaying?: boolean;
excludeTextures?: GPUTexture[];
+ /** Signal values that override the node's own (text animators read them). */
+ signals?: Record<string, unknown>;
renderChild?: (
child: VirtualMediaData,
overrides?: Partial<NodeRenderProps>,
diff --git a/packages/renderer/src/dma-staging-ring.ts b/packages/renderer/src/dma-staging-ring.ts
index 181f5727..702a682a 100644
--- a/packages/renderer/src/dma-staging-ring.ts
+++ b/packages/renderer/src/dma-staging-ring.ts
@@ -216,6 +216,8 @@ export class DmaStagingRing {
frameIndex: number,
timestampUs: number,
colorSpace?: VideoColorSpaceConfig,
+ /** Sees the tightly packed RGBA pixels while they are mapped; must not keep them. */
+ inspect?: (rgba: Uint8Array) => void,
): Promise<VideoFrame> {
this.checkDestroyed();
const slot = this.slots[frameIndex % this.capacity];
@@ -251,6 +253,8 @@ export class DmaStagingRing {
}
}
+ inspect?.(pixelData);
+
const vf = new VideoFrame(pixelData, {
format: "RGBA",
codedWidth: this.width,
diff --git a/packages/renderer/src/headless-webgpu-renderer.ts b/packages/renderer/src/headless-webgpu-renderer.ts
index 3a77193e..7c978ed5 100644
--- a/packages/renderer/src/headless-webgpu-renderer.ts
+++ b/packages/renderer/src/headless-webgpu-renderer.ts
@@ -47,9 +47,15 @@ import {
drawCompositionTree,
mixAudioTracks,
} from "@gitframes/compositions";
-import { getMediaType, updateClockSignals, type VirtualMediaData } from "@gitframes/core";
+import {
+ getMediaType,
+ takeRenderDiagnostics,
+ updateClockSignals,
+ type VirtualMediaData,
+} from "@gitframes/core";
import { rendererLogger } from "@gitframes/server-utils";
import {
+ clearAllVideoCache,
ensureDevice,
getRenderer2D,
initHeadlessWebGPU,
@@ -82,6 +88,15 @@ import sharp from "sharp";
import { Canvas as SkiaCanvas, Image as SkiaImage } from "skia-canvas";
import { preloadFonts } from "./asset-preloader.js";
import { discoverAndRegisterNodeRenderers } from "./dynamic-node-discovery.js";
+import {
+ type AudioQaStats,
+ analyzeAudio,
+ buildQaReport,
+ FrameInspector,
+ type VideoQaOptions,
+ type VideoQaReport,
+ type VideoQaStats,
+} from "./video-qa.js";
import { ZeroCopyWebCodecsPipeline } from "./zero-copy-webcodecs-pipeline.js";
// ─── Bootstrap ─────────────────────────────────────────────────────────────────
@@ -471,7 +486,11 @@ export class HeadlessWebGPURenderer {
surface = new NodeSurfaceProvider(device, width, height);
renderer = getRenderer2D(device, surface.colorFormat);
const ctx: RenderContextValue = { device, renderer, surface };
- updateClockSignals(frame, fps, virtualMedia.metadata.durationMs);
+ updateClockSignals(
+ frame,
+ fps,
+ virtualMedia.metadata.durationMs ?? undefined,
+ );
const needsPrePass =
frame === 0 ||
@@ -604,7 +623,9 @@ export class HeadlessWebGPURenderer {
options: { atMs?: number; frame?: number; fps?: number } = {},
): Promise<Buffer> {
const vm =
- typeof source === "object" && source !== null && "toVirtualMedia" in source
+ typeof source === "object" &&
+ source !== null &&
+ "toVirtualMedia" in source
? (source as { toVirtualMedia(): VirtualMediaData }).toVirtualMedia()
: (source as VirtualMediaData);
const fps = options.fps ?? (vm.metadata?.fps || 30);
@@ -828,8 +849,18 @@ export class HeadlessWebGPURenderer {
audioCodec?: string;
quality?: string;
concurrency?: number;
+ /**
+ * Measure the output while it renders (black and frozen frames,
+ * dropped frames, loudness, clipping, silence, document warnings)
+ * and return the findings as `qa`. `true` uses default thresholds.
+ */
+ qa?: boolean | VideoQaOptions;
},
- ): Promise<{ filePath: string; cleanup: () => Promise<void> }> {
+ ): Promise<{
+ filePath: string;
+ cleanup: () => Promise<void>;
+ qa?: VideoQaReport;
+ }> {
await HeadlessWebGPURenderer.initialize();
const codecOption = options?.codec ?? "h264";
@@ -876,6 +907,11 @@ export class HeadlessWebGPURenderer {
const totalFrames = Math.max(1, Math.round((durationMs / 1000) * fps));
const renderId = `vid-${randomUUID()}`;
+ const qaOptions: VideoQaOptions | undefined =
+ options?.qa === true ? {} : options?.qa ? options.qa : undefined;
+ const inspector = qaOptions ? new FrameInspector(qaOptions) : undefined;
+ let videoQa: VideoQaStats | undefined;
+ let audioQa: AudioQaStats | undefined;
const videoCodec: "avc" | "vp9" | "vp8" =
codecOption === "vp9" ? "vp9" : codecOption === "vp8" ? "vp8" : "avc";
const audioCodec: "aac" | "opus" | "mp3" =
@@ -1014,6 +1050,9 @@ export class HeadlessWebGPURenderer {
fps,
videoSource,
ringCapacity: 2,
+ onFrame: inspector
+ ? (rgba) => inspector.inspect(rgba, width, height)
+ : undefined,
});
const ctx: RenderContextValue | null =
@@ -1114,6 +1153,16 @@ export class HeadlessWebGPURenderer {
if (pipeline) {
await pipeline.drain();
const pipelineStats = pipeline.getStats();
+ if (inspector && qaOptions) {
+ videoQa = inspector.stats(
+ totalFrames,
+ pipelineStats.totalFramesEncoded,
+ fps,
+ width,
+ height,
+ qaOptions,
+ );
+ }
rendererLogger.debug(
{ renderId, totalFrames, ...pipelineStats },
`[HeadlessWebGPURenderer] All frames rendered and encoded (avg cycle: ${pipelineStats.averageCycleTimeMs.toFixed(2)}ms, throughput: ${pipelineStats.throughputFps.toFixed(1)} fps)`,
@@ -1134,6 +1183,8 @@ export class HeadlessWebGPURenderer {
if (numChannels === 0)
throw new Error("mixAudioTracks returned no channels");
+ if (qaOptions) audioQa = analyzeAudio(channels, sampleRate, qaOptions);
+
const totalSamples = channels[0].length;
const chunkSize = Math.round(0.1 * sampleRate);
@@ -1167,6 +1218,15 @@ export class HeadlessWebGPURenderer {
await output.finalize();
outputFinalized = true;
+ const qa = qaOptions
+ ? buildQaReport({
+ video: videoQa,
+ audio: audioQa,
+ diagnostics: takeRenderDiagnostics(renderId),
+ options: qaOptions,
+ })
+ : undefined;
+
const cleanup = async () => {
try {
await fs.unlink(tempFilePath).catch(() => {});
@@ -1181,10 +1241,10 @@ export class HeadlessWebGPURenderer {
"[HeadlessWebGPURenderer] Converting rendered MP4 to high-quality GIF...",
);
await convertMp4ToGif(tempFilePath, tempGifPath);
- return { filePath: tempGifPath, cleanup };
+ return { filePath: tempGifPath, cleanup, ...(qa && { qa }) };
}
- return { filePath: tempFilePath, cleanup };
+ return { filePath: tempFilePath, cleanup, ...(qa && { qa }) };
} catch (error) {
if (outputStarted && !outputFinalized) {
try {
@@ -1227,6 +1287,8 @@ export class HeadlessWebGPURenderer {
try {
shaderStore.clear(renderId);
} catch {}
+ // Drop anything reported for this render that QA did not collect.
+ takeRenderDiagnostics(renderId);
renderSemaphore.release();
rendererLogger.debug(
{ renderId, ...renderSemaphore.stats },
diff --git a/packages/renderer/src/index.ts b/packages/renderer/src/index.ts
index 606869d4..0de6cc97 100644
--- a/packages/renderer/src/index.ts
+++ b/packages/renderer/src/index.ts
@@ -3,4 +3,5 @@ export * from "./dma-staging-ring.js";
export * from "./dynamic-node-discovery.js";
export * from "./headless-webgpu-renderer.js";
export * from "./node-manifest.js";
+export * from "./video-qa.js";
export * from "./zero-copy-webcodecs-pipeline.js";
diff --git a/packages/renderer/src/offscreen-canvas.ts b/packages/renderer/src/offscreen-canvas.ts
index 577c583f..18219784 100644
--- a/packages/renderer/src/offscreen-canvas.ts
+++ b/packages/renderer/src/offscreen-canvas.ts
@@ -1,7 +1,9 @@
import { Canvas as SkiaCanvas } from "skia-canvas";
-let GL: any = null;
-function getGL() { return null; }
+// No WebGL backend is bundled; the WebGL paths below stay inert.
+function getGL(): any {
+ return null;
+}
// Only polyfill WebGLRenderingContext if not provided by the runtime (e.g. standard Node without Electron)
if (typeof globalThis.WebGLRenderingContext === "undefined") {
diff --git a/packages/renderer/src/tests/video-qa.test.ts b/packages/renderer/src/tests/video-qa.test.ts
new file mode 100644
index 00000000..5106e8da
--- /dev/null
+++ b/packages/renderer/src/tests/video-qa.test.ts
@@ -0,0 +1,140 @@
+import { reportRenderDiagnostic, takeRenderDiagnostics } from "@gitframes/core";
+import { describe, expect, it } from "vitest";
+import {
+ analyzeAudio,
+ buildQaReport,
+ FrameInspector,
+ formatQaReport,
+ integratedLoudness,
+} from "../video-qa.js";
+
+const RATE = 48_000;
+
+function sine(amplitude: number, seconds: number, hz = 997): Float32Array {
+ const out = new Float32Array(Math.round(seconds * RATE));
+ for (let i = 0; i < out.length; i++) {
+ out[i] = amplitude * Math.sin((2 * Math.PI * hz * i) / RATE);
+ }
+ return out;
+}
+
+function frame(width: number, height: number, rgb: number): Uint8Array {
+ const px = new Uint8Array(width * height * 4);
+ for (let i = 0; i < px.length; i += 4) {
+ px[i] = rgb;
+ px[i + 1] = rgb;
+ px[i + 2] = rgb;
+ px[i + 3] = 255;
+ }
+ return px;
+}
+
+describe("integratedLoudness (ITU-R BS.1770)", () => {
+ it("reads a -20 dBFS 997 Hz stereo sine as -20 LUFS", () => {
+ const ch = sine(0.1, 5);
+ expect(integratedLoudness([ch, ch], RATE)).toBeCloseTo(-20, 1);
+ });
+
+ it("reads the same sine on one channel 3 LU quieter", () => {
+ const ch = sine(0.1, 5);
+ const lufs = integratedLoudness([ch, new Float32Array(ch.length)], RATE)!;
+ expect(lufs).toBeCloseTo(-23.01, 1);
+ });
+
+ it("gates out silence instead of averaging it in", () => {
+ const loud = sine(0.1, 3);
+ const ch = new Float32Array(loud.length * 3);
+ ch.set(loud); // 3 s of tone, 6 s of silence
+ expect(integratedLoudness([ch, ch], RATE)).toBeCloseTo(-20, 0);
+ });
+});
+
+describe("analyzeAudio", () => {
+ it("measures peak, clipping and silent stretches", () => {
+ const ch = new Float32Array(RATE * 6);
+ ch.set(sine(1.2, 2)); // 2 s overdriven, then 4 s silence
+ const stats = analyzeAudio([ch, ch], RATE);
+ expect(stats.peakDbfs).toBeCloseTo(20 * Math.log10(1.2), 1);
+ expect(stats.clippedSamples).toBeGreaterThan(0);
+ expect(stats.silentSegments).toHaveLength(1);
+ expect(stats.silentSegments[0]!.startSec).toBeCloseTo(2, 1);
+ expect(stats.silentSegments[0]!.endSec).toBeCloseTo(6, 1);
+ });
+
+ it("reports a silent track with no loudness", () => {
+ const stats = analyzeAudio([new Float32Array(RATE * 3)], RATE);
+ expect(stats.integratedLufs).toBeNull();
+ });
+});
+
+describe("FrameInspector", () => {
+ it("finds black and frozen stretches", () => {
+ const fps = 10;
+ const inspector = new FrameInspector();
+ // 1 s of moving gray, 1 s of black, 3 s holding one gray
+ for (let i = 0; i < 10; i++) inspector.inspect(frame(64, 36, 100 + i * 10), 64, 36);
+ for (let i = 0; i < 10; i++) inspector.inspect(frame(64, 36, 0), 64, 36);
+ for (let i = 0; i < 30; i++) inspector.inspect(frame(64, 36, 128), 64, 36);
+
+ const stats = inspector.stats(50, 50, fps, 64, 36);
+ expect(stats.blackSegments).toEqual([{ startSec: 1, endSec: 2 }]);
+ // The black second also holds still, but is shorter than the 2 s minimum.
+ expect(stats.frozenSegments).toEqual([{ startSec: 2, endSec: 5 }]);
+ });
+});
+
+describe("buildQaReport", () => {
+ it("fails on warnings and errors, passes with only info", () => {
+ const inspector = new FrameInspector();
+ for (let i = 0; i < 4; i++) inspector.inspect(frame(8, 8, 200), 8, 8);
+ const video = inspector.stats(5, 4, 30, 8, 8);
+ const ch = sine(0.1, 2);
+ const audio = analyzeAudio([ch, ch], RATE);
+
+ const failing = buildQaReport({
+ video,
+ audio,
+ diagnostics: [
+ {
+ severity: "warning",
+ code: "animation_truncated",
+ message: "tail never plays",
+ layerId: "title",
+ },
+ ],
+ options: { targetLufs: -14 },
+ });
+ expect(failing.passed).toBe(false);
+ expect(failing.issues.map((i) => i.code)).toEqual([
+ "frames_missing",
+ "loudness_off_target",
+ "animation_truncated",
+ ]);
+ expect(formatQaReport(failing)).toContain("QA FAILED");
+
+ const passing = buildQaReport({
+ video: inspector.stats(4, 4, 30, 8, 8),
+ audio,
+ });
+ expect(passing.passed).toBe(true);
+ });
+});
+
+describe("render diagnostics", () => {
+ it("collects a render's own and its nested compositions' reports once", () => {
+ const d = {
+ severity: "warning" as const,
+ code: "animation_truncated",
+ message: "tail never plays",
+ layerId: "a",
+ };
+ reportRenderDiagnostic("vid-1", d);
+ reportRenderDiagnostic("vid-1", d); // repeat: kept once
+ reportRenderDiagnostic("vid-1-c3-Compositor", { ...d, layerId: "b" });
+ reportRenderDiagnostic("vid-10", { ...d, layerId: "other render" });
+
+ expect(takeRenderDiagnostics("vid-1").map((x) => x.layerId)).toEqual(["a", "b"]);
+ expect(takeRenderDiagnostics("vid-1")).toEqual([]);
+ expect(takeRenderDiagnostics("vid-10")).toHaveLength(1);
+ });
+});
diff --git a/packages/renderer/src/video-qa.ts b/packages/renderer/src/video-qa.ts
new file mode 100644
index 00000000..e145e4af
--- /dev/null
+++ b/packages/renderer/src/video-qa.ts
@@ -0,0 +1,466 @@
+import type { RenderDiagnostic } from "@gitframes/core";
+
+/**
+ * Quality checks run on a video while it renders: frames are measured as
+ * they are read back for the encoder and audio as it is mixed, so the report
+ * describes exactly what was encoded without decoding the file again.
+ */
+export interface VideoQaOptions {
+ /** Luma (0-1) at or below which a pixel counts as black. Default 0.1. */
+ readonly blackPixelThreshold?: number;
+ /** Share of black pixels that makes a frame black. Default 0.98. */
+ readonly blackFrameRatio?: number;
+ /** Shortest run of black frames reported. Default 0.5 s. */
+ readonly minBlackSec?: number;
+ /** Shortest run of unchanging frames reported. Default 2 s. */
+ readonly minFrozenSec?: number;
+ /** Level (dBFS, 100 ms windows) below which audio counts as silent. Default -60. */
+ readonly silenceDbfs?: number;
+ /** Shortest silent stretch reported. Default 2 s. */
+ readonly minSilenceSec?: number;
+ /** Integrated loudness to hit, e.g. -14 for most streaming platforms. Unchecked when unset. */
+ readonly targetLufs?: number;
+ /** Allowed distance from `targetLufs`, in LU. Default 1. */
+ readonly lufsTolerance?: number;
+}
+
+export interface QaSegment {
+ readonly startSec: number;
+ readonly endSec: number;
+}
+
+export interface QaIssue {
+ readonly severity: "error" | "warning" | "info";
+ /** Stable machine-readable kind, e.g. "black_frames", "audio_clipping" */
+ readonly code: string;
+ readonly message: string;
+ readonly startSec?: number;
+ readonly endSec?: number;
+ readonly layerId?: string;
+}
+
+export interface VideoQaStats {
+ readonly expectedFrames: number;
+ readonly encodedFrames: number;
+ readonly fps: number;
+ readonly width: number;
+ readonly height: number;
+ /** Average luma over the whole video, 0-1 */
+ readonly meanLuma: number;
+ readonly blackSegments: readonly QaSegment[];
+ readonly frozenSegments: readonly QaSegment[];
+}
+
+export interface AudioQaStats {
+ readonly sampleRate: number;
+ readonly channels: number;
+ readonly durationSec: number;
+ /** Highest sample, dBFS (sample peak, not true peak) */
+ readonly peakDbfs: number;
+ /** Mean level over the whole track, dBFS */
+ readonly rmsDbfs: number;
+ /** ITU-R BS.1770 integrated loudness; null when the track is silent */
+ readonly integratedLufs: number | null;
+ /** Samples at or beyond full scale */
+ readonly clippedSamples: number;
+ readonly silentSegments: readonly QaSegment[];
+}
+
+export interface VideoQaReport {
+ /** No errors or warnings (info issues do not fail a render) */
+ readonly passed: boolean;
+ readonly video?: VideoQaStats;
+ readonly audio?: AudioQaStats;
+ readonly issues: readonly QaIssue[];
+}
+
+const DEFAULTS = {
+ blackPixelThreshold: 0.1,
+ blackFrameRatio: 0.98,
+ minBlackSec: 0.5,
+ minFrozenSec: 2,
+ silenceDbfs: -60,
+ minSilenceSec: 2,
+ lufsTolerance: 1,
+} as const;
+
+// Every 4th pixel of every 4th row: 1/16 of the frame keeps inspection
+// around a millisecond at 1080p while still seeing anything frame-sized.
+const SAMPLE_STEP = 4;
+// Mean absolute luma change (0-255) under which two frames are the same picture.
+const FROZEN_DIFF = 0.25;
+
+/** Measures frames handed to it in order, as RGBA bytes. */
+export class FrameInspector {
+ private readonly blackLuma: number;
+ private readonly blackRatio: number;
+ private readonly black: boolean[] = [];
+ private readonly sameAsPrevious: boolean[] = [];
+ private lumaSum = 0;
+ private previous: Uint8Array | null = null;
+
+ constructor(options: VideoQaOptions = {}) {
+ this.blackLuma =
+ (options.blackPixelThreshold ?? DEFAULTS.blackPixelThreshold) * 255;
+ this.blackRatio = options.blackFrameRatio ?? DEFAULTS.blackFrameRatio;
+ }
+
+ get frameCount(): number {
+ return this.black.length;
+ }
+
+ inspect(rgba: Uint8Array, width: number, height: number): void {
+ const cols = Math.ceil(width / SAMPLE_STEP);
+ const rows = Math.ceil(height / SAMPLE_STEP);
+ const luma = new Uint8Array(cols * rows);
+ let blackCount = 0;
+ let sum = 0;
+ let k = 0;
+ for (let y = 0; y < height; y += SAMPLE_STEP) {
+ let p = y * width * 4;
+ for (let x = 0; x < width; x += SAMPLE_STEP, p += SAMPLE_STEP * 4) {
+ // BT.709 luma from 8-bit RGB
+ const l =
+ 0.2126 * rgba[p]! + 0.7152 * rgba[p + 1]! + 0.0722 * rgba[p + 2]!;
+ luma[k++] = l;
+ sum += l;
+ if (l <= this.blackLuma) blackCount++;
+ }
+ }
+ const n = luma.length;
+ this.lumaSum += sum / n / 255;
+ this.black.push(blackCount / n >= this.blackRatio);
+
+ let same = false;
+ if (this.previous && this.previous.length === n) {
+ let diff = 0;
+ for (let i = 0; i < n; i++)
+ diff += Math.abs(luma[i]! - this.previous[i]!);
+ same = diff / n < FROZEN_DIFF;
+ }
+ this.sameAsPrevious.push(same);
+ this.previous = luma;
+ }
+
+ stats(
+ expectedFrames: number,
+ encodedFrames: number,
+ fps: number,
+ width: number,
+ height: number,
+ options: VideoQaOptions = {},
+ ): VideoQaStats {
+ const minBlack = Math.max(
+ 1,
+ Math.round((options.minBlackSec ?? DEFAULTS.minBlackSec) * fps),
+ );
+ const minFrozen = Math.max(
+ 2,
+ Math.round((options.minFrozenSec ?? DEFAULTS.minFrozenSec) * fps),
+ );
+ const blackSegments = runs(this.black, minBlack).map(([a, b]) => ({
+ startSec: a / fps,
+ endSec: b / fps,
+ }));
+ // A frozen run starts on the frame before its first repeat.
+ const frozenSegments = runs(this.sameAsPrevious, minFrozen - 1).map(
+ ([a, b]) => ({ startSec: (a - 1) / fps, endSec: b / fps }),
+ );
+ return {
+ expectedFrames,
+ encodedFrames,
+ fps,
+ width,
+ height,
+ meanLuma: this.frameCount > 0 ? this.lumaSum / this.frameCount : 0,
+ blackSegments,
+ frozenSegments,
+ };
+ }
+}
+
+/** [start, end) index ranges where flags hold for at least minLength in a row. */
+function runs(
+ flags: readonly boolean[],
+ minLength: number,
+): [number, number][] {
+ const out: [number, number][] = [];
+ let start = -1;
+ for (let i = 0; i <= flags.length; i++) {
+ if (i < flags.length && flags[i]) {
+ if (start < 0) start = i;
+ } else if (start >= 0) {
+ if (i - start >= minLength) out.push([start, i]);
+ start = -1;
+ }
+ }
+ return out;
+}
+
+const toDb = (power: number) =>
+ power > 0 ? 10 * Math.log10(power) : -Infinity;
+
+/** Measures a mixed audio track (one Float32Array per channel). */
+export function analyzeAudio(
+ channels: readonly Float32Array[],
+ sampleRate: number,
+ options: VideoQaOptions = {},
+): AudioQaStats {
+ const length = channels[0]?.length ?? 0;
+ let peak = 0;
+ let sumSquares = 0;
+ let clipped = 0;
+ for (const ch of channels) {
+ for (let i = 0; i < length; i++) {
+ const v = Math.abs(ch[i]!);
+ if (v > peak) peak = v;
+ if (v >= 0.9999) clipped++;
+ sumSquares += v * v;
+ }
+ }
+ const total = length * Math.max(1, channels.length);
+
+ const silenceDbfs = options.silenceDbfs ?? DEFAULTS.silenceDbfs;
+ const window = Math.max(1, Math.round(sampleRate * 0.1));
+ const silent: boolean[] = [];
+ for (let start = 0; start < length; start += window) {
+ const end = Math.min(length, start + window);
+ let s = 0;
+ for (const ch of channels)
+ for (let i = start; i < end; i++) s += ch[i]! * ch[i]!;
+ silent.push(toDb(s / ((end - start) * channels.length)) < silenceDbfs);
+ }
+ const minSilence = Math.max(
+ 1,
+ Math.round((options.minSilenceSec ?? DEFAULTS.minSilenceSec) / 0.1),
+ );
+ const silentSegments = runs(silent, minSilence).map(([a, b]) => ({
+ startSec: (a * window) / sampleRate,
+ endSec: Math.min(length, b * window) / sampleRate,
+ }));
+
+ const peakDbfs = 20 * Math.log10(Math.max(peak, 1e-12));
+ return {
+ sampleRate,
+ channels: channels.length,
+ durationSec: length / sampleRate,
+ peakDbfs,
+ rmsDbfs: toDb(sumSquares / Math.max(1, total)),
+ integratedLufs:
+ peakDbfs < silenceDbfs ? null : integratedLoudness(channels, sampleRate),
+ clippedSamples: clipped,
+ silentSegments,
+ };
+}
+
+interface Biquad {
+ b0: number;
+ b1: number;
+ b2: number;
+ a1: number;
+ a2: number;
+}
+
+/** BS.1770 K-weighting stage 1: +4 dB high shelf around 1.5 kHz (the head). */
+function highShelf(sampleRate: number): Biquad {
+ const A = 10 ** (4 / 40);
+ const w0 = (2 * Math.PI * 1500) / sampleRate;
+ const cos = Math.cos(w0);
+ const alpha = Math.sin(w0) / (2 / Math.SQRT2);
+ const sq = 2 * Math.sqrt(A) * alpha;
+ const a0 = A + 1 - (A - 1) * cos + sq;
+ return {
+ b0: (A * (A + 1 + (A - 1) * cos + sq)) / a0,
+ b1: (-2 * A * (A - 1 + (A + 1) * cos)) / a0,
+ b2: (A * (A + 1 + (A - 1) * cos - sq)) / a0,
+ a1: (2 * (A - 1 - (A + 1) * cos)) / a0,
+ a2: (A + 1 - (A - 1) * cos - sq) / a0,
+ };
+}
+
+/** BS.1770 K-weighting stage 2: 38 Hz high pass, Q 0.5 (the RLB curve). */
+function highPass(sampleRate: number): Biquad {
+ const w0 = (2 * Math.PI * 38) / sampleRate;
+ const cos = Math.cos(w0);
+ const alpha = Math.sin(w0) / (2 * 0.5);
+ const a0 = 1 + alpha;
+ return {
+ b0: (1 + cos) / 2 / a0,
+ b1: -(1 + cos) / a0,
+ b2: (1 + cos) / 2 / a0,
+ a1: (-2 * cos) / a0,
+ a2: (1 - alpha) / a0,
+ };
+}
+
+function filter(input: Float32Array, f: Biquad): Float32Array {
+ const out = new Float32Array(input.length);
+ let x1 = 0;
+ let x2 = 0;
+ let y1 = 0;
+ let y2 = 0;
+ for (let i = 0; i < input.length; i++) {
+ const x = input[i]!;
+ const y = f.b0 * x + f.b1 * x1 + f.b2 * x2 - f.a1 * y1 - f.a2 * y2;
+ out[i] = y;
+ x2 = x1;
+ x1 = x;
+ y2 = y1;
+ y1 = y;
+ }
+ return out;
+}
+
+/**
+ * ITU-R BS.1770-4 integrated loudness (LUFS): K-weighted mean square in
+ * 400 ms blocks with 75% overlap, gated at -70 LUFS and then at 10 LU below
+ * the mean of what passed. Channels weigh 1 (stereo and mono layouts).
+ */
+export function integratedLoudness(
+ channels: readonly Float32Array[],
+ sampleRate: number,
+): number | null {
+ const shelf = highShelf(sampleRate);
+ const hp = highPass(sampleRate);
+ const weighted = channels.map((ch) => filter(filter(ch, shelf), hp));
+ const length = weighted[0]?.length ?? 0;
+ const block = Math.round(0.4 * sampleRate);
+ const hop = Math.round(0.1 * sampleRate);
+ if (length < block) return null;
+
+ const blockPower: number[] = [];
+ for (let start = 0; start + block <= length; start += hop) {
+ let power = 0;
+ for (const ch of weighted) {
+ let s = 0;
+ for (let i = start; i < start + block; i++) s += ch[i]! * ch[i]!;
+ power += s / block;
+ }
+ blockPower.push(power);
+ }
+ const lufs = (power: number) => -0.691 + toDb(power);
+ const mean = (xs: number[]) => xs.reduce((a, b) => a + b, 0) / xs.length;
+
+ const absolute = blockPower.filter((p) => lufs(p) > -70);
+ if (absolute.length === 0) return null;
+ const relativeGate = lufs(mean(absolute)) - 10;
+ const gated = absolute.filter((p) => lufs(p) > relativeGate);
+ return gated.length > 0 ? lufs(mean(gated)) : null;
+}
+
+/** Turn measurements and document diagnostics into a report with issues. */
+export function buildQaReport(parts: {
+ video?: VideoQaStats;
+ audio?: AudioQaStats;
+ diagnostics?: readonly RenderDiagnostic[];
+ options?: VideoQaOptions;
+}): VideoQaReport {
+ const { video, audio, options = {} } = parts;
+ const issues: QaIssue[] = [];
+ const at = (s: QaSegment) =>
+ `${s.startSec.toFixed(2)}–${s.endSec.toFixed(2)} s`;
+
+ if (video) {
+ if (video.encodedFrames !== video.expectedFrames) {
+ issues.push({
+ severity: "error",
+ code: "frames_missing",
+ message: `${video.encodedFrames} of ${video.expectedFrames} frames were encoded`,
+ });
+ }
+ for (const s of video.blackSegments) {
+ issues.push({
+ severity: "warning",
+ code: "black_frames",
+ message: `Black frames at ${at(s)}`,
+ ...s,
+ });
+ }
+ for (const s of video.frozenSegments) {
+ issues.push({
+ severity: "info",
+ code: "frozen_video",
+ message: `Picture does not change at ${at(s)}`,
+ ...s,
+ });
+ }
+ }
+
+ if (audio) {
+ if (audio.integratedLufs === null) {
+ issues.push({
+ severity: "info",
+ code: "audio_silent",
+ message: "The audio track is silent",
+ });
+ } else {
+ if (audio.clippedSamples > 0) {
+ issues.push({
+ severity: "warning",
+ code: "audio_clipping",
+ message: `${audio.clippedSamples} audio sample(s) at or beyond full scale (peak ${audio.peakDbfs.toFixed(1)} dBFS)`,
+ });
+ }
+ for (const s of audio.silentSegments) {
+ issues.push({
+ severity: "info",
+ code: "audio_silence",
+ message: `Silence at ${at(s)}`,
+ ...s,
+ });
+ }
+ if (options.targetLufs !== undefined) {
+ const off = audio.integratedLufs - options.targetLufs;
+ const tolerance = options.lufsTolerance ?? DEFAULTS.lufsTolerance;
+ if (Math.abs(off) > tolerance) {
+ issues.push({
+ severity: "warning",
+ code: "loudness_off_target",
+ message: `Integrated loudness ${audio.integratedLufs.toFixed(1)} LUFS is ${Math.abs(off).toFixed(1)} LU ${off > 0 ? "above" : "below"} the ${options.targetLufs} LUFS target`,
+ });
+ }
+ }
+ }
+ }
+
+ for (const d of parts.diagnostics ?? []) {
+ issues.push({
+ severity: d.severity,
+ code: d.code,
+ message: d.message,
+ ...(d.layerId !== undefined && { layerId: d.layerId }),
+ });
+ }
+
+ return {
+ passed: !issues.some((i) => i.severity !== "info"),
+ ...(video && { video }),
+ ...(audio && { audio }),
+ issues,
+ };
+}
+
+/** A short human- and agent-readable summary of a report. */
+export function formatQaReport(report: VideoQaReport): string {
+ const lines = [`QA ${report.passed ? "passed" : "FAILED"}`];
+ const v = report.video;
+ if (v) {
+ lines.push(
+ ` video: ${v.encodedFrames}/${v.expectedFrames} frames, ${v.width}x${v.height} @ ${v.fps} fps, mean luma ${v.meanLuma.toFixed(2)}`,
+ );
+ }
+ const a = report.audio;
+ if (a) {
+ const lufs =
+ a.integratedLufs === null
+ ? "silent"
+ : `${a.integratedLufs.toFixed(1)} LUFS`;
+ lines.push(
+ ` audio: ${a.durationSec.toFixed(2)} s, ${lufs}, peak ${a.peakDbfs.toFixed(1)} dBFS, rms ${a.rmsDbfs.toFixed(1)} dBFS`,
+ );
+ }
+ for (const i of report.issues) {
+ lines.push(` ${i.severity.padEnd(7)} ${i.code}: ${i.message}`);
+ }
+ return lines.join("\n");
+}
diff --git a/packages/renderer/src/zero-copy-webcodecs-pipeline.ts b/packages/renderer/src/zero-copy-webcodecs-pipeline.ts
index b0826a43..c6a3c784 100644
--- a/packages/renderer/src/zero-copy-webcodecs-pipeline.ts
+++ b/packages/renderer/src/zero-copy-webcodecs-pipeline.ts
@@ -12,6 +12,11 @@ export interface ZeroCopyWebCodecsPipelineOptions {
readonly ringCapacity?: number;
readonly colorSpace?: VideoColorSpaceConfig;
readonly directCanvasSource?: unknown;
+ /**
+ * Called with each frame's RGBA pixels, in frame order, just before it is
+ * encoded (staging-ring path only). The buffer is reused: read, don't keep.
+ */
+ readonly onFrame?: (rgba: Uint8Array, frameIndex: number) => void;
}
export interface WebCodecsPipelineStats {
@@ -46,6 +51,7 @@ export class ZeroCopyWebCodecsPipeline {
private readonly stagingRing: DmaStagingRing;
private readonly colorSpace?: VideoColorSpaceConfig;
private readonly directCanvasSource?: unknown;
+ private readonly onFrame?: (rgba: Uint8Array, frameIndex: number) => void;
private readonly inFlightFrames: number[] = [];
private readonly cycleStartTimes = new Map<number, number>();
@@ -68,6 +74,7 @@ export class ZeroCopyWebCodecsPipeline {
this.ringCapacity = Math.max(2, options.ringCapacity ?? 2);
this.colorSpace = options.colorSpace;
this.directCanvasSource = options.directCanvasSource;
+ this.onFrame = options.onFrame;
this.stagingRing = new DmaStagingRing(device, {
width: this.width,
@@ -209,13 +216,17 @@ export class ZeroCopyWebCodecsPipeline {
sample.close();
}
} else {
+ const onFrame = this.onFrame;
const vf = await this.stagingRing.readFrameToVideoFrame(
frameIndex,
timestampUs,
this.colorSpace,
+ onFrame && ((rgba) => onFrame(rgba, frameIndex)),
);
try {
- const sample = new VideoSample(vf, {
+ // bootstrap-mediabunny installs @napi-rs/webcodecs as the global
+ // VideoFrame, so this is the class mediabunny expects at runtime.
+ const sample = new VideoSample(vf as unknown as globalThis.VideoFrame, {
duration: 1 / this.fps,
timestamp: frameIndex / this.fps,
});
diff --git a/packages/webgpu-renderers/src/loaders/3ds-loader.ts b/packages/webgpu-renderers/src/loaders/3ds-loader.ts
index 0bc0e506..eb991e6f 100644
--- a/packages/webgpu-renderers/src/loaders/3ds-loader.ts
+++ b/packages/webgpu-renderers/src/loaders/3ds-loader.ts
@@ -127,7 +127,7 @@ export class ThreeDSLoader {
startOffset: number,
endOffset: number,
mesh: Raw3DSMesh,
- materials: Record<string, Material3D>,
+ _materials: Record<string, Material3D>,
): void {
let offset = startOffset;
diff --git a/packages/webgpu-renderers/src/loaders/extended-loaders.test.ts b/packages/webgpu-renderers/src/loaders/extended-loaders.test.ts
index d811c0bc..e027b712 100644
--- a/packages/webgpu-renderers/src/loaders/extended-loaders.test.ts
+++ b/packages/webgpu-renderers/src/loaders/extended-loaders.test.ts
@@ -1,10 +1,10 @@
import { describe, expect, it } from "vitest";
-import { parseSTL, StlLoader } from "./stl-loader.js";
-import { parsePLY, PlyLoader } from "./ply-loader.js";
+import { parseSTL } from "./stl-loader.js";
+import { parsePLY } from "./ply-loader.js";
import { parseGLTF, parseGLB, GltfLoader } from "./gltf-loader.js";
import { parseVOX, VoxLoader } from "./vox-loader.js";
import { parse3DS, ThreeDSLoader } from "./3ds-loader.js";
-import { parseOFF, OffLoader } from "./off-loader.js";
+import { parseOFF } from "./off-loader.js";
import { loadModel3D } from "../renderer3d/renderer3d.js";
describe("Extended 3D Model Loaders (STL, PLY, glTF/GLB, VOX, 3DS, OFF)", () => {
diff --git a/packages/webgpu-renderers/src/loaders/gltf-loader.ts b/packages/webgpu-renderers/src/loaders/gltf-loader.ts
index f142c008..ce8a8e21 100644
--- a/packages/webgpu-renderers/src/loaders/gltf-loader.ts
+++ b/packages/webgpu-renderers/src/loaders/gltf-loader.ts
@@ -59,12 +59,6 @@ function componentReader(
}
}
-interface GltfHeader {
- magic: number;
- version: number;
- length: number;
-}
-
interface GltfAccessor {
bufferView?: number;
byteOffset?: number;
@@ -278,7 +272,6 @@ export class GltfLoader {
private static unpackGlb(bytes: Uint8Array): { doc: GltfDocument; binBuffer?: Uint8Array } {
const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
const magic = view.getUint32(0, true);
- const version = view.getUint32(4, true);
const totalLength = view.getUint32(8, true);
if (magic !== GLB_MAGIC) {
diff --git a/packages/webgpu-renderers/src/loaders/loaders.test.ts b/packages/webgpu-renderers/src/loaders/loaders.test.ts
index 8a1d9443..d5ac6a8e 100644
--- a/packages/webgpu-renderers/src/loaders/loaders.test.ts
+++ b/packages/webgpu-renderers/src/loaders/loaders.test.ts
@@ -34,7 +34,7 @@ vt 1.0 1.0
vt 0.0 1.0
f 1/1/1 2/2/1 3/3/1 4/4/1
`;
- const model = parseOBJ(objText, { mtlText });
+ const model = parseOBJ(objText, mtlText);
expect(model.meshes.length).toBe(1);
const mesh = model.meshes[0];
expect(mesh.materialName).toBe("GoldMaterial");
diff --git a/packages/webgpu-renderers/src/loaders/obj-loader.ts b/packages/webgpu-renderers/src/loaders/obj-loader.ts
index e70ff060..ee4dba9c 100644
--- a/packages/webgpu-renderers/src/loaders/obj-loader.ts
+++ b/packages/webgpu-renderers/src/loaders/obj-loader.ts
@@ -1,6 +1,6 @@
import fs from "node:fs";
import path from "node:path";
-import { Matrix4Math, type Vec3, Vector3Math } from "../math3d/index.js";
+import { type Vec3, Vector3Math } from "../math3d/index.js";
import type {
LoadModelOptions,
Material3D,
@@ -8,6 +8,8 @@ import type {
Model3DData,
} from "./types.js";
+type Vec2 = [number, number];
+
export class ObjLoader {
/**
* Parses Wavefront .obj text content (and optional .mtl text) into Model3DData.
diff --git a/packages/webgpu-renderers/src/loaders/types.ts b/packages/webgpu-renderers/src/loaders/types.ts
index 389ee99a..6fe7538c 100644
--- a/packages/webgpu-renderers/src/loaders/types.ts
+++ b/packages/webgpu-renderers/src/loaders/types.ts
@@ -25,6 +25,11 @@ export interface Material3D {
specularColor?: [number, number, number];
ambientColor?: [number, number, number];
shininess?: number;
+ /**
+ * Specular strength parsed from the file (e.g. OBJ `Ks`). The mesh pass
+ * lights with the layer's `specularIntensity`; this is not applied yet.
+ */
+ specularIntensity?: number;
roughness?: number;
metallic?: number;
opacity?: number;
diff --git a/packages/webgpu-renderers/src/loaders/vox-loader.ts b/packages/webgpu-renderers/src/loaders/vox-loader.ts
index c98a5373..47e360db 100644
--- a/packages/webgpu-renderers/src/loaders/vox-loader.ts
+++ b/packages/webgpu-renderers/src/loaders/vox-loader.ts
@@ -51,7 +51,6 @@ export class VoxLoader {
while (offset + 12 <= bytes.byteLength) {
const chunkId = new TextDecoder("ascii").decode(bytes.subarray(offset, offset + 4));
const contentSize = view.getUint32(offset + 4, true);
- const childrenSize = view.getUint32(offset + 8, true);
offset += 12;
const chunkDataOffset = offset;
@@ -95,9 +94,9 @@ export class VoxLoader {
private static buildModel(
voxels: Voxel[],
palette: Array<[number, number, number, number]>,
- sizeX: number,
- sizeY: number,
- sizeZ: number,
+ _sizeX: number,
+ _sizeY: number,
+ _sizeZ: number,
options: LoadModelOptions,
): Model3DData {
const voxelSet = new Set<string>();
diff --git a/packages/webgpu-renderers/src/math3d/camera3d.ts b/packages/webgpu-renderers/src/math3d/camera3d.ts
index 13811a64..b7a7c4d6 100644
--- a/packages/webgpu-renderers/src/math3d/camera3d.ts
+++ b/packages/webgpu-renderers/src/math3d/camera3d.ts
@@ -51,13 +51,6 @@ export interface FrustumPlanes {
far: FrustumPlane;
}
-/** Deterministic pseudo-random gradient noise generator */
-function pseudoNoise(t: number, seed: number): number {
- const n = Math.sin(t * 12.9898 + seed * 78.233) * 43758.5453123;
- const frac = n - Math.floor(n);
- return frac * 2.0 - 1.0;
-}
-
/** Smooth multi-harmonic procedural continuous noise */
function smoothHarmonicNoise(t: number, seed: number, octaves = 3): number {
let total = 0;
diff --git a/packages/webgpu-renderers/src/math3d/path3d.ts b/packages/webgpu-renderers/src/math3d/path3d.ts
index 031a5bd0..e2534f4b 100644
--- a/packages/webgpu-renderers/src/math3d/path3d.ts
+++ b/packages/webgpu-renderers/src/math3d/path3d.ts
@@ -160,7 +160,7 @@ export class Path3D {
/**
* Extracts pitch, yaw, roll Euler angles in degrees from the orthonormal frame.
*/
- private eulerFromFrame(tangent: Vec3, normal: Vec3, binormal: Vec3): Vec3 {
+ private eulerFromFrame(tangent: Vec3, normal: Vec3, _binormal: Vec3): Vec3 {
// Tangent is the forward viewing axis Z_fwd or X_fwd
const pitch = Math.asin(Math.max(-1, Math.min(1, -tangent[1]))) * (180 / Math.PI);
const yaw = Math.atan2(tangent[0], tangent[2]) * (180 / Math.PI);
diff --git a/packages/webgpu-renderers/src/nodes/paragraph.ts b/packages/webgpu-renderers/src/nodes/paragraph.ts
index fc2cc9f8..7292e5ec 100644
--- a/packages/webgpu-renderers/src/nodes/paragraph.ts
+++ b/packages/webgpu-renderers/src/nodes/paragraph.ts
@@ -4,7 +4,11 @@ import type { RenderContextValue } from "../render-context.js";
import type { Color } from "../renderer2d/index.js";
import { signalRegistry } from "../signals/signal-registry.js";
import { parseFontWeight, SlugFontCache } from "../slug/slug-font-cache.js";
-import { SlugGeometry } from "../slug/slug-geometry.js";
+import {
+ SlugGeometry,
+ type SlugGlyphLayout,
+ type SlugLayoutResult,
+} from "../slug/slug-geometry.js";
import type { SlugFont } from "../slug/slug-loader.js";
import {
type AdvancedTextAnimator,
@@ -632,7 +636,9 @@ export function applyGlyphAnimators(
for (const animator of animators) {
if ("selectors" in animator && Array.isArray(animator.selectors)) {
const aeAnim = animator as AETextAnimator | AdvancedTextAnimator;
- const p = aeAnim.props ?? {};
+ // Basic and advanced animators share one props bag; read it as both.
+ const p = (aeAnim.props ?? {}) as ExtendedGlyphProperties &
+ NonNullable<AETextAnimator["props"]>;
const selectorWeight = evaluateAEAnimatorWeight(
aeAnim,
glyph,
@@ -897,14 +903,15 @@ export function drawParagraphNode(
const strokeWidthInFontUnits =
effectiveStrokeWidth * (slugFont.unitsPerEm / fontSize);
- strokeFont = SlugFontCache.getFont(
- fontFamily,
- props.fontWeight,
- ctx.device,
- strokeWidthInFontUnits,
- fontSize,
- props.variableAxes,
- );
+ strokeFont =
+ SlugFontCache.getFont(
+ fontFamily,
+ props.fontWeight,
+ ctx.device,
+ strokeWidthInFontUnits,
+ fontSize,
+ props.variableAxes,
+ ) ?? null;
}
if (!slugFont?.curvesTex || (hasStroke && !strokeFont?.curvesTex)) {
@@ -1075,7 +1082,6 @@ export function drawParagraphNode(
}
}
}
- const passes = 1;
const instanceCount = layout.glyphs.length;
const elapsed =
diff --git a/packages/webgpu-renderers/src/nodes/signal-integration.test.ts b/packages/webgpu-renderers/src/nodes/signal-integration.test.ts
index fbb1f987..11d009b1 100644
--- a/packages/webgpu-renderers/src/nodes/signal-integration.test.ts
+++ b/packages/webgpu-renderers/src/nodes/signal-integration.test.ts
@@ -2,7 +2,6 @@ import fs from "node:fs";
import path from "node:path";
import { describe, expect, it } from "vitest";
import { ensureDevice } from "../device.js";
-import { signalRegistry } from "../signals/signal-registry.js";
import { drawSignalNode } from "./signal.js";
describe("Signal Node Audio Integration", () => {
@@ -84,7 +83,12 @@ describe("Signal Node Audio Integration", () => {
height: 256,
};
- await drawSignalNode(mockCtx as any, encoder, pass, props);
+ await drawSignalNode(
+ mockCtx as any,
+ encoder,
+ pass,
+ props as unknown as Parameters<typeof drawSignalNode>[3],
+ );
pass.end();
// Copy renderTarget to readback buffer
diff --git a/packages/webgpu-renderers/src/nodes/signal.test.ts b/packages/webgpu-renderers/src/nodes/signal.test.ts
index 40d26443..cda91d4e 100644
--- a/packages/webgpu-renderers/src/nodes/signal.test.ts
+++ b/packages/webgpu-renderers/src/nodes/signal.test.ts
@@ -89,7 +89,12 @@ describe("Signal Node", () => {
height: 256,
};
- await drawSignalNode(mockCtx, mockEncoder, mockPass, props);
+ await drawSignalNode(
+ mockCtx,
+ mockEncoder,
+ mockPass,
+ props as unknown as Parameters<typeof drawSignalNode>[3],
+ );
expect(signalRegistry.getOrCreate2DTextureView).toHaveBeenCalledWith(
mockDevice,
@@ -128,7 +133,12 @@ describe("Signal Node", () => {
height: 100,
};
- await drawSignalNode(mockCtx, mockEncoder, mockPass, props);
+ await drawSignalNode(
+ mockCtx,
+ mockEncoder,
+ mockPass,
+ props as unknown as Parameters<typeof drawSignalNode>[3],
+ );
expect(signalRegistry.getOrCreate2DTextureView).not.toHaveBeenCalled();
expect(mockPass.draw).not.toHaveBeenCalled();
@@ -155,7 +165,12 @@ describe("Signal Node", () => {
height: 256,
};
- await drawSignalNode(mockCtx, mockEncoder, mockPass, props);
+ await drawSignalNode(
+ mockCtx,
+ mockEncoder,
+ mockPass,
+ props as unknown as Parameters<typeof drawSignalNode>[3],
+ );
expect(signalRegistry.extractFromAudioSource).toHaveBeenCalledWith(
mockDevice,
@@ -209,7 +224,12 @@ describe("Signal Node", () => {
height: 256,
};
- await drawSignalNode(mockCtx, mockEncoder, mockPass, math2Props);
+ await drawSignalNode(
+ mockCtx,
+ mockEncoder,
+ mockPass,
+ math2Props as unknown as Parameters<typeof drawSignalNode>[3],
+ );
// Audio should ONLY be extracted for audio-extractor-root, NEVER for math-node-1 or math-node-2
expect(signalRegistry.extractFromAudioSource).toHaveBeenCalledTimes(1);
diff --git a/packages/webgpu-renderers/src/nodes/types.ts b/packages/webgpu-renderers/src/nodes/types.ts
index fdcd1fb1..e53eb052 100644
--- a/packages/webgpu-renderers/src/nodes/types.ts
+++ b/packages/webgpu-renderers/src/nodes/types.ts
@@ -1,4 +1,4 @@
-import type { SignalData } from "@gitframes/core";
+import type { SignalData, TextSpan } from "@gitframes/core";
import type { ChartGPUOptions } from "chartgpu";
import type { Rect } from "../renderer2d/index.js";
@@ -128,7 +128,8 @@ export interface ParagraphNodeProps {
opacity?: number;
matrix?: DOMMatrix;
lineHeight?: number;
- align?: TextAlign;
+ /** "start"/"end" follow the line direction (left/right for LTR text) */
+ align?: TextAlign | "start" | "end";
fontWeight?: number | string;
fontStyle?: "normal" | "italic";
letterSpacing?: number;
@@ -173,6 +174,10 @@ export interface ParagraphNodeProps {
durationMs?: number;
elapsedMs?: number;
isCaption?: boolean;
+ /** Rich-text runs, each with its own font, size or color */
+ spans?: TextSpan[];
+ /** Variable-font axis values, e.g. `{ wght: 650 }` */
+ variableAxes?: Record<string, number>;
isVideoMode?: boolean;
renderId?: string;
}
diff --git a/packages/webgpu-renderers/src/renderer2d/motion-graphics.test.ts b/packages/webgpu-renderers/src/renderer2d/motion-graphics.test.ts
index 5e8c6cd3..320bebf7 100644
--- a/packages/webgpu-renderers/src/renderer2d/motion-graphics.test.ts
+++ b/packages/webgpu-renderers/src/renderer2d/motion-graphics.test.ts
@@ -1,9 +1,8 @@
-import { beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
+import { beforeAll, beforeEach, describe, expect, it } from "vitest";
import { Renderer2D } from "./index.js";
import {
createMockCommandEncoder,
createMockDevice,
- createMockRenderPassEncoder,
createMockTexture,
ensureDOMGlobals,
} from "./test-helpers.js";
diff --git a/packages/webgpu-renderers/src/renderer2d/screen-space-relight-pipeline.ts b/packages/webgpu-renderers/src/renderer2d/screen-space-relight-pipeline.ts
index f31d369d..e25a651c 100644
--- a/packages/webgpu-renderers/src/renderer2d/screen-space-relight-pipeline.ts
+++ b/packages/webgpu-renderers/src/renderer2d/screen-space-relight-pipeline.ts
@@ -5,7 +5,6 @@
* and surface normal maps in real-time.
*/
-import type { NormalRelightingOptions } from "@gitframes/core";
import { screenSpaceRelightWgsl } from "../shaders/screen-space-relight.js";
import { BufferPool } from "./buffer-pool.js";
@@ -205,11 +204,7 @@ export class ScreenSpaceRelightPipeline {
matData[14] = 0.0;
matData[15] = 0.0;
- const matBuffer = this.materialPool.getBuffer(
- this.device,
- matData,
- "relight_mat_buffer",
- );
+ const matBuffer = this.materialPool.getBuffer(this.device, matData);
const matBindGroup = this.device.createBindGroup({
layout: this.materialLayout,
diff --git a/packages/webgpu-renderers/src/renderer2d/test-helpers.ts b/packages/webgpu-renderers/src/renderer2d/test-helpers.ts
index 91a90254..8af1ec31 100644
--- a/packages/webgpu-renderers/src/renderer2d/test-helpers.ts
+++ b/packages/webgpu-renderers/src/renderer2d/test-helpers.ts
@@ -13,6 +13,8 @@ export interface MockBuffer {
size: number;
usage: number;
destroy: any;
+ getMappedRange: any;
+ unmap: any;
}
export function createMockTexture(
diff --git a/packages/webgpu-renderers/src/renderer3d/mesh3d-pipeline.ts b/packages/webgpu-renderers/src/renderer3d/mesh3d-pipeline.ts
index 3e1ba988..ca809556 100644
--- a/packages/webgpu-renderers/src/renderer3d/mesh3d-pipeline.ts
+++ b/packages/webgpu-renderers/src/renderer3d/mesh3d-pipeline.ts
@@ -1,4 +1,3 @@
-import type { Mat4 } from "../math3d/index.js";
import { BufferPool } from "../renderer2d/buffer-pool.js";
import { mesh3dWgsl } from "../shaders/mesh3d.js";
@@ -52,6 +51,8 @@ export class Mesh3DPipeline {
depthFormat: GPUTextureFormat = "depth24plus",
cameraLayout?: GPUBindGroupLayout,
lightsLayout?: GPUBindGroupLayout,
+ /** Samples per pixel of the pass this draws into (MSAA). */
+ sampleCount = 1,
) {
this.device = device;
this.modelPool = new BufferPool(
@@ -188,6 +189,7 @@ export class Mesh3DPipeline {
topology: "triangle-list",
cullMode: "none",
},
+ multisample: { count: sampleCount },
});
// MRT pipeline (Color + Linear Depth)
@@ -228,6 +230,7 @@ export class Mesh3DPipeline {
topology: "triangle-list",
cullMode: "none",
},
+ multisample: { count: sampleCount },
});
// 1x1 White dummy texture
diff --git a/packages/webgpu-renderers/src/renderer3d/quad3d-pipeline.ts b/packages/webgpu-renderers/src/renderer3d/quad3d-pipeline.ts
index d7e973fc..67d52c33 100644
--- a/packages/webgpu-renderers/src/renderer3d/quad3d-pipeline.ts
+++ b/packages/webgpu-renderers/src/renderer3d/quad3d-pipeline.ts
@@ -67,6 +67,8 @@ export class Quad3DPipeline {
device: GPUDevice,
format: GPUTextureFormat,
depthFormat: GPUTextureFormat = "depth24plus",
+ /** Samples per pixel of the pass this draws into (MSAA). */
+ sampleCount = 1,
) {
this.device = device;
this.modelPool = new BufferPool(
@@ -187,6 +189,7 @@ export class Quad3DPipeline {
topology: "triangle-list",
cullMode: "none",
},
+ multisample: { count: sampleCount },
});
// 2. MRT pipeline (Color + Linear Depth)
@@ -227,6 +230,7 @@ export class Quad3DPipeline {
topology: "triangle-list",
cullMode: "none",
},
+ multisample: { count: sampleCount },
});
// Unit quad centered at origin: [-0.5, -0.5] to [0.5, 0.5] with normal [0, 0, -1]
diff --git a/packages/webgpu-renderers/src/renderer3d/renderer3d.test.ts b/packages/webgpu-renderers/src/renderer3d/renderer3d.test.ts
index 94cf4c81..1a9408b7 100644
--- a/packages/webgpu-renderers/src/renderer3d/renderer3d.test.ts
+++ b/packages/webgpu-renderers/src/renderer3d/renderer3d.test.ts
@@ -1,4 +1,4 @@
-import { beforeAll, beforeEach, describe, expect, it, vi } from "vitest";
+import { beforeAll, beforeEach, describe, expect, it } from "vitest";
import { Camera3D } from "../math3d/index.js";
import { Renderer3D } from "./renderer3d.js";
import {
@@ -115,8 +115,8 @@ describe("Renderer3D Subsystem", () => {
const { ShadowPipeline } = await import("./shadow-pipeline.js");
const { GlassPipeline } = await import("./glass-pipeline.js");
- const motionBlur = new MotionBlurPipeline(mockDevice, "rgba8unorm");
- const ssao = new SSAOPipeline(mockDevice, "rgba8unorm");
+ const motionBlur = new MotionBlurPipeline(mockDevice);
+ const ssao = new SSAOPipeline(mockDevice);
const shadow = new ShadowPipeline(mockDevice, 1024);
const glass = new GlassPipeline(mockDevice, "rgba8unorm");
diff --git a/packages/webgpu-renderers/src/renderer3d/renderer3d.ts b/packages/webgpu-renderers/src/renderer3d/renderer3d.ts
index a7913570..2bb0587a 100644
--- a/packages/webgpu-renderers/src/renderer3d/renderer3d.ts
+++ b/packages/webgpu-renderers/src/renderer3d/renderer3d.ts
@@ -17,15 +17,13 @@ import {
GltfLoader,
load3DS,
loadFBX,
+ loadGLB,
loadGLTF,
loadOBJ,
loadOFF,
loadPLY,
loadSTL,
loadVOX,
- ObjLoader,
- OffLoader,
- PlyLoader,
StlLoader,
ThreeDSLoader,
VoxLoader,
@@ -37,11 +35,7 @@ import type {
Model3DData,
} from "../loaders/types.js";
import { type Color, parseColor } from "../color.js";
-import {
- type GPUMeshBuffers,
- Mesh3DPipeline,
- type MeshUniformData,
-} from "./mesh3d-pipeline.js";
+import { type GPUMeshBuffers, Mesh3DPipeline } from "./mesh3d-pipeline.js";
import { Quad3DPipeline } from "./quad3d-pipeline.js";
import { Slug3DPipeline } from "./slug3d-pipeline.js";
import type { SlugGlyphBatch } from "../slug/slug-pipeline.js";
@@ -173,6 +167,9 @@ export class Renderer3D {
private samplerCache = new SamplerCache();
private depthTexture: GPUTexture | null = null;
private linearDepthTexture: GPUTexture | null = null;
+ // Multisampled targets the pass draws into; they resolve into the caller's views.
+ private msaaColorTexture: GPUTexture | null = null;
+ private msaaLinearDepthTexture: GPUTexture | null = null;
private cameraUniformData = new Float32Array(80); // 320 bytes = 80 floats
private meshBuffersCache = new WeakMap<Mesh3DData, GPUMeshBuffers>();
private materialTextures = new WeakMap<
@@ -184,16 +181,30 @@ export class Renderer3D {
private lastCameraPose: Camera3DPose | null = null;
private lastCameraBuffer: GPUBuffer | null = null;
- constructor(device: GPUDevice, format: GPUTextureFormat) {
+ /**
+ * Samples per pixel in the 3D pass. Mesh silhouettes (extruded text,
+ * models) are rasterized triangles: without MSAA every edge is a hard
+ * stair-step. 4 is the count every WebGPU device supports; 1 turns it off.
+ */
+ public readonly sampleCount: number;
+
+ constructor(device: GPUDevice, format: GPUTextureFormat, sampleCount = 4) {
this.device = device;
this.format = format;
- this.quad3dPipeline = new Quad3DPipeline(device, format);
+ this.sampleCount = sampleCount;
+ this.quad3dPipeline = new Quad3DPipeline(
+ device,
+ format,
+ "depth24plus",
+ sampleCount,
+ );
this.slug3dPipeline = new Slug3DPipeline(
device,
format,
"depth24plus",
this.quad3dPipeline.cameraLayout,
this.quad3dPipeline.lightsLayout,
+ sampleCount,
);
this.mesh3dPipeline = new Mesh3DPipeline(
device,
@@ -201,6 +212,7 @@ export class Renderer3D {
"depth24plus",
this.quad3dPipeline.cameraLayout,
this.quad3dPipeline.lightsLayout,
+ sampleCount,
);
this.audioMeshDeformPipeline = new AudioMeshDeformPipeline(device);
@@ -286,12 +298,33 @@ export class Renderer3D {
label: "Renderer3DDepthTexture",
size: { width, height },
format: "depth24plus",
+ sampleCount: this.sampleCount,
usage: GPUTextureUsage.RENDER_ATTACHMENT,
});
}
return this.depthTexture;
}
+ private getOrCreateMsaaTexture(
+ current: GPUTexture | null,
+ label: string,
+ format: GPUTextureFormat,
+ width: number,
+ height: number,
+ ): GPUTexture {
+ if (current && current.width === width && current.height === height) {
+ return current;
+ }
+ current?.destroy();
+ return this.device.createTexture({
+ label,
+ size: { width, height },
+ format,
+ sampleCount: this.sampleCount,
+ usage: GPUTextureUsage.RENDER_ATTACHMENT,
+ });
+ }
+
getOrCreateLinearDepthTexture(width: number, height: number): GPUTexture {
if (
!this.linearDepthTexture ||
@@ -430,22 +463,68 @@ export class Renderer3D {
lightsBuffer,
);
+ const msaa = this.sampleCount > 1;
+ if (msaa && loadOp === "load") {
+ // The multisampled target starts empty: it cannot carry the view's contents in.
+ throw new Error(
+ "Renderer3D.beginPass: loadOp 'load' needs a Renderer3D built with sampleCount 1",
+ );
+ }
+
+ // With MSAA each attachment draws into its multisampled twin and
+ // resolves into the caller's view when the pass ends.
+ const attach = (
+ view: GPUTextureView,
+ msaaTex: GPUTexture | null,
+ clearValue: GPUColor,
+ attachmentLoadOp: GPULoadOp,
+ ): GPURenderPassColorAttachment =>
+ msaaTex
+ ? {
+ view: msaaTex.createView(),
+ resolveTarget: view,
+ loadOp: "clear",
+ storeOp: "discard",
+ clearValue,
+ }
+ : { view, loadOp: attachmentLoadOp, storeOp: "store", clearValue };
+
+ if (msaa) {
+ this.msaaColorTexture = this.getOrCreateMsaaTexture(
+ this.msaaColorTexture,
+ "Renderer3DMsaaColor",
+ this.format,
+ surfaceWidth,
+ surfaceHeight,
+ );
+ }
const colorAttachments: GPURenderPassColorAttachment[] = [
- {
- view: colorTargetView,
+ attach(
+ colorTargetView,
+ msaa ? this.msaaColorTexture : null,
+ clearColor,
loadOp,
- storeOp: "store",
- clearValue: clearColor,
- },
+ ),
];
if (linearDepthTargetView) {
- colorAttachments.push({
- view: linearDepthTargetView,
- loadOp: "clear",
- storeOp: "store",
- clearValue: { r: camera.far, g: 0, b: 0, a: 1 },
- });
+ if (msaa) {
+ this.msaaLinearDepthTexture = this.getOrCreateMsaaTexture(
+ this.msaaLinearDepthTexture,
+ "Renderer3DMsaaLinearDepth",
+ "rgba16float",
+ surfaceWidth,
+ surfaceHeight,
+ );
+ }
+ colorAttachments.push(
+ attach(
+ linearDepthTargetView,
+ msaa ? this.msaaLinearDepthTexture : null,
+ { r: camera.far, g: 0, b: 0, a: 1 },
+ "clear",
+ ),
+ );
}
const pass = encoder.beginRenderPass({
@@ -454,7 +533,8 @@ export class Renderer3D {
view: depthTex.createView(),
depthClearValue: 1.0,
depthLoadOp: "clear",
- depthStoreOp: "store",
+ // Nothing reads depth after the pass; the MSAA copy is never resolved.
+ depthStoreOp: msaa ? "discard" : "store",
},
});
@@ -975,6 +1055,8 @@ export class Renderer3D {
destroy(): void {
this.depthTexture?.destroy();
this.linearDepthTexture?.destroy();
+ this.msaaColorTexture?.destroy();
+ this.msaaLinearDepthTexture?.destroy();
this.quad3dPipeline.destroy();
this.slug3dPipeline.destroy();
this.mesh3dPipeline.destroy();
diff --git a/packages/webgpu-renderers/src/renderer3d/shadow-pipeline.ts b/packages/webgpu-renderers/src/renderer3d/shadow-pipeline.ts
index db978bda..fe756b66 100644
--- a/packages/webgpu-renderers/src/renderer3d/shadow-pipeline.ts
+++ b/packages/webgpu-renderers/src/renderer3d/shadow-pipeline.ts
@@ -3,7 +3,7 @@
* Directional and Point Light Shadow Map Generator with Percentage Closer Filtering (PCF).
*/
-import { type Mat4, Matrix4Math, type Vec3, Vector3Math } from "../math3d/index.js";
+import { type Mat4, Matrix4Math, type Vec3 } from "../math3d/index.js";
export interface ShadowMapOptions {
resolution?: number;
@@ -14,14 +14,12 @@ export interface ShadowMapOptions {
}
export class ShadowPipeline {
- private device: GPUDevice;
public shadowDepthTexture: GPUTexture;
public shadowDepthView: GPUTextureView;
public shadowComparisonSampler: GPUSampler;
public lightViewProjMatrix: Mat4;
constructor(device: GPUDevice, resolution = 2048) {
- this.device = device;
this.lightViewProjMatrix = Matrix4Math.identity();
this.shadowDepthTexture = device.createTexture({
@@ -52,7 +50,7 @@ export class ShadowPipeline {
): Mat4 {
const viewMatrix = Matrix4Math.lookAt(lightPos, targetPos, [0, 1, 0]);
const half = orthoSize * 0.5;
- const projMatrix = Matrix4Math.ortho(-half, half, -half, half, near, far);
+ const projMatrix = Matrix4Math.orthographic(-half, half, -half, half, near, far);
this.lightViewProjMatrix = Matrix4Math.multiply(projMatrix, viewMatrix);
return this.lightViewProjMatrix;
}
diff --git a/packages/webgpu-renderers/src/renderer3d/slug3d-pipeline.ts b/packages/webgpu-renderers/src/renderer3d/slug3d-pipeline.ts
index 6ff2a3d0..5a62b3c5 100644
--- a/packages/webgpu-renderers/src/renderer3d/slug3d-pipeline.ts
+++ b/packages/webgpu-renderers/src/renderer3d/slug3d-pipeline.ts
@@ -40,6 +40,8 @@ export class Slug3DPipeline {
depthFormat: GPUTextureFormat,
cameraLayout: GPUBindGroupLayout,
lightsLayout: GPUBindGroupLayout,
+ /** Samples per pixel of the pass this draws into (MSAA). */
+ sampleCount = 1,
) {
this.device = device;
this.modelPool = new BufferPool(
@@ -132,6 +134,7 @@ export class Slug3DPipeline {
topology: "triangle-list",
cullMode: "none",
};
+ const multisample: GPUMultisampleState = { count: sampleCount };
this.pipelineSingle = device.createRenderPipeline({
label: "Slug3DPipelineSingle",
@@ -144,6 +147,7 @@ export class Slug3DPipeline {
},
depthStencil,
primitive,
+ multisample,
});
this.pipelineMrt = device.createRenderPipeline({
@@ -157,6 +161,7 @@ export class Slug3DPipeline {
},
depthStencil,
primitive,
+ multisample,
});
}
diff --git a/packages/webgpu-renderers/src/renderer3d/ssao-pipeline.ts b/packages/webgpu-renderers/src/renderer3d/ssao-pipeline.ts
index 784120c2..c2d056ac 100644
--- a/packages/webgpu-renderers/src/renderer3d/ssao-pipeline.ts
+++ b/packages/webgpu-renderers/src/renderer3d/ssao-pipeline.ts
@@ -168,7 +168,11 @@ export class SSAOPipeline {
finalAOView: GPUTextureView,
options: SSAOPassOptions,
): void {
- const invProj = options.invProjMatrix ?? Matrix4Math.invert(options.projMatrix);
+ // A singular projection has no inverse; identity keeps the pass running.
+ const invProj =
+ options.invProjMatrix ??
+ Matrix4Math.invert(options.projMatrix) ??
+ Matrix4Math.identity();
// Pack SSAO uniforms
const uBuf = new Float32Array(64);
diff --git a/packages/webgpu-renderers/src/renderer3d/text-extruder.test.ts b/packages/webgpu-renderers/src/renderer3d/text-extruder.test.ts
index ecc93a41..449c14ea 100644
--- a/packages/webgpu-renderers/src/renderer3d/text-extruder.test.ts
+++ b/packages/webgpu-renderers/src/renderer3d/text-extruder.test.ts
@@ -1,6 +1,7 @@
import fs from "node:fs";
import path from "node:path";
import * as fontkit from "fontkit";
+import { Canvas, Path2D } from "skia-canvas";
import { describe, expect, it } from "vitest";
import { generateExtrudedTextGeometry } from "./text-extruder.js";
@@ -11,7 +12,7 @@ describe("Extruded 3D Text Polygonal Mesh Generator", () => {
"../../examples/22_gitframes_launch/assets/fonts/Unbounded.ttf",
);
const fontBuf = fs.readFileSync(fontPath);
- const font = fontkit.create(fontBuf);
+ const font = fontkit.create(fontBuf) as fontkit.Font;
const result = generateExtrudedTextGeometry({
text: "GITFRAMES",
@@ -63,7 +64,7 @@ describe("Extruded 3D Text Polygonal Mesh Generator", () => {
"../../examples/22_gitframes_launch/assets/fonts/Inter.ttf",
);
const fontBuf = fs.readFileSync(fontPath);
- const font = fontkit.create(fontBuf);
+ const font = fontkit.create(fontBuf) as fontkit.Font;
const result = generateExtrudedTextGeometry({
text: "GITFRAMES",
@@ -88,7 +89,7 @@ describe("Extruded 3D Text Polygonal Mesh Generator", () => {
"../../examples/22_gitframes_launch/assets/fonts/Unbounded.ttf",
);
const fontBuf = fs.readFileSync(fontPath);
- const font = fontkit.create(fontBuf);
+ const font = fontkit.create(fontBuf) as fontkit.Font;
const result = generateExtrudedTextGeometry({
text: "TFMA",
@@ -108,4 +109,98 @@ describe("Extruded 3D Text Polygonal Mesh Generator", () => {
expect(result.bounds.width).toBeGreaterThan(250);
expect(result.bounds.height).toBeGreaterThan(60);
});
+
+ const loadInter = () =>
+ fontkit.create(
+ fs.readFileSync(
+ path.resolve(
+ process.cwd(),
+ "../../examples/22_gitframes_launch/assets/fonts/Inter.ttf",
+ ),
+ ),
+ ) as fontkit.Font;
+
+ /** Area of the front cap's triangles (vertices at -z, facing the camera). */
+ const frontCapArea = (
+ result: ReturnType<typeof generateExtrudedTextGeometry>,
+ ) => {
+ const front = result.modelData.meshes.find((m) => m.name === "front")!;
+ const p = front.positions;
+ const ix = front.indices;
+ let area = 0;
+ for (let i = 0; i < ix.length; i += 3) {
+ const [a, b, c] = [ix[i]!, ix[i + 1]!, ix[i + 2]!];
+ if (p[a * 3 + 2]! >= 0) continue;
+ area +=
+ Math.abs(
+ (p[b * 3]! - p[a * 3]!) * (p[c * 3 + 1]! - p[a * 3 + 1]!) -
+ (p[c * 3]! - p[a * 3]!) * (p[b * 3 + 1]! - p[a * 3 + 1]!),
+ ) / 2;
+ }
+ return area;
+ };
+
+ it("fills shapes sitting inside a glyph's counter (the R of ®, the C of ©)", () => {
+ const font = loadInter();
+ const fontSize = 200;
+ for (const text of ["O", "®", "©", "@"]) {
+ const glyph = font.layout(text).glyphs[0]!;
+ const scale = fontSize / font.unitsPerEm;
+ // Reference: the glyph filled by skia, measured in covered pixels.
+ const canvas = new Canvas(400, 400);
+ const ctx = canvas.getContext("2d");
+ ctx.translate(100, 300);
+ ctx.scale(scale, -scale);
+ ctx.fill(new Path2D(glyph.path.toSVG()));
+ const data = ctx.getImageData(0, 0, 400, 400).data;
+ let trueArea = 0;
+ for (let i = 3; i < data.length; i += 4) trueArea += data[i]! / 255;
+
+ const result = generateExtrudedTextGeometry({
+ text,
+ font,
+ fontSize,
+ depth: 10,
+ });
+ expect(frontCapArea(result) / trueArea, text).toBeCloseTo(1, 1);
+ }
+ });
+
+ it("maps front UVs across the aligned glyphs", () => {
+ const result = generateExtrudedTextGeometry({
+ text: "GIT",
+ font: loadInter(),
+ fontSize: 120,
+ depth: 20,
+ });
+ const front = result.modelData.meshes.find((m) => m.name === "front")!;
+ let [minU, maxU, minV, maxV] = [1, 0, 1, 0];
+ for (let i = 0; i < front.uvs.length; i += 2) {
+ minU = Math.min(minU, front.uvs[i]!);
+ maxU = Math.max(maxU, front.uvs[i]!);
+ minV = Math.min(minV, front.uvs[i + 1]!);
+ maxV = Math.max(maxV, front.uvs[i + 1]!);
+ }
+ expect(minU).toBeCloseTo(0, 5);
+ expect(maxU).toBeCloseTo(1, 5);
+ expect(minV).toBeCloseTo(0, 5);
+ expect(maxV).toBeCloseTo(1, 5);
+ // Centered text: bounds straddle the origin.
+ expect(result.bounds.minX).toBeCloseTo(-result.bounds.width / 2, 5);
+ });
+
+ it("gives the back cap the fill material, not the bevel color", () => {
+ const result = generateExtrudedTextGeometry({
+ text: "O",
+ font: loadInter(),
+ fontSize: 100,
+ depth: 20,
+ });
+ const front = result.modelData.meshes.find((m) => m.name === "front")!;
+ const side = result.modelData.meshes.find((m) => m.name === "side")!;
+ const zNormals = (n: ArrayLike<number>) =>
+ Array.from({ length: n.length / 3 }, (_, i) => n[i * 3 + 2]!);
+ expect(zNormals(front.normals)).toContain(-1);
+ expect(zNormals(side.normals).every((z) => z === 0)).toBe(true);
+ });
});
diff --git a/packages/webgpu-renderers/src/renderer3d/text-extruder.ts b/packages/webgpu-renderers/src/renderer3d/text-extruder.ts
index cb77f9cc..4a183a42 100644
--- a/packages/webgpu-renderers/src/renderer3d/text-extruder.ts
+++ b/packages/webgpu-renderers/src/renderer3d/text-extruder.ts
@@ -15,17 +15,25 @@ export interface ExtrudedTextGeometryOptions {
font: fontkit.Font;
fontSize?: number;
depth?: number;
+ /** Fixed straight segments per curve; overrides `curveTolerance`. */
curveSegments?: number;
- align?: "left" | "center" | "right";
+ /**
+ * Most a flattened curve may stray from the true outline, in px at
+ * `fontSize`. Each curve gets as many segments as its bend needs, so big
+ * text and tight bowls stay round while straight stems stay cheap.
+ */
+ curveTolerance?: number;
+ /** "start"/"end" read as left/right (the text runs left to right) */
+ align?: "left" | "center" | "right" | "start" | "end";
verticalAlign?: "top" | "middle" | "bottom";
letterSpacing?: number;
+ /** Letter faces: the front cap (toward a default camera, -z) and the back cap */
fill?: string;
+ /** Side walls; defaults to `fill` */
bevelColor?: string;
material?: "lit" | "unlit" | "toon";
shininess?: number;
roughness?: number;
- specularIntensity?: number;
- ambientIntensity?: number;
metallic?: number;
}
@@ -108,13 +116,43 @@ export function isPointInsidePolygon(p: Point2D, poly: Contour): boolean {
return inside;
}
+export interface CurveFlattening {
+ /** Fixed segment count per curve, if set */
+ segments?: number;
+ /** Max distance between curve and polyline, in output units */
+ tolerance: number;
+}
+
+const DEFAULT_CURVE_TOLERANCE = 0.1;
+const MAX_CURVE_SEGMENTS = 64;
+
+/**
+ * Segments needed for a polyline through n uniform steps of a curve to stay
+ * within `tolerance`: the chord error is at most max|B''| / (8 n^2).
+ */
+function curveSegmentCount(
+ maxSecondDerivative: number,
+ flattening: CurveFlattening,
+): number {
+ if (flattening.segments !== undefined) return flattening.segments;
+ const n = Math.ceil(
+ Math.sqrt(maxSecondDerivative / (8 * flattening.tolerance)),
+ );
+ return Math.min(MAX_CURVE_SEGMENTS, Math.max(1, n));
+}
+
function processCurveQuadratic(
p0: Point2D,
cp: Point2D,
end: Point2D,
- segments: number,
+ flattening: CurveFlattening,
out: Contour,
): void {
+ // B'' = 2 (p0 - 2 cp + end), constant along the curve.
+ const segments = curveSegmentCount(
+ 2 * Math.hypot(p0.x - 2 * cp.x + end.x, p0.y - 2 * cp.y + end.y),
+ flattening,
+ );
for (let s = 1; s <= segments; s++) {
const t = s / segments;
const mt = 1 - t;
@@ -130,9 +168,18 @@ function processCurveCubic(
cp1: Point2D,
cp2: Point2D,
end: Point2D,
- segments: number,
+ flattening: CurveFlattening,
out: Contour,
): void {
+ // B'' is linear in t, so its largest magnitude is at an end: 6 x the larger second difference.
+ const segments = curveSegmentCount(
+ 6 *
+ Math.max(
+ Math.hypot(p0.x - 2 * cp1.x + cp2.x, p0.y - 2 * cp1.y + cp2.y),
+ Math.hypot(cp1.x - 2 * cp2.x + end.x, cp1.y - 2 * cp2.y + end.y),
+ ),
+ flattening,
+ );
for (let s = 1; s <= segments; s++) {
const t = s / segments;
const mt = 1 - t;
@@ -156,7 +203,7 @@ function extractGlyphContours(
glyphX: number,
glyphY: number,
scale: number,
- curveSegments: number,
+ flattening: CurveFlattening,
): Contour[] {
const svg = glyph.path?.toSVG();
if (!svg) return [];
@@ -189,7 +236,8 @@ function extractGlyphContours(
for (const edge of edges) {
const name = edge[0];
- const args = edge.slice(1);
+ // skia's edges are [verb, ...coordinates]
+ const args = edge.slice(1) as number[];
if (name === "moveTo") {
finishCurrent();
@@ -213,7 +261,7 @@ function extractGlyphContours(
x: (glyphX + args[2]!) * scale,
y: -(glyphY + args[3]!) * scale,
};
- processCurveQuadratic(curr, cp, end, curveSegments, current);
+ processCurveQuadratic(curr, cp, end, flattening, current);
curr = end;
} else if (name === "bezierCurveTo") {
const cp1: Point2D = {
@@ -228,7 +276,7 @@ function extractGlyphContours(
x: (glyphX + args[4]!) * scale,
y: -(glyphY + args[5]!) * scale,
};
- processCurveCubic(curr, cp1, cp2, end, curveSegments, current);
+ processCurveCubic(curr, cp1, cp2, end, flattening, current);
curr = end;
} else if (name === "closePath") {
finishCurrent();
@@ -238,53 +286,103 @@ function extractGlyphContours(
return contours;
}
+/** Side-wall segments meeting at more than this angle (30°) keep a hard edge. */
+const SIDE_CREASE_COS = Math.cos((30 * Math.PI) / 180);
+
interface PolygonHierarchy {
outer: Contour;
holes: Contour[];
}
+/**
+ * Group a glyph's contours into filled outlines and their holes by nesting
+ * depth: a contour inside an even number of others is an outline (the ring
+ * of "O", or the "R" sitting in the counter of "®"), one inside an odd number
+ * is a hole of the smallest outline around it.
+ */
function organizeGlyphPolygons(glyphContours: Contour[]): PolygonHierarchy[] {
- if (glyphContours.length === 0) return [];
-
- const sorted = [...glyphContours].sort(
- (a, b) =>
- Math.abs(computeContourSignedArea(b)) -
- Math.abs(computeContourSignedArea(a)),
- );
-
- const outers: PolygonHierarchy[] = [];
-
- for (const contour of sorted) {
- const area = Math.abs(computeContourSignedArea(contour));
- let parent: PolygonHierarchy | null = null;
-
- for (const out of outers) {
- const parentArea = Math.abs(computeContourSignedArea(out.outer));
- if (area < parentArea) {
- // Check sample point or mid-point
- const sample = contour[0]!;
- const mid = {
- x: (contour[0]!.x + contour[Math.floor(contour.length / 2)]!.x) / 2,
- y: (contour[0]!.y + contour[Math.floor(contour.length / 2)]!.y) / 2,
- };
- if (
- isPointInsidePolygon(sample, out.outer) ||
- isPointInsidePolygon(mid, out.outer)
- ) {
- parent = out;
- break;
- }
+ const contours = glyphContours.map((contour) => ({
+ contour,
+ area: Math.abs(computeContourSignedArea(contour)),
+ }));
+ // Outermost first, so every contour's container is handled before it.
+ contours.sort((a, b) => b.area - a.area);
+
+ const containers: number[][] = contours.map(() => []);
+ for (let i = 0; i < contours.length; i++) {
+ const inner = contours[i]!.contour;
+ const mid = inner[Math.floor(inner.length / 2)]!;
+ for (let j = 0; j < i; j++) {
+ if (contours[j]!.area <= contours[i]!.area) continue;
+ const outer = contours[j]!.contour;
+ // Contours do not cross (the path is simplified), so a vertex
+ // decides; a second guards a vertex lying on the other outline.
+ if (
+ isPointInsidePolygon(inner[0]!, outer) ||
+ isPointInsidePolygon(mid, outer)
+ ) {
+ containers[i]!.push(j);
}
}
+ }
- if (parent) {
- parent.holes.push(contour);
+ const polygons: PolygonHierarchy[] = [];
+ const polygonOf = new Map<number, PolygonHierarchy>();
+ for (let i = 0; i < contours.length; i++) {
+ const around = containers[i]!;
+ if (around.length % 2 === 0) {
+ const poly = { outer: contours[i]!.contour, holes: [] };
+ polygons.push(poly);
+ polygonOf.set(i, poly);
} else {
- outers.push({ outer: contour, holes: [] });
+ // The innermost container is the latest (smallest) one.
+ polygonOf
+ .get(around[around.length - 1]!)
+ ?.holes.push(contours[i]!.contour);
}
}
+ return polygons;
+}
- return outers;
+/**
+ * A glyph's outlines are flattened and sorted into outlines and holes once per
+ * font, size and flattening, at the glyph origin; each use gets a translated
+ * copy (callers reorder and move the points they are given).
+ */
+const glyphOutlineCache = new WeakMap<
+ object,
+ Map<string, PolygonHierarchy[]>
+>();
+
+function glyphPolygonsAt(
+ font: fontkit.Font,
+ glyph: fontkit.Glyph,
+ glyphX: number,
+ glyphY: number,
+ scale: number,
+ flattening: CurveFlattening,
+): PolygonHierarchy[] {
+ let byGlyph = glyphOutlineCache.get(font);
+ if (!byGlyph) {
+ byGlyph = new Map();
+ glyphOutlineCache.set(font, byGlyph);
+ }
+ const key = `${glyph.id}|${scale}|${flattening.segments ?? ""}|${flattening.tolerance}`;
+ let atOrigin = byGlyph.get(key);
+ if (!atOrigin) {
+ atOrigin = organizeGlyphPolygons(
+ extractGlyphContours(glyph, 0, 0, scale, flattening),
+ );
+ byGlyph.set(key, atOrigin);
+ }
+ const dx = glyphX * scale;
+ const dy = -glyphY * scale;
+ const moved = (c: Contour): Contour =>
+ c.map((p) => ({ x: p.x + dx, y: p.y + dy }));
+ return atOrigin.map((poly) => ({
+ outer: moved(poly.outer),
+ holes: poly.holes.map(moved),
+ }));
}
export function generateExtrudedTextGeometry(
@@ -294,7 +392,16 @@ export function generateExtrudedTextGeometry(
const fontSize = options.fontSize ?? 48;
const depth = options.depth ?? 20;
const halfDepth = depth / 2;
- const curveSegments = Math.max(2, options.curveSegments ?? 5);
+ const flattening: CurveFlattening = {
+ segments:
+ options.curveSegments !== undefined
+ ? Math.max(2, options.curveSegments)
+ : undefined,
+ tolerance: Math.max(
+ 1e-3,
+ options.curveTolerance ?? DEFAULT_CURVE_TOLERANCE,
+ ),
+ };
const unitsPerEm = font.unitsPerEm || 1000;
const scale = fontSize / unitsPerEm;
@@ -308,15 +415,14 @@ export function generateExtrudedTextGeometry(
const glyphX = cursorX + (pos.xOffset ?? 0);
const glyphY = pos.yOffset ?? 0;
- const glyphContours = extractGlyphContours(
+ const glyphPolys = glyphPolygonsAt(
+ font,
glyph,
glyphX,
glyphY,
scale,
- curveSegments,
+ flattening,
);
-
- const glyphPolys = organizeGlyphPolygons(glyphContours);
for (const gp of glyphPolys) {
polygons.push(gp);
}
@@ -359,8 +465,9 @@ export function generateExtrudedTextGeometry(
const h = maxY - minY;
let alignOffsetX = (minX + maxX) / 2;
- if (options.align === "left") alignOffsetX = minX;
- if (options.align === "right") alignOffsetX = maxX;
+ if (options.align === "left" || options.align === "start")
+ alignOffsetX = minX;
+ if (options.align === "right" || options.align === "end") alignOffsetX = maxX;
let alignOffsetY = (minY + maxY) / 2;
if (options.verticalAlign === "top") alignOffsetY = minY;
@@ -378,6 +485,11 @@ export function generateExtrudedTextGeometry(
}
}
}
+ // Bounds (and the UVs built from them) describe the aligned geometry.
+ minX -= alignOffsetX;
+ maxX -= alignOffsetX;
+ minY -= alignOffsetY;
+ maxY -= alignOffsetY;
const frontPositions: number[] = [];
const frontNormals: number[] = [];
@@ -389,35 +501,70 @@ export function generateExtrudedTextGeometry(
const sideUvs: number[] = [];
const sideIndices: number[] = [];
- const addSideQuad = (p0: Point2D, p1: Point2D): void => {
- const dx = p1.x - p0.x;
- const dy = p1.y - p0.y;
- const len = Math.hypot(dx, dy);
- if (len < 1e-6) return;
-
- const nx = dy / len;
- const ny = -dx / len;
-
- const baseIdx = sidePositions.length / 3;
-
- sidePositions.push(p0.x, p0.y, halfDepth);
- sideNormals.push(nx, ny, 0);
- sideUvs.push(0, 0);
-
- sidePositions.push(p1.x, p1.y, halfDepth);
- sideNormals.push(nx, ny, 0);
- sideUvs.push(1, 0);
-
- sidePositions.push(p1.x, p1.y, -halfDepth);
- sideNormals.push(nx, ny, 0);
- sideUvs.push(1, 1);
+ // Side walls are columns of vertex pairs (top at +z, bottom at -z) joined
+ // by quads. Along a smooth run neighbouring quads share their column, so
+ // a flattened curve shades as one surface (averaged normals) at half the
+ // vertices; a corner sharper than the crease angle gets one column per
+ // side and stays crisp. u runs along the outline, v front to back.
+ const addColumn = (p: Point2D, n: Point2D, u: number): number => {
+ const index = sidePositions.length / 3;
+ sidePositions.push(p.x, p.y, halfDepth, p.x, p.y, -halfDepth);
+ sideNormals.push(n.x, n.y, 0, n.x, n.y, 0);
+ sideUvs.push(u, 0, u, 1);
+ return index;
+ };
- sidePositions.push(p0.x, p0.y, -halfDepth);
- sideNormals.push(nx, ny, 0);
- sideUvs.push(0, 1);
+ const addSideWall = (contour: Contour): void => {
+ const pts = contour.filter((p, k) => {
+ const next = contour[(k + 1) % contour.length]!;
+ return Math.hypot(next.x - p.x, next.y - p.y) >= 1e-6;
+ });
+ const count = pts.length;
+ if (count < 2) return;
+ const edgeLengths: number[] = [];
+ const faceNormals = pts.map((p, k) => {
+ const next = pts[(k + 1) % count]!;
+ const len = Math.hypot(next.x - p.x, next.y - p.y);
+ edgeLengths.push(len);
+ return { x: (next.y - p.y) / len, y: -(next.x - p.x) / len };
+ });
+ const perimeter = edgeLengths.reduce((a, b) => a + b, 0);
+
+ // Column each edge starts from, and column the previous edge ends on.
+ const startColumn: number[] = [];
+ const endColumn: number[] = [];
+ let along = 0;
+ for (let k = 0; k < count; k++) {
+ const incoming = faceNormals[(k - 1 + count) % count]!;
+ const outgoing = faceNormals[k]!;
+ const u = along / perimeter;
+ const sx = incoming.x + outgoing.x;
+ const sy = incoming.y + outgoing.y;
+ const smooth =
+ incoming.x * outgoing.x + incoming.y * outgoing.y >= SIDE_CREASE_COS;
+ const averaged = smooth
+ ? { x: sx / Math.hypot(sx, sy), y: sy / Math.hypot(sx, sy) }
+ : null;
+ if (averaged && k > 0) {
+ startColumn[k] = endColumn[k] = addColumn(pts[k]!, averaged, u);
+ } else {
+ // Vertex 0 always splits, smooth or not: its end column closes the outline at u = 1.
+ endColumn[k] = addColumn(
+ pts[k]!,
+ averaged ?? incoming,
+ k === 0 ? 1 : u,
+ );
+ startColumn[k] = addColumn(pts[k]!, averaged ?? outgoing, u);
+ }
+ along += edgeLengths[k]!;
+ }
- sideIndices.push(baseIdx, baseIdx + 2, baseIdx + 1);
- sideIndices.push(baseIdx, baseIdx + 3, baseIdx + 2);
+ for (let k = 0; k < count; k++) {
+ const a = startColumn[k]!; // top; a + 1 is its bottom
+ const b = endColumn[(k + 1) % count]!;
+ sideIndices.push(a, b + 1, b);
+ sideIndices.push(a, a + 1, b + 1);
+ }
};
for (const poly of polygons) {
@@ -447,48 +594,45 @@ export function generateExtrudedTextGeometry(
const triangles = earcut(earcutCoords, holeIndices, 2);
const vertCount = earcutCoords.length / 2;
+ // The front cap faces the default camera, which looks along +z (as
+ // planar layers do, normal -z).
const frontBaseIdx = frontPositions.length / 3;
for (let v = 0; v < vertCount; v++) {
const vx = earcutCoords[v * 2]!;
const vy = earcutCoords[v * 2 + 1]!;
- frontPositions.push(vx, vy, halfDepth);
- frontNormals.push(0, 0, 1);
+ frontPositions.push(vx, vy, -halfDepth);
+ frontNormals.push(0, 0, -1);
frontUvs.push((vx - minX) / (w || 1), (vy - minY) / (h || 1));
}
for (let t = 0; t < triangles.length; t += 3) {
frontIndices.push(
frontBaseIdx + triangles[t]!,
- frontBaseIdx + triangles[t + 1]!,
frontBaseIdx + triangles[t + 2]!,
+ frontBaseIdx + triangles[t + 1]!,
);
}
- const backBaseIdx = sidePositions.length / 3;
+ // The back cap is a face of the letter like the front: it takes the
+ // fill, so text seen from behind is not painted in the bevel color.
+ const backBaseIdx = frontPositions.length / 3;
for (let v = 0; v < vertCount; v++) {
const vx = earcutCoords[v * 2]!;
const vy = earcutCoords[v * 2 + 1]!;
- sidePositions.push(vx, vy, -halfDepth);
- sideNormals.push(0, 0, -1);
- sideUvs.push((vx - minX) / (w || 1), (vy - minY) / (h || 1));
+ frontPositions.push(vx, vy, halfDepth);
+ frontNormals.push(0, 0, 1);
+ frontUvs.push((vx - minX) / (w || 1), (vy - minY) / (h || 1));
}
for (let t = 0; t < triangles.length; t += 3) {
- sideIndices.push(
+ frontIndices.push(
backBaseIdx + triangles[t]!,
- backBaseIdx + triangles[t + 2]!,
backBaseIdx + triangles[t + 1]!,
+ backBaseIdx + triangles[t + 2]!,
);
}
- for (let k = 0; k < outer.length; k++) {
- const next = (k + 1) % outer.length;
- addSideQuad(outer[k]!, outer[next]!);
- }
-
+ addSideWall(outer);
for (const hole of poly.holes) {
- for (let k = 0; k < hole.length; k++) {
- const next = (k + 1) % hole.length;
- addSideQuad(hole[k]!, hole[next]!);
- }
+ addSideWall(hole);
}
}
@@ -503,8 +647,6 @@ export function generateExtrudedTextGeometry(
shading: options.material ?? "lit",
shininess: options.shininess ?? 32,
roughness: options.roughness,
- specularIntensity: options.specularIntensity ?? 0.8,
- ambientIntensity: options.ambientIntensity ?? 1.0,
metallic: options.metallic,
twoSided: true,
};
@@ -515,8 +657,6 @@ export function generateExtrudedTextGeometry(
shading: options.material ?? "lit",
shininess: options.shininess ?? 32,
roughness: options.roughness,
- specularIntensity: options.specularIntensity ?? 0.8,
- ambientIntensity: options.ambientIntensity ?? 1.0,
metallic: options.metallic,
twoSided: true,
};
@@ -561,6 +701,13 @@ export function generateExtrudedTextGeometry(
"side-material": sideMaterial,
},
animations: [],
+ bounds: {
+ min: [minX, minY, -halfDepth],
+ max: [maxX, maxY, halfDepth],
+ center: [(minX + maxX) / 2, (minY + maxY) / 2, 0],
+ size: [w, h, depth],
+ boundingSphereRadius: Math.hypot(w, h, depth) / 2,
+ },
};
return {
diff --git a/packages/webgpu-renderers/src/signals/signal-registry.ts b/packages/webgpu-renderers/src/signals/signal-registry.ts
index 6ee38430..dd0b4a11 100644
--- a/packages/webgpu-renderers/src/signals/signal-registry.ts
+++ b/packages/webgpu-renderers/src/signals/signal-registry.ts
@@ -782,7 +782,7 @@ export class SignalRegistry {
*/
public getOrCreate1DBuffer(
device: GPUDevice,
- _encoder: GPUCommandEncoder,
+ _encoder: GPUCommandEncoder | null,
nodeId: string,
elapsedTime: number,
duration: number,
@@ -1061,7 +1061,7 @@ export class SignalRegistry {
*/
public getOrCreate2DTextureView(
device: GPUDevice,
- _encoder: GPUCommandEncoder,
+ _encoder: GPUCommandEncoder | null,
nodeId: string,
elapsedTime: number,
duration: number,
diff --git a/packages/webgpu-renderers/src/slug/slug-font-cache.ts b/packages/webgpu-renderers/src/slug/slug-font-cache.ts
index 2fc2cb99..83a917f8 100644
--- a/packages/webgpu-renderers/src/slug/slug-font-cache.ts
+++ b/packages/webgpu-renderers/src/slug/slug-font-cache.ts
@@ -3,6 +3,11 @@ import type { FontkitFont } from "./slug-generator.js";
import { SlugGenerator } from "./slug-generator.js";
import type { SlugFont } from "./slug-loader.js";
+/** Per-font load timings are debug output: shown only with LOG_LEVEL=debug or trace. */
+const VERBOSE_FONT_LOGS =
+ typeof process !== "undefined" &&
+ /^(debug|trace)$/i.test(process.env?.LOG_LEVEL ?? "");
+
export type FontLoadListener = (fontFamily: string) => void;
export function parseFontWeight(fontWeight?: string | number): number {
@@ -136,6 +141,11 @@ export class SlugFontCache {
return g[GLOBAL_PARSED_KEY];
}
+ /** Register an already parsed font (e.g. by FontManager) under a family name. */
+ public static setParsed(fontFamily: string, font: FontkitFont): void {
+ SlugFontCache.parsedFontCache.set(fontFamily, font);
+ }
+
public static getParsed(fontFamily: string): FontkitFont | null {
return SlugFontCache.parsedFontCache.get(fontFamily) || null;
}
@@ -700,7 +710,7 @@ export class SlugFontCache {
buffer = nodeBuffer.buffer.slice(
nodeBuffer.byteOffset,
nodeBuffer.byteOffset + nodeBuffer.byteLength,
- );
+ ) as ArrayBuffer;
} else {
// Browser/Web or remote URL environment
const response = await fetch(fontUrl);
@@ -726,12 +736,13 @@ export class SlugFontCache {
const generated = generator.generate(device, font);
const generateTime = performance.now();
- console.log(
- `[SlugFontCache] Preloaded and parsed font "${fontFamily}":\n` +
- ` - fontkit parse: ${(parseTime - startTime).toFixed(2)}ms\n` +
- ` - slug generation: ${(generateTime - parseTime).toFixed(2)}ms\n` +
- ` - total: ${(generateTime - startTime).toFixed(2)}ms`,
- );
+ if (VERBOSE_FONT_LOGS)
+ console.log(
+ `[SlugFontCache] Preloaded and parsed font "${fontFamily}":\n` +
+ ` - fontkit parse: ${(parseTime - startTime).toFixed(2)}ms\n` +
+ ` - slug generation: ${(generateTime - parseTime).toFixed(2)}ms\n` +
+ ` - total: ${(generateTime - startTime).toFixed(2)}ms`,
+ );
generated.fontFamily = fontFamily;
SlugFontCache.cache.set(fontFamily, generated);
diff --git a/packages/webgpu-renderers/src/slug/slug-generator.ts b/packages/webgpu-renderers/src/slug/slug-generator.ts
index b56e2cd7..53d7ce93 100644
--- a/packages/webgpu-renderers/src/slug/slug-generator.ts
+++ b/packages/webgpu-renderers/src/slug/slug-generator.ts
@@ -33,6 +33,11 @@ export interface FontkitFont {
unitsPerEm: number;
variationAxes?: Record<string, unknown>;
getVariation?(settings: Record<string, number>): FontkitFont;
+ /** Shapes a string (kerning, ligatures); fontkit fonts always have it. */
+ layout?(text: string): {
+ glyphs: unknown[];
+ positions: Array<{ xAdvance: number }>;
+ };
}
const TEXTURE_WIDTH = 4096;
diff --git a/packages/webgpu-renderers/src/slug/slug-geometry.ts b/packages/webgpu-renderers/src/slug/slug-geometry.ts
index 8eb9b3ab..5d907c06 100644
--- a/packages/webgpu-renderers/src/slug/slug-geometry.ts
+++ b/packages/webgpu-renderers/src/slug/slug-geometry.ts
@@ -99,7 +99,7 @@ function resolveParsedFont(font: SlugFont): FontkitFont | null {
const parsed = SlugFontCache.getParsed(font.fontFamily);
if (parsed) return parsed;
}
- const globalParsed = (globalThis as Record<string, unknown>)[
+ const globalParsed = (globalThis as Record<symbol, unknown>)[
Symbol.for("gatewai.slugFontCache.parsed")
] as Map<string, FontkitFont> | undefined;
if (globalParsed) {
@@ -124,13 +124,14 @@ export function getKernOffset(font: SlugFont, a: number, b: number): number {
if (src) {
try {
const s = String.fromCodePoint(a) + String.fromCodePoint(b);
- const laid = src.layout(s);
+ const laid = src.layout?.(s);
+ if (!laid) throw new Error("font cannot shape text");
const total = laid.positions.reduce(
(acc: number, p: { xAdvance: number }) => acc + p.xAdvance,
0,
);
- const advA = src.glyphForCodePoint(a)?.advanceWidth ?? 0;
- const advB = src.glyphForCodePoint(b)?.advanceWidth ?? 0;
+ const advA = src.glyphForCodePoint?.(a)?.advanceWidth ?? 0;
+ const advB = src.glyphForCodePoint?.(b)?.advanceWidth ?? 0;
v = laid.glyphs.length === 2 ? total - (advA + advB) : 0;
} catch {
v = 0;
diff --git a/packages/webgpu-renderers/src/text-animations/advanced-types.ts b/packages/webgpu-renderers/src/text-animations/advanced-types.ts
index 08c26a2e..9ec78fc4 100644
--- a/packages/webgpu-renderers/src/text-animations/advanced-types.ts
+++ b/packages/webgpu-renderers/src/text-animations/advanced-types.ts
@@ -340,6 +340,8 @@ export const ExtendedGlyphPropertiesSchema = z.object({
rotationX: z.number().optional(),
rotationY: z.number().optional(),
rotationZ: z.number().optional(),
+ /** Volumetric 3D arrangement (set by `volumetricFormation()`) */
+ formation: VolumetricFormationConfigSchema.optional(),
skew: z.number().optional(),
skewAxis: z.number().optional(),
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index faf36bc7..d2c2a597 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -2086,6 +2086,9 @@ importers:
'@types/react':
specifier: ^19.2.4
version: 19.2.14
+ '@webgpu/types':
+ specifier: ^0.1.70
+ version: 0.1.70
dotenv:
specifier: ^16.4.5
version: 16.6.1
@@ -2117,12 +2120,18 @@ importers:
'@gitframes/tsconfig':
specifier: workspace:*
version: link:../tsconfig
+ '@webgpu/types':
+ specifier: ^0.1.70
+ version: 0.1.70
tsdown:
specifier: ^0.19.0
typescript:
specifier: ^5.8.3
version: 5.8.3
+ vitest:
+ specifier: ^4.1.5
packages/gitframes:
dependencies:
@@ -2420,6 +2429,9 @@ importers:
'@types/node':
specifier: ^24.1.0
version: 24.10.14
+ '@webgpu/types':
+ specifier: ^0.1.70
+ version: 0.1.70
tsdown:
specifier: ^0.19.0
diff --git a/skills/gitframes-compose/SKILL.md b/skills/gitframes-compose/SKILL.md
index f4b8a32e..fb936c30 100644
--- a/skills/gitframes-compose/SKILL.md
+++ b/skills/gitframes-compose/SKILL.md
@@ -116,6 +116,17 @@ const anim = LayerAnimation.create()
// Directions: in, out, inOut (e.g. "expo.out", "back.out(1.7)", "power2.inOut")
```
+For more than two keyframes on one prop, list them with `keys`: each tuple is `[frame, value, ease?]`, where the ease shapes the motion *into* that keyframe. Frames are rounded, so beat math (`at + BEAT / 2`) can go straight in:
+
+```typescript
+const pop = LayerAnimation.create()
+ .keys("scale", [[0, 0], [10, 1.12, "back.out(1.6)"], [16, 1, "power2.out"]])
+ .keys("opacity", [[0, 0], [6, 1]]);
+// Static form: LayerAnimation.keys("rotation", [[0, -8], [20, 0, "expo.out"]])
+```
+
+Keyframes past the layer's `durationFrames` never play; the render warns (`animation_truncated`) and `renderVideo({ qa: true })` reports it.
+
### 2. Kinetic Typography (`TextAnimator`)
Text animators operate at the GPU glyph instancing level, cascading offsets across characters, words, or lines:
diff --git a/skills/gitframes-render/SKILL.md b/skills/gitframes-render/SKILL.md
index d4acb7fe..dde67739 100644
--- a/skills/gitframes-render/SKILL.md
+++ b/skills/gitframes-render/SKILL.md
@@ -127,6 +127,8 @@ const result = await comp.renderVideo({
await result.cleanup?.();
```
+Pass `qa: true` to have the engine check the output while it renders (see **Built-in Video QA** below) and return the findings as `result.qa`.
+
---
## Production Render Script Pattern
@@ -169,6 +171,8 @@ if (mode === "frames") {
process.exit(0);
```
+To run against the engine's TypeScript source instead of built `dist` (while changing the engine itself), give tsx a `tsconfig.dev.json` that extends the project's tsconfig and adds `"../../packages/**/*"` and `"../../nodes/**/*"` to `include` (tsx only applies compiler options, such as the decorators `server-utils` needs, to included files), then run `tsx --tsconfig tsconfig.dev.json --conditions=development src/render.ts`. `examples/22_gitframes_launch` has it as `pnpm render:src`.
+
CLI Usage:
```bash
# Render specific keyframes for instant review:
@@ -180,9 +184,39 @@ tsx src/render.ts
---
-## Video QA & Automated Inspection
+## Built-in Video QA (`renderVideo({ qa })`)
+
+The engine measures the video while it renders: every frame as it is read back for the encoder, and the audio as it is mixed. No second decode pass, and the numbers describe exactly what was encoded.
+
+```typescript
+import { formatQaReport } from "gitframes";
+
+const result = await comp.renderVideo({
+ outputPath: "output/final-video.mp4",
+ qa: { targetLufs: -14 }, // or `qa: true` for defaults (no loudness target)
+});
+console.log(formatQaReport(result.qa!));
+if (!result.qa!.passed) process.exitCode = 1; // fail CI / the agent loop
+```
+
+`result.qa` is structured, so branch on it instead of parsing logs:
+
+| `issues[].code` | Severity | Meaning |
+|---|---|---|
+| `frames_missing` | error | Fewer frames encoded than the composition has |
+| `black_frames` | warning | ≥ `minBlackSec` (0.5 s) of black frames; has `startSec`/`endSec` |
+| `audio_clipping` | warning | Samples at or beyond full scale |
+| `loudness_off_target` | warning | Integrated loudness more than `lufsTolerance` (1 LU) from `targetLufs` |
+| `animation_truncated` | warning | Keyframes past a layer's clip window never play; has `layerId` |
+| `frozen_video` | info | Picture unchanged for ≥ `minFrozenSec` (2 s); fine for held title cards |
+| `audio_silence` | info | ≥ `minSilenceSec` (2 s) below `silenceDbfs` (-60) |
+| `audio_silent` | info | The whole track is silent |
+
+`passed` is false when any error or warning is present. `qa.audio.integratedLufs` is ITU-R BS.1770 loudness (streaming platforms normalize to about -14 LUFS); `qa.audio.peakDbfs` is the sample peak.
+
+## Inspecting a Finished File with ffmpeg
-After rendering, inspect the output with `ffmpeg`/`ffprobe` to verify audio presence, duration, and absence of black frames:
+For files rendered elsewhere, or to compare against a previous render, inspect with `ffmpeg`/`ffprobe`:
```bash
# 1. Check video and audio stream properties