Skip to content

Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.

feat(effects): check an effect's code off the main thread, and price the tiers on an M1

Etienne Lescot3759811cbb05
diff --git a/crates/compositor-view-napi/src/lib.rs b/crates/compositor-view-napi/src/lib.rs
index 447df805c..b25b0dfe0 100644
--- a/crates/compositor-view-napi/src/lib.rs
+++ b/crates/compositor-view-napi/src/lib.rs
@@ -401,20 +401,41 @@ pub struct ShaderEffectCheck {
pub iterations: Option<f64>,
}
+pub struct CheckShaderEffectTask {
+ code: String,
+ params_json: String,
+}
+
+impl Task for CheckShaderEffectTask {
+ type Output = ShaderEffectCheck;
+ type JsValue = ShaderEffectCheck;
+
+ fn compute(&mut self) -> Result<Self::Output> {
+ Ok(match openscreen_compositor::shader_fx::check_for_agent_json(&self.code, &self.params_json) {
+ Ok(report) => ShaderEffectCheck {
+ error: None,
+ tier: Some(report.tier.name().to_string()),
+ frame_reads: Some(report.frame_reads as f64),
+ iterations: Some(report.iterations as f64),
+ },
+ Err(error) => ShaderEffectCheck { error: Some(error), tier: None, frame_reads: None, iterations: None },
+ })
+ }
+
+ fn resolve(&mut self, _env: Env, check: Self::Output) -> Result<Self::JsValue> {
+ Ok(check)
+ }
+}
+
/// Vérifie le WGSL d'un effet shader : validation, bornes des boucles, compilation réelle pour le
-/// backend de la machine quand elle existe (FXC sur Windows), et palier de coût. `params_json` :
-/// les paramètres déclarés, `[{ "name": "strength", "type": "number" }, …]`.
+/// backend de la machine (FXC sur Windows, Metal sur macOS, SPIR-V sur Linux), et palier de coût.
+/// `params_json` : les paramètres déclarés, `[{ "name": "strength", "type": "number" }, …]`.
+///
+/// `AsyncTask`, comme `segment_frame` : la compilation Metal d'une source neuve prend 120 ms, et
+/// jusqu'à 375 ms pour la première du processus. Ce temps n'a rien à faire sur le thread principal.
#[napi]
-pub fn check_shader_effect(code: String, params_json: String) -> ShaderEffectCheck {
- match openscreen_compositor::shader_fx::check_for_agent_json(&code, &params_json) {
- Ok(report) => ShaderEffectCheck {
- error: None,
- tier: Some(report.tier.name().to_string()),
- frame_reads: Some(report.frame_reads as f64),
- iterations: Some(report.iterations as f64),
- },
- Err(error) => ShaderEffectCheck { error: Some(error), tier: None, frame_reads: None, iterations: None },
- }
+pub fn check_shader_effect(code: String, params_json: String) -> AsyncTask<CheckShaderEffectTask> {
+ AsyncTask::new(CheckShaderEffectTask { code, params_json })
}
/// Installe la scène de l'app (JSON `SceneDescription`) sur la vue : layout preset piloté par
diff --git a/docs/shader-effects.md b/docs/shader-effects.md
index 24104cf4c..c7c55341a 100644
--- a/docs/shader-effects.md
+++ b/docs/shader-effects.md
@@ -421,6 +421,9 @@ Il y a deux compilations, à deux moments :
`setEffectDefinition`), le processus principal appelle `checkShaderEffect(code, params)` de l'addon.
- **La compilation est réelle,** pour le backend de la machine, en plus de la validation
naga. « Accepté » veut donc dire « se dessinera ».
+ - **Elle tourne sur le pool de libuv** (une `AsyncTask`) : la compilation Metal d'un code neuf
+ prend 120 ms, et jusqu'à 375 ms pour la première du processus. `runDocumentTool` attend la
+ réponse, puis la passe à l'exécuteur des outils, qui reste synchrone.
- **Le coût est estimé** depuis le code (C.7), sans rendu ni mesure sur la machine.
- **L'erreur revient à l'agent** avec la ligne et la colonne de SON code.
2. **À la pose de la scène** (`Compositor::set_scene`). Chaque code nouveau part compiler sur un
@@ -748,9 +751,8 @@ module et pipeline compris ; les paliers en C.7.
| Vue live contre MP4 (BT.709), image entière | 3,1 / 255 avec effet, 3,0 sans |
| Vue live contre MP4, sur les pixels où l'effet agit | 13 à 16 / 255 avec effet, 14 à 16 sans effet sur les mêmes pixels : le bruit du codec sur des bords nets |
-La compilation, dix fois plus lente qu'avec FXC, reste hors du rendu. `checkShaderEffect`, lui,
-compile sur le thread qui l'appelle : le processus principal y passe ces 120 ms à chaque code
-neuf.
+La compilation, dix fois plus lente qu'avec FXC, reste hors du rendu. Elle tourne aussi à
+l'écriture (`checkShaderEffect`), d'où le pool de libuv plutôt que le thread principal (D.4).
Le détail est dans
[rendering-performance.md](../technical-documentation/engineering/rendering-performance.md).
diff --git a/electron/ai-edition/agent-tools.test.ts b/electron/ai-edition/agent-tools.test.ts
index ca4261cf2..764467566 100644
--- a/electron/ai-edition/agent-tools.test.ts
+++ b/electron/ai-edition/agent-tools.test.ts
@@ -13,6 +13,8 @@ import {
executeAgentTool,
isMutatingTool,
MUTATING_TOOL_NAMES,
+ shaderEffectCheckLookup,
+ shaderEffectToCheck,
} from "./agent-tools";
import { measureTextInShippedFonts } from "./text-measure";
@@ -3924,3 +3926,37 @@ describe("shader effects", () => {
expect(JSON.parse(added.resultJson)).toMatchObject({ validated: false, cost: null });
});
});
+
+describe("the executor's view of a check asked beforehand", () => {
+ const source = {
+ code: "fn effect(uv: vec2<f32>) -> vec4<f32> { return frame(uv); }",
+ params: [{ name: "level", type: "number" as const }],
+ };
+ const check = { tier: "light" as const, frameReads: 1, iterations: 0 };
+
+ it("is the answer for the same code and layout, and nothing for any other", () => {
+ const lookup = shaderEffectCheckLookup(source, check);
+ expect(lookup(source.code, [{ name: "level", type: "number" }])).toBe(check);
+ expect(lookup(`${source.code}\n`, source.params)).toBeNull();
+ expect(lookup(source.code, [{ name: "level", type: "color" }])).toBeNull();
+ expect(lookup(source.code, [])).toBeNull();
+ });
+
+ it("is asked only by the two tools that compile", () => {
+ const document = fixtureDocument();
+ expect(
+ shaderEffectToCheck(document, "addEffect", { startSec: 0, endSec: 1, effectId: "fx_1" }),
+ ).toBeNull();
+ expect(shaderEffectToCheck(document, "addTrim", { startSec: 0, endSec: 1 })).toBeNull();
+ expect(
+ shaderEffectToCheck(document, "addEffect", {
+ startSec: 0,
+ endSec: 1,
+ effect: { name: "A", description: "", code: source.code },
+ }),
+ ).toEqual({ code: source.code, params: [] });
+ expect(
+ shaderEffectToCheck(document, "setEffectDefinition", { effectId: "fx_missing", code: "x" }),
+ ).toBeNull();
+ });
+});
diff --git a/electron/ai-edition/agent-tools.ts b/electron/ai-edition/agent-tools.ts
index 91c18d756..ffa381069 100644
--- a/electron/ai-edition/agent-tools.ts
+++ b/electron/ai-edition/agent-tools.ts
@@ -61,7 +61,7 @@ import {
findShaderEffect,
SHADER_EFFECT_CONTRACT,
type ShaderEffectCheck,
- type ShaderEffectChecker,
+ type ShaderEffectCheckLookup,
shaderEffectCostAdvice,
shaderEffectTierOf,
storedShaderEffectValues,
@@ -1526,7 +1526,7 @@ export function documentSnapshotForModel(
* wired, its cost cached. An error reads as the agent must act on it. */
function checkedEffectDefinition(
candidate: unknown,
- checker: ShaderEffectChecker | undefined,
+ checker: ShaderEffectCheckLookup | undefined,
): { definition: AxcutShaderEffect; check: ShaderEffectCheck | null } | { error: string } {
const parsed = shaderEffectSchema.safeParse(candidate);
if (!parsed.success) {
@@ -1546,6 +1546,53 @@ function checkedEffectDefinition(
return { definition: { ...definition, cost }, check };
}
+type ShaderEffectSource = {
+ code: string;
+ params: Array<{ name: string; type: AxcutShaderEffect["params"][number]["type"] }>;
+};
+
+/** The code and parameter layout `addEffect` or `setEffectDefinition` will have the compositor
+ * check, read from the same arguments the executor reads; `null` for a call that compiles nothing.
+ * The check is asynchronous (a Metal compile takes 120 ms and more) and the executor is not: the
+ * async wrapper asks first, then hands the answer over (`shaderEffectCheckLookup`). */
+export function shaderEffectToCheck(
+ document: AxcutDocument,
+ name: string,
+ args: unknown,
+): ShaderEffectSource | null {
+ const layout = (
+ params: ReadonlyArray<Pick<AxcutShaderEffect["params"][number], "name" | "type">>,
+ ) => params.map(({ name, type }) => ({ name, type }));
+ if (name === "addEffect") {
+ const parsed = addEffectArgs.safeParse(args);
+ if (!parsed.success || !parsed.data.effect) return null;
+ return { code: parsed.data.effect.code, params: layout(parsed.data.effect.params ?? []) };
+ }
+ if (name === "setEffectDefinition") {
+ const parsed = setEffectDefinitionArgs.safeParse(args);
+ if (!parsed.success) return null;
+ const { effectId, code, params } = parsed.data;
+ const existing = findShaderEffect(document, effectId);
+ if (!existing || (code === undefined && params === undefined)) return null;
+ return { code: code ?? existing.code, params: layout(params ?? existing.params) };
+ }
+ return null;
+}
+
+/** The executor's view of an answer asked for `source`: that answer for the same code and
+ * layout, `null` (not checked) for anything else. */
+export function shaderEffectCheckLookup(
+ source: ShaderEffectSource,
+ check: ShaderEffectCheck | null,
+): ShaderEffectCheckLookup {
+ return (code, params) =>
+ code === source.code &&
+ params.length === source.params.length &&
+ params.every((p, i) => p.name === source.params[i].name && p.type === source.params[i].type)
+ ? check
+ : null;
+}
+
/** Two effects of one lane cannot overlap: the refusal names what is in the way, and the way out. */
function effectLaneBlocked(
ranges: AxcutShaderEffectRange[],
@@ -1733,9 +1780,10 @@ export interface AgentToolOptions {
/** Whether this machine can cut out the camera background, asked only by the two tools that
* report it. Absent where nothing can ask the compositor: reported as not checked. */
cameraSegmentation?: () => SegmentationSupport;
- /** Asks the compositor to check an effect's WGSL (`checkShaderEffect`). Absent where nothing
- * can: the effect is stored unchecked, and the result says `validated: false`. */
- checkShaderEffect?: ShaderEffectChecker;
+ /** The compositor's check of the WGSL this call writes (`checkShaderEffect`), asked beforehand
+ * by the async wrapper (`shaderEffectToCheck`). Absent where nothing can: the effect is stored
+ * unchecked, and the result says `validated: false`. */
+ checkShaderEffect?: ShaderEffectCheckLookup;
/** `addAutoZooms`' plan, made by the async wrapper from the recorded clicks. Absent means
* no cursor reader is wired here, which the tool reports as "could not look". */
autoZooms?: AutoZoomPlan;
diff --git a/electron/ai-edition/deep-agent/service.test.ts b/electron/ai-edition/deep-agent/service.test.ts
index 22d508fb7..fffbeb4c5 100644
--- a/electron/ai-edition/deep-agent/service.test.ts
+++ b/electron/ai-edition/deep-agent/service.test.ts
@@ -780,3 +780,104 @@ describe("addAutoZooms reads the clicks through the runtime's cursor reader", ()
expect(result.resultJson).toContain("could not be read");
});
});
+
+describe("an effect's code is checked off the main thread, before the executor reads it", () => {
+ const LEVEL = { type: "number", name: "level", label: "Level", default: 1, min: 0, max: 2 };
+ const effect = (code: string) => ({
+ name: "Dim",
+ description: "Dims the picture.",
+ code,
+ params: [LEVEL],
+ });
+ // The compositor's check takes a while (a Metal compile is 120 ms and more): a promise that
+ // settles later, refusing `nope` and pricing the rest at 20 reads.
+ const slowChecker = (asked: Array<{ code: string; params: unknown }>) => ({
+ checkShaderEffect: async (
+ code: string,
+ params: ReadonlyArray<{ name: string; type: string }>,
+ ) => {
+ asked.push({ code, params });
+ await new Promise((resolve) => setTimeout(resolve, 5));
+ return code.includes("nope")
+ ? { error: "error: no definition in scope for identifier: `nope`" }
+ : { tier: "medium" as const, frameReads: 20, iterations: 20 };
+ },
+ });
+
+ it("waits for the compositor's answer, accepting or refusing on it", async () => {
+ const asked: Array<{ code: string; params: unknown }> = [];
+ const runtime = slowChecker(asked);
+ const accepted = await runDocumentTool(
+ fixtureDocument(),
+ "addEffect",
+ {
+ startSec: 1,
+ endSec: 2,
+ effect: effect("fn effect(uv: vec2<f32>) -> vec4<f32> { return frame(uv); }"),
+ },
+ true,
+ runtime,
+ );
+ expect(accepted.ok).toBe(true);
+ const result = JSON.parse(accepted.resultJson);
+ expect(result.validated).toBe(true);
+ expect(result.cost).toMatchObject({ tier: "medium", frameReads: 20 });
+ expect(asked[0].params).toEqual([{ name: "level", type: "number" }]);
+
+ const refused = await runDocumentTool(
+ fixtureDocument(),
+ "addEffect",
+ { startSec: 1, endSec: 2, effect: effect("fn effect(uv: vec2<f32>) -> vec4<f32> { nope }") },
+ true,
+ runtime,
+ );
+ expect(refused.ok).toBe(false);
+ expect(refused.resultJson).toContain("does not compile");
+ });
+
+ it("checks a new code against the definition's parameters when only the code changes", async () => {
+ const asked: Array<{ code: string; params: unknown }> = [];
+ const runtime = slowChecker(asked);
+ const added = await runDocumentTool(
+ fixtureDocument(),
+ "addEffect",
+ {
+ startSec: 1,
+ endSec: 2,
+ effect: effect("fn effect(uv: vec2<f32>) -> vec4<f32> { return frame(uv); }"),
+ },
+ true,
+ runtime,
+ );
+ const { effectId } = JSON.parse(added.resultJson);
+ const document = added.document as AxcutDocument;
+ const code = "fn effect(uv: vec2<f32>) -> vec4<f32> { return frame(uv) * params.level; }";
+ const changed = await runDocumentTool(
+ document,
+ "setEffectDefinition",
+ { effectId, code },
+ true,
+ runtime,
+ );
+ expect(changed.ok).toBe(true);
+ expect(asked.at(-1)).toEqual({ code, params: [{ name: "level", type: "number" }] });
+ });
+
+ it("asks nothing when edits are off, or for a tool that compiles nothing", async () => {
+ const asked: Array<{ code: string; params: unknown }> = [];
+ const runtime = slowChecker(asked);
+ await runDocumentTool(
+ fixtureDocument(),
+ "addEffect",
+ {
+ startSec: 1,
+ endSec: 2,
+ effect: effect("fn effect(uv: vec2<f32>) -> vec4<f32> { return frame(uv); }"),
+ },
+ false,
+ runtime,
+ );
+ await runDocumentTool(fixtureDocument(), "addTrim", { startSec: 1, endSec: 2 }, true, runtime);
+ expect(asked).toEqual([]);
+ });
+});
diff --git a/electron/ai-edition/deep-agent/service.ts b/electron/ai-edition/deep-agent/service.ts
index b3e2f978b..88b7333d0 100644
--- a/electron/ai-edition/deep-agent/service.ts
+++ b/electron/ai-edition/deep-agent/service.ts
@@ -89,6 +89,8 @@ import {
setTrimArgs,
setWordTextArgs,
setZoomArgs,
+ shaderEffectCheckLookup,
+ shaderEffectToCheck,
} from "../agent-tools";
import {
getCaptionsArgs,
@@ -263,7 +265,7 @@ export const TOOL_DESCRIPTIONS: Record<string, string> = {
addCameraFullscreen:
"Add a camera-fullscreen region over a span of the edited timeline (virtual seconds): the webcam fills the frame for that span. This only does something when the footage under that span comes from an asset with a linked webcam — check assets[].hasCameraTrack (or hasAnyCamera) in getCurrentDocument first. On footage with no camera the call is refused rather than storing a region that would render nothing; say so instead of retrying.",
addEffect:
- "Write a visual effect as a shader and pose it over a span of the edited timeline (virtual seconds), or pose one of this project's effects again (effectId). Use it for a look the settings do not offer. `effect.code` is WGSL that defines `fn effect(uv: vec2<f32>) -> vec4<f32>`, called for every pixel; uv is 0..1 with (0,0) top-left. Available inside it: `frame(uv)` samples the composed picture; `params.<name>` reads each declared parameter; `ctx.time` is programme seconds; `ctx.region.x` goes 0 to 1 across the region (y its length in s, z seconds since it began); `ctx.resolution` and `ctx.aspect`; `ctx.cursor` (xy in uv, z = 1 when on screen); `ctx.clicks[0..3]`, the last four clicks, newest first (xy in uv where it happened, z seconds since, w = 1 when the slot holds one); `ctx.footage` and `ctx.camera`, the recording's and the webcam's rects (x, y, w, h in uv; camera zero-sized when absent) — all already placed through zoom and layout; `fx_to_footage(uv)` and `fx_from_footage(p)` map between output uv and the recording's own surface (0..1 across the recording) through the warp the compositor draws it with, zoom, layout and 3D camera included, and `ctx.footage_aspect` is its width over height: the effect works in whichever of the two spaces fits it; `fx_px(n)`, n pixels of a 1080p frame as a fraction of the height; `fx_hash(vec2<u32>)`, a random number in [0,1) identical on every GPU. Rules: the effect is a pure function of those inputs (no state between frames), so the preview and the export show the same pixels; reason in uv or fx_px, never raw pixels; every loop is a `for` with literal bounds (`for (var i = 0; i < 16; i++)`), at most 4096 iterations per pixel — the one hard rule. `params` declares up to 8 (recommended) parameters, each `{type: number|toggle|color|point, name, label, default}` (number also `min`, `max`); declare one only when moving it visibly changes the effect over its whole range. `target`: `image` (default: the picture under the annotations and captions), `text` (the annotations and captions alone, premultiplied alpha), `both` (everything); use text or both only when asked. Effects of one `lane` (0–7) cannot overlap; lanes apply in order, lane 0 first. The result gives the cost tier (frame() reads per pixel: light ≤ 8, about 0.06 ms per 1080p frame on a fast GPU; medium ≤ 32, 0.14 ms; heavy ≤ 128, 0.54 ms; very heavy beyond) and a recommendation when the tier runs long: light anywhere, medium up to a minute, heavy up to 10 s, very heavy a few seconds. It is a recommendation, never a refusal. Code that does not compile is refused with the compiler's error at the line of your code: fix it and call again. Check the result with getFrames.",
+ "Write a visual effect as a shader and pose it over a span of the edited timeline (virtual seconds), or pose one of this project's effects again (effectId). Use it for a look the settings do not offer. `effect.code` is WGSL that defines `fn effect(uv: vec2<f32>) -> vec4<f32>`, called for every pixel; uv is 0..1 with (0,0) top-left. Available inside it: `frame(uv)` samples the composed picture; `params.<name>` reads each declared parameter; `ctx.time` is programme seconds; `ctx.region.x` goes 0 to 1 across the region (y its length in s, z seconds since it began); `ctx.resolution` and `ctx.aspect`; `ctx.cursor` (xy in uv, z = 1 when on screen); `ctx.clicks[0..3]`, the last four clicks, newest first (xy in uv where it happened, z seconds since, w = 1 when the slot holds one); `ctx.footage` and `ctx.camera`, the recording's and the webcam's rects (x, y, w, h in uv; camera zero-sized when absent) — all already placed through zoom and layout; `fx_to_footage(uv)` and `fx_from_footage(p)` map between output uv and the recording's own surface (0..1 across the recording) through the warp the compositor draws it with, zoom, layout and 3D camera included, and `ctx.footage_aspect` is its width over height: the effect works in whichever of the two spaces fits it; `fx_px(n)`, n pixels of a 1080p frame as a fraction of the height; `fx_hash(vec2<u32>)`, a random number in [0,1) identical on every GPU. Rules: the effect is a pure function of those inputs (no state between frames), so the preview and the export show the same pixels; reason in uv or fx_px, never raw pixels; every loop is a `for` with literal bounds (`for (var i = 0; i < 16; i++)`), at most 4096 iterations per pixel — the one hard rule. `params` declares up to 8 (recommended) parameters, each `{type: number|toggle|color|point, name, label, default}` (number also `min`, `max`); declare one only when moving it visibly changes the effect over its whole range. `target`: `image` (default: the picture under the annotations and captions), `text` (the annotations and captions alone, premultiplied alpha), `both` (everything); use text or both only when asked. Effects of one `lane` (0–7) cannot overlap; lanes apply in order, lane 0 first. The result gives the cost tier, in frame() reads per pixel, with what a 1080p frame pays on a fast discrete GPU (RTX 4070 Ti) and on an integrated one (Apple M1): light ≤ 8, up to 0.06 ms and 1.3 ms; medium ≤ 32, 0.14 ms and 4.3 ms; heavy ≤ 128, 0.54 ms and 16 ms; very heavy beyond, 2 ms and 49 ms at 512 reads. It also gives a recommendation when the tier runs long: light anywhere, medium up to a minute, heavy up to 10 s, very heavy a few seconds. It is a recommendation, never a refusal. Code that does not compile is refused with the compiler's error at the line of your code: fix it and call again. Check the result with getFrames.",
setEffect:
"Move, resize or re-tune an effect region by id (virtual seconds): its span, its parameter `values`, its `target` or its `lane`. Only the fields you pass change; a value equal to the default is dropped. Refused if the new span runs into another effect of the same lane.",
setEffectDefinition:
@@ -405,11 +407,22 @@ export async function runDocumentTool(
: undefined;
const autoZooms =
name === "addAutoZooms" && editsAllowed ? await planAutoZooms(document, runtime) : undefined;
+ // The compositor compiles an effect's code for real, off the main thread: asked here, before
+ // the synchronous executor reads the answer.
+ const effectSource =
+ editsAllowed && runtime.checkShaderEffect ? shaderEffectToCheck(document, name, args) : null;
+ const checkShaderEffect =
+ effectSource && runtime.checkShaderEffect
+ ? shaderEffectCheckLookup(
+ effectSource,
+ await runtime.checkShaderEffect(effectSource.code, effectSource.params),
+ )
+ : undefined;
return executeAgentTool(document, name, JSON.stringify(args ?? {}), {
editsAllowed,
cursorTelemetry: { availableByAssetId: runtime.availableByAssetId, load },
cameraSegmentation: runtime.cameraSegmentation,
- checkShaderEffect: runtime.checkShaderEffect,
+ checkShaderEffect,
autoZooms,
});
}
diff --git a/electron/mcp/effects-contract.ts b/electron/mcp/effects-contract.ts
index 47c28e37d..96c24d764 100644
--- a/electron/mcp/effects-contract.ts
+++ b/electron/mcp/effects-contract.ts
@@ -91,16 +91,17 @@ two lanes.
## Cost
The result of addEffect gives the cost in frame() reads per pixel, counted from the code (loops
-multiplied by their bound), and its tier. Measured on a fast GPU (RTX 4070 Ti) at 1080p:
+multiplied by their bound), and its tier. Per 1080p frame, measured on a fast discrete GPU (RTX
+4070 Ti) and on an integrated one (Apple M1):
-| Tier | Reads | Per frame | Recommended for |
-|---|---|---|---|
-| light | ≤ 8 | ≤ 0.06 ms | anywhere, the whole video included |
-| medium | ≤ 32 | ≤ 0.14 ms | up to a minute |
-| heavy | ≤ 128 | ≤ 0.54 ms | up to 10 s: a transition, a key moment |
-| very heavy | more | 2 ms at 512 | a few seconds |
+| Tier | Reads | Per frame, RTX 4070 Ti | Per frame, M1 | Recommended for |
+|---|---|---|---|---|
+| light | ≤ 8 | ≤ 0.06 ms | ≤ 1.3 ms | anywhere, the whole video included |
+| medium | ≤ 32 | ≤ 0.14 ms | ≤ 4.3 ms | up to a minute |
+| heavy | ≤ 128 | ≤ 0.54 ms | ≤ 16 ms | up to 10 s: a transition, a key moment |
+| very heavy | more | 2 ms at 512 | 49 ms at 512 | a few seconds |
-An integrated GPU pays several times more. The tiers are recommendations, never refusals.
+A 4K frame costs about four times as much. The tiers are recommendations, never refusals.
## Example: a ripple on the recording, on every click
diff --git a/electron/native-bridge/services/compositorViewService.test.ts b/electron/native-bridge/services/compositorViewService.test.ts
index 01817e826..40987fe62 100644
--- a/electron/native-bridge/services/compositorViewService.test.ts
+++ b/electron/native-bridge/services/compositorViewService.test.ts
@@ -271,6 +271,34 @@ function writeCargoConfig(root: string, body: string): void {
);
}
+describe("CompositorViewService checks an effect's code off the main thread", () => {
+ const service = (checkShaderEffect: CompositorViewAddon["checkShaderEffect"]) =>
+ new CompositorViewService({ addon: { checkShaderEffect } as unknown as CompositorViewAddon });
+
+ it("passes the addon's answer through", async () => {
+ const check = { tier: "light" as const, frameReads: 1, iterations: 0 };
+ const checker = vi.fn(async () => check);
+ expect(
+ await service(checker).checkShaderEffect("code", [{ name: "level", type: "number" }]),
+ ).toEqual(check);
+ expect(checker).toHaveBeenCalledWith(
+ "code",
+ JSON.stringify([{ name: "level", type: "number" }]),
+ );
+ });
+
+ it("reads an addon that fails, whether it throws or rejects, as not checked", async () => {
+ const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined);
+ const rejects = vi.fn(() => Promise.reject(new Error("native")));
+ const throws = vi.fn(() => {
+ throw new Error("native");
+ });
+ expect(await service(rejects).checkShaderEffect("code", [])).toBeNull();
+ expect(await service(throws).checkShaderEffect("code", [])).toBeNull();
+ warn.mockRestore();
+ });
+});
+
describe("ffmpegSharedBinCandidates", () => {
let tmpRoot: string;
diff --git a/electron/native-bridge/services/compositorViewService.ts b/electron/native-bridge/services/compositorViewService.ts
index fd0a25b52..95bff7ad0 100644
--- a/electron/native-bridge/services/compositorViewService.ts
+++ b/electron/native-bridge/services/compositorViewService.ts
@@ -592,14 +592,16 @@ export class CompositorViewService {
/** Checks an agent-written effect's WGSL in the compositor (`docs/shader-effects.md`, E.3): the
* compile error, or the cost tier. `null` when this addon cannot answer, which the agent tools
* report as "not checked" rather than as a compile failure. */
- checkShaderEffect(
+ async checkShaderEffect(
code: string,
params: ReadonlyArray<{ name: string; type: string }>,
- ): ShaderEffectCheck | null {
+ ): Promise<ShaderEffectCheck | null> {
const addon = this.ensureAddon();
if (!addon || typeof addon.checkShaderEffect !== "function") return null;
try {
- return addon.checkShaderEffect(
+ // `await`, not a bare return: the check is a promise now, and a rejection must land in
+ // this catch rather than escape it.
+ return await addon.checkShaderEffect(
code,
JSON.stringify(params.map(({ name, type }) => ({ name, type }))),
);
diff --git a/electron/native/compositor-view/addon.d.ts b/electron/native/compositor-view/addon.d.ts
index de1bb08d7..501a41c73 100644
--- a/electron/native/compositor-view/addon.d.ts
+++ b/electron/native/compositor-view/addon.d.ts
@@ -156,9 +156,10 @@ export interface CompositorViewAddon {
* needs the answer before any preview exists. Cached native-side. */
probeBackend(): CompositorBackend;
/** Checks an effect's WGSL: validation, literal loop bounds, the real compile for this machine's
- * backend where there is one (FXC on Windows), and the cost tier. `paramsJson` is the declared
- * parameters, `[{ "name": "strength", "type": "number" }, …]`. Absent from older builds. */
- checkShaderEffect?(code: string, paramsJson: string): ShaderEffectCheck;
+ * backend (FXC on Windows, Metal on macOS, SPIR-V on Linux), and the cost tier. `paramsJson`
+ * is the declared parameters, `[{ "name": "strength", "type": "number" }, …]`. Runs on the
+ * libuv pool: a Metal compile takes 120 ms and more. Absent from older builds. */
+ checkShaderEffect?(code: string, paramsJson: string): Promise<ShaderEffectCheck>;
/** Whether the ONNX Runtime library is where the app staged it, and therefore whether a
* segmentation mask can be produced at all. Asked of the machine rather than guessed from
diff --git a/src/lib/ai-edition/shaderEffects.ts b/src/lib/ai-edition/shaderEffects.ts
index 8aa973128..6d0f29515 100644
--- a/src/lib/ai-edition/shaderEffects.ts
+++ b/src/lib/ai-edition/shaderEffects.ts
@@ -25,10 +25,19 @@ export interface ShaderEffectCheck {
iterations?: number | null;
}
-/** Asks the compositor about a code; `null` where nothing can answer (no addon, a test). */
+/** Asks the compositor about a code; `null` where nothing can answer (no addon, a test). A
+ * promise: the compositor compiles for real, which takes 120 ms and more on Metal, off the main
+ * thread. */
export type ShaderEffectChecker = (
code: string,
params: ReadonlyArray<Pick<AxcutShaderEffectParam, "name" | "type">>,
+) => Promise<ShaderEffectCheck | null>;
+
+/** The checker's answer as the tool executor reads it, synchronously: `runDocumentTool` asked
+ * first (`shaderEffectToCheck`). `null` for a code that was not asked about. */
+export type ShaderEffectCheckLookup = (
+ code: string,
+ params: ReadonlyArray<Pick<AxcutShaderEffectParam, "name" | "type">>,
) => ShaderEffectCheck | null;
export function findShaderEffect(
diff --git a/technical-documentation/engineering/rendering-performance.md b/technical-documentation/engineering/rendering-performance.md
index b88bedc35..78c68bf18 100644
--- a/technical-documentation/engineering/rendering-performance.md
+++ b/technical-documentation/engineering/rendering-performance.md
@@ -474,7 +474,7 @@ An agent-written effect (`crates/compositor/src/shader_fx.rs`, [docs/shader-effe
- **About 0.12 ms per read at 1080p and 0.5 ms at 4K,** over a fixed 0.3 ms: thirty times the RTX. The fixed part is the copy into `ann_copy` and the full-frame pass, which are bandwidth on a 68 GB/s unified memory.
- **The reads dominate here too.** The bench effect computes a `sin` and a `cos` per read; with integer offsets instead, 128 reads cost 10.5 ms rather than 16.2.
- **naga's `force_loop_bounding` is off** (`shader_fx::to_msl`): its guard, a `volatile` read per iteration, cost 7 to 10 % of every row above. The validation has already proved each loop bounded.
-- **Compiling is ten times slower than FXC.** naga, `new_library_with_source` and the pipeline state take 120 ms in the median for a source Metal has not seen (the C.6 ripple), and 160 to 375 ms for the first one in a process. A source Metal has seen comes back in 0.4 ms in the same process, 3 ms in another. All of it runs on the background thread; the preview shows the effect once it lands. `checkShaderEffect` compiles on the calling thread, so an agent's `addEffect` took 152 ms through the `mcp` CLI.
+- **Compiling is ten times slower than FXC.** naga, `new_library_with_source` and the pipeline state take 120 ms in the median for a source Metal has not seen (the C.6 ripple), and 160 to 375 ms for the first one in a process. A source Metal has seen comes back in 0.4 ms in the same process, 3 ms in another. All of it runs on the background thread; the preview shows the effect once it lands. An agent's `addEffect` took 152 ms through the `mcp` CLI, the compile included; `checkShaderEffect` runs on the libuv pool, off the main thread.
- **The export does not notice an effect:** 180 frames at 1752×1080 (`exportMulti`, three alternated runs after a warm-up) took 2469 ms without, 2428 ms with the ripple over the whole video, and 2375 ms with the spike's heavier effect. The M1 export is encoder-bound. The live view held 29.7 fps against a 30 fps source with and without the effect.
- **Not measured yet:** the Radeon 610M.