Record your screen, ship a demo. Free and open-source, GPU-accelerated, no watermarks, no subscriptions. Windows, macOS, Linux. Actively maintained.
TypeScript cross-platformelectronffmpegopen-sourceproduct-demorustscreen-capturescreen-recorderscreen-recordingscreen-studioscreencastspeech-to-texttypescriptvideo-editingvideo-editorwaylandwgpuwhisper
feat(effects): check an effect's code off the main thread, and price the tiers on an M1
Etienne Lescot3759811cbb05
diff --git a/crates/compositor-view-napi/src/lib.rs b/crates/compositor-view-napi/src/lib.rsindex 447df805c..b25b0dfe0 100644--- a/crates/compositor-view-napi/src/lib.rs+++ b/crates/compositor-view-napi/src/lib.rs@@ -401,20 +401,41 @@ pub struct ShaderEffectCheck {pub iterations: Option<f64>,}+pub struct CheckShaderEffectTask {+ code: String,+ params_json: String,+}++impl Task for CheckShaderEffectTask {+ type Output = ShaderEffectCheck;+ type JsValue = ShaderEffectCheck;++ fn compute(&mut self) -> Result<Self::Output> {+ Ok(match openscreen_compositor::shader_fx::check_for_agent_json(&self.code, &self.params_json) {+ Ok(report) => ShaderEffectCheck {+ error: None,+ tier: Some(report.tier.name().to_string()),+ frame_reads: Some(report.frame_reads as f64),+ iterations: Some(report.iterations as f64),+ },+ Err(error) => ShaderEffectCheck { error: Some(error), tier: None, frame_reads: None, iterations: None },+ })+ }++ fn resolve(&mut self, _env: Env, check: Self::Output) -> Result<Self::JsValue> {+ Ok(check)+ }+}+/// Vérifie le WGSL d'un effet shader : validation, bornes des boucles, compilation réelle pour le-/// backend de la machine quand elle existe (FXC sur Windows), et palier de coût. `params_json` :-/// les paramètres déclarés, `[{ "name": "strength", "type": "number" }, …]`.+/// backend de la machine (FXC sur Windows, Metal sur macOS, SPIR-V sur Linux), et palier de coût.+/// `params_json` : les paramètres déclarés, `[{ "name": "strength", "type": "number" }, …]`.+///+/// `AsyncTask`, comme `segment_frame` : la compilation Metal d'une source neuve prend 120 ms, et+/// jusqu'à 375 ms pour la première du processus. Ce temps n'a rien à faire sur le thread principal.#[napi]-pub fn check_shader_effect(code: String, params_json: String) -> ShaderEffectCheck {- match openscreen_compositor::shader_fx::check_for_agent_json(&code, ¶ms_json) {- Ok(report) => ShaderEffectCheck {- error: None,- tier: Some(report.tier.name().to_string()),- frame_reads: Some(report.frame_reads as f64),- iterations: Some(report.iterations as f64),- },- Err(error) => ShaderEffectCheck { error: Some(error), tier: None, frame_reads: None, iterations: None },- }+pub fn check_shader_effect(code: String, params_json: String) -> AsyncTask<CheckShaderEffectTask> {+ AsyncTask::new(CheckShaderEffectTask { code, params_json })}/// Installe la scène de l'app (JSON `SceneDescription`) sur la vue : layout preset piloté pardiff --git a/docs/shader-effects.md b/docs/shader-effects.mdindex 24104cf4c..c7c55341a 100644--- a/docs/shader-effects.md+++ b/docs/shader-effects.md@@ -421,6 +421,9 @@ Il y a deux compilations, à deux moments :`setEffectDefinition`), le processus principal appelle `checkShaderEffect(code, params)` de l'addon.- **La compilation est réelle,** pour le backend de la machine, en plus de la validationnaga. « Accepté » veut donc dire « se dessinera ».+ - **Elle tourne sur le pool de libuv** (une `AsyncTask`) : la compilation Metal d'un code neuf+ prend 120 ms, et jusqu'à 375 ms pour la première du processus. `runDocumentTool` attend la+ réponse, puis la passe à l'exécuteur des outils, qui reste synchrone.- **Le coût est estimé** depuis le code (C.7), sans rendu ni mesure sur la machine.- **L'erreur revient à l'agent** avec la ligne et la colonne de SON code.2. **À la pose de la scène** (`Compositor::set_scene`). Chaque code nouveau part compiler sur un@@ -748,9 +751,8 @@ module et pipeline compris ; les paliers en C.7.| Vue live contre MP4 (BT.709), image entière | 3,1 / 255 avec effet, 3,0 sans || Vue live contre MP4, sur les pixels où l'effet agit | 13 à 16 / 255 avec effet, 14 à 16 sans effet sur les mêmes pixels : le bruit du codec sur des bords nets |-La compilation, dix fois plus lente qu'avec FXC, reste hors du rendu. `checkShaderEffect`, lui,-compile sur le thread qui l'appelle : le processus principal y passe ces 120 ms à chaque code-neuf.+La compilation, dix fois plus lente qu'avec FXC, reste hors du rendu. Elle tourne aussi à+l'écriture (`checkShaderEffect`), d'où le pool de libuv plutôt que le thread principal (D.4).Le détail est dans[rendering-performance.md](../technical-documentation/engineering/rendering-performance.md).diff --git a/electron/ai-edition/agent-tools.test.ts b/electron/ai-edition/agent-tools.test.tsindex ca4261cf2..764467566 100644--- a/electron/ai-edition/agent-tools.test.ts+++ b/electron/ai-edition/agent-tools.test.ts@@ -13,6 +13,8 @@ import {executeAgentTool,isMutatingTool,MUTATING_TOOL_NAMES,+ shaderEffectCheckLookup,+ shaderEffectToCheck,} from "./agent-tools";import { measureTextInShippedFonts } from "./text-measure";@@ -3924,3 +3926,37 @@ describe("shader effects", () => {expect(JSON.parse(added.resultJson)).toMatchObject({ validated: false, cost: null });});});++describe("the executor's view of a check asked beforehand", () => {+ const source = {+ code: "fn effect(uv: vec2<f32>) -> vec4<f32> { return frame(uv); }",+ params: [{ name: "level", type: "number" as const }],+ };+ const check = { tier: "light" as const, frameReads: 1, iterations: 0 };++ it("is the answer for the same code and layout, and nothing for any other", () => {+ const lookup = shaderEffectCheckLookup(source, check);+ expect(lookup(source.code, [{ name: "level", type: "number" }])).toBe(check);+ expect(lookup(`${source.code}\n`, source.params)).toBeNull();+ expect(lookup(source.code, [{ name: "level", type: "color" }])).toBeNull();+ expect(lookup(source.code, [])).toBeNull();+ });++ it("is asked only by the two tools that compile", () => {+ const document = fixtureDocument();+ expect(+ shaderEffectToCheck(document, "addEffect", { startSec: 0, endSec: 1, effectId: "fx_1" }),+ ).toBeNull();+ expect(shaderEffectToCheck(document, "addTrim", { startSec: 0, endSec: 1 })).toBeNull();+ expect(+ shaderEffectToCheck(document, "addEffect", {+ startSec: 0,+ endSec: 1,+ effect: { name: "A", description: "", code: source.code },+ }),+ ).toEqual({ code: source.code, params: [] });+ expect(+ shaderEffectToCheck(document, "setEffectDefinition", { effectId: "fx_missing", code: "x" }),+ ).toBeNull();+ });+});diff --git a/electron/ai-edition/agent-tools.ts b/electron/ai-edition/agent-tools.tsindex 91c18d756..ffa381069 100644--- a/electron/ai-edition/agent-tools.ts+++ b/electron/ai-edition/agent-tools.ts@@ -61,7 +61,7 @@ import {findShaderEffect,SHADER_EFFECT_CONTRACT,type ShaderEffectCheck,- type ShaderEffectChecker,+ type ShaderEffectCheckLookup,shaderEffectCostAdvice,shaderEffectTierOf,storedShaderEffectValues,@@ -1526,7 +1526,7 @@ export function documentSnapshotForModel(* wired, its cost cached. An error reads as the agent must act on it. */function checkedEffectDefinition(candidate: unknown,- checker: ShaderEffectChecker | undefined,+ checker: ShaderEffectCheckLookup | undefined,): { definition: AxcutShaderEffect; check: ShaderEffectCheck | null } | { error: string } {const parsed = shaderEffectSchema.safeParse(candidate);if (!parsed.success) {@@ -1546,6 +1546,53 @@ function checkedEffectDefinition(return { definition: { ...definition, cost }, check };}+type ShaderEffectSource = {+ code: string;+ params: Array<{ name: string; type: AxcutShaderEffect["params"][number]["type"] }>;+};++/** The code and parameter layout `addEffect` or `setEffectDefinition` will have the compositor+ * check, read from the same arguments the executor reads; `null` for a call that compiles nothing.+ * The check is asynchronous (a Metal compile takes 120 ms and more) and the executor is not: the+ * async wrapper asks first, then hands the answer over (`shaderEffectCheckLookup`). */+export function shaderEffectToCheck(+ document: AxcutDocument,+ name: string,+ args: unknown,+): ShaderEffectSource | null {+ const layout = (+ params: ReadonlyArray<Pick<AxcutShaderEffect["params"][number], "name" | "type">>,+ ) => params.map(({ name, type }) => ({ name, type }));+ if (name === "addEffect") {+ const parsed = addEffectArgs.safeParse(args);+ if (!parsed.success || !parsed.data.effect) return null;+ return { code: parsed.data.effect.code, params: layout(parsed.data.effect.params ?? []) };+ }+ if (name === "setEffectDefinition") {+ const parsed = setEffectDefinitionArgs.safeParse(args);+ if (!parsed.success) return null;+ const { effectId, code, params } = parsed.data;+ const existing = findShaderEffect(document, effectId);+ if (!existing || (code === undefined && params === undefined)) return null;+ return { code: code ?? existing.code, params: layout(params ?? existing.params) };+ }+ return null;+}++/** The executor's view of an answer asked for `source`: that answer for the same code and+ * layout, `null` (not checked) for anything else. */+export function shaderEffectCheckLookup(+ source: ShaderEffectSource,+ check: ShaderEffectCheck | null,+): ShaderEffectCheckLookup {+ return (code, params) =>+ code === source.code &&+ params.length === source.params.length &&+ params.every((p, i) => p.name === source.params[i].name && p.type === source.params[i].type)+ ? check+ : null;+}+/** Two effects of one lane cannot overlap: the refusal names what is in the way, and the way out. */function effectLaneBlocked(ranges: AxcutShaderEffectRange[],@@ -1733,9 +1780,10 @@ export interface AgentToolOptions {/** Whether this machine can cut out the camera background, asked only by the two tools that* report it. Absent where nothing can ask the compositor: reported as not checked. */cameraSegmentation?: () => SegmentationSupport;- /** Asks the compositor to check an effect's WGSL (`checkShaderEffect`). Absent where nothing- * can: the effect is stored unchecked, and the result says `validated: false`. */- checkShaderEffect?: ShaderEffectChecker;+ /** The compositor's check of the WGSL this call writes (`checkShaderEffect`), asked beforehand+ * by the async wrapper (`shaderEffectToCheck`). Absent where nothing can: the effect is stored+ * unchecked, and the result says `validated: false`. */+ checkShaderEffect?: ShaderEffectCheckLookup;/** `addAutoZooms`' plan, made by the async wrapper from the recorded clicks. Absent means* no cursor reader is wired here, which the tool reports as "could not look". */autoZooms?: AutoZoomPlan;diff --git a/electron/ai-edition/deep-agent/service.test.ts b/electron/ai-edition/deep-agent/service.test.tsindex 22d508fb7..fffbeb4c5 100644--- a/electron/ai-edition/deep-agent/service.test.ts+++ b/electron/ai-edition/deep-agent/service.test.ts@@ -780,3 +780,104 @@ describe("addAutoZooms reads the clicks through the runtime's cursor reader", ()expect(result.resultJson).toContain("could not be read");});});++describe("an effect's code is checked off the main thread, before the executor reads it", () => {+ const LEVEL = { type: "number", name: "level", label: "Level", default: 1, min: 0, max: 2 };+ const effect = (code: string) => ({+ name: "Dim",+ description: "Dims the picture.",+ code,+ params: [LEVEL],+ });+ // The compositor's check takes a while (a Metal compile is 120 ms and more): a promise that+ // settles later, refusing `nope` and pricing the rest at 20 reads.+ const slowChecker = (asked: Array<{ code: string; params: unknown }>) => ({+ checkShaderEffect: async (+ code: string,+ params: ReadonlyArray<{ name: string; type: string }>,+ ) => {+ asked.push({ code, params });+ await new Promise((resolve) => setTimeout(resolve, 5));+ return code.includes("nope")+ ? { error: "error: no definition in scope for identifier: `nope`" }+ : { tier: "medium" as const, frameReads: 20, iterations: 20 };+ },+ });++ it("waits for the compositor's answer, accepting or refusing on it", async () => {+ const asked: Array<{ code: string; params: unknown }> = [];+ const runtime = slowChecker(asked);+ const accepted = await runDocumentTool(+ fixtureDocument(),+ "addEffect",+ {+ startSec: 1,+ endSec: 2,+ effect: effect("fn effect(uv: vec2<f32>) -> vec4<f32> { return frame(uv); }"),+ },+ true,+ runtime,+ );+ expect(accepted.ok).toBe(true);+ const result = JSON.parse(accepted.resultJson);+ expect(result.validated).toBe(true);+ expect(result.cost).toMatchObject({ tier: "medium", frameReads: 20 });+ expect(asked[0].params).toEqual([{ name: "level", type: "number" }]);++ const refused = await runDocumentTool(+ fixtureDocument(),+ "addEffect",+ { startSec: 1, endSec: 2, effect: effect("fn effect(uv: vec2<f32>) -> vec4<f32> { nope }") },+ true,+ runtime,+ );+ expect(refused.ok).toBe(false);+ expect(refused.resultJson).toContain("does not compile");+ });++ it("checks a new code against the definition's parameters when only the code changes", async () => {+ const asked: Array<{ code: string; params: unknown }> = [];+ const runtime = slowChecker(asked);+ const added = await runDocumentTool(+ fixtureDocument(),+ "addEffect",+ {+ startSec: 1,+ endSec: 2,+ effect: effect("fn effect(uv: vec2<f32>) -> vec4<f32> { return frame(uv); }"),+ },+ true,+ runtime,+ );+ const { effectId } = JSON.parse(added.resultJson);+ const document = added.document as AxcutDocument;+ const code = "fn effect(uv: vec2<f32>) -> vec4<f32> { return frame(uv) * params.level; }";+ const changed = await runDocumentTool(+ document,+ "setEffectDefinition",+ { effectId, code },+ true,+ runtime,+ );+ expect(changed.ok).toBe(true);+ expect(asked.at(-1)).toEqual({ code, params: [{ name: "level", type: "number" }] });+ });++ it("asks nothing when edits are off, or for a tool that compiles nothing", async () => {+ const asked: Array<{ code: string; params: unknown }> = [];+ const runtime = slowChecker(asked);+ await runDocumentTool(+ fixtureDocument(),+ "addEffect",+ {+ startSec: 1,+ endSec: 2,+ effect: effect("fn effect(uv: vec2<f32>) -> vec4<f32> { return frame(uv); }"),+ },+ false,+ runtime,+ );+ await runDocumentTool(fixtureDocument(), "addTrim", { startSec: 1, endSec: 2 }, true, runtime);+ expect(asked).toEqual([]);+ });+});diff --git a/electron/ai-edition/deep-agent/service.ts b/electron/ai-edition/deep-agent/service.tsindex b3e2f978b..88b7333d0 100644--- a/electron/ai-edition/deep-agent/service.ts+++ b/electron/ai-edition/deep-agent/service.ts@@ -89,6 +89,8 @@ import {setTrimArgs,setWordTextArgs,setZoomArgs,+ shaderEffectCheckLookup,+ shaderEffectToCheck,} from "../agent-tools";import {getCaptionsArgs,@@ -263,7 +265,7 @@ export const TOOL_DESCRIPTIONS: Record<string, string> = {addCameraFullscreen:"Add a camera-fullscreen region over a span of the edited timeline (virtual seconds): the webcam fills the frame for that span. This only does something when the footage under that span comes from an asset with a linked webcam — check assets[].hasCameraTrack (or hasAnyCamera) in getCurrentDocument first. On footage with no camera the call is refused rather than storing a region that would render nothing; say so instead of retrying.",addEffect:- "Write a visual effect as a shader and pose it over a span of the edited timeline (virtual seconds), or pose one of this project's effects again (effectId). Use it for a look the settings do not offer. `effect.code` is WGSL that defines `fn effect(uv: vec2<f32>) -> vec4<f32>`, called for every pixel; uv is 0..1 with (0,0) top-left. Available inside it: `frame(uv)` samples the composed picture; `params.<name>` reads each declared parameter; `ctx.time` is programme seconds; `ctx.region.x` goes 0 to 1 across the region (y its length in s, z seconds since it began); `ctx.resolution` and `ctx.aspect`; `ctx.cursor` (xy in uv, z = 1 when on screen); `ctx.clicks[0..3]`, the last four clicks, newest first (xy in uv where it happened, z seconds since, w = 1 when the slot holds one); `ctx.footage` and `ctx.camera`, the recording's and the webcam's rects (x, y, w, h in uv; camera zero-sized when absent) — all already placed through zoom and layout; `fx_to_footage(uv)` and `fx_from_footage(p)` map between output uv and the recording's own surface (0..1 across the recording) through the warp the compositor draws it with, zoom, layout and 3D camera included, and `ctx.footage_aspect` is its width over height: the effect works in whichever of the two spaces fits it; `fx_px(n)`, n pixels of a 1080p frame as a fraction of the height; `fx_hash(vec2<u32>)`, a random number in [0,1) identical on every GPU. Rules: the effect is a pure function of those inputs (no state between frames), so the preview and the export show the same pixels; reason in uv or fx_px, never raw pixels; every loop is a `for` with literal bounds (`for (var i = 0; i < 16; i++)`), at most 4096 iterations per pixel — the one hard rule. `params` declares up to 8 (recommended) parameters, each `{type: number|toggle|color|point, name, label, default}` (number also `min`, `max`); declare one only when moving it visibly changes the effect over its whole range. `target`: `image` (default: the picture under the annotations and captions), `text` (the annotations and captions alone, premultiplied alpha), `both` (everything); use text or both only when asked. Effects of one `lane` (0–7) cannot overlap; lanes apply in order, lane 0 first. The result gives the cost tier (frame() reads per pixel: light ≤ 8, about 0.06 ms per 1080p frame on a fast GPU; medium ≤ 32, 0.14 ms; heavy ≤ 128, 0.54 ms; very heavy beyond) and a recommendation when the tier runs long: light anywhere, medium up to a minute, heavy up to 10 s, very heavy a few seconds. It is a recommendation, never a refusal. Code that does not compile is refused with the compiler's error at the line of your code: fix it and call again. Check the result with getFrames.",+ "Write a visual effect as a shader and pose it over a span of the edited timeline (virtual seconds), or pose one of this project's effects again (effectId). Use it for a look the settings do not offer. `effect.code` is WGSL that defines `fn effect(uv: vec2<f32>) -> vec4<f32>`, called for every pixel; uv is 0..1 with (0,0) top-left. Available inside it: `frame(uv)` samples the composed picture; `params.<name>` reads each declared parameter; `ctx.time` is programme seconds; `ctx.region.x` goes 0 to 1 across the region (y its length in s, z seconds since it began); `ctx.resolution` and `ctx.aspect`; `ctx.cursor` (xy in uv, z = 1 when on screen); `ctx.clicks[0..3]`, the last four clicks, newest first (xy in uv where it happened, z seconds since, w = 1 when the slot holds one); `ctx.footage` and `ctx.camera`, the recording's and the webcam's rects (x, y, w, h in uv; camera zero-sized when absent) — all already placed through zoom and layout; `fx_to_footage(uv)` and `fx_from_footage(p)` map between output uv and the recording's own surface (0..1 across the recording) through the warp the compositor draws it with, zoom, layout and 3D camera included, and `ctx.footage_aspect` is its width over height: the effect works in whichever of the two spaces fits it; `fx_px(n)`, n pixels of a 1080p frame as a fraction of the height; `fx_hash(vec2<u32>)`, a random number in [0,1) identical on every GPU. Rules: the effect is a pure function of those inputs (no state between frames), so the preview and the export show the same pixels; reason in uv or fx_px, never raw pixels; every loop is a `for` with literal bounds (`for (var i = 0; i < 16; i++)`), at most 4096 iterations per pixel — the one hard rule. `params` declares up to 8 (recommended) parameters, each `{type: number|toggle|color|point, name, label, default}` (number also `min`, `max`); declare one only when moving it visibly changes the effect over its whole range. `target`: `image` (default: the picture under the annotations and captions), `text` (the annotations and captions alone, premultiplied alpha), `both` (everything); use text or both only when asked. Effects of one `lane` (0–7) cannot overlap; lanes apply in order, lane 0 first. The result gives the cost tier, in frame() reads per pixel, with what a 1080p frame pays on a fast discrete GPU (RTX 4070 Ti) and on an integrated one (Apple M1): light ≤ 8, up to 0.06 ms and 1.3 ms; medium ≤ 32, 0.14 ms and 4.3 ms; heavy ≤ 128, 0.54 ms and 16 ms; very heavy beyond, 2 ms and 49 ms at 512 reads. It also gives a recommendation when the tier runs long: light anywhere, medium up to a minute, heavy up to 10 s, very heavy a few seconds. It is a recommendation, never a refusal. Code that does not compile is refused with the compiler's error at the line of your code: fix it and call again. Check the result with getFrames.",setEffect:"Move, resize or re-tune an effect region by id (virtual seconds): its span, its parameter `values`, its `target` or its `lane`. Only the fields you pass change; a value equal to the default is dropped. Refused if the new span runs into another effect of the same lane.",setEffectDefinition:@@ -405,11 +407,22 @@ export async function runDocumentTool(: undefined;const autoZooms =name === "addAutoZooms" && editsAllowed ? await planAutoZooms(document, runtime) : undefined;+ // The compositor compiles an effect's code for real, off the main thread: asked here, before+ // the synchronous executor reads the answer.+ const effectSource =+ editsAllowed && runtime.checkShaderEffect ? shaderEffectToCheck(document, name, args) : null;+ const checkShaderEffect =+ effectSource && runtime.checkShaderEffect+ ? shaderEffectCheckLookup(+ effectSource,+ await runtime.checkShaderEffect(effectSource.code, effectSource.params),+ )+ : undefined;return executeAgentTool(document, name, JSON.stringify(args ?? {}), {editsAllowed,cursorTelemetry: { availableByAssetId: runtime.availableByAssetId, load },cameraSegmentation: runtime.cameraSegmentation,- checkShaderEffect: runtime.checkShaderEffect,+ checkShaderEffect,autoZooms,});}diff --git a/electron/mcp/effects-contract.ts b/electron/mcp/effects-contract.tsindex 47c28e37d..96c24d764 100644--- a/electron/mcp/effects-contract.ts+++ b/electron/mcp/effects-contract.ts@@ -91,16 +91,17 @@ two lanes.## CostThe result of addEffect gives the cost in frame() reads per pixel, counted from the code (loops-multiplied by their bound), and its tier. Measured on a fast GPU (RTX 4070 Ti) at 1080p:+multiplied by their bound), and its tier. Per 1080p frame, measured on a fast discrete GPU (RTX+4070 Ti) and on an integrated one (Apple M1):-| Tier | Reads | Per frame | Recommended for |-|---|---|---|---|-| light | ≤ 8 | ≤ 0.06 ms | anywhere, the whole video included |-| medium | ≤ 32 | ≤ 0.14 ms | up to a minute |-| heavy | ≤ 128 | ≤ 0.54 ms | up to 10 s: a transition, a key moment |-| very heavy | more | 2 ms at 512 | a few seconds |+| Tier | Reads | Per frame, RTX 4070 Ti | Per frame, M1 | Recommended for |+|---|---|---|---|---|+| light | ≤ 8 | ≤ 0.06 ms | ≤ 1.3 ms | anywhere, the whole video included |+| medium | ≤ 32 | ≤ 0.14 ms | ≤ 4.3 ms | up to a minute |+| heavy | ≤ 128 | ≤ 0.54 ms | ≤ 16 ms | up to 10 s: a transition, a key moment |+| very heavy | more | 2 ms at 512 | 49 ms at 512 | a few seconds |-An integrated GPU pays several times more. The tiers are recommendations, never refusals.+A 4K frame costs about four times as much. The tiers are recommendations, never refusals.## Example: a ripple on the recording, on every clickdiff --git a/electron/native-bridge/services/compositorViewService.test.ts b/electron/native-bridge/services/compositorViewService.test.tsindex 01817e826..40987fe62 100644--- a/electron/native-bridge/services/compositorViewService.test.ts+++ b/electron/native-bridge/services/compositorViewService.test.ts@@ -271,6 +271,34 @@ function writeCargoConfig(root: string, body: string): void {);}+describe("CompositorViewService checks an effect's code off the main thread", () => {+ const service = (checkShaderEffect: CompositorViewAddon["checkShaderEffect"]) =>+ new CompositorViewService({ addon: { checkShaderEffect } as unknown as CompositorViewAddon });++ it("passes the addon's answer through", async () => {+ const check = { tier: "light" as const, frameReads: 1, iterations: 0 };+ const checker = vi.fn(async () => check);+ expect(+ await service(checker).checkShaderEffect("code", [{ name: "level", type: "number" }]),+ ).toEqual(check);+ expect(checker).toHaveBeenCalledWith(+ "code",+ JSON.stringify([{ name: "level", type: "number" }]),+ );+ });++ it("reads an addon that fails, whether it throws or rejects, as not checked", async () => {+ const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined);+ const rejects = vi.fn(() => Promise.reject(new Error("native")));+ const throws = vi.fn(() => {+ throw new Error("native");+ });+ expect(await service(rejects).checkShaderEffect("code", [])).toBeNull();+ expect(await service(throws).checkShaderEffect("code", [])).toBeNull();+ warn.mockRestore();+ });+});+describe("ffmpegSharedBinCandidates", () => {let tmpRoot: string;diff --git a/electron/native-bridge/services/compositorViewService.ts b/electron/native-bridge/services/compositorViewService.tsindex fd0a25b52..95bff7ad0 100644--- a/electron/native-bridge/services/compositorViewService.ts+++ b/electron/native-bridge/services/compositorViewService.ts@@ -592,14 +592,16 @@ export class CompositorViewService {/** Checks an agent-written effect's WGSL in the compositor (`docs/shader-effects.md`, E.3): the* compile error, or the cost tier. `null` when this addon cannot answer, which the agent tools* report as "not checked" rather than as a compile failure. */- checkShaderEffect(+ async checkShaderEffect(code: string,params: ReadonlyArray<{ name: string; type: string }>,- ): ShaderEffectCheck | null {+ ): Promise<ShaderEffectCheck | null> {const addon = this.ensureAddon();if (!addon || typeof addon.checkShaderEffect !== "function") return null;try {- return addon.checkShaderEffect(+ // `await`, not a bare return: the check is a promise now, and a rejection must land in+ // this catch rather than escape it.+ return await addon.checkShaderEffect(code,JSON.stringify(params.map(({ name, type }) => ({ name, type }))),);diff --git a/electron/native/compositor-view/addon.d.ts b/electron/native/compositor-view/addon.d.tsindex de1bb08d7..501a41c73 100644--- a/electron/native/compositor-view/addon.d.ts+++ b/electron/native/compositor-view/addon.d.ts@@ -156,9 +156,10 @@ export interface CompositorViewAddon {* needs the answer before any preview exists. Cached native-side. */probeBackend(): CompositorBackend;/** Checks an effect's WGSL: validation, literal loop bounds, the real compile for this machine's- * backend where there is one (FXC on Windows), and the cost tier. `paramsJson` is the declared- * parameters, `[{ "name": "strength", "type": "number" }, …]`. Absent from older builds. */- checkShaderEffect?(code: string, paramsJson: string): ShaderEffectCheck;+ * backend (FXC on Windows, Metal on macOS, SPIR-V on Linux), and the cost tier. `paramsJson`+ * is the declared parameters, `[{ "name": "strength", "type": "number" }, …]`. Runs on the+ * libuv pool: a Metal compile takes 120 ms and more. Absent from older builds. */+ checkShaderEffect?(code: string, paramsJson: string): Promise<ShaderEffectCheck>;/** Whether the ONNX Runtime library is where the app staged it, and therefore whether a* segmentation mask can be produced at all. Asked of the machine rather than guessed fromdiff --git a/src/lib/ai-edition/shaderEffects.ts b/src/lib/ai-edition/shaderEffects.tsindex 8aa973128..6d0f29515 100644--- a/src/lib/ai-edition/shaderEffects.ts+++ b/src/lib/ai-edition/shaderEffects.ts@@ -25,10 +25,19 @@ export interface ShaderEffectCheck {iterations?: number | null;}-/** Asks the compositor about a code; `null` where nothing can answer (no addon, a test). */+/** Asks the compositor about a code; `null` where nothing can answer (no addon, a test). A+ * promise: the compositor compiles for real, which takes 120 ms and more on Metal, off the main+ * thread. */export type ShaderEffectChecker = (code: string,params: ReadonlyArray<Pick<AxcutShaderEffectParam, "name" | "type">>,+) => Promise<ShaderEffectCheck | null>;++/** The checker's answer as the tool executor reads it, synchronously: `runDocumentTool` asked+ * first (`shaderEffectToCheck`). `null` for a code that was not asked about. */+export type ShaderEffectCheckLookup = (+ code: string,+ params: ReadonlyArray<Pick<AxcutShaderEffectParam, "name" | "type">>,) => ShaderEffectCheck | null;export function findShaderEffect(diff --git a/technical-documentation/engineering/rendering-performance.md b/technical-documentation/engineering/rendering-performance.mdindex b88bedc35..78c68bf18 100644--- a/technical-documentation/engineering/rendering-performance.md+++ b/technical-documentation/engineering/rendering-performance.md@@ -474,7 +474,7 @@ An agent-written effect (`crates/compositor/src/shader_fx.rs`, [docs/shader-effe- **About 0.12 ms per read at 1080p and 0.5 ms at 4K,** over a fixed 0.3 ms: thirty times the RTX. The fixed part is the copy into `ann_copy` and the full-frame pass, which are bandwidth on a 68 GB/s unified memory.- **The reads dominate here too.** The bench effect computes a `sin` and a `cos` per read; with integer offsets instead, 128 reads cost 10.5 ms rather than 16.2.- **naga's `force_loop_bounding` is off** (`shader_fx::to_msl`): its guard, a `volatile` read per iteration, cost 7 to 10 % of every row above. The validation has already proved each loop bounded.-- **Compiling is ten times slower than FXC.** naga, `new_library_with_source` and the pipeline state take 120 ms in the median for a source Metal has not seen (the C.6 ripple), and 160 to 375 ms for the first one in a process. A source Metal has seen comes back in 0.4 ms in the same process, 3 ms in another. All of it runs on the background thread; the preview shows the effect once it lands. `checkShaderEffect` compiles on the calling thread, so an agent's `addEffect` took 152 ms through the `mcp` CLI.+- **Compiling is ten times slower than FXC.** naga, `new_library_with_source` and the pipeline state take 120 ms in the median for a source Metal has not seen (the C.6 ripple), and 160 to 375 ms for the first one in a process. A source Metal has seen comes back in 0.4 ms in the same process, 3 ms in another. All of it runs on the background thread; the preview shows the effect once it lands. An agent's `addEffect` took 152 ms through the `mcp` CLI, the compile included; `checkShaderEffect` runs on the libuv pool, off the main thread.- **The export does not notice an effect:** 180 frames at 1752×1080 (`exportMulti`, three alternated runs after a warm-up) took 2469 ms without, 2428 ms with the ripple over the whole video, and 2375 ms with the spike's heavier effect. The M1 export is encoder-bound. The live view held 29.7 fps against a 30 fps source with and without the effect.- **Not measured yet:** the Radeon 610M.