Code-first video & image, rendered natively on WebGPU.
perf(vision): share sessions across layers, run on WebGPU by default
okanasl60c35a0d8392
diff --git a/.changeset/vision-gpu-shared-sessions.md b/.changeset/vision-gpu-shared-sessions.mdnew file mode 100644index 00000000..94827d9f--- /dev/null+++ b/.changeset/vision-gpu-shared-sessions.md@@ -0,0 +1,15 @@+---+"framefields": minor+"@framefields/vision": minor+"@framefields/node-vision": patch+---++Faster on-device vision, on the GPU by default:++- Vision layers no longer reload their models every frame. Layers whose options differed (e.g. one with `classes: ["person"]` and one without) used to close and recreate their ONNX sessions on each frame; they now share sessions through `VisionRunner.derive()`. This also fixes the `thread constructor failed: Resource temporarily unavailable` crash with `onnxruntime-web` installed.+- Layers that hand vision the same pixels (two panels of one video) share one RTMDet-Ins pass.+- `onnxruntime-web` is now an optional dependency of `framefields`, so inference runs on WebGPU when there is a hardware adapter. Dawn's `GPUSupportedFeatures` gains the `forEach` that onnxruntime-web's WebGPU build needs. Software adapters (lavapipe, SwiftShader) use the CPU instead.+- CPU sessions on Linux size their thread pool from the CPUs the process may use, cgroup quota included, instead of the host's cores. `FRAMEFIELDS_VISION_THREADS` overrides it.+- `FRAMEFIELDS_VISION_PROVIDER` (`auto`, `webgpu`, `cpu`, `cuda`) forces a backend; `cuda` uses onnxruntime-node's CUDA EP on Linux x64.++The smoke project's `vision` export (three vision layers, 60 frames of 1280×720) on an Apple Silicon Mac: 84.9 s before; 41–43 s on the CPU and 26–29 s on the GPU after.diff --git a/nodes/node-vision/src/renderers/webgpu-renderer.ts b/nodes/node-vision/src/renderers/webgpu-renderer.tsindex 57d63dfc..1624d6c6 100644--- a/nodes/node-vision/src/renderers/webgpu-renderer.ts+++ b/nodes/node-vision/src/renderers/webgpu-renderer.ts@@ -21,7 +21,16 @@ import {} from "./frame-cache.js";import { MatteCompositor } from "./matte-compositor.js";-let sharedRunner: { key: string; runner: VisionRunner } | null = null;+/**+ * Runners by model options (variant, models dir, mirror), each with the runners+ * derived from it by decode options — layers that differ only in classes or+ * confidence share sessions instead of reloading the models.+ */+const sharedRunners = new Map<+ string,+ { runner: VisionRunner; derived: Map<string, VisionRunner> }+>();+const MAX_SHARED_RUNNERS = 4;let sharedSkeletonRenderer: PoseSkeletonRenderer | null = null;let sharedCompositor: MatteCompositor | null = null;let sharedCompositorFormat: GPUTextureFormat | null = null;@@ -175,24 +184,42 @@ function selectSubject(/*** Lazy shared runner — `create()` is a pure constructor (zero I/O); the first frame that* requests a task triggers that task's model download at inference time (never at init).- * Recreated only when an option that changes inference changes.+ * Nodes with the same model options share sessions; decode options (classes,+ * confidence, …) get a derived runner over the same sessions.*/function getSharedRunner(op: VisionOperation): VisionRunner {- const options = {+ const modelOptions = {variant: op.variant,+ modelsDir: op.modelsDir,+ baseUrl: op.baseUrl,+ };+ const modelKey = JSON.stringify(modelOptions);+ let entry = sharedRunners.get(modelKey);+ if (!entry) {+ if (sharedRunners.size >= MAX_SHARED_RUNNERS) {+ const [oldestKey, oldest] = sharedRunners.entries().next().value as [+ string,+ { runner: VisionRunner },+ ];+ oldest.runner.close();+ sharedRunners.delete(oldestKey);+ }+ entry = { runner: VisionRunner.create(modelOptions), derived: new Map() };+ sharedRunners.set(modelKey, entry);+ }+ const decodeOptions = {confidence: op.confidence,classes: op.classes,maskThreshold: op.maskThreshold,featherRadius: op.featherRadius,- modelsDir: op.modelsDir,- baseUrl: op.baseUrl,};- const key = JSON.stringify(options);- if (sharedRunner?.key !== key) {- sharedRunner?.runner.close();- sharedRunner = { key, runner: VisionRunner.create(options) };+ const decodeKey = JSON.stringify(decodeOptions);+ let runner = entry.derived.get(decodeKey);+ if (!runner) {+ runner = entry.runner.derive(decodeOptions);+ entry.derived.set(decodeKey, runner);}- return sharedRunner.runner;+ return runner;}function getObjectTracker(diff --git a/packages/framefields/package.json b/packages/framefields/package.jsonindex 76a02bd4..5af0202e 100644--- a/packages/framefields/package.json+++ b/packages/framefields/package.json@@ -108,6 +108,9 @@"yoga-layout": "^3.2.1","zod": "^4.3.6"},+ "optionalDependencies": {+ "onnxruntime-web": "^1.24.0"+ },"devDependencies": {"@framefields/client-utils": "workspace:*","@framefields/compositions": "workspace:*",diff --git a/packages/vision/src/analysis/analyze-sequence.test.ts b/packages/vision/src/analysis/analyze-sequence.test.tsindex f8c1f6c3..6e069767 100644--- a/packages/vision/src/analysis/analyze-sequence.test.ts+++ b/packages/vision/src/analysis/analyze-sequence.test.ts@@ -36,7 +36,8 @@ class MovingProvider implements SessionProvider {async function* frames(count: number): AsyncIterable<VisionImageInput> {for (let i = 0; i < count; i++) {yield {- data: new Uint8ClampedArray(640 * 640 * 4).fill(128),+ // A distinct picture per frame: equal pixels share one model pass.+ data: new Uint8ClampedArray(640 * 640 * 4).fill(100 + i),width: 640,height: 640,};diff --git a/packages/vision/src/runner/vision-runner.test.ts b/packages/vision/src/runner/vision-runner.test.tsindex 798655ca..d158c1ee 100644--- a/packages/vision/src/runner/vision-runner.test.ts+++ b/packages/vision/src/runner/vision-runner.test.ts@@ -53,11 +53,40 @@ describe("VisionRunner (lazy + decode integration)", () => {expect(m.mask[180 * 640 + 230]).toBe(0); // inside the box, outside the blobexpect(m.coverage).toBeGreaterThan(0);- // A new frame buffer runs the model again.+ // The same pixels in another buffer (another layer) reuse the pass…await runner.segment(grayImage());+ expect(provider.created[0].inputs).toHaveLength(1);++ // …a different frame runs the model again.+ const next = grayImage();+ next.data[0] = 0;+ await runner.segment(next);expect(provider.created[0].inputs).toHaveLength(2);});+ it("derived runners share sessions and passes with their base", async () => {+ const { store } = fakeStore();+ const provider = new ScriptedProvider([rtmdetInsOutput()]);+ const base = VisionRunner.create({ store, provider });+ const people = base.derive({ classes: ["person"], confidence: 0.5 });+ expect(people.classes).toEqual(["person"]);+ expect(people.confidence).toBe(0.5);+ expect(people.variant).toBe(base.variant);++ const frame = grayImage();+ await base.detect(frame);+ await people.segment(frame);+ expect(provider.created).toHaveLength(1);+ expect(provider.created[0].inputs).toHaveLength(1);++ // Closing a derived runner leaves the shared session alone.+ people.close();+ const next = grayImage();+ next.data[0] = 0;+ await base.detect(next);+ expect(provider.created).toHaveLength(1);+ });+it("maps boxes and masks back through a top-left letterbox on wide frames", async () => {const { store } = fakeStore();const runner = VisionRunner.create({diff --git a/packages/vision/src/runner/vision-runner.ts b/packages/vision/src/runner/vision-runner.tsindex 82c5dd5e..06167b15 100644--- a/packages/vision/src/runner/vision-runner.ts+++ b/packages/vision/src/runner/vision-runner.ts@@ -46,7 +46,7 @@ export interface VisionRunnerOptions {/** Mirror origin for model downloads (see VisionModelStore). */readonly baseUrl?: string;readonly timeoutMs?: number;- /** Session backend. Default: onnxruntime-node (CPU). Browser: `createWebGPUProvider()`. */+ /** Session backend. Default: WebGPU when possible, else onnxruntime-node (CPU); see `AutoSessionProvider`. Browser: `createWebGPUProvider()`. */readonly provider?: SessionProvider;/** Inject a model store (tests, shared caches). Overrides modelsDir/baseUrl/timeoutMs. */readonly store?: VisionModelStore;@@ -57,6 +57,35 @@ export interface VisionRunnerOptions {) => void;}+/** Options that only change how outputs are decoded — not which model runs. */+export type VisionDecodeOptions = Pick<+ VisionRunnerOptions,+ "confidence" | "classes" | "maskThreshold" | "featherRadius"+>;++/**+ * Model-side state: sessions, buffers and the per-frame RTMDet-Ins pass. Shared+ * by a runner and every runner `derive()`d from it.+ */+interface RunnerCore {+ readonly store: VisionModelStore;+ readonly provider: SessionProvider;+ readonly sessions: Map<VisionModelKey, Promise<VisionSession>>;+ readonly tensorScratch: Map<VisionModelKey, Float32Array>;+ /** Tail of each model's run queue: one inference per session at a time. */+ readonly runQueue: Map<VisionModelKey, Promise<unknown>>;+ /**+ * Recent RTMDet-Ins passes, newest last, shared by detect() and segment()+ * and matched by content — layers showing the same pixels share one pass.+ */+ readonly instancePasses: {+ readonly content: string;+ readonly result: Promise<InstancePass>;+ }[];+}++const MAX_INSTANCE_PASSES = 4;+/*** Lazy vision runner.*@@ -72,34 +101,32 @@ export class VisionRunner {public readonly maskThreshold: number;public readonly featherRadius?: number;- private readonly _store: VisionModelStore;- private readonly _provider: SessionProvider;- private readonly _sessions = new Map<- VisionModelKey,- Promise<VisionSession>- >();- private readonly _tensorScratch = new Map<VisionModelKey, Float32Array>();- /** One RTMDet-Ins pass per frame buffer, shared by detect() and segment(). */- private _instancePass?: {- readonly frame: VisionImageInput["data"];- readonly result: Promise<InstancePass>;- };+ private readonly _core: RunnerCore;+ /** False for derived runners: their sessions belong to the runner they came from. */+ private readonly _ownsCore: boolean;- private constructor(options: VisionRunnerOptions) {+ private constructor(options: VisionRunnerOptions, core?: RunnerCore) {this.variant = options.variant ?? "s";this.confidence = options.confidence ?? 0.3;this.classes = options.classes;this.maskThreshold = options.maskThreshold ?? 0.5;this.featherRadius = options.featherRadius;- this._store =- options.store ??- new VisionModelStore({- modelsDir: options.modelsDir,- baseUrl: options.baseUrl,- timeoutMs: options.timeoutMs,- onProgress: options.onProgress,- });- this._provider = options.provider ?? createDefaultSessionProvider();+ this._ownsCore = core === undefined;+ this._core = core ?? {+ store:+ options.store ??+ new VisionModelStore({+ modelsDir: options.modelsDir,+ baseUrl: options.baseUrl,+ timeoutMs: options.timeoutMs,+ onProgress: options.onProgress,+ }),+ provider: options.provider ?? createDefaultSessionProvider(),+ sessions: new Map(),+ tensorScratch: new Map(),+ runQueue: new Map(),+ instancePasses: [],+ };}/** Cheap: no downloads, no sessions, no side effects. */@@ -107,8 +134,30 @@ export class VisionRunner {return new VisionRunner(options);}+ /**+ * A runner with other decode options (confidence, classes, mask threshold,+ * feather) that shares this runner's models, sessions and per-frame RTMDet-Ins+ * pass — e.g. one per layer, without loading a model twice. Its `close()` is a+ * no-op; closing this runner releases the shared sessions.+ */+ public derive(options: VisionDecodeOptions): VisionRunner {+ return new VisionRunner(+ {+ variant: this.variant,+ confidence: options.confidence ?? this.confidence,+ classes: "classes" in options ? options.classes : this.classes,+ maskThreshold: options.maskThreshold ?? this.maskThreshold,+ featherRadius:+ "featherRadius" in options+ ? options.featherRadius+ : this.featherRadius,+ },+ this._core,+ );+ }+public get modelsDir(): string {- return this._store.modelsDir;+ return this._core.store.modelsDir;}/** Every registry model keyed to its current download state (downloaded → "ready"). */@@ -120,7 +169,7 @@ export class VisionRunner {for (const key of Object.keys(VISION_MODELS) as VisionModelKey[]) {statuses.set(key, "pending");}- for (const [key, status] of this._store.statuses()) {+ for (const [key, status] of this._core.store.statuses()) {statuses.set(key, status);}return statuses;@@ -147,7 +196,7 @@ export class VisionRunner {keys.add(this.modelKeyFor(task));}}- await this._store.preload([...keys]);+ await this._core.store.preload([...keys]);await Promise.all([...keys].map((key) => this.session(key)));}@@ -210,41 +259,46 @@ export class VisionRunner {}public close(): void {- for (const promise of this._sessions.values()) {+ if (!this._ownsCore) return;+ const core = this._core;+ for (const promise of core.sessions.values()) {void promise.then((session) => session.release()).catch(() => undefined);}- this._sessions.clear();- this._instancePass = undefined;- this._store.clearMemoryCache();+ core.sessions.clear();+ core.runQueue.clear();+ core.instancePasses.length = 0;+ core.store.clearMemoryCache();}// ── internals ──────────────────────────────────────────────────────────private instancePass(image: VisionImageInput): Promise<InstancePass> {- if (this._instancePass?.frame === image.data) {- return this._instancePass.result;- }+ const passes = this._core.instancePasses;+ const content = contentKey(image);+ const byContent = passes.find((pass) => pass.content === content);+ if (byContent) return byContent.result;const key = this.modelKeyFor("segment");- const result = this.infer(key, image).then(- ({ outputs, transform }) => {- const dets = output(outputs, "dets", key);- const masks = output(outputs, "masks", key);- return {- transform,- outputs: {- dets: dets.data as Float32Array,- labels: output(outputs, "labels", key).data,- masks: masks.data as Float32Array,- count: dets.dims[1] ?? 0,- maskHeight: masks.dims[masks.dims.length - 2] ?? 0,- maskWidth: masks.dims[masks.dims.length - 1] ?? 0,- },- };- },- );- this._instancePass = { frame: image.data, result };+ const result = this.infer(key, image).then(({ outputs, transform }) => {+ const dets = output(outputs, "dets", key);+ const masks = output(outputs, "masks", key);+ return {+ transform,+ outputs: {+ dets: dets.data as Float32Array,+ labels: output(outputs, "labels", key).data,+ masks: masks.data as Float32Array,+ count: dets.dims[1] ?? 0,+ maskHeight: masks.dims[masks.dims.length - 2] ?? 0,+ maskWidth: masks.dims[masks.dims.length - 1] ?? 0,+ },+ };+ });+ const pass = { content, result };+ passes.push(pass);+ if (passes.length > MAX_INSTANCE_PASSES) passes.shift();result.catch(() => {- if (this._instancePass?.result === result) this._instancePass = undefined;+ const index = passes.indexOf(pass);+ if (index >= 0) passes.splice(index, 1);});return result;}@@ -267,7 +321,29 @@ export class VisionRunner {};}- private async infer(+ /**+ * Runs one model on one image. Runs of the same model are queued: they share+ * the input scratch buffer, and one run already uses the session's threads.+ */+ private infer(+ key: VisionModelKey,+ image: VisionImageInput,+ ): Promise<{ outputs: VisionTensorOutputs; transform: InputTransform }> {+ const queue = this._core.runQueue;+ const previous = queue.get(key) ?? Promise.resolve();+ const run = previous.then(+ () => this.inferNow(key, image),+ () => this.inferNow(key, image),+ );+ const tail = run.catch(() => undefined);+ queue.set(key, tail);+ void tail.then(() => {+ if (queue.get(key) === tail) queue.delete(key);+ });+ return run;+ }++ private async inferNow(key: VisionModelKey,image: VisionImageInput,): Promise<{ outputs: VisionTensorOutputs; transform: InputTransform }> {@@ -275,10 +351,11 @@ export class VisionRunner {const desc = VISION_MODELS[key];const spec = preprocessSpec(desc.family, desc.input);const size = 3 * spec.width * spec.height;- let scratch = this._tensorScratch.get(key);+ const scratchByKey = this._core.tensorScratch;+ let scratch = scratchByKey.get(key);if (!scratch || scratch.length < size) {scratch = new Float32Array(size);- this._tensorScratch.set(key, scratch);+ scratchByKey.set(key, scratch);}const { tensor, transform } = imageToTensor(image, spec, scratch);const outputs = await session.run({@@ -290,20 +367,21 @@ export class VisionRunner {}private session(key: VisionModelKey): Promise<VisionSession> {- let promise = this._sessions.get(key);+ const { sessions, store, provider } = this._core;+ let promise = sessions.get(key);if (!promise) {- promise = this._store+ promise = store.ensure(key)- .then((bytes) => this._provider.createSession(bytes))+ .then((bytes) => provider.createSession(bytes)).catch((err: unknown) => {// Drop both the session cache entry AND the cached file: a parse// failure means the on-disk bytes are corrupt — the next call must// re-download instead of replaying the same bad file.- this._sessions.delete(key);- void this._store.evict(key);+ sessions.delete(key);+ void store.evict(key);throw err;});- this._sessions.set(key, promise);+ sessions.set(key, promise);}return promise;}@@ -321,6 +399,24 @@ interface InstancePass {};}+/** Size plus a 64-bit hash of the pixels: equal keys mean the same picture. */+function contentKey(image: VisionImageInput): string {+ const { data } = image;+ const words = data.byteOffset % 4 === 0 ? data.byteLength >>> 2 : 0;+ const view = new Uint32Array(data.buffer, data.byteOffset, words);+ let h1 = 0x811c9dc5;+ let h2 = 0x01000193 ^ data.byteLength;+ for (let i = 0; i < words; i++) {+ const v = view[i] as number;+ h1 = Math.imul(h1 ^ v, 0x9e3779b1);+ h2 = Math.imul(h2 ^ v, 0x85ebca77) ^ (h1 >>> 15);+ }+ for (let i = words * 4; i < data.length; i++) {+ h1 = Math.imul(h1 ^ (data[i] as number), 0x9e3779b1);+ }+ return `${image.width}x${image.height}:${(h1 >>> 0).toString(36)}:${(h2 >>> 0).toString(36)}`;+}+function output(outputs: VisionTensorOutputs,name: string,diff --git a/packages/vision/src/runtime/node-webgpu-provider.ts b/packages/vision/src/runtime/node-webgpu-provider.tsindex 6dead1ea..2ebe9900 100644--- a/packages/vision/src/runtime/node-webgpu-provider.ts+++ b/packages/vision/src/runtime/node-webgpu-provider.ts@@ -22,29 +22,74 @@ const WEBGPU_SPECIFIER = "webgpu";let nodeWebGPUReady: Promise<void> | undefined;+interface NodeGPUAdapter {+ readonly features: object;+ readonly info?: {+ readonly vendor?: string;+ readonly architecture?: string;+ readonly device?: string;+ readonly description?: string;+ readonly isFallbackAdapter?: boolean;+ };+ readonly isFallbackAdapter?: boolean;+}++interface NodeGPU {+ requestAdapter(options?: {+ powerPreference?: "low-power" | "high-performance";+ }): Promise<NodeGPUAdapter | null>;+}++/** CPU rasterizers (lavapipe, llvmpipe, SwiftShader): slower than the CPU EP. */+const SOFTWARE_ADAPTER = /llvmpipe|lavapipe|swiftshader|software/i;+/**- * Ensure `globalThis.navigator.gpu` exists. In the renderer this is already the- * Dawn device; otherwise imports the `webgpu` (Dawn) package and installs its- * globals plus an adapter, which is what onnxruntime-web's WebGPU EP needs.+ * Ensure `globalThis.navigator.gpu` exists and has a hardware adapter that+ * onnxruntime-web's WebGPU EP can use. In the renderer this is already the Dawn+ * device; otherwise imports the `webgpu` (Dawn) package and installs its+ * globals. Rejects when the only adapter is a software rasterizer.*/export function ensureNodeWebGPU(): Promise<void> {if (!nodeWebGPUReady) {nodeWebGPUReady = (async () => {- const g = globalThis as { navigator?: { gpu?: unknown } };- if (g.navigator?.gpu) return;- const { create, globals } = (await import(- /* @vite-ignore */ WEBGPU_SPECIFIER- )) as {- create: (flags: string[]) => unknown;- globals: Record<string, unknown>;- };- Object.assign(globalThis, globals);- const nav = (g.navigator ??= {});- Object.defineProperty(nav, "gpu", {- value: create([]),- configurable: true,- writable: true,+ const g = globalThis as { navigator?: { gpu?: NodeGPU } };+ if (!g.navigator?.gpu) {+ const { create, globals } = (await import(+ /* @vite-ignore */ WEBGPU_SPECIFIER+ )) as {+ create: (flags: string[]) => NodeGPU;+ globals: Record<string, unknown>;+ };+ Object.assign(globalThis, globals);+ g.navigator ??= {};+ Object.defineProperty(g.navigator, "gpu", {+ value: create([]),+ configurable: true,+ writable: true,+ });+ }+ const gpu = g.navigator?.gpu as NodeGPU;+ const adapter = await gpu.requestAdapter({+ powerPreference: "high-performance",});+ if (!adapter) throw new Error("No WebGPU adapter.");+ const info = adapter.info ?? {};+ const name = [+ info.vendor,+ info.architecture,+ info.device,+ info.description,+ ]+ .filter(Boolean)+ .join(" ");+ if (+ adapter.isFallbackAdapter === true ||+ info.isFallbackAdapter === true ||+ SOFTWARE_ADAPTER.test(name)+ ) {+ throw new Error(`WebGPU adapter is a software rasterizer (${name}).`);+ }+ patchSupportedFeatures(adapter.features);})();nodeWebGPUReady.catch(() => {nodeWebGPUReady = undefined;@@ -53,6 +98,54 @@ export function ensureNodeWebGPU(): Promise<void> {return nodeWebGPUReady;}+/**+ * Dawn's Node `GPUSupportedFeatures` has `has`/`keys`/`size` but no `forEach`,+ * which onnxruntime-web's WebGPU build calls on adapter and device features+ * (they share the class). Adds the missing set-like methods once.+ */+function patchSupportedFeatures(features: object): void {+ const proto = Object.getPrototypeOf(features) as {+ keys?: () => IterableIterator<string>;+ forEach?: unknown;+ values?: unknown;+ entries?: unknown;+ [Symbol.iterator]?: unknown;+ } | null;+ if (!proto || typeof proto.keys !== "function") return;+ type SetLike = { keys(): IterableIterator<string> };+ if (typeof proto.forEach !== "function") {+ Object.defineProperty(proto, "forEach", {+ value(+ this: SetLike,+ callback: (value: string, key: string, set: SetLike) => void,+ thisArg?: unknown,+ ) {+ for (const key of this.keys()) callback.call(thisArg, key, key, this);+ },+ configurable: true,+ writable: true,+ });+ }+ if (typeof proto.values !== "function") {+ Object.defineProperty(proto, "values", {+ value(this: SetLike) {+ return this.keys();+ },+ configurable: true,+ writable: true,+ });+ }+ if (typeof proto.entries !== "function") {+ Object.defineProperty(proto, "entries", {+ *value(this: SetLike) {+ for (const key of this.keys()) yield [key, key];+ },+ configurable: true,+ writable: true,+ });+ }+}+/** Options mirror the browser provider; the loader is overridden for Node. */export type NodeWebGPUProviderOptions = WebGPUProviderOptions;@@ -69,11 +162,16 @@ export class NodeWebGPUProvider implements SessionProvider {const mod = (await import(/* @vite-ignore */ ORT_WEBGPU_SPECIFIER)) as OrtWebModule & {- env?: { wasm?: { numThreads?: number } };+ env?: {+ wasm?: { numThreads?: number };+ webgpu?: { powerPreference?: string };+ };};// Single-threaded WASM: Node has no cross-origin isolation, and the// WASM path is only a fallback behind the WebGPU EP.if (mod.env?.wasm) mod.env.wasm.numThreads = 1;+ if (mod.env?.webgpu)+ mod.env.webgpu.powerPreference = "high-performance";return mod;});// No WASM fallback inside the WebGPU provider: when WebGPU isn't possiblediff --git a/packages/vision/src/runtime/session-provider.test.ts b/packages/vision/src/runtime/session-provider.test.tsnew file mode 100644index 00000000..ebfa282c--- /dev/null+++ b/packages/vision/src/runtime/session-provider.test.ts@@ -0,0 +1,42 @@+import { afterEach, describe, expect, it, vi } from "vitest";+import {+ availableCpus,+ cpuSessionThreads,+ createDefaultSessionProvider,+ requestedVisionBackend,+ setDefaultSessionProvider,+} from "./session-provider.js";++afterEach(() => {+ vi.unstubAllEnvs();+ setDefaultSessionProvider(undefined);+});++describe("session provider selection", () => {+ it("reads FRAMEFIELDS_VISION_PROVIDER, defaulting to auto", () => {+ expect(requestedVisionBackend()).toBe("auto");+ vi.stubEnv("FRAMEFIELDS_VISION_PROVIDER", " CUDA ");+ expect(requestedVisionBackend()).toBe("cuda");+ vi.stubEnv("FRAMEFIELDS_VISION_PROVIDER", "tpu");+ expect(requestedVisionBackend()).toBe("auto");+ });++ it("shares one default provider per process", () => {+ expect(createDefaultSessionProvider()).toBe(createDefaultSessionProvider());+ });+});++describe("CPU session threads", () => {+ it("honours FRAMEFIELDS_VISION_THREADS", async () => {+ vi.stubEnv("FRAMEFIELDS_VISION_THREADS", "3");+ expect(await cpuSessionThreads()).toBe(3);+ });++ it("never exceeds the CPUs the process may use", async () => {+ const cpus = await availableCpus();+ expect(cpus).toBeGreaterThanOrEqual(1);+ const threads = await cpuSessionThreads();+ if (process.platform === "linux") expect(threads).toBe(cpus);+ else expect(threads).toBeUndefined();+ });+});diff --git a/packages/vision/src/runtime/session-provider.ts b/packages/vision/src/runtime/session-provider.tsindex fcf0d42b..1232bb42 100644--- a/packages/vision/src/runtime/session-provider.ts+++ b/packages/vision/src/runtime/session-provider.ts@@ -43,47 +43,167 @@ export function setDefaultSessionProvider(defaultProvider = factory;}+/** Inference backend: `FRAMEFIELDS_VISION_PROVIDER`, default "auto". */+export type VisionBackend = "auto" | "webgpu" | "cuda" | "cpu";++export function requestedVisionBackend(): VisionBackend {+ const value =+ typeof process !== "undefined"+ ? process.env.FRAMEFIELDS_VISION_PROVIDER?.trim().toLowerCase()+ : undefined;+ return value === "webgpu" || value === "cuda" || value === "cpu"+ ? value+ : "auto";+}+/*** Prefers the GPU (onnxruntime-web's WebGPU EP on the process's WebGPU device —* the Dawn device the renderer already creates, or one this ensures) and falls* back to the CPU provider (onnxruntime-node) when a WebGPU session cannot be- * created: no device, no Dawn, or the EP unavailable. The choice is made on the- * first session and reused for the runner's lifetime.+ * created: no hardware adapter, no Dawn, or onnxruntime-web not installed. The+ * choice is made on the first session and kept.+ *+ * `FRAMEFIELDS_VISION_PROVIDER` forces one backend instead: `webgpu`, `cpu`, or+ * `cuda` (onnxruntime-node's CUDA EP on Linux x64; needs the CUDA 12 and cuDNN 9+ * libraries). A forced backend that fails throws instead of falling back.*/export class AutoSessionProvider implements SessionProvider {private _resolved?: SessionProvider;+ private _resolving?: Promise<{+ provider: SessionProvider;+ first?: VisionSession;+ }>;public get kind(): "node" | "webgpu" {return this._resolved?.kind ?? "webgpu";}public async createSession(modelBytes: Uint8Array): Promise<VisionSession> {- if (!this._resolved) {- const webgpu = new NodeWebGPUProvider({ wasmFallback: false });- try {- const session = await webgpu.createSession(modelBytes);- this._resolved = webgpu;- return session;- } catch {- // WebGPU isn't possible here — use CPU from now on.- this._resolved = new NodeSessionProvider();- }+ if (this._resolved) return this._resolved.createSession(modelBytes);+ if (!this._resolving) {+ this._resolving = this.resolve(modelBytes);+ this._resolving.catch(() => {+ this._resolving = undefined;+ });+ const { provider, first } = await this._resolving;+ this._resolved = provider;+ if (first) return first;+ return provider.createSession(modelBytes);+ }+ const { provider } = await this._resolving;+ return provider.createSession(modelBytes);+ }++ private async resolve(+ modelBytes: Uint8Array,+ ): Promise<{ provider: SessionProvider; first?: VisionSession }> {+ const backend = requestedVisionBackend();+ if (backend === "cpu") return { provider: new NodeSessionProvider() };+ if (backend === "cuda") {+ return {+ provider: new NodeSessionProvider({+ executionProviders: ["cuda", "cpu"],+ }),+ };+ }+ const webgpu = new NodeWebGPUProvider({ wasmFallback: false });+ try {+ return {+ provider: webgpu,+ first: await webgpu.createSession(modelBytes),+ };+ } catch (err) {+ if (backend === "webgpu") throw err;+ // WebGPU isn't possible here — use CPU from now on.+ return { provider: new NodeSessionProvider() };}- return this._resolved.createSession(modelBytes);}}+let processProvider: AutoSessionProvider | undefined;+/*** The provider for a runner created without one: GPU if possible, otherwise CPU.+ * One per process, so the GPU probe runs once however many runners there are.*/export function createDefaultSessionProvider(): SessionProvider {- return defaultProvider?.() ?? new AutoSessionProvider();+ if (defaultProvider) return defaultProvider();+ processProvider ??= new AutoSessionProvider();+ return processProvider;+}++/**+ * CPUs this process may use: the affinity mask, capped by a cgroup CPU quota+ * (containers). ONNX Runtime sizes its thread pool from the host's cores, which+ * oversubscribes a container limited to fewer CPUs than the host has.+ */+export async function availableCpus(): Promise<number> {+ const os = await import("node:os");+ const cpus =+ typeof os.availableParallelism === "function"+ ? os.availableParallelism()+ : os.cpus().length;+ const quota = await cgroupCpuQuota();+ return Math.max(1, quota !== undefined ? Math.min(cpus, quota) : cpus);+}++/** cgroup v2 `cpu.max` or v1 `cpu.cfs_quota_us`, in CPUs (rounded up); undefined when unlimited. */+async function cgroupCpuQuota(): Promise<number | undefined> {+ if (typeof process === "undefined" || process.platform !== "linux") {+ return undefined;+ }+ const { readFile } = await import("node:fs/promises");+ const read = (path: string) =>+ readFile(path, "utf8").then(+ (text) => text.trim(),+ () => undefined,+ );+ const toCpus = (quota: number, period: number) =>+ quota > 0 && period > 0 ? Math.ceil(quota / period) : undefined;++ const v2 = await read("/sys/fs/cgroup/cpu.max");+ if (v2 !== undefined) {+ const [quota, period] = v2.split(/\s+/);+ return quota === "max" ? undefined : toCpus(Number(quota), Number(period));+ }+ const v1Quota = await read("/sys/fs/cgroup/cpu/cpu.cfs_quota_us");+ const v1Period = await read("/sys/fs/cgroup/cpu/cpu.cfs_period_us");+ if (v1Quota === undefined || v1Period === undefined) return undefined;+ return toCpus(Number(v1Quota), Number(v1Period));+}++/**+ * Intra-op threads for CPU sessions: `FRAMEFIELDS_VISION_THREADS` when set,+ * else the available CPUs on Linux (where containers cap them), else ONNX+ * Runtime's own default (undefined).+ */+export async function cpuSessionThreads(): Promise<number | undefined> {+ const env =+ typeof process !== "undefined"+ ? Number.parseInt(process.env.FRAMEFIELDS_VISION_THREADS ?? "", 10)+ : Number.NaN;+ if (Number.isFinite(env) && env > 0) return env;+ if (typeof process === "undefined" || process.platform !== "linux") {+ return undefined;+ }+ return availableCpus();+}++export interface NodeSessionProviderOptions {+ /** onnxruntime-node execution providers in priority order. Default `["cpu"]`. */+ readonly executionProviders?: readonly string[];}/** onnxruntime-node provider. The native module is imported lazily on first session. */export class NodeSessionProvider implements SessionProvider {public readonly kind = "node" as const;+ private readonly _executionProviders: readonly string[];++ constructor(options: NodeSessionProviderOptions = {}) {+ this._executionProviders = options.executionProviders ?? ["cpu"];+ }+private ortPromise?: Promise<typeof import("onnxruntime-node")>;private async ort(): Promise<typeof import("onnxruntime-node")> {@@ -101,9 +221,13 @@ export class NodeSessionProvider implements SessionProvider {public async createSession(modelBytes: Uint8Array): Promise<VisionSession> {const ort = await this.ort();+ const threads = await cpuSessionThreads();const session = await ort.InferenceSession.create(modelBytes, {- executionProviders: ["cpu"],+ executionProviders: [...this._executionProviders],graphOptimizationLevel: "all",+ ...(threads !== undefined+ ? { intraOpNumThreads: threads, interOpNumThreads: 1 }+ : {}),});type OrtTensor = InstanceType<typeof ort.Tensor>;diff --git a/plugins/framefields/skills/framefields-effects/SKILL.md b/plugins/framefields/skills/framefields-effects/SKILL.mdindex 52792de5..500a85a7 100644--- a/plugins/framefields/skills/framefields-effects/SKILL.md+++ b/plugins/framefields/skills/framefields-effects/SKILL.md@@ -160,6 +160,15 @@ Measured per 1280–2048 px frame on CPU (`onnxruntime-node`):- Raise `confidence` (e.g. `0.5`) on abstract or stylized footage — at the default `0.3` the model will put loose labels ("teddy bear", "donut") on smoke, eyes and planets.- Restrict `classes` when you only care about one thing; it also stops the subject from switching to another object when the person leaves frame.+### GPU or CPU++Inference runs on the GPU through `onnxruntime-web`'s WebGPU backend (an optional dependency of `framefields`, installed by default) on a hardware adapter, and on the CPU through `onnxruntime-node` otherwise: when `onnxruntime-web` is missing, the only adapter is a software one (lavapipe, SwiftShader), or WebGPU fails. On an Apple Silicon Mac the GPU roughly halves render time against the CPU. Layers that use the same `variant` share their models, and layers showing the same pixels share one detect/segment pass.++| Variable | Effect |+| --- | --- |+| `FRAMEFIELDS_VISION_PROVIDER` | `auto` (default), `webgpu`, `cpu`, or `cuda` (onnxruntime-node's CUDA EP on Linux x64; needs the CUDA 12 and cuDNN 9 libraries). A forced backend that fails throws instead of falling back. |+| `FRAMEFIELDS_VISION_THREADS` | CPU threads per model. Default on Linux: the CPUs the process may use, cgroup quota included; elsewhere ONNX Runtime's default. |+### How it behaves at render time- **One-frame delay.** Vision reads the layer's *previous* rendered frame, so the very first frame has no results (a cutout renders transparent) and masks trail the plate by one frame. This is invisible at normal playback speed.diff --git a/pnpm-lock.yaml b/pnpm-lock.yamlindex 2ec4f4b3..f4f15e55 100644--- a/pnpm-lock.yaml+++ b/pnpm-lock.yaml@@ -2514,6 +2514,10 @@ importers:vitest:specifier: ^4.1.11version: 4.1.11(@opentelemetry/[email protected])(@types/[email protected])(@vitest/[email protected])([email protected](@noble/[email protected])([email protected]))([email protected](@types/[email protected])([email protected])([email protected])([email protected])([email protected])([email protected]))+ optionalDependencies:+ onnxruntime-web:+ specifier: ^1.24.0+ version: 1.30.0packages/media:dependencies: