Skip to content

Code-first video & image, rendered natively on WebGPU.

perf(vision): share sessions across layers, run on WebGPU by default

okanasl60c35a0d8392
diff --git a/.changeset/vision-gpu-shared-sessions.md b/.changeset/vision-gpu-shared-sessions.md
new file mode 100644
index 00000000..94827d9f
--- /dev/null
+++ b/.changeset/vision-gpu-shared-sessions.md
@@ -0,0 +1,15 @@
+---
+"framefields": minor
+"@framefields/vision": minor
+"@framefields/node-vision": patch
+---
+
+Faster on-device vision, on the GPU by default:
+
+- Vision layers no longer reload their models every frame. Layers whose options differed (e.g. one with `classes: ["person"]` and one without) used to close and recreate their ONNX sessions on each frame; they now share sessions through `VisionRunner.derive()`. This also fixes the `thread constructor failed: Resource temporarily unavailable` crash with `onnxruntime-web` installed.
+- Layers that hand vision the same pixels (two panels of one video) share one RTMDet-Ins pass.
+- `onnxruntime-web` is now an optional dependency of `framefields`, so inference runs on WebGPU when there is a hardware adapter. Dawn's `GPUSupportedFeatures` gains the `forEach` that onnxruntime-web's WebGPU build needs. Software adapters (lavapipe, SwiftShader) use the CPU instead.
+- CPU sessions on Linux size their thread pool from the CPUs the process may use, cgroup quota included, instead of the host's cores. `FRAMEFIELDS_VISION_THREADS` overrides it.
+- `FRAMEFIELDS_VISION_PROVIDER` (`auto`, `webgpu`, `cpu`, `cuda`) forces a backend; `cuda` uses onnxruntime-node's CUDA EP on Linux x64.
+
+The smoke project's `vision` export (three vision layers, 60 frames of 1280×720) on an Apple Silicon Mac: 84.9 s before; 41–43 s on the CPU and 26–29 s on the GPU after.
diff --git a/nodes/node-vision/src/renderers/webgpu-renderer.ts b/nodes/node-vision/src/renderers/webgpu-renderer.ts
index 57d63dfc..1624d6c6 100644
--- a/nodes/node-vision/src/renderers/webgpu-renderer.ts
+++ b/nodes/node-vision/src/renderers/webgpu-renderer.ts
@@ -21,7 +21,16 @@ import {
} from "./frame-cache.js";
import { MatteCompositor } from "./matte-compositor.js";
-let sharedRunner: { key: string; runner: VisionRunner } | null = null;
+/**
+ * Runners by model options (variant, models dir, mirror), each with the runners
+ * derived from it by decode options — layers that differ only in classes or
+ * confidence share sessions instead of reloading the models.
+ */
+const sharedRunners = new Map<
+ string,
+ { runner: VisionRunner; derived: Map<string, VisionRunner> }
+>();
+const MAX_SHARED_RUNNERS = 4;
let sharedSkeletonRenderer: PoseSkeletonRenderer | null = null;
let sharedCompositor: MatteCompositor | null = null;
let sharedCompositorFormat: GPUTextureFormat | null = null;
@@ -175,24 +184,42 @@ function selectSubject(
/**
* Lazy shared runner — `create()` is a pure constructor (zero I/O); the first frame that
* requests a task triggers that task's model download at inference time (never at init).
- * Recreated only when an option that changes inference changes.
+ * Nodes with the same model options share sessions; decode options (classes,
+ * confidence, …) get a derived runner over the same sessions.
*/
function getSharedRunner(op: VisionOperation): VisionRunner {
- const options = {
+ const modelOptions = {
variant: op.variant,
+ modelsDir: op.modelsDir,
+ baseUrl: op.baseUrl,
+ };
+ const modelKey = JSON.stringify(modelOptions);
+ let entry = sharedRunners.get(modelKey);
+ if (!entry) {
+ if (sharedRunners.size >= MAX_SHARED_RUNNERS) {
+ const [oldestKey, oldest] = sharedRunners.entries().next().value as [
+ string,
+ { runner: VisionRunner },
+ ];
+ oldest.runner.close();
+ sharedRunners.delete(oldestKey);
+ }
+ entry = { runner: VisionRunner.create(modelOptions), derived: new Map() };
+ sharedRunners.set(modelKey, entry);
+ }
+ const decodeOptions = {
confidence: op.confidence,
classes: op.classes,
maskThreshold: op.maskThreshold,
featherRadius: op.featherRadius,
- modelsDir: op.modelsDir,
- baseUrl: op.baseUrl,
};
- const key = JSON.stringify(options);
- if (sharedRunner?.key !== key) {
- sharedRunner?.runner.close();
- sharedRunner = { key, runner: VisionRunner.create(options) };
+ const decodeKey = JSON.stringify(decodeOptions);
+ let runner = entry.derived.get(decodeKey);
+ if (!runner) {
+ runner = entry.runner.derive(decodeOptions);
+ entry.derived.set(decodeKey, runner);
}
- return sharedRunner.runner;
+ return runner;
}
function getObjectTracker(
diff --git a/packages/framefields/package.json b/packages/framefields/package.json
index 76a02bd4..5af0202e 100644
--- a/packages/framefields/package.json
+++ b/packages/framefields/package.json
@@ -108,6 +108,9 @@
"yoga-layout": "^3.2.1",
"zod": "^4.3.6"
},
+ "optionalDependencies": {
+ "onnxruntime-web": "^1.24.0"
+ },
"devDependencies": {
"@framefields/client-utils": "workspace:*",
"@framefields/compositions": "workspace:*",
diff --git a/packages/vision/src/analysis/analyze-sequence.test.ts b/packages/vision/src/analysis/analyze-sequence.test.ts
index f8c1f6c3..6e069767 100644
--- a/packages/vision/src/analysis/analyze-sequence.test.ts
+++ b/packages/vision/src/analysis/analyze-sequence.test.ts
@@ -36,7 +36,8 @@ class MovingProvider implements SessionProvider {
async function* frames(count: number): AsyncIterable<VisionImageInput> {
for (let i = 0; i < count; i++) {
yield {
- data: new Uint8ClampedArray(640 * 640 * 4).fill(128),
+ // A distinct picture per frame: equal pixels share one model pass.
+ data: new Uint8ClampedArray(640 * 640 * 4).fill(100 + i),
width: 640,
height: 640,
};
diff --git a/packages/vision/src/runner/vision-runner.test.ts b/packages/vision/src/runner/vision-runner.test.ts
index 798655ca..d158c1ee 100644
--- a/packages/vision/src/runner/vision-runner.test.ts
+++ b/packages/vision/src/runner/vision-runner.test.ts
@@ -53,11 +53,40 @@ describe("VisionRunner (lazy + decode integration)", () => {
expect(m.mask[180 * 640 + 230]).toBe(0); // inside the box, outside the blob
expect(m.coverage).toBeGreaterThan(0);
- // A new frame buffer runs the model again.
+ // The same pixels in another buffer (another layer) reuse the pass…
await runner.segment(grayImage());
+ expect(provider.created[0].inputs).toHaveLength(1);
+
+ // …a different frame runs the model again.
+ const next = grayImage();
+ next.data[0] = 0;
+ await runner.segment(next);
expect(provider.created[0].inputs).toHaveLength(2);
});
+ it("derived runners share sessions and passes with their base", async () => {
+ const { store } = fakeStore();
+ const provider = new ScriptedProvider([rtmdetInsOutput()]);
+ const base = VisionRunner.create({ store, provider });
+ const people = base.derive({ classes: ["person"], confidence: 0.5 });
+ expect(people.classes).toEqual(["person"]);
+ expect(people.confidence).toBe(0.5);
+ expect(people.variant).toBe(base.variant);
+
+ const frame = grayImage();
+ await base.detect(frame);
+ await people.segment(frame);
+ expect(provider.created).toHaveLength(1);
+ expect(provider.created[0].inputs).toHaveLength(1);
+
+ // Closing a derived runner leaves the shared session alone.
+ people.close();
+ const next = grayImage();
+ next.data[0] = 0;
+ await base.detect(next);
+ expect(provider.created).toHaveLength(1);
+ });
+
it("maps boxes and masks back through a top-left letterbox on wide frames", async () => {
const { store } = fakeStore();
const runner = VisionRunner.create({
diff --git a/packages/vision/src/runner/vision-runner.ts b/packages/vision/src/runner/vision-runner.ts
index 82c5dd5e..06167b15 100644
--- a/packages/vision/src/runner/vision-runner.ts
+++ b/packages/vision/src/runner/vision-runner.ts
@@ -46,7 +46,7 @@ export interface VisionRunnerOptions {
/** Mirror origin for model downloads (see VisionModelStore). */
readonly baseUrl?: string;
readonly timeoutMs?: number;
- /** Session backend. Default: onnxruntime-node (CPU). Browser: `createWebGPUProvider()`. */
+ /** Session backend. Default: WebGPU when possible, else onnxruntime-node (CPU); see `AutoSessionProvider`. Browser: `createWebGPUProvider()`. */
readonly provider?: SessionProvider;
/** Inject a model store (tests, shared caches). Overrides modelsDir/baseUrl/timeoutMs. */
readonly store?: VisionModelStore;
@@ -57,6 +57,35 @@ export interface VisionRunnerOptions {
) => void;
}
+/** Options that only change how outputs are decoded — not which model runs. */
+export type VisionDecodeOptions = Pick<
+ VisionRunnerOptions,
+ "confidence" | "classes" | "maskThreshold" | "featherRadius"
+>;
+
+/**
+ * Model-side state: sessions, buffers and the per-frame RTMDet-Ins pass. Shared
+ * by a runner and every runner `derive()`d from it.
+ */
+interface RunnerCore {
+ readonly store: VisionModelStore;
+ readonly provider: SessionProvider;
+ readonly sessions: Map<VisionModelKey, Promise<VisionSession>>;
+ readonly tensorScratch: Map<VisionModelKey, Float32Array>;
+ /** Tail of each model's run queue: one inference per session at a time. */
+ readonly runQueue: Map<VisionModelKey, Promise<unknown>>;
+ /**
+ * Recent RTMDet-Ins passes, newest last, shared by detect() and segment()
+ * and matched by content — layers showing the same pixels share one pass.
+ */
+ readonly instancePasses: {
+ readonly content: string;
+ readonly result: Promise<InstancePass>;
+ }[];
+}
+
+const MAX_INSTANCE_PASSES = 4;
+
/**
* Lazy vision runner.
*
@@ -72,34 +101,32 @@ export class VisionRunner {
public readonly maskThreshold: number;
public readonly featherRadius?: number;
- private readonly _store: VisionModelStore;
- private readonly _provider: SessionProvider;
- private readonly _sessions = new Map<
- VisionModelKey,
- Promise<VisionSession>
- >();
- private readonly _tensorScratch = new Map<VisionModelKey, Float32Array>();
- /** One RTMDet-Ins pass per frame buffer, shared by detect() and segment(). */
- private _instancePass?: {
- readonly frame: VisionImageInput["data"];
- readonly result: Promise<InstancePass>;
- };
+ private readonly _core: RunnerCore;
+ /** False for derived runners: their sessions belong to the runner they came from. */
+ private readonly _ownsCore: boolean;
- private constructor(options: VisionRunnerOptions) {
+ private constructor(options: VisionRunnerOptions, core?: RunnerCore) {
this.variant = options.variant ?? "s";
this.confidence = options.confidence ?? 0.3;
this.classes = options.classes;
this.maskThreshold = options.maskThreshold ?? 0.5;
this.featherRadius = options.featherRadius;
- this._store =
- options.store ??
- new VisionModelStore({
- modelsDir: options.modelsDir,
- baseUrl: options.baseUrl,
- timeoutMs: options.timeoutMs,
- onProgress: options.onProgress,
- });
- this._provider = options.provider ?? createDefaultSessionProvider();
+ this._ownsCore = core === undefined;
+ this._core = core ?? {
+ store:
+ options.store ??
+ new VisionModelStore({
+ modelsDir: options.modelsDir,
+ baseUrl: options.baseUrl,
+ timeoutMs: options.timeoutMs,
+ onProgress: options.onProgress,
+ }),
+ provider: options.provider ?? createDefaultSessionProvider(),
+ sessions: new Map(),
+ tensorScratch: new Map(),
+ runQueue: new Map(),
+ instancePasses: [],
+ };
}
/** Cheap: no downloads, no sessions, no side effects. */
@@ -107,8 +134,30 @@ export class VisionRunner {
return new VisionRunner(options);
}
+ /**
+ * A runner with other decode options (confidence, classes, mask threshold,
+ * feather) that shares this runner's models, sessions and per-frame RTMDet-Ins
+ * pass — e.g. one per layer, without loading a model twice. Its `close()` is a
+ * no-op; closing this runner releases the shared sessions.
+ */
+ public derive(options: VisionDecodeOptions): VisionRunner {
+ return new VisionRunner(
+ {
+ variant: this.variant,
+ confidence: options.confidence ?? this.confidence,
+ classes: "classes" in options ? options.classes : this.classes,
+ maskThreshold: options.maskThreshold ?? this.maskThreshold,
+ featherRadius:
+ "featherRadius" in options
+ ? options.featherRadius
+ : this.featherRadius,
+ },
+ this._core,
+ );
+ }
+
public get modelsDir(): string {
- return this._store.modelsDir;
+ return this._core.store.modelsDir;
}
/** Every registry model keyed to its current download state (downloaded → "ready"). */
@@ -120,7 +169,7 @@ export class VisionRunner {
for (const key of Object.keys(VISION_MODELS) as VisionModelKey[]) {
statuses.set(key, "pending");
}
- for (const [key, status] of this._store.statuses()) {
+ for (const [key, status] of this._core.store.statuses()) {
statuses.set(key, status);
}
return statuses;
@@ -147,7 +196,7 @@ export class VisionRunner {
keys.add(this.modelKeyFor(task));
}
}
- await this._store.preload([...keys]);
+ await this._core.store.preload([...keys]);
await Promise.all([...keys].map((key) => this.session(key)));
}
@@ -210,41 +259,46 @@ export class VisionRunner {
}
public close(): void {
- for (const promise of this._sessions.values()) {
+ if (!this._ownsCore) return;
+ const core = this._core;
+ for (const promise of core.sessions.values()) {
void promise.then((session) => session.release()).catch(() => undefined);
}
- this._sessions.clear();
- this._instancePass = undefined;
- this._store.clearMemoryCache();
+ core.sessions.clear();
+ core.runQueue.clear();
+ core.instancePasses.length = 0;
+ core.store.clearMemoryCache();
}
// ── internals ──────────────────────────────────────────────────────────
private instancePass(image: VisionImageInput): Promise<InstancePass> {
- if (this._instancePass?.frame === image.data) {
- return this._instancePass.result;
- }
+ const passes = this._core.instancePasses;
+ const content = contentKey(image);
+ const byContent = passes.find((pass) => pass.content === content);
+ if (byContent) return byContent.result;
const key = this.modelKeyFor("segment");
- const result = this.infer(key, image).then(
- ({ outputs, transform }) => {
- const dets = output(outputs, "dets", key);
- const masks = output(outputs, "masks", key);
- return {
- transform,
- outputs: {
- dets: dets.data as Float32Array,
- labels: output(outputs, "labels", key).data,
- masks: masks.data as Float32Array,
- count: dets.dims[1] ?? 0,
- maskHeight: masks.dims[masks.dims.length - 2] ?? 0,
- maskWidth: masks.dims[masks.dims.length - 1] ?? 0,
- },
- };
- },
- );
- this._instancePass = { frame: image.data, result };
+ const result = this.infer(key, image).then(({ outputs, transform }) => {
+ const dets = output(outputs, "dets", key);
+ const masks = output(outputs, "masks", key);
+ return {
+ transform,
+ outputs: {
+ dets: dets.data as Float32Array,
+ labels: output(outputs, "labels", key).data,
+ masks: masks.data as Float32Array,
+ count: dets.dims[1] ?? 0,
+ maskHeight: masks.dims[masks.dims.length - 2] ?? 0,
+ maskWidth: masks.dims[masks.dims.length - 1] ?? 0,
+ },
+ };
+ });
+ const pass = { content, result };
+ passes.push(pass);
+ if (passes.length > MAX_INSTANCE_PASSES) passes.shift();
result.catch(() => {
- if (this._instancePass?.result === result) this._instancePass = undefined;
+ const index = passes.indexOf(pass);
+ if (index >= 0) passes.splice(index, 1);
});
return result;
}
@@ -267,7 +321,29 @@ export class VisionRunner {
};
}
- private async infer(
+ /**
+ * Runs one model on one image. Runs of the same model are queued: they share
+ * the input scratch buffer, and one run already uses the session's threads.
+ */
+ private infer(
+ key: VisionModelKey,
+ image: VisionImageInput,
+ ): Promise<{ outputs: VisionTensorOutputs; transform: InputTransform }> {
+ const queue = this._core.runQueue;
+ const previous = queue.get(key) ?? Promise.resolve();
+ const run = previous.then(
+ () => this.inferNow(key, image),
+ () => this.inferNow(key, image),
+ );
+ const tail = run.catch(() => undefined);
+ queue.set(key, tail);
+ void tail.then(() => {
+ if (queue.get(key) === tail) queue.delete(key);
+ });
+ return run;
+ }
+
+ private async inferNow(
key: VisionModelKey,
image: VisionImageInput,
): Promise<{ outputs: VisionTensorOutputs; transform: InputTransform }> {
@@ -275,10 +351,11 @@ export class VisionRunner {
const desc = VISION_MODELS[key];
const spec = preprocessSpec(desc.family, desc.input);
const size = 3 * spec.width * spec.height;
- let scratch = this._tensorScratch.get(key);
+ const scratchByKey = this._core.tensorScratch;
+ let scratch = scratchByKey.get(key);
if (!scratch || scratch.length < size) {
scratch = new Float32Array(size);
- this._tensorScratch.set(key, scratch);
+ scratchByKey.set(key, scratch);
}
const { tensor, transform } = imageToTensor(image, spec, scratch);
const outputs = await session.run({
@@ -290,20 +367,21 @@ export class VisionRunner {
}
private session(key: VisionModelKey): Promise<VisionSession> {
- let promise = this._sessions.get(key);
+ const { sessions, store, provider } = this._core;
+ let promise = sessions.get(key);
if (!promise) {
- promise = this._store
+ promise = store
.ensure(key)
- .then((bytes) => this._provider.createSession(bytes))
+ .then((bytes) => provider.createSession(bytes))
.catch((err: unknown) => {
// Drop both the session cache entry AND the cached file: a parse
// failure means the on-disk bytes are corrupt — the next call must
// re-download instead of replaying the same bad file.
- this._sessions.delete(key);
- void this._store.evict(key);
+ sessions.delete(key);
+ void store.evict(key);
throw err;
});
- this._sessions.set(key, promise);
+ sessions.set(key, promise);
}
return promise;
}
@@ -321,6 +399,24 @@ interface InstancePass {
};
}
+/** Size plus a 64-bit hash of the pixels: equal keys mean the same picture. */
+function contentKey(image: VisionImageInput): string {
+ const { data } = image;
+ const words = data.byteOffset % 4 === 0 ? data.byteLength >>> 2 : 0;
+ const view = new Uint32Array(data.buffer, data.byteOffset, words);
+ let h1 = 0x811c9dc5;
+ let h2 = 0x01000193 ^ data.byteLength;
+ for (let i = 0; i < words; i++) {
+ const v = view[i] as number;
+ h1 = Math.imul(h1 ^ v, 0x9e3779b1);
+ h2 = Math.imul(h2 ^ v, 0x85ebca77) ^ (h1 >>> 15);
+ }
+ for (let i = words * 4; i < data.length; i++) {
+ h1 = Math.imul(h1 ^ (data[i] as number), 0x9e3779b1);
+ }
+ return `${image.width}x${image.height}:${(h1 >>> 0).toString(36)}:${(h2 >>> 0).toString(36)}`;
+}
+
function output(
outputs: VisionTensorOutputs,
name: string,
diff --git a/packages/vision/src/runtime/node-webgpu-provider.ts b/packages/vision/src/runtime/node-webgpu-provider.ts
index 6dead1ea..2ebe9900 100644
--- a/packages/vision/src/runtime/node-webgpu-provider.ts
+++ b/packages/vision/src/runtime/node-webgpu-provider.ts
@@ -22,29 +22,74 @@ const WEBGPU_SPECIFIER = "webgpu";
let nodeWebGPUReady: Promise<void> | undefined;
+interface NodeGPUAdapter {
+ readonly features: object;
+ readonly info?: {
+ readonly vendor?: string;
+ readonly architecture?: string;
+ readonly device?: string;
+ readonly description?: string;
+ readonly isFallbackAdapter?: boolean;
+ };
+ readonly isFallbackAdapter?: boolean;
+}
+
+interface NodeGPU {
+ requestAdapter(options?: {
+ powerPreference?: "low-power" | "high-performance";
+ }): Promise<NodeGPUAdapter | null>;
+}
+
+/** CPU rasterizers (lavapipe, llvmpipe, SwiftShader): slower than the CPU EP. */
+const SOFTWARE_ADAPTER = /llvmpipe|lavapipe|swiftshader|software/i;
+
/**
- * Ensure `globalThis.navigator.gpu` exists. In the renderer this is already the
- * Dawn device; otherwise imports the `webgpu` (Dawn) package and installs its
- * globals plus an adapter, which is what onnxruntime-web's WebGPU EP needs.
+ * Ensure `globalThis.navigator.gpu` exists and has a hardware adapter that
+ * onnxruntime-web's WebGPU EP can use. In the renderer this is already the Dawn
+ * device; otherwise imports the `webgpu` (Dawn) package and installs its
+ * globals. Rejects when the only adapter is a software rasterizer.
*/
export function ensureNodeWebGPU(): Promise<void> {
if (!nodeWebGPUReady) {
nodeWebGPUReady = (async () => {
- const g = globalThis as { navigator?: { gpu?: unknown } };
- if (g.navigator?.gpu) return;
- const { create, globals } = (await import(
- /* @vite-ignore */ WEBGPU_SPECIFIER
- )) as {
- create: (flags: string[]) => unknown;
- globals: Record<string, unknown>;
- };
- Object.assign(globalThis, globals);
- const nav = (g.navigator ??= {});
- Object.defineProperty(nav, "gpu", {
- value: create([]),
- configurable: true,
- writable: true,
+ const g = globalThis as { navigator?: { gpu?: NodeGPU } };
+ if (!g.navigator?.gpu) {
+ const { create, globals } = (await import(
+ /* @vite-ignore */ WEBGPU_SPECIFIER
+ )) as {
+ create: (flags: string[]) => NodeGPU;
+ globals: Record<string, unknown>;
+ };
+ Object.assign(globalThis, globals);
+ g.navigator ??= {};
+ Object.defineProperty(g.navigator, "gpu", {
+ value: create([]),
+ configurable: true,
+ writable: true,
+ });
+ }
+ const gpu = g.navigator?.gpu as NodeGPU;
+ const adapter = await gpu.requestAdapter({
+ powerPreference: "high-performance",
});
+ if (!adapter) throw new Error("No WebGPU adapter.");
+ const info = adapter.info ?? {};
+ const name = [
+ info.vendor,
+ info.architecture,
+ info.device,
+ info.description,
+ ]
+ .filter(Boolean)
+ .join(" ");
+ if (
+ adapter.isFallbackAdapter === true ||
+ info.isFallbackAdapter === true ||
+ SOFTWARE_ADAPTER.test(name)
+ ) {
+ throw new Error(`WebGPU adapter is a software rasterizer (${name}).`);
+ }
+ patchSupportedFeatures(adapter.features);
})();
nodeWebGPUReady.catch(() => {
nodeWebGPUReady = undefined;
@@ -53,6 +98,54 @@ export function ensureNodeWebGPU(): Promise<void> {
return nodeWebGPUReady;
}
+/**
+ * Dawn's Node `GPUSupportedFeatures` has `has`/`keys`/`size` but no `forEach`,
+ * which onnxruntime-web's WebGPU build calls on adapter and device features
+ * (they share the class). Adds the missing set-like methods once.
+ */
+function patchSupportedFeatures(features: object): void {
+ const proto = Object.getPrototypeOf(features) as {
+ keys?: () => IterableIterator<string>;
+ forEach?: unknown;
+ values?: unknown;
+ entries?: unknown;
+ [Symbol.iterator]?: unknown;
+ } | null;
+ if (!proto || typeof proto.keys !== "function") return;
+ type SetLike = { keys(): IterableIterator<string> };
+ if (typeof proto.forEach !== "function") {
+ Object.defineProperty(proto, "forEach", {
+ value(
+ this: SetLike,
+ callback: (value: string, key: string, set: SetLike) => void,
+ thisArg?: unknown,
+ ) {
+ for (const key of this.keys()) callback.call(thisArg, key, key, this);
+ },
+ configurable: true,
+ writable: true,
+ });
+ }
+ if (typeof proto.values !== "function") {
+ Object.defineProperty(proto, "values", {
+ value(this: SetLike) {
+ return this.keys();
+ },
+ configurable: true,
+ writable: true,
+ });
+ }
+ if (typeof proto.entries !== "function") {
+ Object.defineProperty(proto, "entries", {
+ *value(this: SetLike) {
+ for (const key of this.keys()) yield [key, key];
+ },
+ configurable: true,
+ writable: true,
+ });
+ }
+}
+
/** Options mirror the browser provider; the loader is overridden for Node. */
export type NodeWebGPUProviderOptions = WebGPUProviderOptions;
@@ -69,11 +162,16 @@ export class NodeWebGPUProvider implements SessionProvider {
const mod = (await import(
/* @vite-ignore */ ORT_WEBGPU_SPECIFIER
)) as OrtWebModule & {
- env?: { wasm?: { numThreads?: number } };
+ env?: {
+ wasm?: { numThreads?: number };
+ webgpu?: { powerPreference?: string };
+ };
};
// Single-threaded WASM: Node has no cross-origin isolation, and the
// WASM path is only a fallback behind the WebGPU EP.
if (mod.env?.wasm) mod.env.wasm.numThreads = 1;
+ if (mod.env?.webgpu)
+ mod.env.webgpu.powerPreference = "high-performance";
return mod;
});
// No WASM fallback inside the WebGPU provider: when WebGPU isn't possible
diff --git a/packages/vision/src/runtime/session-provider.test.ts b/packages/vision/src/runtime/session-provider.test.ts
new file mode 100644
index 00000000..ebfa282c
--- /dev/null
+++ b/packages/vision/src/runtime/session-provider.test.ts
@@ -0,0 +1,42 @@
+import { afterEach, describe, expect, it, vi } from "vitest";
+import {
+ availableCpus,
+ cpuSessionThreads,
+ createDefaultSessionProvider,
+ requestedVisionBackend,
+ setDefaultSessionProvider,
+} from "./session-provider.js";
+
+afterEach(() => {
+ vi.unstubAllEnvs();
+ setDefaultSessionProvider(undefined);
+});
+
+describe("session provider selection", () => {
+ it("reads FRAMEFIELDS_VISION_PROVIDER, defaulting to auto", () => {
+ expect(requestedVisionBackend()).toBe("auto");
+ vi.stubEnv("FRAMEFIELDS_VISION_PROVIDER", " CUDA ");
+ expect(requestedVisionBackend()).toBe("cuda");
+ vi.stubEnv("FRAMEFIELDS_VISION_PROVIDER", "tpu");
+ expect(requestedVisionBackend()).toBe("auto");
+ });
+
+ it("shares one default provider per process", () => {
+ expect(createDefaultSessionProvider()).toBe(createDefaultSessionProvider());
+ });
+});
+
+describe("CPU session threads", () => {
+ it("honours FRAMEFIELDS_VISION_THREADS", async () => {
+ vi.stubEnv("FRAMEFIELDS_VISION_THREADS", "3");
+ expect(await cpuSessionThreads()).toBe(3);
+ });
+
+ it("never exceeds the CPUs the process may use", async () => {
+ const cpus = await availableCpus();
+ expect(cpus).toBeGreaterThanOrEqual(1);
+ const threads = await cpuSessionThreads();
+ if (process.platform === "linux") expect(threads).toBe(cpus);
+ else expect(threads).toBeUndefined();
+ });
+});
diff --git a/packages/vision/src/runtime/session-provider.ts b/packages/vision/src/runtime/session-provider.ts
index fcf0d42b..1232bb42 100644
--- a/packages/vision/src/runtime/session-provider.ts
+++ b/packages/vision/src/runtime/session-provider.ts
@@ -43,47 +43,167 @@ export function setDefaultSessionProvider(
defaultProvider = factory;
}
+/** Inference backend: `FRAMEFIELDS_VISION_PROVIDER`, default "auto". */
+export type VisionBackend = "auto" | "webgpu" | "cuda" | "cpu";
+
+export function requestedVisionBackend(): VisionBackend {
+ const value =
+ typeof process !== "undefined"
+ ? process.env.FRAMEFIELDS_VISION_PROVIDER?.trim().toLowerCase()
+ : undefined;
+ return value === "webgpu" || value === "cuda" || value === "cpu"
+ ? value
+ : "auto";
+}
+
/**
* Prefers the GPU (onnxruntime-web's WebGPU EP on the process's WebGPU device —
* the Dawn device the renderer already creates, or one this ensures) and falls
* back to the CPU provider (onnxruntime-node) when a WebGPU session cannot be
- * created: no device, no Dawn, or the EP unavailable. The choice is made on the
- * first session and reused for the runner's lifetime.
+ * created: no hardware adapter, no Dawn, or onnxruntime-web not installed. The
+ * choice is made on the first session and kept.
+ *
+ * `FRAMEFIELDS_VISION_PROVIDER` forces one backend instead: `webgpu`, `cpu`, or
+ * `cuda` (onnxruntime-node's CUDA EP on Linux x64; needs the CUDA 12 and cuDNN 9
+ * libraries). A forced backend that fails throws instead of falling back.
*/
export class AutoSessionProvider implements SessionProvider {
private _resolved?: SessionProvider;
+ private _resolving?: Promise<{
+ provider: SessionProvider;
+ first?: VisionSession;
+ }>;
public get kind(): "node" | "webgpu" {
return this._resolved?.kind ?? "webgpu";
}
public async createSession(modelBytes: Uint8Array): Promise<VisionSession> {
- if (!this._resolved) {
- const webgpu = new NodeWebGPUProvider({ wasmFallback: false });
- try {
- const session = await webgpu.createSession(modelBytes);
- this._resolved = webgpu;
- return session;
- } catch {
- // WebGPU isn't possible here — use CPU from now on.
- this._resolved = new NodeSessionProvider();
- }
+ if (this._resolved) return this._resolved.createSession(modelBytes);
+ if (!this._resolving) {
+ this._resolving = this.resolve(modelBytes);
+ this._resolving.catch(() => {
+ this._resolving = undefined;
+ });
+ const { provider, first } = await this._resolving;
+ this._resolved = provider;
+ if (first) return first;
+ return provider.createSession(modelBytes);
+ }
+ const { provider } = await this._resolving;
+ return provider.createSession(modelBytes);
+ }
+
+ private async resolve(
+ modelBytes: Uint8Array,
+ ): Promise<{ provider: SessionProvider; first?: VisionSession }> {
+ const backend = requestedVisionBackend();
+ if (backend === "cpu") return { provider: new NodeSessionProvider() };
+ if (backend === "cuda") {
+ return {
+ provider: new NodeSessionProvider({
+ executionProviders: ["cuda", "cpu"],
+ }),
+ };
+ }
+ const webgpu = new NodeWebGPUProvider({ wasmFallback: false });
+ try {
+ return {
+ provider: webgpu,
+ first: await webgpu.createSession(modelBytes),
+ };
+ } catch (err) {
+ if (backend === "webgpu") throw err;
+ // WebGPU isn't possible here — use CPU from now on.
+ return { provider: new NodeSessionProvider() };
}
- return this._resolved.createSession(modelBytes);
}
}
+let processProvider: AutoSessionProvider | undefined;
+
/**
* The provider for a runner created without one: GPU if possible, otherwise CPU.
+ * One per process, so the GPU probe runs once however many runners there are.
*/
export function createDefaultSessionProvider(): SessionProvider {
- return defaultProvider?.() ?? new AutoSessionProvider();
+ if (defaultProvider) return defaultProvider();
+ processProvider ??= new AutoSessionProvider();
+ return processProvider;
+}
+
+/**
+ * CPUs this process may use: the affinity mask, capped by a cgroup CPU quota
+ * (containers). ONNX Runtime sizes its thread pool from the host's cores, which
+ * oversubscribes a container limited to fewer CPUs than the host has.
+ */
+export async function availableCpus(): Promise<number> {
+ const os = await import("node:os");
+ const cpus =
+ typeof os.availableParallelism === "function"
+ ? os.availableParallelism()
+ : os.cpus().length;
+ const quota = await cgroupCpuQuota();
+ return Math.max(1, quota !== undefined ? Math.min(cpus, quota) : cpus);
+}
+
+/** cgroup v2 `cpu.max` or v1 `cpu.cfs_quota_us`, in CPUs (rounded up); undefined when unlimited. */
+async function cgroupCpuQuota(): Promise<number | undefined> {
+ if (typeof process === "undefined" || process.platform !== "linux") {
+ return undefined;
+ }
+ const { readFile } = await import("node:fs/promises");
+ const read = (path: string) =>
+ readFile(path, "utf8").then(
+ (text) => text.trim(),
+ () => undefined,
+ );
+ const toCpus = (quota: number, period: number) =>
+ quota > 0 && period > 0 ? Math.ceil(quota / period) : undefined;
+
+ const v2 = await read("/sys/fs/cgroup/cpu.max");
+ if (v2 !== undefined) {
+ const [quota, period] = v2.split(/\s+/);
+ return quota === "max" ? undefined : toCpus(Number(quota), Number(period));
+ }
+ const v1Quota = await read("/sys/fs/cgroup/cpu/cpu.cfs_quota_us");
+ const v1Period = await read("/sys/fs/cgroup/cpu/cpu.cfs_period_us");
+ if (v1Quota === undefined || v1Period === undefined) return undefined;
+ return toCpus(Number(v1Quota), Number(v1Period));
+}
+
+/**
+ * Intra-op threads for CPU sessions: `FRAMEFIELDS_VISION_THREADS` when set,
+ * else the available CPUs on Linux (where containers cap them), else ONNX
+ * Runtime's own default (undefined).
+ */
+export async function cpuSessionThreads(): Promise<number | undefined> {
+ const env =
+ typeof process !== "undefined"
+ ? Number.parseInt(process.env.FRAMEFIELDS_VISION_THREADS ?? "", 10)
+ : Number.NaN;
+ if (Number.isFinite(env) && env > 0) return env;
+ if (typeof process === "undefined" || process.platform !== "linux") {
+ return undefined;
+ }
+ return availableCpus();
+}
+
+export interface NodeSessionProviderOptions {
+ /** onnxruntime-node execution providers in priority order. Default `["cpu"]`. */
+ readonly executionProviders?: readonly string[];
}
/** onnxruntime-node provider. The native module is imported lazily on first session. */
export class NodeSessionProvider implements SessionProvider {
public readonly kind = "node" as const;
+ private readonly _executionProviders: readonly string[];
+
+ constructor(options: NodeSessionProviderOptions = {}) {
+ this._executionProviders = options.executionProviders ?? ["cpu"];
+ }
+
private ortPromise?: Promise<typeof import("onnxruntime-node")>;
private async ort(): Promise<typeof import("onnxruntime-node")> {
@@ -101,9 +221,13 @@ export class NodeSessionProvider implements SessionProvider {
public async createSession(modelBytes: Uint8Array): Promise<VisionSession> {
const ort = await this.ort();
+ const threads = await cpuSessionThreads();
const session = await ort.InferenceSession.create(modelBytes, {
- executionProviders: ["cpu"],
+ executionProviders: [...this._executionProviders],
graphOptimizationLevel: "all",
+ ...(threads !== undefined
+ ? { intraOpNumThreads: threads, interOpNumThreads: 1 }
+ : {}),
});
type OrtTensor = InstanceType<typeof ort.Tensor>;
diff --git a/plugins/framefields/skills/framefields-effects/SKILL.md b/plugins/framefields/skills/framefields-effects/SKILL.md
index 52792de5..500a85a7 100644
--- a/plugins/framefields/skills/framefields-effects/SKILL.md
+++ b/plugins/framefields/skills/framefields-effects/SKILL.md
@@ -160,6 +160,15 @@ Measured per 1280–2048 px frame on CPU (`onnxruntime-node`):
- Raise `confidence` (e.g. `0.5`) on abstract or stylized footage — at the default `0.3` the model will put loose labels ("teddy bear", "donut") on smoke, eyes and planets.
- Restrict `classes` when you only care about one thing; it also stops the subject from switching to another object when the person leaves frame.
+### GPU or CPU
+
+Inference runs on the GPU through `onnxruntime-web`'s WebGPU backend (an optional dependency of `framefields`, installed by default) on a hardware adapter, and on the CPU through `onnxruntime-node` otherwise: when `onnxruntime-web` is missing, the only adapter is a software one (lavapipe, SwiftShader), or WebGPU fails. On an Apple Silicon Mac the GPU roughly halves render time against the CPU. Layers that use the same `variant` share their models, and layers showing the same pixels share one detect/segment pass.
+
+| Variable | Effect |
+| --- | --- |
+| `FRAMEFIELDS_VISION_PROVIDER` | `auto` (default), `webgpu`, `cpu`, or `cuda` (onnxruntime-node's CUDA EP on Linux x64; needs the CUDA 12 and cuDNN 9 libraries). A forced backend that fails throws instead of falling back. |
+| `FRAMEFIELDS_VISION_THREADS` | CPU threads per model. Default on Linux: the CPUs the process may use, cgroup quota included; elsewhere ONNX Runtime's default. |
+
### How it behaves at render time
- **One-frame delay.** Vision reads the layer's *previous* rendered frame, so the very first frame has no results (a cutout renders transparent) and masks trail the plate by one frame. This is invisible at normal playback speed.
diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml
index 2ec4f4b3..f4f15e55 100644
--- a/pnpm-lock.yaml
+++ b/pnpm-lock.yaml
@@ -2514,6 +2514,10 @@ importers:
vitest:
specifier: ^4.1.11
+ optionalDependencies:
+ onnxruntime-web:
+ specifier: ^1.24.0
+ version: 1.30.0
packages/media:
dependencies: