Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Document WebGPU model lessons and add audit and long-prompt checks

Claude4ac89edc8c0d
diff --git a/AGENT.md b/AGENT.md
index 64c2f8f0c..9889292c1 100644
--- a/AGENT.md
+++ b/AGENT.md
@@ -61,6 +61,8 @@ vanilla JS/CSS unless there is a very strong reason to add a framework.
- `docs/site-adapters.md`: adapter shape, prompt style, and testing guidance.
- `docs/providers-and-models.md`: provider configuration and model capability
assumptions.
+- `docs/webgpu-models.md`: required checklist before adding or changing an
+ in-browser WebGPU model or its ONNX graphs.
- `docs/localization.md`: UI locale workflow and mirroring expectations.
- `docs/privacy-and-data-flow.md` and `docs/security-model.md`: data handling,
permissions, and user-risk model.
diff --git a/docs/providers-and-models.md b/docs/providers-and-models.md
index 1ce140ffd..1053290ae 100644
--- a/docs/providers-and-models.md
+++ b/docs/providers-and-models.md
@@ -453,6 +453,13 @@ approximately one megapixel per image. Build sources, tensor mapping checks,
and the hardware browser verification command are documented in
[`scripts/models/bonsai/README.md`](../scripts/models/bonsai/README.md).
+Before adding, replacing or re-exporting any in-browser model, follow
+[`docs/webgpu-models.md`](webgpu-models.md). It records why the first Bonsai 2
+graphs prefilled on the CPU and froze the desktop, and the required checks:
+`scripts/models/audit_webgpu_onnx.py` for CPU-only ops and full-sequence
+logits, node placement, long-prompt timing with
+`scripts/check-webgpu-long-prompts.mjs`, and how to update a published graph.
+
To verify the published text presets through the packaged production worker,
run these commands from the repository root with its Playwright Chromium
installed:
diff --git a/docs/webgpu-models.md b/docs/webgpu-models.md
new file mode 100644
index 000000000..726da9160
--- /dev/null
+++ b/docs/webgpu-models.md
@@ -0,0 +1,169 @@
+# In-browser WebGPU models: checklist and lessons
+
+Read this before adding, replacing or re-exporting a model for the Chromium
+**WebGPU (In-browser)** provider (`src/chrome/src/providers/webgpu.js`,
+`src/chrome/src/offscreen/inference-worker.js`). It records what made Ternary
+Bonsai 2 unusable in the browser and the checks that would have caught it.
+
+## What went wrong with Bonsai 2
+
+Asking Bonsai 2 2B Vision to "summarize this page" took many minutes and froze
+the desktop, on an RTX 5090 and on an M4 Max alike. Five independent causes
+stacked up:
+
+1. **Token-by-token prefill on the CPU.** The upstream language graph
+ (`danyalmalik/Ternary-Bonsai-2-*-ONNX-WebGPU`, the same pattern as
+ `onnx-community/Qwen3.5-*-ONNX`) computes every GatedDeltaNet prefill with an
+ ONNX `Scan` over tokens. ONNX Runtime Web's WebGPU execution provider has no
+ `Scan` (or `Loop`) kernel, so the node is placed on the CPU and its body runs
+ once per token, with a GPU round trip, in each of 18 layers (24 for 4B).
+ That costs about 6.5 ms per prompt token per layer: 0.13 s per prompt
+ token for 2B and 0.19 s for 4B on an RTX 5090. A faster GPU barely helps,
+ because the time goes to synchronisation, not compute.
+2. **Logits for every prompt token.** The graph's `logits` output was
+ `[batch, sequence_length, 248320]`, so prefill ran the LM head over the
+ whole prompt and downloaded about 0.5 MB of FP16 scores per prompt token.
+ Above roughly 4.3k tokens that output also exceeds the 2 GB maximum buffer
+ size many WebGPU adapters report.
+3. **The wrong GPU.** Chrome runs WebGPU on the adapter its GPU process uses.
+ On a Windows machine with an RTX 5090 and a small display card (T400), every
+ `powerPreference` returned the T400, which also drives the monitors. The model
+ saturated it and the whole desktop stalled while the 5090 sat idle.
+4. **One giant prefill pass.** Even with a fixed graph, a ~9k-token prompt in a
+ single decoder pass keeps a GPU that is shared with the display (Apple
+ silicon, a display-attached card) busy for seconds, and the desktop stutters.
+5. **Large prompts.** Ask mode sends the full Ask system prompt (~3k tokens)
+ plus 17 tool schemas (~6k tokens) to in-browser models on every call,
+ because the compact tier currently only changes Act mode.
+
+Two process gaps let this ship. The production harness
+(`scripts/check-bonsai-webgpu.mjs`) only uses prompts of a few dozen tokens,
+which hide prefill problems entirely. And graph files were replaced under
+an already-cached URL, which installed copies never re-fetch.
+
+## Checklist for a new or re-exported model
+
+Do all of these on the exact files you will publish.
+
+1. **Static graph audit.**
+
+ ```bash
+ python scripts/models/audit_webgpu_onnx.py path/to/decoder.onnx
+ ```
+
+ It exits non-zero for `Scan`/`Loop` anywhere in the graph (including `If`
+ branches) and for a `logits` output that keeps the sequence dimension.
+ `If` is fine; it has a WebGPU kernel. The original Bonsai 2 and
+ `onnx-community/Qwen3.5` decoders fail this check. Compass Tiny,
+ `Mike0021/Ternary-Bonsai-2-4B-text-ONNX` (which uses the
+ `com.microsoft::LinearAttention` and `CausalConvWithState` contrib ops) and
+ the rewritten Bonsai 2 Vision graphs pass.
+ - A `num_logits_to_keep` input is not enough on its own. On the
+ multimodal forward path Transformers.js passes `0` (keep all), so slice to
+ the last position inside the graph, as `chunk_gdn_prefill.py` does.
+2. **Node placement on real WebGPU.** Create the session in a browser with
+ `ort.env.logLevel = 'verbose'` and `logSeverityLevel: 0`, then read the
+ `Node placements` block. Small shape ops (`Shape`, `Gather`, `Equal` on
+ int64) belong on the CPU. Any compute op reported as
+ `webgpu kernel not found in registries` is a red flag.
+3. **Long prompts and prefill scaling.** The short harness prompts cannot show
+ prefill problems. Run real prompts of about 1k, 4k and 8k tokens, plus a
+ long prompt with an image for vision models, through the production
+ worker on hardware WebGPU:
+
+ ```bash
+ node scripts/check-webgpu-long-prompts.mjs --model <owner/repo> --vision \
+ --profile <cached profile> --offline --lengths 1000,4000,8000 --slices 0,512
+ ```
+
+ It prints wall time and seconds per 1k prompt tokens for each prompt, and
+ fails if the greedy answer changes between slice sizes. Seconds per 1k
+ tokens should stay roughly flat and around a second or less on a desktop
+ GPU. Tens of seconds per 1k tokens that scale linearly mean a CPU
+ fallback. Reference for rewritten Bonsai 2 2B on an RTX 5090 (48 new
+ tokens, including decode): 3k tokens in 2.5 s and 8k in 5.6 s in one pass,
+ 7.8 s in 512-token slices. In a decoder-only benchmark, 4B prefilled 8,192
+ tokens in 16 s. The original Scan graph took 128 s for 1,024 tokens.
+4. **Production harness.** Run `scripts/check-bonsai-webgpu.mjs` online and
+ `--offline` (with `--vision --image-history` for vision models) and record
+ the evidence with `scripts/models/bonsai/record_browser_validation.py`.
+5. **Memory.** Budget weights, plus KV and recurrent cache for the browser
+ context, plus transient prefill buffers. Keep every single tensor under
+ 2 GB, the per-buffer limit on many adapters.
+6. **Rollout.** Browsers keep files cached by URL. If a published file
+ changes in place, existing installs never see it unless the worker
+ refreshes it explicitly (see "Updating a published graph").
+
+## Fixing a Qwen3.5 / GatedDeltaNet export
+
+Use the tools in `scripts/models/bonsai/`:
+
+- `chunk_gdn_prefill.py` replaces every GatedDeltaNet `Scan` with the
+ chunked gated delta rule and slices the LM head to the last position.
+ Weights and the `.onnx_data` file stay byte-identical; only the small graph
+ file changes. Decode keeps the original single-token branch.
+ - 128-token chunks. Inside a chunk, the unit lower-triangular WY inverse uses
+ **recursive block inversion**. A Neumann-product inverse is shorter but
+ overflows FP32 when keys in a chunk are strongly correlated.
+ - Across chunks, a Hillis-Steele prefix scan over the affine state maps runs
+ with `If`-skipped steps. The step count is derived from
+ `text_config.max_position_embeddings`, so every accepted prompt length
+ stays correct. Chunks must be powers of two from 16 to 1024.
+- `validate_chunked_gdn.py` compares old and new graphs on CPU with real text.
+ It covers chunk-boundary lengths, a multi-chunk prompt, a cached continuation,
+ logits, every cache output (whole-tensor and worst-element error) and
+ finiteness. It reads the chunk size from the rewritten graph.
+- `make_chunked_bundle.py` derives a sealed bundle. It writes model-card
+ parity and timing claims only from evidence that passes the verifier (both
+ graph hashes, chunk coverage, top-1 and decode equality, FP16 tolerances).
+ It drops browser evidence recorded against the previous graph and keeps
+ native vision parity only while its encoder hashes still match.
+- `build_vision.py` applies the rewrite on full rebuilds.
+
+## Runtime behaviour to keep
+
+- **Sliced prefill.** For the Bonsai 2 Vision presets the worker prefills
+ long prompts in 512-token slices that carry the recurrent, conv and KV
+ cache, then lets `generate()` continue from that cache. Greedy answers are
+ identical to a single pass. Each extra slice adds roughly one decoder run of
+ fixed overhead (about 0.14 s on an RTX 5090). The `chrome.storage.local` key
+ `webgpuPrefillSliceTokens` overrides the slice (64–8192, `0` disables).
+ Apply the same pattern to any new model that serves long prompts on shared
+ GPUs.
+- **Cache-only inference.** Chat never downloads. Only explicit download or
+ preload flows fetch model files.
+
+## Updating a published graph
+
+Prefer a new repository or file name for a changed graph. If a file must
+change in place, as the Bonsai 2 Vision decoders did:
+
+- Pin the new file's SHA-256 in the worker (`BONSAI2_DECODER_GRAPH_SHA256`).
+- Inference only detects a stale cached graph and keeps using it.
+- An explicit download or preload replaces just that file, verified by hash,
+ with a 15 s deadline and no attempt while known offline. It then drops the
+ resident session built from the old graph.
+- Never block or break offline use to force an update.
+
+## Diagnosing a slow or stuttering user machine
+
+- In a normal tab, run
+ `(await navigator.gpu.requestAdapter({powerPreference: 'high-performance'})).info`
+ to see the adapter Chrome uses. On Windows, compare it with `nvidia-smi` or
+ Task Manager.
+- To move Chrome to another GPU on Windows, use Settings β†’ System β†’ Display β†’
+ Graphics β†’ Chrome, or start Chrome with `--use-adapter-luid=<high>,<low>`.
+ Use **decimal** values: the hex form gave no adapter in Chromium 147. Find a
+ LUID with PowerShell, matching adapters by dedicated memory use:
+ `(Get-Counter '\GPU Adapter Memory(*)\Dedicated Usage').CounterSamples`.
+- Stutter while the GPU is shared with the display is expected for long
+ single passes. Lower `webgpuPrefillSliceTokens` (for example to 256) and keep
+ prompts short.
+
+## Open items
+
+- The Bonsai 2 2B **text** preset still points at
+ `danyalmalik/Ternary-Bonsai-2-2B-ONNX-WebGPU`, which fails the audit. Publish
+ a rewritten text repository or point the preset at a passing graph.
+- Ask mode should use a compact prompt and a smaller tool set for in-browser
+ models.
diff --git a/scripts/check-webgpu-long-prompts.mjs b/scripts/check-webgpu-long-prompts.mjs
new file mode 100644
index 000000000..f17e220bd
--- /dev/null
+++ b/scripts/check-webgpu-long-prompts.mjs
@@ -0,0 +1,170 @@
+// Long-prompt check for in-browser WebGPU models through the production worker.
+//
+// The short prompts in check-bonsai-webgpu.mjs cannot reveal prefill problems
+// (a CPU-fallback recurrence or full-sequence logits only show up with
+// thousands of prompt tokens). This runs real text of several lengths (and,
+// with --vision, a long prompt that includes an image) through
+// src/chrome/src/offscreen/inference-worker.js, reports wall time per prompt
+// and seconds per 1k prompt tokens, and checks that greedy answers are the same
+// for every requested prefill slice size. See docs/webgpu-models.md.
+//
+// node scripts/check-webgpu-long-prompts.mjs --model webbrain-one/Ternary-Bonsai-2-2B-ONNX-WebGPU-Vision \
+// --vision --profile .build/bonsai/chunked-2b-profile --offline --lengths 1000,4000,8000 --slices 0,512
+//
+// Options mirror check-bonsai-webgpu.mjs: --local-repo owner/repo=dir serves
+// unpublished files, --offline requires a cached model and blocks Hugging Face,
+// --adapter-luid high,low (decimal) selects a Windows GPU.
+import assert from 'node:assert/strict';
+import fs from 'node:fs/promises';
+import { createReadStream } from 'node:fs';
+import http from 'node:http';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { chromium } from 'playwright';
+
+const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');
+const args = process.argv.slice(2);
+const option = (name, fallback) => { const i = args.indexOf(name); return i < 0 ? fallback : args[i + 1]; };
+const model = option('--model', 'webbrain-one/Ternary-Bonsai-2-2B-ONNX-WebGPU-Vision');
+const vision = args.includes('--vision');
+const offline = args.includes('--offline');
+const dtype = option('--dtype', null)
+ ? JSON.parse(option('--dtype'))
+ : vision ? { embed_tokens: 'q2f16', vision_encoder: 'fp16', decoder_model_merged: 'q2f16' } : 'q2f16';
+const lengths = option('--lengths', '1000,4000,8000').split(',').map(Number);
+const slices = option('--slices', '0,512').split(',').map(Number);
+const maxTokens = Number(option('--max-tokens', 48));
+const profile = path.resolve(option('--profile', '.build/webgpu-long-prompts/profile'));
+const output = path.resolve(option('--output', '.build/webgpu-long-prompts/result.json'));
+const localRepos = {};
+for (let i = 0; i < args.length; i++) {
+ if (args[i] !== '--local-repo') continue;
+ const entry = args[++i] || '';
+ const split = entry.indexOf('=');
+ assert(split > 0, '--local-repo requires owner/repo=directory');
+ localRepos[entry.slice(0, split)] = path.resolve(entry.slice(split + 1));
+}
+const mime = file => ({ '.mjs': 'text/javascript', '.js': 'text/javascript', '.json': 'application/json', '.wasm': 'application/wasm' }[path.extname(file)] || 'application/octet-stream');
+const workerBootstrap = `
+const nativeFetch = globalThis.fetch.bind(globalThis);
+const mapped = ${JSON.stringify(Object.keys(localRepos))};
+globalThis.fetch = (input, init) => {
+ const url = new URL(typeof input === 'string' ? input : input.url || String(input), self.location.href);
+ if (url.hostname === 'huggingface.co') {
+ if (${offline}) throw new Error('Offline check blocked a Hugging Face fetch: ' + url.pathname);
+ const match = url.pathname.match(/^\\/([^/]+\\/[^/]+)\\/resolve\\/[^/]+\\/(.+)$/);
+ if (match && mapped.includes(match[1])) return nativeFetch('/models/' + match[1] + '/' + match[2], init);
+ }
+ return nativeFetch(input, init);
+};
+await import('/extension/src/offscreen/inference-worker.js');
+self.postMessage({ type: 'harness-ready' });`;
+const server = http.createServer(async (request, response) => {
+ try {
+ const pathname = decodeURIComponent(new URL(request.url, 'http://localhost').pathname);
+ if (pathname === '/') { response.setHeader('Content-Type', 'text/html'); response.end('<!doctype html><title>WebGPU long prompts</title>'); return; }
+ if (pathname === '/worker.mjs') { response.setHeader('Content-Type', 'text/javascript'); response.end(workerBootstrap); return; }
+ let base, relative;
+ if (pathname.startsWith('/extension/')) { base = path.join(root, 'src/chrome'); relative = pathname.slice('/extension/'.length); }
+ else if (pathname.startsWith('/models/')) {
+ const match = pathname.match(/^\/models\/([^/]+\/[^/]+)\/(.+)$/);
+ base = match && localRepos[match[1]]; relative = match?.[2];
+ }
+ if (!base || !relative) { response.writeHead(404); response.end(); return; }
+ const file = path.resolve(base, relative);
+ if (!file.startsWith(path.resolve(base) + path.sep)) { response.writeHead(403); response.end(); return; }
+ const stat = await fs.stat(file);
+ response.setHeader('Content-Type', mime(file)); response.setHeader('Content-Length', stat.size);
+ createReadStream(file).on('error', () => response.destroy()).pipe(response);
+ } catch (error) {
+ response.writeHead(error.code === 'ENOENT' ? 404 : 500); response.end();
+ }
+});
+await fs.mkdir(path.dirname(output), { recursive: true });
+await new Promise((resolve, reject) => { server.once('error', reject); server.listen(Number(option('--port', 8769)), '127.0.0.1', resolve); });
+const context = await chromium.launchPersistentContext(profile, {
+ headless: true,
+ channel: 'chromium',
+ args: ['--enable-unsafe-webgpu', '--force-high-performance-gpu',
+ ...(option('--adapter-luid', '') ? [`--use-adapter-luid=${option('--adapter-luid')}`] : [])],
+});
+const result = { model, vision, dtype, startedAt: new Date().toISOString(), runs: [], mismatches: [] };
+try {
+ const page = await context.newPage();
+ page.on('console', message => { if (message.type() === 'error') console.log(`[browser:error] ${message.text().slice(0, 300)}`); });
+ await page.goto(`http://127.0.0.1:${server.address().port}/`);
+ await page.evaluate(() => new Promise((ready, failed) => {
+ const pending = new Map(); let id = 0;
+ const worker = new Worker('/worker.mjs', { type: 'module' });
+ worker.onmessage = ({ data }) => {
+ if (data.type === 'harness-ready') ready();
+ else if (data.id && pending.has(data.id)) {
+ const { resolve, reject } = pending.get(data.id); pending.delete(data.id);
+ if (data.ok === false) reject(new Error(data.error)); else resolve(data);
+ }
+ };
+ worker.onerror = event => failed(new Error(event.message));
+ window.workerCall = (type, payload) => new Promise((resolve, reject) => {
+ const requestId = ++id; pending.set(requestId, { resolve, reject }); worker.postMessage({ id: requestId, type, payload });
+ });
+ }));
+ const origin = new URL(page.url()).origin;
+ const call = (type, payload = {}) => page.evaluate(({ type, payload }) => window.workerCall(type, payload), { type, payload });
+ await call('init', {
+ transformersUrl: `${origin}/extension/vendor/transformers/transformers.web.js`,
+ wasmMjsUrl: `${origin}/extension/vendor/transformers/ort-wasm-simd-threaded.asyncify.mjs`,
+ wasmUrl: `${origin}/extension/vendor/transformers/ort-wasm-simd-threaded.asyncify.wasm`,
+ });
+ result.adapter = await page.evaluate(async () => {
+ const adapter = await navigator.gpu.requestAdapter({ powerPreference: 'high-performance' });
+ return { vendor: adapter?.info.vendor, architecture: adapter?.info.architecture };
+ });
+ const download = await call(offline ? 'text-download-status' : 'download-text', { modelId: model, dtype, device: 'webgpu' });
+ assert(download.ready, `${model} is not ready: ${JSON.stringify(download)}`);
+ const corpus = await fs.readFile(path.join(root, 'CHANGELOG.md'), 'utf8');
+ // About 3.3 characters per token for this English changelog.
+ const textFor = tokens => corpus.slice(0, Math.round(tokens * 3.3));
+ const prompts = lengths.map(tokens => ({
+ name: `text-${tokens}`, tokens,
+ messages: [{ role: 'user', content: `${textFor(tokens)}\n\nIn one sentence, what is the most recent change above?` }],
+ }));
+ if (vision) {
+ const image = await page.evaluate(() => {
+ const canvas = document.createElement('canvas'); canvas.width = 448; canvas.height = 448;
+ const ctx = canvas.getContext('2d'); ctx.fillStyle = 'white'; ctx.fillRect(0, 0, 448, 448);
+ ctx.fillStyle = 'red'; ctx.fillRect(64, 40, 320, 180);
+ return canvas.toDataURL('image/png');
+ });
+ prompts.push({ name: `image-text-${lengths[0]}`, tokens: lengths[0], messages: [{ role: 'user', content: [
+ { type: 'text', text: textFor(lengths[0]) },
+ { type: 'image_url', image_url: { url: image } },
+ { type: 'text', text: 'Ignore the release notes. What color is the large shape in the image, and what is its shape?' },
+ ] }] });
+ }
+ const type = vision ? 'multimodal-text-chat' : 'text-chat';
+ await call(type, { modelId: model, dtype, device: 'webgpu', messages: [{ role: 'user', content: 'Hi' }], options: { maxTokens: 4 } });
+ for (const prompt of prompts) {
+ const answers = new Map();
+ for (const slice of slices) {
+ const started = Date.now();
+ const reply = await call(type, { modelId: model, dtype, device: 'webgpu', messages: prompt.messages, options: { maxTokens, prefillSliceTokens: slice } });
+ const seconds = (Date.now() - started) / 1000;
+ const run = { prompt: prompt.name, slice, seconds, secondsPer1kPromptTokens: +(seconds / (prompt.tokens / 1000)).toFixed(3), content: reply.content };
+ result.runs.push(run);
+ answers.set(slice, reply.content);
+ console.log(JSON.stringify(run).slice(0, 300));
+ }
+ const reference = answers.get(slices[0]);
+ for (const [slice, content] of answers) {
+ if (content !== reference) result.mismatches.push({ prompt: prompt.name, slice, reference: slices[0] });
+ }
+ }
+ result.passed = result.mismatches.length === 0;
+} finally {
+ result.finishedAt = new Date().toISOString();
+ await fs.writeFile(output, JSON.stringify(result, null, 2));
+ await context.close();
+ server.close();
+}
+console.log(`${result.passed ? 'PASSED' : 'FAILED'}: ${output}`);
+if (!result.passed) process.exitCode = 1;
diff --git a/scripts/models/audit_webgpu_onnx.py b/scripts/models/audit_webgpu_onnx.py
new file mode 100644
index 000000000..b78b4ee92
--- /dev/null
+++ b/scripts/models/audit_webgpu_onnx.py
@@ -0,0 +1,137 @@
+"""Static audit of an ONNX decoder graph before shipping it on WebGPU.
+
+Catches the problems that made Ternary Bonsai 2 prefill take minutes in the
+browser (see docs/webgpu-models.md):
+
+* control-flow ops that ONNX Runtime Web's WebGPU execution provider does not
+ implement (`Scan`, `Loop`), including inside `If` branches. ONNX Runtime
+ places them on the CPU and their bodies then run once per iteration with a
+ GPU round trip, so a per-token recurrence becomes a per-token CPU loop;
+* a `logits` output that keeps the sequence dimension, so prefill computes and
+ downloads `[sequence_length, vocab]` scores (0.5 MB per prompt token for a
+ 248k vocabulary in FP16) and can exceed a 2 GB WebGPU buffer limit.
+
+Only the graph structure is read (external weights are not loaded). Exit code
+1 means at least one blocker was found. This is a static check: also confirm
+node placement and prefill scaling on hardware WebGPU as the doc describes.
+
+ python scripts/models/audit_webgpu_onnx.py path/to/decoder_model_merged_q4f16.onnx [--json]
+"""
+
+import argparse
+import collections
+import json
+import sys
+from pathlib import Path
+
+import onnx
+
+# Verified with onnxruntime-web 1.27: the WebGPU EP logs "webgpu kernel not found
+# in registries for Op type: Scan" (and the same for Loop) and places them on
+# the CPU. `If` has a WebGPU kernel and is fine.
+CPU_ONLY_CONTROL_FLOW = {"Scan", "Loop"}
+LOGITS_NAMES = {"logits"}
+
+
+def walk(graph, location=""):
+ for node in graph.node:
+ here = f"{location}/{node.name or node.op_type}"
+ yield node, here
+ for attribute in node.attribute:
+ if attribute.type == onnx.AttributeProto.GRAPH:
+ yield from walk(attribute.g, f"{here}:{attribute.name}")
+ elif attribute.type == onnx.AttributeProto.GRAPHS:
+ for index, subgraph in enumerate(attribute.graphs):
+ yield from walk(subgraph, f"{here}:{attribute.name}[{index}]")
+
+
+def dims(value_info):
+ return [d.dim_param or d.dim_value for d in value_info.type.tensor_type.shape.dim]
+
+
+def audit(path):
+ model = onnx.load(path, load_external_data=False)
+ graph = model.graph
+ blockers, warnings, info = [], [], {}
+
+ control_flow = collections.defaultdict(list)
+ op_counts = collections.Counter()
+ for node, location in walk(graph):
+ domain = node.domain or "ai.onnx"
+ op_counts[f"{domain}::{node.op_type}"] += 1
+ if node.op_type in CPU_ONLY_CONTROL_FLOW and domain == "ai.onnx":
+ control_flow[node.op_type].append(location)
+ for op_type, locations in control_flow.items():
+ blockers.append({
+ "check": "cpu-only-control-flow",
+ "op": op_type,
+ "count": len(locations),
+ "examples": locations[:3],
+ "why": f"No WebGPU kernel for {op_type}: it runs on the CPU and its body is dispatched per iteration "
+ "with a GPU round trip (Bonsai 2: ~6.5 ms per prompt token per layer on an RTX 5090).",
+ })
+
+ inputs = {i.name for i in graph.input}
+ for output in graph.output:
+ if output.name not in LOGITS_NAMES:
+ continue
+ shape = dims(output)
+ info["logits_shape"] = shape
+ if len(shape) == 3 and shape[1] != 1:
+ vocab = shape[2] if isinstance(shape[2], int) and shape[2] else None
+ entry = {
+ "check": "full-sequence-logits",
+ "shape": shape,
+ "why": "Prefill computes and downloads logits for every prompt token"
+ + (f" ({vocab * 2 / 1e6:.2f} MB per token in FP16)" if vocab else "")
+ + "; generation reads only the last position.",
+ }
+ if "num_logits_to_keep" in inputs:
+ entry["why"] += (" The graph accepts num_logits_to_keep, but Transformers.js passes 0 (keep all) "
+ "on its multimodal forward path, so slice to the last position in the graph.")
+ warnings.append(entry)
+ else:
+ blockers.append(entry)
+
+ contrib = sorted(k for k in op_counts if not k.startswith("ai.onnx::"))
+ info.update({
+ "opsets": {o.domain or "ai.onnx": o.version for o in model.opset_import},
+ "nodes": sum(op_counts.values()),
+ "if_nodes": op_counts.get("ai.onnx::If", 0),
+ "contrib_ops": {k: op_counts[k] for k in contrib},
+ "external_data_files": sorted({
+ entry.value for tensor in graph.initializer if tensor.data_location == onnx.TensorProto.EXTERNAL
+ for entry in tensor.external_data if entry.key == "location"
+ }),
+ "inputs": [i.name for i in graph.input if not i.name.startswith(("past", "present"))],
+ })
+ return {"graph": str(path), "blockers": blockers, "warnings": warnings, "info": info}
+
+
+def main():
+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ parser.add_argument("graphs", type=Path, nargs="+")
+ parser.add_argument("--json", action="store_true", help="print the full report as JSON")
+ args = parser.parse_args()
+ failed = False
+ for path in args.graphs:
+ report = audit(path)
+ failed = failed or bool(report["blockers"])
+ if args.json:
+ print(json.dumps(report, indent=2))
+ continue
+ status = "BLOCKED" if report["blockers"] else "ok"
+ print(f"{status}: {path}")
+ for item in report["blockers"]:
+ print(f" blocker {item['check']}: {item.get('op') or item.get('shape')} "
+ f"(x{item.get('count', 1)}) - {item['why']}")
+ for item in report["warnings"]:
+ print(f" warning {item['check']}: {item.get('shape')} - {item['why']}")
+ info = report["info"]
+ print(f" logits {info.get('logits_shape')}, {info['nodes']} nodes, {info['if_nodes']} If, "
+ f"contrib {', '.join(info['contrib_ops']) or 'none'}")
+ sys.exit(1 if failed else 0)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/scripts/models/bonsai/README.md b/scripts/models/bonsai/README.md
index aafa9a2c9..4c2960ea5 100644
--- a/scripts/models/bonsai/README.md
+++ b/scripts/models/bonsai/README.md
@@ -82,6 +82,11 @@ the worker compares the cached graph with a pinned SHA-256. Inference stays
cache-only and keeps using a stale graph; an explicit download of the model
replaces just the 3 MB graph file and keeps the cached weights.
+The general lessons and the checklist for any in-browser model are in
+[`docs/webgpu-models.md`](../../../docs/webgpu-models.md). Run
+`scripts/models/audit_webgpu_onnx.py` on every decoder graph and
+`scripts/check-webgpu-long-prompts.mjs` on every bundle before publishing.
+
## Runtime contract
Load `AutoProcessor` and `AutoModelForImageTextToText` with: