Open-source AI browser agent for Chrome and Firefox (monorepo) π§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Document WebGPU model lessons and add audit and long-prompt checks
Claude4ac89edc8c0d
diff --git a/AGENT.md b/AGENT.mdindex 64c2f8f0c..9889292c1 100644--- a/AGENT.md+++ b/AGENT.md@@ -61,6 +61,8 @@ vanilla JS/CSS unless there is a very strong reason to add a framework.- `docs/site-adapters.md`: adapter shape, prompt style, and testing guidance.- `docs/providers-and-models.md`: provider configuration and model capabilityassumptions.+- `docs/webgpu-models.md`: required checklist before adding or changing an+ in-browser WebGPU model or its ONNX graphs.- `docs/localization.md`: UI locale workflow and mirroring expectations.- `docs/privacy-and-data-flow.md` and `docs/security-model.md`: data handling,permissions, and user-risk model.diff --git a/docs/providers-and-models.md b/docs/providers-and-models.mdindex 1ce140ffd..1053290ae 100644--- a/docs/providers-and-models.md+++ b/docs/providers-and-models.md@@ -453,6 +453,13 @@ approximately one megapixel per image. Build sources, tensor mapping checks,and the hardware browser verification command are documented in[`scripts/models/bonsai/README.md`](../scripts/models/bonsai/README.md).+Before adding, replacing or re-exporting any in-browser model, follow+[`docs/webgpu-models.md`](webgpu-models.md). It records why the first Bonsai 2+graphs prefilled on the CPU and froze the desktop, and the required checks:+`scripts/models/audit_webgpu_onnx.py` for CPU-only ops and full-sequence+logits, node placement, long-prompt timing with+`scripts/check-webgpu-long-prompts.mjs`, and how to update a published graph.+To verify the published text presets through the packaged production worker,run these commands from the repository root with its Playwright Chromiuminstalled:diff --git a/docs/webgpu-models.md b/docs/webgpu-models.mdnew file mode 100644index 000000000..726da9160--- /dev/null+++ b/docs/webgpu-models.md@@ -0,0 +1,169 @@+# In-browser WebGPU models: checklist and lessons++Read this before adding, replacing or re-exporting a model for the Chromium+**WebGPU (In-browser)** provider (`src/chrome/src/providers/webgpu.js`,+`src/chrome/src/offscreen/inference-worker.js`). It records what made Ternary+Bonsai 2 unusable in the browser and the checks that would have caught it.++## What went wrong with Bonsai 2++Asking Bonsai 2 2B Vision to "summarize this page" took many minutes and froze+the desktop, on an RTX 5090 and on an M4 Max alike. Five independent causes+stacked up:++1. **Token-by-token prefill on the CPU.** The upstream language graph+ (`danyalmalik/Ternary-Bonsai-2-*-ONNX-WebGPU`, the same pattern as+ `onnx-community/Qwen3.5-*-ONNX`) computes every GatedDeltaNet prefill with an+ ONNX `Scan` over tokens. ONNX Runtime Web's WebGPU execution provider has no+ `Scan` (or `Loop`) kernel, so the node is placed on the CPU and its body runs+ once per token, with a GPU round trip, in each of 18 layers (24 for 4B).+ That costs about 6.5 ms per prompt token per layer: 0.13 s per prompt+ token for 2B and 0.19 s for 4B on an RTX 5090. A faster GPU barely helps,+ because the time goes to synchronisation, not compute.+2. **Logits for every prompt token.** The graph's `logits` output was+ `[batch, sequence_length, 248320]`, so prefill ran the LM head over the+ whole prompt and downloaded about 0.5 MB of FP16 scores per prompt token.+ Above roughly 4.3k tokens that output also exceeds the 2 GB maximum buffer+ size many WebGPU adapters report.+3. **The wrong GPU.** Chrome runs WebGPU on the adapter its GPU process uses.+ On a Windows machine with an RTX 5090 and a small display card (T400), every+ `powerPreference` returned the T400, which also drives the monitors. The model+ saturated it and the whole desktop stalled while the 5090 sat idle.+4. **One giant prefill pass.** Even with a fixed graph, a ~9k-token prompt in a+ single decoder pass keeps a GPU that is shared with the display (Apple+ silicon, a display-attached card) busy for seconds, and the desktop stutters.+5. **Large prompts.** Ask mode sends the full Ask system prompt (~3k tokens)+ plus 17 tool schemas (~6k tokens) to in-browser models on every call,+ because the compact tier currently only changes Act mode.++Two process gaps let this ship. The production harness+(`scripts/check-bonsai-webgpu.mjs`) only uses prompts of a few dozen tokens,+which hide prefill problems entirely. And graph files were replaced under+an already-cached URL, which installed copies never re-fetch.++## Checklist for a new or re-exported model++Do all of these on the exact files you will publish.++1. **Static graph audit.**++ ```bash+ python scripts/models/audit_webgpu_onnx.py path/to/decoder.onnx+ ```++ It exits non-zero for `Scan`/`Loop` anywhere in the graph (including `If`+ branches) and for a `logits` output that keeps the sequence dimension.+ `If` is fine; it has a WebGPU kernel. The original Bonsai 2 and+ `onnx-community/Qwen3.5` decoders fail this check. Compass Tiny,+ `Mike0021/Ternary-Bonsai-2-4B-text-ONNX` (which uses the+ `com.microsoft::LinearAttention` and `CausalConvWithState` contrib ops) and+ the rewritten Bonsai 2 Vision graphs pass.+ - A `num_logits_to_keep` input is not enough on its own. On the+ multimodal forward path Transformers.js passes `0` (keep all), so slice to+ the last position inside the graph, as `chunk_gdn_prefill.py` does.+2. **Node placement on real WebGPU.** Create the session in a browser with+ `ort.env.logLevel = 'verbose'` and `logSeverityLevel: 0`, then read the+ `Node placements` block. Small shape ops (`Shape`, `Gather`, `Equal` on+ int64) belong on the CPU. Any compute op reported as+ `webgpu kernel not found in registries` is a red flag.+3. **Long prompts and prefill scaling.** The short harness prompts cannot show+ prefill problems. Run real prompts of about 1k, 4k and 8k tokens, plus a+ long prompt with an image for vision models, through the production+ worker on hardware WebGPU:++ ```bash+ node scripts/check-webgpu-long-prompts.mjs --model <owner/repo> --vision \+ --profile <cached profile> --offline --lengths 1000,4000,8000 --slices 0,512+ ```++ It prints wall time and seconds per 1k prompt tokens for each prompt, and+ fails if the greedy answer changes between slice sizes. Seconds per 1k+ tokens should stay roughly flat and around a second or less on a desktop+ GPU. Tens of seconds per 1k tokens that scale linearly mean a CPU+ fallback. Reference for rewritten Bonsai 2 2B on an RTX 5090 (48 new+ tokens, including decode): 3k tokens in 2.5 s and 8k in 5.6 s in one pass,+ 7.8 s in 512-token slices. In a decoder-only benchmark, 4B prefilled 8,192+ tokens in 16 s. The original Scan graph took 128 s for 1,024 tokens.+4. **Production harness.** Run `scripts/check-bonsai-webgpu.mjs` online and+ `--offline` (with `--vision --image-history` for vision models) and record+ the evidence with `scripts/models/bonsai/record_browser_validation.py`.+5. **Memory.** Budget weights, plus KV and recurrent cache for the browser+ context, plus transient prefill buffers. Keep every single tensor under+ 2 GB, the per-buffer limit on many adapters.+6. **Rollout.** Browsers keep files cached by URL. If a published file+ changes in place, existing installs never see it unless the worker+ refreshes it explicitly (see "Updating a published graph").++## Fixing a Qwen3.5 / GatedDeltaNet export++Use the tools in `scripts/models/bonsai/`:++- `chunk_gdn_prefill.py` replaces every GatedDeltaNet `Scan` with the+ chunked gated delta rule and slices the LM head to the last position.+ Weights and the `.onnx_data` file stay byte-identical; only the small graph+ file changes. Decode keeps the original single-token branch.+ - 128-token chunks. Inside a chunk, the unit lower-triangular WY inverse uses+ **recursive block inversion**. A Neumann-product inverse is shorter but+ overflows FP32 when keys in a chunk are strongly correlated.+ - Across chunks, a Hillis-Steele prefix scan over the affine state maps runs+ with `If`-skipped steps. The step count is derived from+ `text_config.max_position_embeddings`, so every accepted prompt length+ stays correct. Chunks must be powers of two from 16 to 1024.+- `validate_chunked_gdn.py` compares old and new graphs on CPU with real text.+ It covers chunk-boundary lengths, a multi-chunk prompt, a cached continuation,+ logits, every cache output (whole-tensor and worst-element error) and+ finiteness. It reads the chunk size from the rewritten graph.+- `make_chunked_bundle.py` derives a sealed bundle. It writes model-card+ parity and timing claims only from evidence that passes the verifier (both+ graph hashes, chunk coverage, top-1 and decode equality, FP16 tolerances).+ It drops browser evidence recorded against the previous graph and keeps+ native vision parity only while its encoder hashes still match.+- `build_vision.py` applies the rewrite on full rebuilds.++## Runtime behaviour to keep++- **Sliced prefill.** For the Bonsai 2 Vision presets the worker prefills+ long prompts in 512-token slices that carry the recurrent, conv and KV+ cache, then lets `generate()` continue from that cache. Greedy answers are+ identical to a single pass. Each extra slice adds roughly one decoder run of+ fixed overhead (about 0.14 s on an RTX 5090). The `chrome.storage.local` key+ `webgpuPrefillSliceTokens` overrides the slice (64β8192, `0` disables).+ Apply the same pattern to any new model that serves long prompts on shared+ GPUs.+- **Cache-only inference.** Chat never downloads. Only explicit download or+ preload flows fetch model files.++## Updating a published graph++Prefer a new repository or file name for a changed graph. If a file must+change in place, as the Bonsai 2 Vision decoders did:++- Pin the new file's SHA-256 in the worker (`BONSAI2_DECODER_GRAPH_SHA256`).+- Inference only detects a stale cached graph and keeps using it.+- An explicit download or preload replaces just that file, verified by hash,+ with a 15 s deadline and no attempt while known offline. It then drops the+ resident session built from the old graph.+- Never block or break offline use to force an update.++## Diagnosing a slow or stuttering user machine++- In a normal tab, run+ `(await navigator.gpu.requestAdapter({powerPreference: 'high-performance'})).info`+ to see the adapter Chrome uses. On Windows, compare it with `nvidia-smi` or+ Task Manager.+- To move Chrome to another GPU on Windows, use Settings β System β Display β+ Graphics β Chrome, or start Chrome with `--use-adapter-luid=<high>,<low>`.+ Use **decimal** values: the hex form gave no adapter in Chromium 147. Find a+ LUID with PowerShell, matching adapters by dedicated memory use:+ `(Get-Counter '\GPU Adapter Memory(*)\Dedicated Usage').CounterSamples`.+- Stutter while the GPU is shared with the display is expected for long+ single passes. Lower `webgpuPrefillSliceTokens` (for example to 256) and keep+ prompts short.++## Open items++- The Bonsai 2 2B **text** preset still points at+ `danyalmalik/Ternary-Bonsai-2-2B-ONNX-WebGPU`, which fails the audit. Publish+ a rewritten text repository or point the preset at a passing graph.+- Ask mode should use a compact prompt and a smaller tool set for in-browser+ models.diff --git a/scripts/check-webgpu-long-prompts.mjs b/scripts/check-webgpu-long-prompts.mjsnew file mode 100644index 000000000..f17e220bd--- /dev/null+++ b/scripts/check-webgpu-long-prompts.mjs@@ -0,0 +1,170 @@+// Long-prompt check for in-browser WebGPU models through the production worker.+//+// The short prompts in check-bonsai-webgpu.mjs cannot reveal prefill problems+// (a CPU-fallback recurrence or full-sequence logits only show up with+// thousands of prompt tokens). This runs real text of several lengths (and,+// with --vision, a long prompt that includes an image) through+// src/chrome/src/offscreen/inference-worker.js, reports wall time per prompt+// and seconds per 1k prompt tokens, and checks that greedy answers are the same+// for every requested prefill slice size. See docs/webgpu-models.md.+//+// node scripts/check-webgpu-long-prompts.mjs --model webbrain-one/Ternary-Bonsai-2-2B-ONNX-WebGPU-Vision \+// --vision --profile .build/bonsai/chunked-2b-profile --offline --lengths 1000,4000,8000 --slices 0,512+//+// Options mirror check-bonsai-webgpu.mjs: --local-repo owner/repo=dir serves+// unpublished files, --offline requires a cached model and blocks Hugging Face,+// --adapter-luid high,low (decimal) selects a Windows GPU.+import assert from 'node:assert/strict';+import fs from 'node:fs/promises';+import { createReadStream } from 'node:fs';+import http from 'node:http';+import path from 'node:path';+import { fileURLToPath } from 'node:url';+import { chromium } from 'playwright';++const root = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..');+const args = process.argv.slice(2);+const option = (name, fallback) => { const i = args.indexOf(name); return i < 0 ? fallback : args[i + 1]; };+const model = option('--model', 'webbrain-one/Ternary-Bonsai-2-2B-ONNX-WebGPU-Vision');+const vision = args.includes('--vision');+const offline = args.includes('--offline');+const dtype = option('--dtype', null)+ ? JSON.parse(option('--dtype'))+ : vision ? { embed_tokens: 'q2f16', vision_encoder: 'fp16', decoder_model_merged: 'q2f16' } : 'q2f16';+const lengths = option('--lengths', '1000,4000,8000').split(',').map(Number);+const slices = option('--slices', '0,512').split(',').map(Number);+const maxTokens = Number(option('--max-tokens', 48));+const profile = path.resolve(option('--profile', '.build/webgpu-long-prompts/profile'));+const output = path.resolve(option('--output', '.build/webgpu-long-prompts/result.json'));+const localRepos = {};+for (let i = 0; i < args.length; i++) {+ if (args[i] !== '--local-repo') continue;+ const entry = args[++i] || '';+ const split = entry.indexOf('=');+ assert(split > 0, '--local-repo requires owner/repo=directory');+ localRepos[entry.slice(0, split)] = path.resolve(entry.slice(split + 1));+}+const mime = file => ({ '.mjs': 'text/javascript', '.js': 'text/javascript', '.json': 'application/json', '.wasm': 'application/wasm' }[path.extname(file)] || 'application/octet-stream');+const workerBootstrap = `+const nativeFetch = globalThis.fetch.bind(globalThis);+const mapped = ${JSON.stringify(Object.keys(localRepos))};+globalThis.fetch = (input, init) => {+ const url = new URL(typeof input === 'string' ? input : input.url || String(input), self.location.href);+ if (url.hostname === 'huggingface.co') {+ if (${offline}) throw new Error('Offline check blocked a Hugging Face fetch: ' + url.pathname);+ const match = url.pathname.match(/^\\/([^/]+\\/[^/]+)\\/resolve\\/[^/]+\\/(.+)$/);+ if (match && mapped.includes(match[1])) return nativeFetch('/models/' + match[1] + '/' + match[2], init);+ }+ return nativeFetch(input, init);+};+await import('/extension/src/offscreen/inference-worker.js');+self.postMessage({ type: 'harness-ready' });`;+const server = http.createServer(async (request, response) => {+ try {+ const pathname = decodeURIComponent(new URL(request.url, 'http://localhost').pathname);+ if (pathname === '/') { response.setHeader('Content-Type', 'text/html'); response.end('<!doctype html><title>WebGPU long prompts</title>'); return; }+ if (pathname === '/worker.mjs') { response.setHeader('Content-Type', 'text/javascript'); response.end(workerBootstrap); return; }+ let base, relative;+ if (pathname.startsWith('/extension/')) { base = path.join(root, 'src/chrome'); relative = pathname.slice('/extension/'.length); }+ else if (pathname.startsWith('/models/')) {+ const match = pathname.match(/^\/models\/([^/]+\/[^/]+)\/(.+)$/);+ base = match && localRepos[match[1]]; relative = match?.[2];+ }+ if (!base || !relative) { response.writeHead(404); response.end(); return; }+ const file = path.resolve(base, relative);+ if (!file.startsWith(path.resolve(base) + path.sep)) { response.writeHead(403); response.end(); return; }+ const stat = await fs.stat(file);+ response.setHeader('Content-Type', mime(file)); response.setHeader('Content-Length', stat.size);+ createReadStream(file).on('error', () => response.destroy()).pipe(response);+ } catch (error) {+ response.writeHead(error.code === 'ENOENT' ? 404 : 500); response.end();+ }+});+await fs.mkdir(path.dirname(output), { recursive: true });+await new Promise((resolve, reject) => { server.once('error', reject); server.listen(Number(option('--port', 8769)), '127.0.0.1', resolve); });+const context = await chromium.launchPersistentContext(profile, {+ headless: true,+ channel: 'chromium',+ args: ['--enable-unsafe-webgpu', '--force-high-performance-gpu',+ ...(option('--adapter-luid', '') ? [`--use-adapter-luid=${option('--adapter-luid')}`] : [])],+});+const result = { model, vision, dtype, startedAt: new Date().toISOString(), runs: [], mismatches: [] };+try {+ const page = await context.newPage();+ page.on('console', message => { if (message.type() === 'error') console.log(`[browser:error] ${message.text().slice(0, 300)}`); });+ await page.goto(`http://127.0.0.1:${server.address().port}/`);+ await page.evaluate(() => new Promise((ready, failed) => {+ const pending = new Map(); let id = 0;+ const worker = new Worker('/worker.mjs', { type: 'module' });+ worker.onmessage = ({ data }) => {+ if (data.type === 'harness-ready') ready();+ else if (data.id && pending.has(data.id)) {+ const { resolve, reject } = pending.get(data.id); pending.delete(data.id);+ if (data.ok === false) reject(new Error(data.error)); else resolve(data);+ }+ };+ worker.onerror = event => failed(new Error(event.message));+ window.workerCall = (type, payload) => new Promise((resolve, reject) => {+ const requestId = ++id; pending.set(requestId, { resolve, reject }); worker.postMessage({ id: requestId, type, payload });+ });+ }));+ const origin = new URL(page.url()).origin;+ const call = (type, payload = {}) => page.evaluate(({ type, payload }) => window.workerCall(type, payload), { type, payload });+ await call('init', {+ transformersUrl: `${origin}/extension/vendor/transformers/transformers.web.js`,+ wasmMjsUrl: `${origin}/extension/vendor/transformers/ort-wasm-simd-threaded.asyncify.mjs`,+ wasmUrl: `${origin}/extension/vendor/transformers/ort-wasm-simd-threaded.asyncify.wasm`,+ });+ result.adapter = await page.evaluate(async () => {+ const adapter = await navigator.gpu.requestAdapter({ powerPreference: 'high-performance' });+ return { vendor: adapter?.info.vendor, architecture: adapter?.info.architecture };+ });+ const download = await call(offline ? 'text-download-status' : 'download-text', { modelId: model, dtype, device: 'webgpu' });+ assert(download.ready, `${model} is not ready: ${JSON.stringify(download)}`);+ const corpus = await fs.readFile(path.join(root, 'CHANGELOG.md'), 'utf8');+ // About 3.3 characters per token for this English changelog.+ const textFor = tokens => corpus.slice(0, Math.round(tokens * 3.3));+ const prompts = lengths.map(tokens => ({+ name: `text-${tokens}`, tokens,+ messages: [{ role: 'user', content: `${textFor(tokens)}\n\nIn one sentence, what is the most recent change above?` }],+ }));+ if (vision) {+ const image = await page.evaluate(() => {+ const canvas = document.createElement('canvas'); canvas.width = 448; canvas.height = 448;+ const ctx = canvas.getContext('2d'); ctx.fillStyle = 'white'; ctx.fillRect(0, 0, 448, 448);+ ctx.fillStyle = 'red'; ctx.fillRect(64, 40, 320, 180);+ return canvas.toDataURL('image/png');+ });+ prompts.push({ name: `image-text-${lengths[0]}`, tokens: lengths[0], messages: [{ role: 'user', content: [+ { type: 'text', text: textFor(lengths[0]) },+ { type: 'image_url', image_url: { url: image } },+ { type: 'text', text: 'Ignore the release notes. What color is the large shape in the image, and what is its shape?' },+ ] }] });+ }+ const type = vision ? 'multimodal-text-chat' : 'text-chat';+ await call(type, { modelId: model, dtype, device: 'webgpu', messages: [{ role: 'user', content: 'Hi' }], options: { maxTokens: 4 } });+ for (const prompt of prompts) {+ const answers = new Map();+ for (const slice of slices) {+ const started = Date.now();+ const reply = await call(type, { modelId: model, dtype, device: 'webgpu', messages: prompt.messages, options: { maxTokens, prefillSliceTokens: slice } });+ const seconds = (Date.now() - started) / 1000;+ const run = { prompt: prompt.name, slice, seconds, secondsPer1kPromptTokens: +(seconds / (prompt.tokens / 1000)).toFixed(3), content: reply.content };+ result.runs.push(run);+ answers.set(slice, reply.content);+ console.log(JSON.stringify(run).slice(0, 300));+ }+ const reference = answers.get(slices[0]);+ for (const [slice, content] of answers) {+ if (content !== reference) result.mismatches.push({ prompt: prompt.name, slice, reference: slices[0] });+ }+ }+ result.passed = result.mismatches.length === 0;+} finally {+ result.finishedAt = new Date().toISOString();+ await fs.writeFile(output, JSON.stringify(result, null, 2));+ await context.close();+ server.close();+}+console.log(`${result.passed ? 'PASSED' : 'FAILED'}: ${output}`);+if (!result.passed) process.exitCode = 1;diff --git a/scripts/models/audit_webgpu_onnx.py b/scripts/models/audit_webgpu_onnx.pynew file mode 100644index 000000000..b78b4ee92--- /dev/null+++ b/scripts/models/audit_webgpu_onnx.py@@ -0,0 +1,137 @@+"""Static audit of an ONNX decoder graph before shipping it on WebGPU.++Catches the problems that made Ternary Bonsai 2 prefill take minutes in the+browser (see docs/webgpu-models.md):++* control-flow ops that ONNX Runtime Web's WebGPU execution provider does not+ implement (`Scan`, `Loop`), including inside `If` branches. ONNX Runtime+ places them on the CPU and their bodies then run once per iteration with a+ GPU round trip, so a per-token recurrence becomes a per-token CPU loop;+* a `logits` output that keeps the sequence dimension, so prefill computes and+ downloads `[sequence_length, vocab]` scores (0.5 MB per prompt token for a+ 248k vocabulary in FP16) and can exceed a 2 GB WebGPU buffer limit.++Only the graph structure is read (external weights are not loaded). Exit code+1 means at least one blocker was found. This is a static check: also confirm+node placement and prefill scaling on hardware WebGPU as the doc describes.++ python scripts/models/audit_webgpu_onnx.py path/to/decoder_model_merged_q4f16.onnx [--json]+"""++import argparse+import collections+import json+import sys+from pathlib import Path++import onnx++# Verified with onnxruntime-web 1.27: the WebGPU EP logs "webgpu kernel not found+# in registries for Op type: Scan" (and the same for Loop) and places them on+# the CPU. `If` has a WebGPU kernel and is fine.+CPU_ONLY_CONTROL_FLOW = {"Scan", "Loop"}+LOGITS_NAMES = {"logits"}+++def walk(graph, location=""):+ for node in graph.node:+ here = f"{location}/{node.name or node.op_type}"+ yield node, here+ for attribute in node.attribute:+ if attribute.type == onnx.AttributeProto.GRAPH:+ yield from walk(attribute.g, f"{here}:{attribute.name}")+ elif attribute.type == onnx.AttributeProto.GRAPHS:+ for index, subgraph in enumerate(attribute.graphs):+ yield from walk(subgraph, f"{here}:{attribute.name}[{index}]")+++def dims(value_info):+ return [d.dim_param or d.dim_value for d in value_info.type.tensor_type.shape.dim]+++def audit(path):+ model = onnx.load(path, load_external_data=False)+ graph = model.graph+ blockers, warnings, info = [], [], {}++ control_flow = collections.defaultdict(list)+ op_counts = collections.Counter()+ for node, location in walk(graph):+ domain = node.domain or "ai.onnx"+ op_counts[f"{domain}::{node.op_type}"] += 1+ if node.op_type in CPU_ONLY_CONTROL_FLOW and domain == "ai.onnx":+ control_flow[node.op_type].append(location)+ for op_type, locations in control_flow.items():+ blockers.append({+ "check": "cpu-only-control-flow",+ "op": op_type,+ "count": len(locations),+ "examples": locations[:3],+ "why": f"No WebGPU kernel for {op_type}: it runs on the CPU and its body is dispatched per iteration "+ "with a GPU round trip (Bonsai 2: ~6.5 ms per prompt token per layer on an RTX 5090).",+ })++ inputs = {i.name for i in graph.input}+ for output in graph.output:+ if output.name not in LOGITS_NAMES:+ continue+ shape = dims(output)+ info["logits_shape"] = shape+ if len(shape) == 3 and shape[1] != 1:+ vocab = shape[2] if isinstance(shape[2], int) and shape[2] else None+ entry = {+ "check": "full-sequence-logits",+ "shape": shape,+ "why": "Prefill computes and downloads logits for every prompt token"+ + (f" ({vocab * 2 / 1e6:.2f} MB per token in FP16)" if vocab else "")+ + "; generation reads only the last position.",+ }+ if "num_logits_to_keep" in inputs:+ entry["why"] += (" The graph accepts num_logits_to_keep, but Transformers.js passes 0 (keep all) "+ "on its multimodal forward path, so slice to the last position in the graph.")+ warnings.append(entry)+ else:+ blockers.append(entry)++ contrib = sorted(k for k in op_counts if not k.startswith("ai.onnx::"))+ info.update({+ "opsets": {o.domain or "ai.onnx": o.version for o in model.opset_import},+ "nodes": sum(op_counts.values()),+ "if_nodes": op_counts.get("ai.onnx::If", 0),+ "contrib_ops": {k: op_counts[k] for k in contrib},+ "external_data_files": sorted({+ entry.value for tensor in graph.initializer if tensor.data_location == onnx.TensorProto.EXTERNAL+ for entry in tensor.external_data if entry.key == "location"+ }),+ "inputs": [i.name for i in graph.input if not i.name.startswith(("past", "present"))],+ })+ return {"graph": str(path), "blockers": blockers, "warnings": warnings, "info": info}+++def main():+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ parser.add_argument("graphs", type=Path, nargs="+")+ parser.add_argument("--json", action="store_true", help="print the full report as JSON")+ args = parser.parse_args()+ failed = False+ for path in args.graphs:+ report = audit(path)+ failed = failed or bool(report["blockers"])+ if args.json:+ print(json.dumps(report, indent=2))+ continue+ status = "BLOCKED" if report["blockers"] else "ok"+ print(f"{status}: {path}")+ for item in report["blockers"]:+ print(f" blocker {item['check']}: {item.get('op') or item.get('shape')} "+ f"(x{item.get('count', 1)}) - {item['why']}")+ for item in report["warnings"]:+ print(f" warning {item['check']}: {item.get('shape')} - {item['why']}")+ info = report["info"]+ print(f" logits {info.get('logits_shape')}, {info['nodes']} nodes, {info['if_nodes']} If, "+ f"contrib {', '.join(info['contrib_ops']) or 'none'}")+ sys.exit(1 if failed else 0)+++if __name__ == "__main__":+ main()diff --git a/scripts/models/bonsai/README.md b/scripts/models/bonsai/README.mdindex aafa9a2c9..4c2960ea5 100644--- a/scripts/models/bonsai/README.md+++ b/scripts/models/bonsai/README.md@@ -82,6 +82,11 @@ the worker compares the cached graph with a pinned SHA-256. Inference stayscache-only and keeps using a stale graph; an explicit download of the modelreplaces just the 3 MB graph file and keeps the cached weights.+The general lessons and the checklist for any in-browser model are in+[`docs/webgpu-models.md`](../../../docs/webgpu-models.md). Run+`scripts/models/audit_webgpu_onnx.py` on every decoder graph and+`scripts/check-webgpu-long-prompts.mjs` on every bundle before publishing.+## Runtime contractLoad `AutoProcessor` and `AutoModelForImageTextToText` with: