Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Long-prompt check: require hardware WebGPU, add deadlines, time text models only
Claudee03345556439
diff --git a/docs/webgpu-models.md b/docs/webgpu-models.mdindex 726da9160..fb164945c 100644--- a/docs/webgpu-models.md+++ b/docs/webgpu-models.md@@ -76,14 +76,28 @@ Do all of these on the exact files you will publish.--profile <cached profile> --offline --lengths 1000,4000,8000 --slices 0,512```- It prints wall time and seconds per 1k prompt tokens for each prompt, and- fails if the greedy answer changes between slice sizes. Seconds per 1k- tokens should stay roughly flat and around a second or less on a desktop- GPU. Tens of seconds per 1k tokens that scale linearly mean a CPU- fallback. Reference for rewritten Bonsai 2 2B on an RTX 5090 (48 new- tokens, including decode): 3k tokens in 2.5 s and 8k in 5.6 s in one pass,- 7.8 s in 512-token slices. In a decoder-only benchmark, 4B prefilled 8,192- tokens in 16 s. The original Scan graph took 128 s for 1,024 tokens.+ It prints wall time and seconds per 1k prompt tokens for each prompt. It+ rejects software WebGPU adapters, and it fails when a text prompt of 4k or+ more tokens takes longer than `--max-seconds-per-1k` (default 10). A+ CPU fallback costs 100 seconds or more per 1k tokens.+ - With `--vision`, it also compares each sliced answer with the first+ `--slices` value. It fails if they diverge within the first eight words,+ which is what a broken cache handoff does. A later divergence is reported+ but expected (see "Sliced prefill" below).+ - Without `--vision`, each prompt runs once, for timing only.++ Seconds per 1k tokens should stay roughly flat, around a second on a+ desktop GPU. Reference for rewritten Bonsai 2 2B Vision on an RTX 5090+ (48 new tokens, including decode):++ | Prompt tokens | One pass | 512-token slices | 256-token slices |+ |---|---|---|---|+ | 1k | 1.3 s | 1.4 s | 1.4 s |+ | 4k | 3.3 s | 4.0 s | 4.7 s |+ | 8k | 5.5 s | 7.7 s | 10.0 s |++ In a decoder-only benchmark, 4B prefilled 8,192 tokens in 16 s. The+ original Scan graph took 128 s for 1,024 tokens.4. **Production harness.** Run `scripts/check-bonsai-webgpu.mjs` online and`--offline` (with `--vision --image-history` for vision models) and recordthe evidence with `scripts/models/bonsai/record_browser_validation.py`.@@ -124,12 +138,22 @@ Use the tools in `scripts/models/bonsai/`:- **Sliced prefill.** For the Bonsai 2 Vision presets the worker prefillslong prompts in 512-token slices that carry the recurrent, conv and KV- cache, then lets `generate()` continue from that cache. Greedy answers are- identical to a single pass. Each extra slice adds roughly one decoder run of- fixed overhead (about 0.14 s on an RTX 5090). The `chrome.storage.local` key- `webgpuPrefillSliceTokens` overrides the slice (64–8192, `0` disables).- Apply the same pattern to any new model that serves long prompts on shared- GPUs.+ cache, then lets `generate()` continue from that cache.+ - Answers match a single pass up to FP16 rounding. In the RTX 5090 runs,+ most sliced greedy answers were identical over 48 tokens. The rest+ matched for the first 30–39 words, then picked a different but+ equivalent phrasing: slicing changes the order of floating-point sums,+ which can flip a near-tied token. The results were reproducible run to+ run, and a smaller slice was not always further off.+ - A broken cache handoff looks different: the answer loses the prompt right+ away.+ - Each extra slice adds roughly one decoder run of fixed overhead (about+ 0.14 s on an RTX 5090).++ - The `chrome.storage.local` key `webgpuPrefillSliceTokens` overrides the+ slice (64–8192, `0` disables).+ - Apply the same pattern to any new model that serves long prompts on+ shared GPUs.- **Cache-only inference.** Chat never downloads. Only explicit download orpreload flows fetch model files.diff --git a/scripts/check-webgpu-long-prompts.mjs b/scripts/check-webgpu-long-prompts.mjsindex deb6adc11..fb8b69b90 100644--- a/scripts/check-webgpu-long-prompts.mjs+++ b/scripts/check-webgpu-long-prompts.mjs@@ -12,14 +12,18 @@// (default 8). A broken cache handoff loses the prompt and diverges at once,// while FP16 rounding differences between slice sizes can change a near-tied// token later in a long greedy answer; the common prefix is reported either-// way. Slicing applies to the Bonsai 2 Vision path. See docs/webgpu-models.md.+// way. Prefill slicing applies only to the vision path (Bonsai 2 Vision), so+// without --vision each prompt runs once, for timing only: some text presets+// sample, and their answers are not comparable. Software (fallback) WebGPU+// adapters are rejected. See docs/webgpu-models.md.//// node scripts/check-webgpu-long-prompts.mjs --model webbrain-one/Ternary-Bonsai-2-2B-ONNX-WebGPU-Vision \// --vision --profile .build/bonsai/chunked-2b-profile --offline --lengths 1000,4000,8000 --slices 0,512//// Options mirror check-bonsai-webgpu.mjs: --local-repo owner/repo=dir serves// unpublished files, --offline requires a cached model and blocks Hugging Face,-// --adapter-luid high,low (decimal) selects a Windows GPU.+// --adapter-luid high,low (decimal) selects a Windows GPU, and --timeout-ms+// sets the per-operation deadline (default 30 minutes).import assert from 'node:assert/strict';import fs from 'node:fs/promises';import { createReadStream } from 'node:fs';@@ -42,6 +46,7 @@ const slices = option('--slices', '0,512').split(',').map(Number);const maxTokens = Number(option('--max-tokens', 48));const maxSecondsPer1k = Number(option('--max-seconds-per-1k', 10));const minCommonWords = Number(option('--min-common-words', 8));+const timeout = Number(option('--timeout-ms', 30 * 60_000));const profile = path.resolve(option('--profile', '.build/webgpu-long-prompts/profile'));const output = path.resolve(option('--output', '.build/webgpu-long-prompts/result.json'));const localRepos = {};@@ -117,22 +122,33 @@ try {worker.onmessage = ({ data }) => {if (data.type === 'harness-ready') ready();else if (data.id && pending.has(data.id)) {- const { resolve, reject } = pending.get(data.id); pending.delete(data.id);+ const { resolve, reject, timer } = pending.get(data.id);+ clearTimeout(timer); pending.delete(data.id);if (data.ok === false) reject(new Error(data.error)); else resolve(data);}};- worker.onerror = event => failed(new Error(event.message));- window.workerCall = (type, payload) => new Promise((resolve, reject) => {- const requestId = ++id; pending.set(requestId, { resolve, reject }); worker.postMessage({ id: requestId, type, payload });+ worker.onerror = event => {+ failed(new Error(event.message));+ for (const { reject, timer } of pending.values()) { clearTimeout(timer); reject(new Error(event.message)); }+ pending.clear();+ };+ window.workerCall = (type, payload, timeout) => new Promise((resolve, reject) => {+ const requestId = ++id;+ const timer = setTimeout(() => { pending.delete(requestId); reject(new Error(`Timeout: ${type}`)); }, timeout);+ pending.set(requestId, { resolve, reject, timer });+ worker.postMessage({ id: requestId, type, payload });});}));const origin = new URL(page.url()).origin;- const call = (type, payload = {}) => page.evaluate(({ type, payload }) => window.workerCall(type, payload), { type, payload });+ const call = (type, payload = {}) => page.evaluate(({ type, payload, timeout }) => window.workerCall(type, payload, timeout), { type, payload, timeout });await call('init', {transformersUrl: `${origin}/extension/vendor/transformers/transformers.web.js`,wasmMjsUrl: `${origin}/extension/vendor/transformers/ort-wasm-simd-threaded.asyncify.mjs`,wasmUrl: `${origin}/extension/vendor/transformers/ort-wasm-simd-threaded.asyncify.wasm`,});+ result.probe = await call('probe');+ assert(result.probe.hasWebGPU, 'Hardware WebGPU required');+ assert(!result.probe.isFallbackAdapter, 'Software WebGPU cannot validate prefill speed');result.adapter = await page.evaluate(async () => {const adapter = await navigator.gpu.requestAdapter({ powerPreference: 'high-performance' });return { vendor: adapter?.info.vendor, architecture: adapter?.info.architecture };@@ -161,11 +177,14 @@ try {}const type = vision ? 'multimodal-text-chat' : 'text-chat';await call(type, { modelId: model, dtype, device: 'webgpu', messages: [{ role: 'user', content: 'Hi' }], options: { maxTokens: 4 } });+ // Only the vision path slices its prefill and decodes greedily.+ const runSlices = vision ? slices : [null];for (const prompt of prompts) {const answers = new Map();- for (const slice of slices) {+ for (const slice of runSlices) {const started = Date.now();- const reply = await call(type, { modelId: model, dtype, device: 'webgpu', messages: prompt.messages, options: { maxTokens, prefillSliceTokens: slice } });+ const options = { maxTokens, ...(slice == null ? {} : { prefillSliceTokens: slice }) };+ const reply = await call(type, { modelId: model, dtype, device: 'webgpu', messages: prompt.messages, options });const seconds = (Date.now() - started) / 1000;const run = { prompt: prompt.name, slice, seconds, secondsPer1kPromptTokens: +(seconds / (prompt.tokens / 1000)).toFixed(3), content: reply.content };result.runs.push(run);@@ -175,11 +194,11 @@ try {result.failures.push({ prompt: prompt.name, slice, check: 'prefill-speed', secondsPer1kPromptTokens: run.secondsPer1kPromptTokens });}}- const reference = answers.get(slices[0]);+ const reference = answers.get(runSlices[0]);for (const [slice, content] of answers) {if (content === reference) continue;const common = commonPrefixWords(reference, content);- const divergence = { prompt: prompt.name, slice, reference: slices[0], commonPrefixWords: common, referenceWords: words(reference).length };+ const divergence = { prompt: prompt.name, slice, reference: runSlices[0], commonPrefixWords: common, referenceWords: words(reference).length };result.divergences.push(divergence);if (common < Math.min(minCommonWords, words(reference).length)) result.failures.push({ ...divergence, check: 'early-divergence' });}