Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Long-prompt check: require hardware WebGPU, add deadlines, time text models only

Claudee03345556439
diff --git a/docs/webgpu-models.md b/docs/webgpu-models.md
index 726da9160..fb164945c 100644
--- a/docs/webgpu-models.md
+++ b/docs/webgpu-models.md
@@ -76,14 +76,28 @@ Do all of these on the exact files you will publish.
--profile <cached profile> --offline --lengths 1000,4000,8000 --slices 0,512
```
- It prints wall time and seconds per 1k prompt tokens for each prompt, and
- fails if the greedy answer changes between slice sizes. Seconds per 1k
- tokens should stay roughly flat and around a second or less on a desktop
- GPU. Tens of seconds per 1k tokens that scale linearly mean a CPU
- fallback. Reference for rewritten Bonsai 2 2B on an RTX 5090 (48 new
- tokens, including decode): 3k tokens in 2.5 s and 8k in 5.6 s in one pass,
- 7.8 s in 512-token slices. In a decoder-only benchmark, 4B prefilled 8,192
- tokens in 16 s. The original Scan graph took 128 s for 1,024 tokens.
+ It prints wall time and seconds per 1k prompt tokens for each prompt. It
+ rejects software WebGPU adapters, and it fails when a text prompt of 4k or
+ more tokens takes longer than `--max-seconds-per-1k` (default 10). A
+ CPU fallback costs 100 seconds or more per 1k tokens.
+ - With `--vision`, it also compares each sliced answer with the first
+ `--slices` value. It fails if they diverge within the first eight words,
+ which is what a broken cache handoff does. A later divergence is reported
+ but expected (see "Sliced prefill" below).
+ - Without `--vision`, each prompt runs once, for timing only.
+
+ Seconds per 1k tokens should stay roughly flat, around a second on a
+ desktop GPU. Reference for rewritten Bonsai 2 2B Vision on an RTX 5090
+ (48 new tokens, including decode):
+
+ | Prompt tokens | One pass | 512-token slices | 256-token slices |
+ |---|---|---|---|
+ | 1k | 1.3 s | 1.4 s | 1.4 s |
+ | 4k | 3.3 s | 4.0 s | 4.7 s |
+ | 8k | 5.5 s | 7.7 s | 10.0 s |
+
+ In a decoder-only benchmark, 4B prefilled 8,192 tokens in 16 s. The
+ original Scan graph took 128 s for 1,024 tokens.
4. **Production harness.** Run `scripts/check-bonsai-webgpu.mjs` online and
`--offline` (with `--vision --image-history` for vision models) and record
the evidence with `scripts/models/bonsai/record_browser_validation.py`.
@@ -124,12 +138,22 @@ Use the tools in `scripts/models/bonsai/`:
- **Sliced prefill.** For the Bonsai 2 Vision presets the worker prefills
long prompts in 512-token slices that carry the recurrent, conv and KV
- cache, then lets `generate()` continue from that cache. Greedy answers are
- identical to a single pass. Each extra slice adds roughly one decoder run of
- fixed overhead (about 0.14 s on an RTX 5090). The `chrome.storage.local` key
- `webgpuPrefillSliceTokens` overrides the slice (64–8192, `0` disables).
- Apply the same pattern to any new model that serves long prompts on shared
- GPUs.
+ cache, then lets `generate()` continue from that cache.
+ - Answers match a single pass up to FP16 rounding. In the RTX 5090 runs,
+ most sliced greedy answers were identical over 48 tokens. The rest
+ matched for the first 30–39 words, then picked a different but
+ equivalent phrasing: slicing changes the order of floating-point sums,
+ which can flip a near-tied token. The results were reproducible run to
+ run, and a smaller slice was not always further off.
+ - A broken cache handoff looks different: the answer loses the prompt right
+ away.
+ - Each extra slice adds roughly one decoder run of fixed overhead (about
+ 0.14 s on an RTX 5090).
+
+ - The `chrome.storage.local` key `webgpuPrefillSliceTokens` overrides the
+ slice (64–8192, `0` disables).
+ - Apply the same pattern to any new model that serves long prompts on
+ shared GPUs.
- **Cache-only inference.** Chat never downloads. Only explicit download or
preload flows fetch model files.
diff --git a/scripts/check-webgpu-long-prompts.mjs b/scripts/check-webgpu-long-prompts.mjs
index deb6adc11..fb8b69b90 100644
--- a/scripts/check-webgpu-long-prompts.mjs
+++ b/scripts/check-webgpu-long-prompts.mjs
@@ -12,14 +12,18 @@
// (default 8). A broken cache handoff loses the prompt and diverges at once,
// while FP16 rounding differences between slice sizes can change a near-tied
// token later in a long greedy answer; the common prefix is reported either
-// way. Slicing applies to the Bonsai 2 Vision path. See docs/webgpu-models.md.
+// way. Prefill slicing applies only to the vision path (Bonsai 2 Vision), so
+// without --vision each prompt runs once, for timing only: some text presets
+// sample, and their answers are not comparable. Software (fallback) WebGPU
+// adapters are rejected. See docs/webgpu-models.md.
//
// node scripts/check-webgpu-long-prompts.mjs --model webbrain-one/Ternary-Bonsai-2-2B-ONNX-WebGPU-Vision \
// --vision --profile .build/bonsai/chunked-2b-profile --offline --lengths 1000,4000,8000 --slices 0,512
//
// Options mirror check-bonsai-webgpu.mjs: --local-repo owner/repo=dir serves
// unpublished files, --offline requires a cached model and blocks Hugging Face,
-// --adapter-luid high,low (decimal) selects a Windows GPU.
+// --adapter-luid high,low (decimal) selects a Windows GPU, and --timeout-ms
+// sets the per-operation deadline (default 30 minutes).
import assert from 'node:assert/strict';
import fs from 'node:fs/promises';
import { createReadStream } from 'node:fs';
@@ -42,6 +46,7 @@ const slices = option('--slices', '0,512').split(',').map(Number);
const maxTokens = Number(option('--max-tokens', 48));
const maxSecondsPer1k = Number(option('--max-seconds-per-1k', 10));
const minCommonWords = Number(option('--min-common-words', 8));
+const timeout = Number(option('--timeout-ms', 30 * 60_000));
const profile = path.resolve(option('--profile', '.build/webgpu-long-prompts/profile'));
const output = path.resolve(option('--output', '.build/webgpu-long-prompts/result.json'));
const localRepos = {};
@@ -117,22 +122,33 @@ try {
worker.onmessage = ({ data }) => {
if (data.type === 'harness-ready') ready();
else if (data.id && pending.has(data.id)) {
- const { resolve, reject } = pending.get(data.id); pending.delete(data.id);
+ const { resolve, reject, timer } = pending.get(data.id);
+ clearTimeout(timer); pending.delete(data.id);
if (data.ok === false) reject(new Error(data.error)); else resolve(data);
}
};
- worker.onerror = event => failed(new Error(event.message));
- window.workerCall = (type, payload) => new Promise((resolve, reject) => {
- const requestId = ++id; pending.set(requestId, { resolve, reject }); worker.postMessage({ id: requestId, type, payload });
+ worker.onerror = event => {
+ failed(new Error(event.message));
+ for (const { reject, timer } of pending.values()) { clearTimeout(timer); reject(new Error(event.message)); }
+ pending.clear();
+ };
+ window.workerCall = (type, payload, timeout) => new Promise((resolve, reject) => {
+ const requestId = ++id;
+ const timer = setTimeout(() => { pending.delete(requestId); reject(new Error(`Timeout: ${type}`)); }, timeout);
+ pending.set(requestId, { resolve, reject, timer });
+ worker.postMessage({ id: requestId, type, payload });
});
}));
const origin = new URL(page.url()).origin;
- const call = (type, payload = {}) => page.evaluate(({ type, payload }) => window.workerCall(type, payload), { type, payload });
+ const call = (type, payload = {}) => page.evaluate(({ type, payload, timeout }) => window.workerCall(type, payload, timeout), { type, payload, timeout });
await call('init', {
transformersUrl: `${origin}/extension/vendor/transformers/transformers.web.js`,
wasmMjsUrl: `${origin}/extension/vendor/transformers/ort-wasm-simd-threaded.asyncify.mjs`,
wasmUrl: `${origin}/extension/vendor/transformers/ort-wasm-simd-threaded.asyncify.wasm`,
});
+ result.probe = await call('probe');
+ assert(result.probe.hasWebGPU, 'Hardware WebGPU required');
+ assert(!result.probe.isFallbackAdapter, 'Software WebGPU cannot validate prefill speed');
result.adapter = await page.evaluate(async () => {
const adapter = await navigator.gpu.requestAdapter({ powerPreference: 'high-performance' });
return { vendor: adapter?.info.vendor, architecture: adapter?.info.architecture };
@@ -161,11 +177,14 @@ try {
}
const type = vision ? 'multimodal-text-chat' : 'text-chat';
await call(type, { modelId: model, dtype, device: 'webgpu', messages: [{ role: 'user', content: 'Hi' }], options: { maxTokens: 4 } });
+ // Only the vision path slices its prefill and decodes greedily.
+ const runSlices = vision ? slices : [null];
for (const prompt of prompts) {
const answers = new Map();
- for (const slice of slices) {
+ for (const slice of runSlices) {
const started = Date.now();
- const reply = await call(type, { modelId: model, dtype, device: 'webgpu', messages: prompt.messages, options: { maxTokens, prefillSliceTokens: slice } });
+ const options = { maxTokens, ...(slice == null ? {} : { prefillSliceTokens: slice }) };
+ const reply = await call(type, { modelId: model, dtype, device: 'webgpu', messages: prompt.messages, options });
const seconds = (Date.now() - started) / 1000;
const run = { prompt: prompt.name, slice, seconds, secondsPer1kPromptTokens: +(seconds / (prompt.tokens / 1000)).toFixed(3), content: reply.content };
result.runs.push(run);
@@ -175,11 +194,11 @@ try {
result.failures.push({ prompt: prompt.name, slice, check: 'prefill-speed', secondsPer1kPromptTokens: run.secondsPer1kPromptTokens });
}
}
- const reference = answers.get(slices[0]);
+ const reference = answers.get(runSlices[0]);
for (const [slice, content] of answers) {
if (content === reference) continue;
const common = commonPrefixWords(reference, content);
- const divergence = { prompt: prompt.name, slice, reference: slices[0], commonPrefixWords: common, referenceWords: words(reference).length };
+ const divergence = { prompt: prompt.name, slice, reference: runSlices[0], commonPrefixWords: common, referenceWords: words(reference).length };
result.divergences.push(divergence);
if (common < Math.min(minCommonWords, words(reference).length)) result.failures.push({ ...divergence, check: 'early-divergence' });
}