Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

d1: pin the FP16 package revision with hardware parity; document decision results

Claude9dd40289b568
diff --git a/docs/webgpu-models.md b/docs/webgpu-models.md
index 0bb148053..6ab302476 100644
--- a/docs/webgpu-models.md
+++ b/docs/webgpu-models.md
@@ -205,6 +205,70 @@ Use the tools in `scripts/models/bonsai/`:
- **Cache-only inference.** Chat never downloads. Only explicit download or
preload flows fetch model files.
+## Decisions on WebGPU
+
+Two on-device paths answer typed decision questions (`choice`, `noul`,
+`score`) without generating text. `scripts/check-webgpu-decisions.mjs` runs
+both through the real extension on the same labelled questions:
+
+```bash
+node scripts/check-webgpu-decisions.mjs --test <final_test.json> --images <dir> \
+ --engines d1,bonsai --stress --download --profile <fresh profile> \
+ --adapter-luid 0,82094 --output result.json
+```
+
+The labelled set is d1's balanced-V3 held-out test (72 synthetic screenshots,
+144 questions). It is not in this repository.
+
+- **d1 (the decision model).** A bidirectional encoder that scores option
+ markers, published as
+ [`webbrain-one/d1-browser-decision-fp16`](https://huggingface.co/webbrain-one/d1-browser-decision-fp16).
+ - Its graphs pass the static audit, but its six full-attention layers are
+ dense: each row materializes a 16 × length × length FP32 score tensor.
+ The old batching budgeted tokens (65,536), not that tensor, and every
+ question is its own row with the whole state. On the 5090, 16 questions
+ on a ~2k-token state needed a 2.6 GB buffer and failed; 4k overflowed.
+ - `d1-runtime.js` now bounds batches by the score tensor (256 MiB, and never
+ more than the adapter's storage-binding limit), caps text-only rows at
+ 2,048 tokens, runs tiled screenshots through the vision tower in crop
+ groups under the same budget, and keeps screenshot features on the GPU
+ between graphs.
+ - The FP16 package stores the FP32 release's weights rounded once and
+ computes in FP32. Against FP32 it agrees on 144/144 decisions with a
+ largest probability difference of 0.010, on the CPU and on WebGPU.
+ - Package checks for a re-export: `scripts/models/d1/convert_fp16.py`,
+ `scripts/models/d1/parity-cpu.mjs` (the extension's runtime on
+ onnxruntime-node, both packages, scored against the labels) and
+ `scripts/models/d1/package_fp16.py`.
+- **The chat model's decision readout** (`src/chrome/src/offscreen/webgpu-decisions.js`).
+ Bonsai 2 Vision prefills the state and screenshots once, appends each
+ question as a short continuation of that cache and reads the probabilities
+ of the option-code tokens (A, B, …; Yes/No; 0–9). This is the readout
+ Perplexity's decider uses; it needs no training. It is reachable only
+ through the Settings-only `webgpu_decide_fixture` diagnostic action, not as
+ a decision provider.
+
+Results on the held-out test, RTX 5090, through the extension:
+
+| | d1 FP32, main | d1 FP32, new runtime | d1 FP16, new runtime | Bonsai 2 2B Vision readout |
+|---|---:|---:|---:|---:|
+| Categorical | 92/126 | 92/126 | 92/126 | 74/126 |
+| Completion found | 22/24 | 22/24 | 22/24 | 24/24 |
+| False completions | 1/48 | 1/48 | 1/48 | 30/48 |
+| Other choice / yes-no | 10/36, 13/18 | 10/36, 13/18 | 10/36, 13/18 | 18/36, 14/18 |
+| Score MAE | 0.574 | 0.574 | 0.573 | 0.459 |
+| Latency p50 / p95 (s per request) | 0.83 / 0.97 | 0.74 / 0.84 | 0.75 / 0.88 | 1.65 / 2.16 |
+| 16 questions on 1k / 2k / 4k / 8k-token states | ok / fail / fail / fail | all ok | all ok | not run |
+
+- The readout followed the answer format (median 99.9% of the next-token
+ probability on the option codes), and on the auxiliary questions it beat
+ d1. It never chose "unknown" for the completion question and called 30 of
+ 48 pending or failed views completed, so it cannot verify completion
+ without training. A stricter prompt, chosen on the validation split, only
+ moved false completions from 30 to 26 on the test.
+- Bonsai's cost is the prefill: about 1.1 s for a screenshot of about 1,000
+ tokens, then about 0.15 s per question from the shared cache.
+
## Updating a published graph
Prefer a new repository or file name for a changed graph. If a file must
diff --git a/scripts/models/d1/package_fp16.py b/scripts/models/d1/package_fp16.py
index a42464cc3..8a6d9421c 100644
--- a/scripts/models/d1/package_fp16.py
+++ b/scripts/models/d1/package_fp16.py
@@ -109,6 +109,8 @@ def main():
provenance = out / "provenance"
provenance.mkdir()
shutil.copy2(args.converted / "conversion-report.json", provenance / "conversion-report.json")
+ if (args.converted / "hardware-eval.json").exists():
+ shutil.copy2(args.converted / "hardware-eval.json", provenance / "hardware-eval.json")
shutil.copy2(ROOT / "scripts/models/d1/convert_fp16.py", provenance / "convert_fp16.py")
shutil.copy2(ROOT / "scripts/models/d1/parity-cpu.mjs", provenance / "parity-cpu.mjs")
shutil.copy2(ROOT / "scripts/models/d1/decision-metrics.mjs", provenance / "decision-metrics.mjs")
diff --git a/src/chrome/src/providers/d1-config.js b/src/chrome/src/providers/d1-config.js
index 635368344..e952956b5 100644
--- a/src/chrome/src/providers/d1-config.js
+++ b/src/chrome/src/providers/d1-config.js
@@ -3,7 +3,7 @@ export const D1_CONSENT_VERSION = 1;
export const D1_CONSENT_KEY = 'd1WebgpuConsentVersion';
export const D1_CACHE_NAME = 'webbrain-d1-fp16-v1';
// Immutable release package; downloaded data is verified before local loading.
-export const D1_RELEASE = Object.freeze({ revision: 'd83c32b1db90861948fa5ce2bf926b46445fc7bd', manifestSha256: '3e0f61917b8e21cb2b7603cbc37680d225934cabb35e9dbd0db0c23c1aebaf1c' });
+export const D1_RELEASE = Object.freeze({ revision: 'd4916f0c47d46f16e7876e425eb12a122671d3eb', manifestSha256: 'a94fb80ba47cc6c3e05cc3dbda45f9f2b99f4f1999b0770a0a09ba59e0c861ed' });
export const D1_REQUIRED_FILES = Object.freeze(['config.json', 'tokenizer.json', 'tokenizer_config.json', 'tiling-ratios.json', 'onnx/decision.onnx', 'onnx/decision.data', 'onnx/vision.onnx', 'onnx/vision.data', 'onnx/projector.onnx', 'onnx/projector.data']);
export function d1BaseURL() {
diff --git a/src/firefox/src/providers/d1-config.js b/src/firefox/src/providers/d1-config.js
index 635368344..e952956b5 100644
--- a/src/firefox/src/providers/d1-config.js
+++ b/src/firefox/src/providers/d1-config.js
@@ -3,7 +3,7 @@ export const D1_CONSENT_VERSION = 1;
export const D1_CONSENT_KEY = 'd1WebgpuConsentVersion';
export const D1_CACHE_NAME = 'webbrain-d1-fp16-v1';
// Immutable release package; downloaded data is verified before local loading.
-export const D1_RELEASE = Object.freeze({ revision: 'd83c32b1db90861948fa5ce2bf926b46445fc7bd', manifestSha256: '3e0f61917b8e21cb2b7603cbc37680d225934cabb35e9dbd0db0c23c1aebaf1c' });
+export const D1_RELEASE = Object.freeze({ revision: 'd4916f0c47d46f16e7876e425eb12a122671d3eb', manifestSha256: 'a94fb80ba47cc6c3e05cc3dbda45f9f2b99f4f1999b0770a0a09ba59e0c861ed' });
export const D1_REQUIRED_FILES = Object.freeze(['config.json', 'tokenizer.json', 'tokenizer_config.json', 'tiling-ratios.json', 'onnx/decision.onnx', 'onnx/decision.data', 'onnx/vision.onnx', 'onnx/vision.data', 'onnx/projector.onnx', 'onnx/projector.data']);
export function d1BaseURL() {