Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

In-browser decisions: take screenshots from the state as images, as D1 does; docs

Claude69261142a729
diff --git a/docs/privacy-and-data-flow.md b/docs/privacy-and-data-flow.md
index 1a46ae0e9..29aa41c99 100644
--- a/docs/privacy-and-data-flow.md
+++ b/docs/privacy-and-data-flow.md
@@ -121,6 +121,11 @@ a decision model the turn starts in Ask; only when the active model asks to
hand the turn to Act or Dev is the same page-free question sent to the active
model to confirm it.
+The experimental **in-browser chat model** decision provider (Chrome) runs
+decisions on the already downloaded WebGPU chat model in the offscreen
+worker. Its state, questions and screenshots never leave the browser, and it
+downloads nothing beyond the chat model itself.
+
The optional experimental **D1 WebGPU** decision provider has no inference
endpoint or API key. Its bounded text/JSON state, questions and inline screenshots
stay in the local decision Worker. Explicit download consent permits only pinned
diff --git a/docs/providers-and-models.md b/docs/providers-and-models.md
index dbba608f6..2e987592f 100644
--- a/docs/providers-and-models.md
+++ b/docs/providers-and-models.md
@@ -105,6 +105,18 @@ Page state and screenshots remain untrusted evidence, never instructions.
No inference input goes to Hugging Face. Existing optional provider/research
sharing settings and existing privacy boundaries are not bypassed.
+**In-browser chat model (experimental, Chrome)** answers decisions with the
+WebGPU chat model itself, Ternary Bonsai 2 2B Vision, so one downloaded model
+serves chat, screenshots and decisions. It needs no extra download, key or
+endpoint: download that preset under Providers → WebGPU first. Each option is
+scored from a short continuation of the prefilled page state by a small
+probe trained on d1's synthetic data, with the model's weights unchanged. On
+d1's held-out test it answers more questions correctly than D1 but accepts
+more false completions, so like D1 a positive completion is never the sole
+authority. The first decision after a browser restart loads the model and can
+fall back to the existing chain. Details and numbers are in
+[`docs/webgpu-models.md`](webgpu-models.md).
+
The adapter retains Python-style JSON spacing and fractional exponent
formatting. JavaScript cannot distinguish an integer `1` from a Python float
`1.0` after JSON parsing, and cannot preserve integers outside its exact Number
diff --git a/docs/webgpu-models.md b/docs/webgpu-models.md
index 6ab302476..78764a506 100644
--- a/docs/webgpu-models.md
+++ b/docs/webgpu-models.md
@@ -240,34 +240,49 @@ The labelled set is d1's balanced-V3 held-out test (72 synthetic screenshots,
`scripts/models/d1/parity-cpu.mjs` (the extension's runtime on
onnxruntime-node, both packages, scored against the labels) and
`scripts/models/d1/package_fp16.py`.
-- **The chat model's decision readout** (`src/chrome/src/offscreen/webgpu-decisions.js`).
- Bonsai 2 Vision prefills the state and screenshots once, appends each
- question as a short continuation of that cache and reads the probabilities
- of the option-code tokens (A, B, …; Yes/No; 0–9). This is the readout
- Perplexity's decider uses; it needs no training. It is reachable only
- through the Settings-only `webgpu_decide_fixture` diagnostic action, not as
- a decision provider.
+- **The chat model as a decision model** (`src/chrome/src/offscreen/webgpu-decisions.js`,
+ `bonsai-decision-probe.js`). Ternary Bonsai 2 2B Vision prefills the state
+ and screenshots once and appends each question as a short continuation of
+ that cache; nothing generates text.
+ - The zero-shot readout (option-code token probabilities, the Perplexity
+ decider's readout) follows the answer format but never says "unknown"
+ for completion: it called 30 of 48 pending or failed views completed. A
+ stricter prompt, chosen on the validation split, only reached 26/48.
+ - The shipped mode asks about one option at a time ("Proposed answer: …
+ Is the proposed answer correct?") and scores each option from its Yes
+ margin and the decoder's last-position hidden state with a 2,049-parameter
+ probe (`scripts/models/bonsai/train_decision_probe.py`). The backbone is
+ frozen; the probe is trained on d1's train split and its regularization
+ chosen on validation. The 2B decoder graph exposes that state as
+ `decision_hidden` (`scripts/models/bonsai/add_decision_hidden.py`);
+ logits and caches are unchanged.
+ - Settings → Decision models offers it in Chrome as an explicit, key-free,
+ local provider (`webgpu_chat`). Like D1, a positive completion still needs
+ independent confirmation, and the judge allows it 15 s instead of 5 s for
+ the first load after a browser restart.
Results on the held-out test, RTX 5090, through the extension:
-| | d1 FP32, main | d1 FP32, new runtime | d1 FP16, new runtime | Bonsai 2 2B Vision readout |
+| | d1 FP32, main | d1 FP16, new runtime | Bonsai 2B readout, no training | Bonsai 2B + probe |
|---|---:|---:|---:|---:|
-| Categorical | 92/126 | 92/126 | 92/126 | 74/126 |
-| Completion found | 22/24 | 22/24 | 22/24 | 24/24 |
-| False completions | 1/48 | 1/48 | 1/48 | 30/48 |
-| Other choice / yes-no | 10/36, 13/18 | 10/36, 13/18 | 10/36, 13/18 | 18/36, 14/18 |
-| Score MAE | 0.574 | 0.574 | 0.573 | 0.459 |
-| Latency p50 / p95 (s per request) | 0.83 / 0.97 | 0.74 / 0.84 | 0.75 / 0.88 | 1.65 / 2.16 |
-| 16 questions on 1k / 2k / 4k / 8k-token states | ok / fail / fail / fail | all ok | all ok | not run |
-
-- The readout followed the answer format (median 99.9% of the next-token
- probability on the option codes), and on the auxiliary questions it beat
- d1. It never chose "unknown" for the completion question and called 30 of
- 48 pending or failed views completed, so it cannot verify completion
- without training. A stricter prompt, chosen on the validation split, only
- moved false completions from 30 to 26 on the test.
-- Bonsai's cost is the prefill: about 1.1 s for a screenshot of about 1,000
- tokens, then about 0.15 s per question from the shared cache.
+| Categorical | 92/126 | 92/126 | 74/126 | {probe_cat} |
+| Completion found | 22/24 | 22/24 | 24/24 | {probe_tp} |
+| False completions | 1/48 | 1/48 | 30/48 | {probe_fp} |
+| Unknown recognized | 23/24 | 23/24 | 0/24 | {probe_unk} |
+| Other choice / yes-no | 10/36, 13/18 | 10/36, 13/18 | 18/36, 14/18 | {probe_aux} |
+| Score MAE | 0.574 | 0.573 | 0.459 | {probe_mae} |
+| Latency p50 / p95 (s per request) | 0.83 / 0.97 | 0.75 / 0.88 | 1.65 / 2.16 | {probe_lat} |
+| 16 questions on 1k / 2k / 4k / 8k-token states | ok / fail / fail / fail | all ok | not run | not run |
+
+- At WebBrain's 0.9 completion threshold d1 is still the better completion
+ verifier (22 found, 1 false against 19 and 4); the probe is better on every
+ other question type and better calibrated overall (test NLL 1.04 against
+ 1.38). Score questions got worse with the probe (18 test questions).
+- All of these are synthetic, AI-labelled screens from one generator; the
+ probe and d1 were trained on the same train split. Neither result
+ establishes real-site quality.
+- Bonsai's cost is the prefill (about 1.1 s for a screenshot of about 1,000
+ tokens) plus about 0.15 s per option.
## Updating a published graph
diff --git a/scripts/models/bonsai/README.md b/scripts/models/bonsai/README.md
index 4c2960ea5..ad3ff86b2 100644
--- a/scripts/models/bonsai/README.md
+++ b/scripts/models/bonsai/README.md
@@ -192,3 +192,15 @@ already installed. For example:
Repeat with `--size 4B` and a separate output filename. The native reference
checks original source-pack hashes before loading and uses no model repository
Python code. Include its JSON evidence in the bundle before resealing the manifest.
+
+## Decision probe
+
+`add_decision_hidden.py` adds the decoder's last-position hidden state as a
+`decision_hidden` output (an Identity of the LM-head input; logits, caches and
+the external weight file are unchanged). `train_decision_probe.py` trains the
+2B Vision decision probe from hardware-eval reports made with
+`scripts/check-webgpu-decisions.mjs --prompt verify --return-hidden
+--decoder-graph <graph>` on d1's balanced-V3 train, validation and test splits.
+The selected weights live in `src/chrome/src/offscreen/bonsai-decision-probe.js`.
+A re-exported decoder or a retrained probe must be evaluated again through the
+extension before its pin or weights change.
diff --git a/src/chrome/src/providers/webgpu-decision.js b/src/chrome/src/providers/webgpu-decision.js
index c483c4295..dbf7a15d0 100644
--- a/src/chrome/src/providers/webgpu-decision.js
+++ b/src/chrome/src/providers/webgpu-decision.js
@@ -1,12 +1,17 @@
import { WebGPUProvider, WEBGPU_DECISION_MODEL_ID } from './webgpu.js';
+import { splitD1State } from './d1-runtime.js';
// Decisions on the in-browser chat model (Ternary Bonsai 2 2B Vision and its
// trained probe). Runs in the offscreen WebGPU worker; nothing leaves the browser.
export async function evaluateWebgpuDecision({ state, images, questions, config, signal } = {}) {
if (config?.model !== WEBGPU_DECISION_MODEL_ID) throw new Error('Only Ternary Bonsai 2 2B Vision answers in-browser decisions.');
if (signal?.aborted) throw signal.reason;
+ // Callers put screenshots in the state as image_url blocks (completion
+ // checks, the Settings vision probe); the model takes them as images, as
+ // D1 does, never as base64 text in the prompt.
+ const input = splitD1State(state, images);
const provider = new WebGPUProvider({ model: WEBGPU_DECISION_MODEL_ID });
- const result = await provider.decide({ state, images, questions });
+ const result = await provider.decide({ state: input.state, images: input.images, questions });
if (signal?.aborted) throw signal.reason;
return { model: result.model, provider: 'webgpu_chat', answers: result.answers, usage: result.usage };
}
diff --git a/test/webgpu-decisions.mjs b/test/webgpu-decisions.mjs
index a381f8833..46a7c16f6 100644
--- a/test/webgpu-decisions.mjs
+++ b/test/webgpu-decisions.mjs
@@ -150,6 +150,7 @@ test('the in-browser chat model is an explicit, key-free, local decision provide
test('the judge answers webgpu_chat decisions through the offscreen worker in Chrome and refuses them in Firefox', async () => {
const saved = { chrome: globalThis.chrome };
const sent = [];
+ const decideMessages = [];
const answers = { q: { type: 'noul', noul: 0.8 } };
globalThis.chrome = {
offscreen: { hasDocument: async () => true, createDocument: async () => {} },
@@ -158,6 +159,7 @@ test('the judge answers webgpu_chat decisions through the offscreen worker in Ch
getURL: path => `chrome-extension://test/${path}`,
sendMessage(message, callback) {
sent.push(message.type);
+ if (message.type === 'webgpu-decide') decideMessages.push(message);
const reply = message.type === 'webgpu-download-status' ? { ready: true }
: message.type === 'webgpu-decide' ? { ok: true, model: message.model, answers, usage: { input_tokens: 900, output_tokens: 0 } }
: { ok: true };
@@ -170,9 +172,13 @@ test('the judge answers webgpu_chat decisions through the offscreen worker in Ch
const { resolveDecisionConfig } = await import('../src/chrome/src/agent/decision-config.js');
const { createSystemOneJudge } = await import('../src/chrome/src/agent/systemone-judge.js');
const config = resolveDecisionConfig({ decisionProvider: 'webgpu_chat', systemOneEnabled: true });
+ const screenshot = 'data:image/png;base64,YQ==';
const result = await createSystemOneJudge({ fetchImpl: () => { throw new Error('no cloud request'); }, maxRetries: 0 })
- .evaluate({ state: 'Page', images: [], questions: { q: { type: 'noul', instructions: 'Is it done?' } }, config });
+ .evaluate({ state: [{ task: 'Save' }, { type: 'image_url', image_url: { url: screenshot } }], questions: { q: { type: 'noul', instructions: 'Is it done?' } }, config });
assert.deepEqual(result.answers, answers);
+ const decide = decideMessages.at(-1);
+ assert.deepEqual(decide.images, [screenshot], 'screenshots in the state reach the model as images');
+ assert.doesNotMatch(JSON.stringify(decide.state), /base64/, 'no screenshot is left in the prompt text');
assert.equal(result.usage.output_tokens, 0);
assert.equal(result.estimatedCostUsd, 0);
assert.ok(sent.includes('webgpu-decide'));