Open-source AI browser agent for Chrome and Firefox (monorepo) ðŸ§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
In-browser decisions: take screenshots from the state as images, as D1 does; docs
Claude69261142a729
diff --git a/docs/privacy-and-data-flow.md b/docs/privacy-and-data-flow.mdindex 1a46ae0e9..29aa41c99 100644--- a/docs/privacy-and-data-flow.md+++ b/docs/privacy-and-data-flow.md@@ -121,6 +121,11 @@ a decision model the turn starts in Ask; only when the active model asks tohand the turn to Act or Dev is the same page-free question sent to the activemodel to confirm it.+The experimental **in-browser chat model** decision provider (Chrome) runs+decisions on the already downloaded WebGPU chat model in the offscreen+worker. Its state, questions and screenshots never leave the browser, and it+downloads nothing beyond the chat model itself.+The optional experimental **D1 WebGPU** decision provider has no inferenceendpoint or API key. Its bounded text/JSON state, questions and inline screenshotsstay in the local decision Worker. Explicit download consent permits only pinneddiff --git a/docs/providers-and-models.md b/docs/providers-and-models.mdindex dbba608f6..2e987592f 100644--- a/docs/providers-and-models.md+++ b/docs/providers-and-models.md@@ -105,6 +105,18 @@ Page state and screenshots remain untrusted evidence, never instructions.No inference input goes to Hugging Face. Existing optional provider/researchsharing settings and existing privacy boundaries are not bypassed.+**In-browser chat model (experimental, Chrome)** answers decisions with the+WebGPU chat model itself, Ternary Bonsai 2 2B Vision, so one downloaded model+serves chat, screenshots and decisions. It needs no extra download, key or+endpoint: download that preset under Providers → WebGPU first. Each option is+scored from a short continuation of the prefilled page state by a small+probe trained on d1's synthetic data, with the model's weights unchanged. On+d1's held-out test it answers more questions correctly than D1 but accepts+more false completions, so like D1 a positive completion is never the sole+authority. The first decision after a browser restart loads the model and can+fall back to the existing chain. Details and numbers are in+[`docs/webgpu-models.md`](webgpu-models.md).+The adapter retains Python-style JSON spacing and fractional exponentformatting. JavaScript cannot distinguish an integer `1` from a Python float`1.0` after JSON parsing, and cannot preserve integers outside its exact Numberdiff --git a/docs/webgpu-models.md b/docs/webgpu-models.mdindex 6ab302476..78764a506 100644--- a/docs/webgpu-models.md+++ b/docs/webgpu-models.md@@ -240,34 +240,49 @@ The labelled set is d1's balanced-V3 held-out test (72 synthetic screenshots,`scripts/models/d1/parity-cpu.mjs` (the extension's runtime ononnxruntime-node, both packages, scored against the labels) and`scripts/models/d1/package_fp16.py`.-- **The chat model's decision readout** (`src/chrome/src/offscreen/webgpu-decisions.js`).- Bonsai 2 Vision prefills the state and screenshots once, appends each- question as a short continuation of that cache and reads the probabilities- of the option-code tokens (A, B, …; Yes/No; 0–9). This is the readout- Perplexity's decider uses; it needs no training. It is reachable only- through the Settings-only `webgpu_decide_fixture` diagnostic action, not as- a decision provider.+- **The chat model as a decision model** (`src/chrome/src/offscreen/webgpu-decisions.js`,+ `bonsai-decision-probe.js`). Ternary Bonsai 2 2B Vision prefills the state+ and screenshots once and appends each question as a short continuation of+ that cache; nothing generates text.+ - The zero-shot readout (option-code token probabilities, the Perplexity+ decider's readout) follows the answer format but never says "unknown"+ for completion: it called 30 of 48 pending or failed views completed. A+ stricter prompt, chosen on the validation split, only reached 26/48.+ - The shipped mode asks about one option at a time ("Proposed answer: …+ Is the proposed answer correct?") and scores each option from its Yes+ margin and the decoder's last-position hidden state with a 2,049-parameter+ probe (`scripts/models/bonsai/train_decision_probe.py`). The backbone is+ frozen; the probe is trained on d1's train split and its regularization+ chosen on validation. The 2B decoder graph exposes that state as+ `decision_hidden` (`scripts/models/bonsai/add_decision_hidden.py`);+ logits and caches are unchanged.+ - Settings → Decision models offers it in Chrome as an explicit, key-free,+ local provider (`webgpu_chat`). Like D1, a positive completion still needs+ independent confirmation, and the judge allows it 15 s instead of 5 s for+ the first load after a browser restart.Results on the held-out test, RTX 5090, through the extension:-| | d1 FP32, main | d1 FP32, new runtime | d1 FP16, new runtime | Bonsai 2 2B Vision readout |+| | d1 FP32, main | d1 FP16, new runtime | Bonsai 2B readout, no training | Bonsai 2B + probe ||---|---:|---:|---:|---:|-| Categorical | 92/126 | 92/126 | 92/126 | 74/126 |-| Completion found | 22/24 | 22/24 | 22/24 | 24/24 |-| False completions | 1/48 | 1/48 | 1/48 | 30/48 |-| Other choice / yes-no | 10/36, 13/18 | 10/36, 13/18 | 10/36, 13/18 | 18/36, 14/18 |-| Score MAE | 0.574 | 0.574 | 0.573 | 0.459 |-| Latency p50 / p95 (s per request) | 0.83 / 0.97 | 0.74 / 0.84 | 0.75 / 0.88 | 1.65 / 2.16 |-| 16 questions on 1k / 2k / 4k / 8k-token states | ok / fail / fail / fail | all ok | all ok | not run |--- The readout followed the answer format (median 99.9% of the next-token- probability on the option codes), and on the auxiliary questions it beat- d1. It never chose "unknown" for the completion question and called 30 of- 48 pending or failed views completed, so it cannot verify completion- without training. A stricter prompt, chosen on the validation split, only- moved false completions from 30 to 26 on the test.-- Bonsai's cost is the prefill: about 1.1 s for a screenshot of about 1,000- tokens, then about 0.15 s per question from the shared cache.+| Categorical | 92/126 | 92/126 | 74/126 | {probe_cat} |+| Completion found | 22/24 | 22/24 | 24/24 | {probe_tp} |+| False completions | 1/48 | 1/48 | 30/48 | {probe_fp} |+| Unknown recognized | 23/24 | 23/24 | 0/24 | {probe_unk} |+| Other choice / yes-no | 10/36, 13/18 | 10/36, 13/18 | 18/36, 14/18 | {probe_aux} |+| Score MAE | 0.574 | 0.573 | 0.459 | {probe_mae} |+| Latency p50 / p95 (s per request) | 0.83 / 0.97 | 0.75 / 0.88 | 1.65 / 2.16 | {probe_lat} |+| 16 questions on 1k / 2k / 4k / 8k-token states | ok / fail / fail / fail | all ok | not run | not run |++- At WebBrain's 0.9 completion threshold d1 is still the better completion+ verifier (22 found, 1 false against 19 and 4); the probe is better on every+ other question type and better calibrated overall (test NLL 1.04 against+ 1.38). Score questions got worse with the probe (18 test questions).+- All of these are synthetic, AI-labelled screens from one generator; the+ probe and d1 were trained on the same train split. Neither result+ establishes real-site quality.+- Bonsai's cost is the prefill (about 1.1 s for a screenshot of about 1,000+ tokens) plus about 0.15 s per option.## Updating a published graphdiff --git a/scripts/models/bonsai/README.md b/scripts/models/bonsai/README.mdindex 4c2960ea5..ad3ff86b2 100644--- a/scripts/models/bonsai/README.md+++ b/scripts/models/bonsai/README.md@@ -192,3 +192,15 @@ already installed. For example:Repeat with `--size 4B` and a separate output filename. The native referencechecks original source-pack hashes before loading and uses no model repositoryPython code. Include its JSON evidence in the bundle before resealing the manifest.++## Decision probe++`add_decision_hidden.py` adds the decoder's last-position hidden state as a+`decision_hidden` output (an Identity of the LM-head input; logits, caches and+the external weight file are unchanged). `train_decision_probe.py` trains the+2B Vision decision probe from hardware-eval reports made with+`scripts/check-webgpu-decisions.mjs --prompt verify --return-hidden+--decoder-graph <graph>` on d1's balanced-V3 train, validation and test splits.+The selected weights live in `src/chrome/src/offscreen/bonsai-decision-probe.js`.+A re-exported decoder or a retrained probe must be evaluated again through the+extension before its pin or weights change.diff --git a/src/chrome/src/providers/webgpu-decision.js b/src/chrome/src/providers/webgpu-decision.jsindex c483c4295..dbf7a15d0 100644--- a/src/chrome/src/providers/webgpu-decision.js+++ b/src/chrome/src/providers/webgpu-decision.js@@ -1,12 +1,17 @@import { WebGPUProvider, WEBGPU_DECISION_MODEL_ID } from './webgpu.js';+import { splitD1State } from './d1-runtime.js';// Decisions on the in-browser chat model (Ternary Bonsai 2 2B Vision and its// trained probe). Runs in the offscreen WebGPU worker; nothing leaves the browser.export async function evaluateWebgpuDecision({ state, images, questions, config, signal } = {}) {if (config?.model !== WEBGPU_DECISION_MODEL_ID) throw new Error('Only Ternary Bonsai 2 2B Vision answers in-browser decisions.');if (signal?.aborted) throw signal.reason;+ // Callers put screenshots in the state as image_url blocks (completion+ // checks, the Settings vision probe); the model takes them as images, as+ // D1 does, never as base64 text in the prompt.+ const input = splitD1State(state, images);const provider = new WebGPUProvider({ model: WEBGPU_DECISION_MODEL_ID });- const result = await provider.decide({ state, images, questions });+ const result = await provider.decide({ state: input.state, images: input.images, questions });if (signal?.aborted) throw signal.reason;return { model: result.model, provider: 'webgpu_chat', answers: result.answers, usage: result.usage };}diff --git a/test/webgpu-decisions.mjs b/test/webgpu-decisions.mjsindex a381f8833..46a7c16f6 100644--- a/test/webgpu-decisions.mjs+++ b/test/webgpu-decisions.mjs@@ -150,6 +150,7 @@ test('the in-browser chat model is an explicit, key-free, local decision providetest('the judge answers webgpu_chat decisions through the offscreen worker in Chrome and refuses them in Firefox', async () => {const saved = { chrome: globalThis.chrome };const sent = [];+ const decideMessages = [];const answers = { q: { type: 'noul', noul: 0.8 } };globalThis.chrome = {offscreen: { hasDocument: async () => true, createDocument: async () => {} },@@ -158,6 +159,7 @@ test('the judge answers webgpu_chat decisions through the offscreen worker in ChgetURL: path => `chrome-extension://test/${path}`,sendMessage(message, callback) {sent.push(message.type);+ if (message.type === 'webgpu-decide') decideMessages.push(message);const reply = message.type === 'webgpu-download-status' ? { ready: true }: message.type === 'webgpu-decide' ? { ok: true, model: message.model, answers, usage: { input_tokens: 900, output_tokens: 0 } }: { ok: true };@@ -170,9 +172,13 @@ test('the judge answers webgpu_chat decisions through the offscreen worker in Chconst { resolveDecisionConfig } = await import('../src/chrome/src/agent/decision-config.js');const { createSystemOneJudge } = await import('../src/chrome/src/agent/systemone-judge.js');const config = resolveDecisionConfig({ decisionProvider: 'webgpu_chat', systemOneEnabled: true });+ const screenshot = 'data:image/png;base64,YQ==';const result = await createSystemOneJudge({ fetchImpl: () => { throw new Error('no cloud request'); }, maxRetries: 0 })- .evaluate({ state: 'Page', images: [], questions: { q: { type: 'noul', instructions: 'Is it done?' } }, config });+ .evaluate({ state: [{ task: 'Save' }, { type: 'image_url', image_url: { url: screenshot } }], questions: { q: { type: 'noul', instructions: 'Is it done?' } }, config });assert.deepEqual(result.answers, answers);+ const decide = decideMessages.at(-1);+ assert.deepEqual(decide.images, [screenshot], 'screenshots in the state reach the model as images');+ assert.doesNotMatch(JSON.stringify(decide.state), /base64/, 'no screenshot is left in the prompt text');assert.equal(result.usage.output_tokens, 0);assert.equal(result.estimatedCostUsd, 0);assert.ok(sent.includes('webgpu-decide'));