Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Let the decision model decide a finished task fast

Claude433697504ab3
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 69e3fe18c..9c22c2cbb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -7,6 +7,7 @@ This changelog was generated from the repository Git history and release tags. V
## [Unreleased]
### Fixed
+- Finishing a task no longer waits long on a slow model. The decision model decides when it is sure; when it answers but is not sure, the active model's second look now has eight seconds in all instead of up to ten seconds for each of two looks, and a look that could not finish in the time left is not started. When the decision model is off or unavailable, the active model keeps its full checks. The active model's reading of the page's status messages at that point is bounded to six seconds. Traces now show where a finished task spent its time and what each verification attempt decided, without any page or model text.
- Deleting your own Mastodon post through its "…" menu works again on current Mastodon (4.4): the menu fades in a moment after the click, so WebBrain lost track of which post it belonged to and refused "Delete" ("Sil" on a Turkish server) as not belonging to any post. The menu the agent's own click opens is now kept for up to three seconds while it appears, and the menu on a post's own page counts as that post's, since Mastodon draws it beside the post rather than inside it.
- Deleting an item through its own "…" or "More" menu now works on sites that open that menu elsewhere in the page without linking it to its button, as many apps do. The menu the agent's own click opened, alone and within three seconds, belongs to that item, even while it fades in or opens inside a window already on the page. Before, Delete in such a menu was refused as belonging to no item. A menu opened by a script, by another item's button, or together with a second menu still borrows nothing, and the request still has to name the item or you confirm it on the card.
- Preserve exact editor identity and private values through owned focus and scroll preparation, while permitting the resulting geometry changes.
diff --git a/docs/privacy-and-data-flow.md b/docs/privacy-and-data-flow.md
index e49e15890..2a3846f6f 100644
--- a/docs/privacy-and-data-flow.md
+++ b/docs/privacy-and-data-flow.md
@@ -339,6 +339,14 @@ counts, outcome categories, and safety-status booleans. URL, target labels,
accessibility references, and other page content are excluded in both tiers,
including lossless traces.
+`done_timing` trace notes retain only the milliseconds a completion check spent
+preparing its evidence, reading the page's status texts and reaching a verdict,
+with the verdict's outcome and engine names. Completion verification attempts
+in `system_one` notes retain the engine, modality (`vision` or `ax`), outcome,
+a reason code and their capture and evaluation times; a model's own words, the
+probabilities, the task, the summary and page content are excluded in both
+tiers, including lossless traces.
+
Default trace redaction does not disable `/workflow --save`. While a traced run
is active, the recorder keeps bounded raw tool payloads in memory only. On
successful completion, the agent immediately compiles them into the existing
diff --git a/docs/providers-and-models.md b/docs/providers-and-models.md
index 56266b7fb..4c7e71a22 100644
--- a/docs/providers-and-models.md
+++ b/docs/providers-and-models.md
@@ -92,7 +92,14 @@ existing checks. Strict Secret Mode disables these additional calls.
| Other provider, outsourcing disabled or unconfigured | Active LLM | Existing checks |
Each judge prefers a fresh screenshot and tries AX if images
-are unavailable or inconclusive. Confident pending/failed results require
+are unavailable or inconclusive. The decision model decides when it is sure.
+When it answers but is not sure, the active LLM's second look has eight seconds
+in all, and a look that could not finish in the time left is not started, so a
+slow model cannot hold a finished task; when the decision model is off or
+unavailable, or a D1 success needs confirmation, the active LLM keeps its full
+checks. At completion the active LLM's reading of the page's status texts, when
+the decision model is not sure of them, is bounded to six seconds; a text it did
+not read proves nothing either way. Confident pending/failed results require
recovery. Accepted fresh proof can establish generic publication evidence;
explicit recipient, payment, download, authorization and workflow contracts
remain required. Future votes and moderation outcomes are not certified.
diff --git a/src/chrome/src/agent/agent.js b/src/chrome/src/agent/agent.js
index 9183de435..4dbbbc3b4 100644
--- a/src/chrome/src/agent/agent.js
+++ b/src/chrome/src/agent/agent.js
@@ -268,6 +268,10 @@ const VISION_SUB_CALL_TIMEOUT_MS = 90_000;
const CONTENT_ACTION_TIMEOUT_MS = 60_000;
const RUN_STARTUP_TIMEOUT_MS = 15_000;
const CONTENT_ACTION_RESPONSE_GRACE_MS = 5_000;
+// At completion the active model reads what a page's status texts say only
+// for this long, so a slow model cannot hold a finished task; texts it did
+// not read prove nothing either way.
+const COMPLETION_READING_TIMEOUT_MS = 6_000;
// The agent's own hover may start a transition on a certified native button.
// Its geometry is sampled (read-only) at this interval until it settles, for
// at most this long, before the single press.
@@ -1646,7 +1650,7 @@ export class Agent extends LoopDetector {
const needsProof = !!submit || guard?.requiresSubmission === true
|| (guard?.requiresSubmission == null && guard?.requiresStateChange === true) || !!guard?.siteWorkflow?.job;
if (needsProof && (live.length || observed.length)) {
- await this._readPageTexts(tabId, 'status_message', [...live, ...observed]);
+ await this._readPageTexts(tabId, 'status_message', [...live, ...observed], {}, { mainTimeoutMs: COMPLETION_READING_TIMEOUT_MS });
}
if (pageState && typeof pageState === 'object') {
pageState.successMessages = live.filter(text => this._pageTextReading('status_message', text) === 'success').slice(0, 5);
@@ -2163,7 +2167,7 @@ export class Agent extends LoopDetector {
async _completionIframeEvidence(tabId) {
const frame = await this._completionIframeSnapshot(tabId);
if (Array.isArray(frame?.messages) && frame.messages.length) {
- await this._readPageTexts(tabId, 'status_message', frame.messages);
+ await this._readPageTexts(tabId, 'status_message', frame.messages, {}, { mainTimeoutMs: COMPLETION_READING_TIMEOUT_MS });
}
return frame;
}
@@ -3039,7 +3043,8 @@ export class Agent extends LoopDetector {
? this._workflowTerminalTextPlan(verificationKind, guard.siteWorkflow, pageState, submissionEvidence, messageProbe)
: null;
if (plan?.texts.length) {
- await this._readPageTexts(tabId, 'workflow_state', plan.texts, { state: WORKFLOW_STATES[plan.state] });
+ await this._readPageTexts(tabId, 'workflow_state', plan.texts, { state: WORKFLOW_STATES[plan.state] },
+ { mainTimeoutMs: COMPLETION_READING_TIMEOUT_MS });
}
}
@@ -7320,7 +7325,7 @@ export class Agent extends LoopDetector {
* choice is null where no sure answer was available. Each item's answer is
* remembered on its own, so a later batch asks only about new items.
*/
- async _semanticDecisions(tabId, kind, { request = '', items = [] } = {}, { provider = this._activeProvider(tabId), signal = null, mainModel = true } = {}) {
+ async _semanticDecisions(tabId, kind, { request = '', items = [] } = {}, { provider = this._activeProvider(tabId), signal = null, mainModel = true, mainTimeoutMs = 15000 } = {}) {
const spec = semanticQuestion(kind);
const trusted = String(request ?? '').slice(0, 6000);
const data = (Array.isArray(items) ? items : []).map(item => boundedSemanticData(item));
@@ -7377,7 +7382,7 @@ export class Agent extends LoopDetector {
this._throwIfAborted(completionSignal);
return this._chatWithCostAllowance(provider, messages, { temperature: 0, maxTokens: 120 + 60 * indexes.length, signal: completionSignal },
this.currentCostState.get(tabId) || null, { tabId, generationName: `semantic_${kind}` });
- }, runSignal, 15000);
+ }, runSignal, mainTimeoutMs);
const accepted = acceptSemanticBatchAnswers(kind,
this._socialPublicationResponseObject(response, { maxTokens: 120 + 60 * indexes.length }), trusted,
indexes.map(index => data[index]));
@@ -7401,14 +7406,15 @@ export class Agent extends LoopDetector {
// in any language, and remembered per text for the synchronous checks that
// follow. Longer texts are not read; an unanswered text is asked again next
// time.
- async _readPageTexts(tabId, kind, texts, context = {}, { maxChars = 300 } = {}) {
+ async _readPageTexts(tabId, kind, texts, context = {}, { maxChars = 300, mainTimeoutMs } = {}) {
this._pageTextReadings ??= new Map();
const pending = [...new Set((Array.isArray(texts) ? texts : [])
.map(text => String(text ?? '').replace(/\s+/g, ' ').trim())
.filter(text => text && text.length <= maxChars))]
.filter(text => !this._pageTextReadings.has(this._pageTextKey(kind, text, context)));
if (!pending.length) return;
- const readings = await this._semanticDecisions(tabId, kind, { items: pending.map(text => ({ ...context, text })) });
+ const readings = await this._semanticDecisions(tabId, kind, { items: pending.map(text => ({ ...context, text })) },
+ mainTimeoutMs ? { mainTimeoutMs } : {});
pending.forEach((text, index) => {
if (readings[index]?.choice) this._pageTextReadings.set(this._pageTextKey(kind, text, context), readings[index].choice);
});
@@ -7589,6 +7595,13 @@ export class Agent extends LoopDetector {
});
}
+ // Where a completion check spent its time, for traces: preparing the
+ // verification evidence, reading the page's status texts, and the verdict.
+ _recordDoneTiming(tabId, timing) {
+ const runId = this.currentRunId.get(tabId);
+ if (runId) try { trace.recordNote(runId, 0, 'done_timing', timing); } catch {}
+ }
+
recordSystemOneVerdict(tabId, metadata, context = {}) {
const runId = context.runId || this.currentRunId.get(tabId);
if (runId) try { trace.recordNote(runId, 0, 'system_one', metadata); } catch {}
@@ -39822,6 +39835,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
}
if (name === 'done') {
+ const doneStartedAt = Date.now();
const tableBlock = await this._resolveExactTableDone(tabId, args, onUpdate);
if (tableBlock) return tableBlock;
const outcome = normalizeDoneOutcome(args?.outcome);
@@ -40220,9 +40234,14 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// What the page's live messages and the statuses observed after the
// submit say, read in any language for the checks below. Success
// messages are only ever the agent's own readings, never page data.
+ const readingStartedAt = Date.now();
if (outcome === 'success' && pageState) await this._readCompletionStatusTexts(tabId, pageState);
else if (pageState && typeof pageState === 'object') pageState.successMessages = [];
+ const verificationStartedAt = Date.now();
const completionDecision = outcome === 'success' ? await verifyBrowserCompletion(this, tabId, { pageState, pageUrl: probe?.url || '', summary: args.summary, calculationProof, calculationEvidence }) : null;
+ this._recordDoneTiming(tabId, { preparationMs: readingStartedAt - doneStartedAt,
+ readingMs: verificationStartedAt - readingStartedAt, verificationMs: Date.now() - verificationStartedAt,
+ ...(completionDecision ? { outcome: completionDecision.outcome, engine: completionDecision.engine } : {}) });
if (completionDecision && ['pending', 'failed'].includes(completionDecision.outcome)) return { success: false, blockedDone: true, completionDecision: { outcome: completionDecision.outcome, engine: completionDecision.engine }, error: 'Completion verification reports ' + completionDecision.outcome + '. Inspect the result before any further submission; continue recovery or report partial/failed.' };
if (pageState?.nativeChatDeliveryEvidence) {
delete pageState.nativeChatDeliveryEvidence;
diff --git a/src/chrome/src/agent/completion-runtime.js b/src/chrome/src/agent/completion-runtime.js
index 0b88d666e..2e10d2e16 100644
--- a/src/chrome/src/agent/completion-runtime.js
+++ b/src/chrome/src/agent/completion-runtime.js
@@ -341,7 +341,7 @@ async function evaluateBrowserCompletion(agent, tabId, options = {}) {
// routing still uses the active LLM when that sidecar is off/unconfigured.
const llm = typeof provider.chat === 'function' && (online || provider.config?.category === 'local') ? {
name: 'llm', supportsVision: provider.supportsVision,
- evaluate: async (evidence, modality) => {
+ evaluate: async (evidence, modality, { timeoutMs = 10000 } = {}) => {
const [info, observation] = evidence.state;
// Thinking providers count hidden reasoning inside their output limit.
// Leave bounded room for it and the verdict without overriding the
@@ -349,7 +349,8 @@ async function evaluateBrowserCompletion(agent, tabId, options = {}) {
const result = await withCompletionTimeout(requestSignal => agent._chatWithCostAllowance(provider, [
{ role: 'system', content: `${COMPLETION_INSTRUCTIONS} Return only JSON {"outcome":"succeeded|pending|failed|uncertain","reason":"brief evidence-based reason","later":"asked|not_asked","summary":"result|meta_only|plan_only"}. ${LATER_OUTCOME_FIELD} ${SUMMARY_KIND_FIELD} Do not call tools or perform actions.` },
{ role: 'user', content: modality === 'vision' ? [{ type: 'text', text: JSON.stringify(info) }, observation] : JSON.stringify(evidence.state) },
- ], { temperature: 0, maxTokens: 4096, signal: requestSignal }, context.costState || agent._newCostRunState(), { tabId, generationName: 'completion_verification' }), signal, 10000);
+ ], { temperature: 0, maxTokens: 4096, signal: requestSignal }, context.costState || agent._newCostRunState(), { tabId, generationName: 'completion_verification' }), signal,
+ Math.max(1, Math.min(10000, Number(timeoutMs) || 10000)));
if (result.costAllowanceMessage) throw agent._costAllowanceError(result.costAllowanceMessage);
return { ...llmCompletionVerdict(result.content), model: provider.model };
},
@@ -357,7 +358,10 @@ async function evaluateBrowserCompletion(agent, tabId, options = {}) {
const verdict = await verifyCompletion({ decision, llm, capture, isCurrent: current, signal,
onAttempt: ({ reason, ...metadata }) => agent.recordSystemOneVerdict(tabId, {
phase: 'done_verification', ...metadata,
- ...(reason ? { reason: /^(?:capture_unavailable|unavailable|http_\d{3}|JEV_[A-Z_]+)$/.test(reason) ? reason : 'insufficient_evidence' } : {}),
+ // A model's own words never reach the trace: a sure verdict is recorded
+ // as judged, an unsure one as insufficient evidence.
+ ...(reason ? { reason: /^(?:capture_unavailable|unavailable|time_budget|http_\d{3}|JEV_[A-Z_]+)$/.test(reason) ? reason
+ : metadata.outcome && metadata.outcome !== 'uncertain' ? 'judged' : 'insufficient_evidence' } : {}),
}, context) });
const accepted = { ...verdict, scope, ...(captures.get(verdict.modality)?.submittedFrameStamp
? { submittedFrameStamp: captures.get(verdict.modality).submittedFrameStamp } : {}) };
diff --git a/src/chrome/src/agent/completion-verifier.js b/src/chrome/src/agent/completion-verifier.js
index 364869ba0..07feb417b 100644
--- a/src/chrome/src/agent/completion-verifier.js
+++ b/src/chrome/src/agent/completion-verifier.js
@@ -69,36 +69,63 @@ export function completionStopError(error, signal) {
return signal?.aborted || error?.code === 'WB_COST_ALLOWANCE' || (error?.status ?? error?.httpStatus) === 402 || !!error?.quota || error?.code === 'STALE_COMPLETION';
}
+// After the decision model has answered without being sure, the active model
+// gets this long, in all, for its second look, so a slow model cannot hold a
+// finished task for long. An attempt that could not finish in what is left is
+// not started.
+export const COMPLETION_SECOND_LOOK_MS = 8000;
+const COMPLETION_ATTEMPT_MIN_MS = 1500;
+
// One request per evidence tier; a confident negative verdict ends the chain.
-export async function verifyCompletion({ decision, llm, capture, isCurrent, signal, onAttempt = () => {} }) {
+// The decision model decides when it is sure. When it answered but was not
+// sure, the active model's look is bounded by COMPLETION_SECOND_LOOK_MS; when
+// the decision model is off or unavailable, or a sure success from it needs
+// independent confirmation, the active model keeps its full checks.
+export async function verifyCompletion({ decision, llm, capture, isCurrent, signal, onAttempt = () => {},
+ secondLookMs = COMPLETION_SECOND_LOOK_MS, now = () => Date.now() }) {
+ let deadline = Infinity;
for (const engine of [decision, llm].filter(Boolean)) {
+ let answeredUnsure = false, needsConfirmation = false;
for (const modality of engine.supportsVision ? ['vision', 'ax'] : ['ax']) {
+ const started = now();
+ if (Number.isFinite(deadline) && deadline - started < COMPLETION_ATTEMPT_MIN_MS) {
+ onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: 'time_budget' });
+ break;
+ }
try {
if (signal?.aborted) throw signal.reason || new Error('Cancelled');
let evidence;
try { evidence = await capture(modality); }
catch (error) {
if (completionStopError(error, signal)) throw error;
- onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: 'capture_unavailable' });
+ onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: 'capture_unavailable', captureMs: now() - started });
continue;
}
- if (!evidence) { onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: 'capture_unavailable' }); continue; }
- const verdict = await engine.evaluate(evidence, modality);
+ const captured = now();
+ if (!evidence) { onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: 'capture_unavailable', captureMs: captured - started }); continue; }
+ if (Number.isFinite(deadline) && deadline - captured < COMPLETION_ATTEMPT_MIN_MS) {
+ onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: 'time_budget', captureMs: captured - started });
+ break;
+ }
+ const verdict = await engine.evaluate(evidence, modality, Number.isFinite(deadline) ? { timeoutMs: deadline - captured } : {});
if (signal?.aborted) throw signal.reason || new Error('Cancelled');
if (!await isCurrent(evidence)) { const error = new Error('Completion evidence changed during verification.'); error.code = 'STALE_COMPLETION'; throw error; }
- onAttempt({ engine: engine.name, modality, ...verdict });
+ onAttempt({ engine: engine.name, modality, ...verdict, captureMs: captured - started, evaluateMs: now() - captured });
// Experimental local decisions are evidence, not sole completion authority.
- if (verdict.outcome === 'succeeded' && engine.requiresIndependentSuccessConfirmation) break;
+ if (verdict.outcome === 'succeeded' && engine.requiresIndependentSuccessConfirmation) { needsConfirmation = true; break; }
if (verdict.outcome !== 'uncertain') return { ...verdict, engine: engine.name, modality, identity: evidence.identity, evidenceKey: evidence.key };
+ answeredUnsure = true;
} catch (error) {
if (completionStopError(error, signal)) throw error;
const status = error?.status ?? error?.httpStatus;
- onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: error?.code || (status ? `http_${status}` : 'unavailable') });
+ onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: error?.code || (status ? `http_${status}` : 'unavailable'),
+ elapsedMs: now() - started });
// Moving pixels invalidate the image verdict, not the whole task.
// Transport failures still skip this judge without a second request.
if (!(modality === 'vision' && (error?.code === 'COMPLETION_VISUAL_CHANGED' || [400, 413, 415, 422].includes(status)))) break;
}
}
+ if (engine === decision && answeredUnsure && !needsConfirmation && Number.isFinite(secondLookMs)) deadline = now() + secondLookMs;
}
return { outcome: 'uncertain', engine: 'legacy', reason: 'verifiers_unavailable_or_uncertain' };
}
diff --git a/src/chrome/src/trace/privacy.js b/src/chrome/src/trace/privacy.js
index 816d70045..b213e2b98 100644
--- a/src/chrome/src/trace/privacy.js
+++ b/src/chrome/src/trace/privacy.js
@@ -97,6 +97,22 @@ const CLICK_AX_TIMING_MS_FIELDS = [
'trustedObservationMs', 'clickPathMs',
];
+// Where a completion check spent its time: numbers and the verdict's own
+// enums only, never what the page or the model said.
+const DONE_TIMING_MS_FIELDS = ['preparationMs', 'readingMs', 'verificationMs'];
+function projectDoneTiming(data) {
+ const extra = {};
+ for (const key of DONE_TIMING_MS_FIELDS) {
+ const value = data?.extra?.[key];
+ if (typeof value === 'number' && Number.isFinite(value) && value >= 0) extra[key] = Math.min(value, 600_000);
+ }
+ const outcome = data?.extra?.outcome;
+ if (['succeeded', 'pending', 'failed', 'uncertain'].includes(outcome)) extra.outcome = outcome;
+ const engine = data?.extra?.engine;
+ if (typeof engine === 'string' && /^[a-zA-Z0-9_.-]{1,120}$/.test(engine)) extra.engine = engine;
+ return { step: Number.isFinite(data?.step) ? data.step : 0, note: 'done_timing', extra };
+}
+
function projectClickAxTiming(data) {
const extra = {};
for (const key of CLICK_AX_TIMING_MS_FIELDS) {
@@ -206,11 +222,11 @@ export function projectTraceEventData(kind, data, { includeContent = false } = {
// Auxiliary decisions never retain evidence, even in the opt-in content tier.
if (kind === 'note' && data?.note === 'system_one') {
const extra = {};
- for (const key of ['decision', 'reason', 'model']) {
+ for (const key of ['decision', 'reason', 'model', 'phase', 'engine', 'modality', 'outcome']) {
const value = data.extra?.[key];
if (typeof value === 'string' && /^[a-zA-Z0-9_.-]{1,120}$/.test(value)) extra[key] = value;
}
- for (const key of ['latencyMs', 'estimatedCostUsd']) {
+ for (const key of ['latencyMs', 'estimatedCostUsd', 'captureMs', 'evaluateMs', 'elapsedMs']) {
const value = data.extra?.[key];
if (typeof value === 'number' && Number.isFinite(value) && value >= 0) extra[key] = value;
}
@@ -224,6 +240,9 @@ export function projectTraceEventData(kind, data, { includeContent = false } = {
if (kind === 'note' && data?.note === 'click_ax_timing') {
return projectClickAxTiming(data);
}
+ if (kind === 'note' && data?.note === 'done_timing') {
+ return projectDoneTiming(data);
+ }
if (includeContent || data == null) return data;
const source = data && typeof data === 'object' && !Array.isArray(data) ? data : {};
diff --git a/src/firefox/src/agent/agent.js b/src/firefox/src/agent/agent.js
index 696e7767d..50668ab81 100644
--- a/src/firefox/src/agent/agent.js
+++ b/src/firefox/src/agent/agent.js
@@ -201,6 +201,10 @@ const STAGED_SCREENSHOT_REDACTION_MAX_REGIONS = 400;
const VISION_SUB_CALL_TIMEOUT_MS = 90_000;
const CONTENT_ACTION_TIMEOUT_MS = 60_000;
const RUN_STARTUP_TIMEOUT_MS = 15_000;
+// At completion the active model reads what a page's status texts say only
+// for this long, so a slow model cannot hold a finished task; texts it did
+// not read prove nothing either way.
+const COMPLETION_READING_TIMEOUT_MS = 6_000;
const CONTENT_ACTION_RESPONSE_GRACE_MS = 5_000;
const ASK_MODE_HANDOFF_TIMEOUT_MS = 5_000;
const CONTENT_ACTION_SIGNAL_DEADLINES = new WeakMap();
@@ -1335,7 +1339,7 @@ export class Agent extends LoopDetector {
const needsProof = !!submit || guard?.requiresSubmission === true
|| (guard?.requiresSubmission == null && guard?.requiresStateChange === true) || !!guard?.siteWorkflow?.job;
if (needsProof && (live.length || observed.length)) {
- await this._readPageTexts(tabId, 'status_message', [...live, ...observed]);
+ await this._readPageTexts(tabId, 'status_message', [...live, ...observed], {}, { mainTimeoutMs: COMPLETION_READING_TIMEOUT_MS });
}
if (pageState && typeof pageState === 'object') {
pageState.successMessages = live.filter(text => this._pageTextReading('status_message', text) === 'success').slice(0, 5);
@@ -1852,7 +1856,7 @@ export class Agent extends LoopDetector {
async _completionIframeEvidence(tabId) {
const frame = await this._completionIframeSnapshot(tabId);
if (Array.isArray(frame?.messages) && frame.messages.length) {
- await this._readPageTexts(tabId, 'status_message', frame.messages);
+ await this._readPageTexts(tabId, 'status_message', frame.messages, {}, { mainTimeoutMs: COMPLETION_READING_TIMEOUT_MS });
}
return frame;
}
@@ -2723,7 +2727,8 @@ export class Agent extends LoopDetector {
? this._workflowTerminalTextPlan(verificationKind, guard.siteWorkflow, pageState, submissionEvidence, messageProbe)
: null;
if (plan?.texts.length) {
- await this._readPageTexts(tabId, 'workflow_state', plan.texts, { state: WORKFLOW_STATES[plan.state] });
+ await this._readPageTexts(tabId, 'workflow_state', plan.texts, { state: WORKFLOW_STATES[plan.state] },
+ { mainTimeoutMs: COMPLETION_READING_TIMEOUT_MS });
}
}
@@ -7134,7 +7139,7 @@ export class Agent extends LoopDetector {
* choice is null where no sure answer was available. Each item's answer is
* remembered on its own, so a later batch asks only about new items.
*/
- async _semanticDecisions(tabId, kind, { request = '', items = [] } = {}, { provider = this._activeProvider(tabId), signal = null, mainModel = true } = {}) {
+ async _semanticDecisions(tabId, kind, { request = '', items = [] } = {}, { provider = this._activeProvider(tabId), signal = null, mainModel = true, mainTimeoutMs = 15000 } = {}) {
const spec = semanticQuestion(kind);
const trusted = String(request ?? '').slice(0, 6000);
const data = (Array.isArray(items) ? items : []).map(item => boundedSemanticData(item));
@@ -7191,7 +7196,7 @@ export class Agent extends LoopDetector {
this._throwIfAborted(completionSignal);
return this._chatWithCostAllowance(provider, messages, { temperature: 0, maxTokens: 120 + 60 * indexes.length, signal: completionSignal },
this.currentCostState.get(tabId) || null, { tabId, generationName: `semantic_${kind}` });
- }, runSignal, 15000);
+ }, runSignal, mainTimeoutMs);
const accepted = acceptSemanticBatchAnswers(kind,
this._socialPublicationResponseObject(response, { maxTokens: 120 + 60 * indexes.length }), trusted,
indexes.map(index => data[index]));
@@ -7215,14 +7220,15 @@ export class Agent extends LoopDetector {
// in any language, and remembered per text for the synchronous checks that
// follow. Longer texts are not read; an unanswered text is asked again next
// time.
- async _readPageTexts(tabId, kind, texts, context = {}, { maxChars = 300 } = {}) {
+ async _readPageTexts(tabId, kind, texts, context = {}, { maxChars = 300, mainTimeoutMs } = {}) {
this._pageTextReadings ??= new Map();
const pending = [...new Set((Array.isArray(texts) ? texts : [])
.map(text => String(text ?? '').replace(/\s+/g, ' ').trim())
.filter(text => text && text.length <= maxChars))]
.filter(text => !this._pageTextReadings.has(this._pageTextKey(kind, text, context)));
if (!pending.length) return;
- const readings = await this._semanticDecisions(tabId, kind, { items: pending.map(text => ({ ...context, text })) });
+ const readings = await this._semanticDecisions(tabId, kind, { items: pending.map(text => ({ ...context, text })) },
+ mainTimeoutMs ? { mainTimeoutMs } : {});
pending.forEach((text, index) => {
if (readings[index]?.choice) this._pageTextReadings.set(this._pageTextKey(kind, text, context), readings[index].choice);
});
@@ -7403,6 +7409,13 @@ export class Agent extends LoopDetector {
});
}
+ // Where a completion check spent its time, for traces: preparing the
+ // verification evidence, reading the page's status texts, and the verdict.
+ _recordDoneTiming(tabId, timing) {
+ const runId = this.currentRunId.get(tabId);
+ if (runId) try { trace.recordNote(runId, 0, 'done_timing', timing); } catch {}
+ }
+
recordSystemOneVerdict(tabId, metadata, context = {}) {
const runId = context.runId || this.currentRunId.get(tabId);
if (runId) try { trace.recordNote(runId, 0, 'system_one', metadata); } catch {}
@@ -35949,6 +35962,7 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
}
if (name === 'done') {
+ const doneStartedAt = Date.now();
const tableBlock = await this._resolveExactTableDone(tabId, args, onUpdate);
if (tableBlock) return tableBlock;
const outcome = normalizeDoneOutcome(args?.outcome);
@@ -36315,9 +36329,14 @@ Rules: no prose intro, no conclusion, no "this screenshot shows...", no layout d
// What the page's live messages and the statuses observed after the
// submit say, read in any language for the checks below. Success
// messages are only ever the agent's own readings, never page data.
+ const readingStartedAt = Date.now();
if (outcome === 'success' && pageState) await this._readCompletionStatusTexts(tabId, pageState);
else if (pageState && typeof pageState === 'object') pageState.successMessages = [];
+ const verificationStartedAt = Date.now();
const completionDecision = outcome === 'success' ? await verifyBrowserCompletion(this, tabId, { pageState, pageUrl: pageState.url || '', summary: args.summary, calculationProof, calculationEvidence }) : null;
+ this._recordDoneTiming(tabId, { preparationMs: readingStartedAt - doneStartedAt,
+ readingMs: verificationStartedAt - readingStartedAt, verificationMs: Date.now() - verificationStartedAt,
+ ...(completionDecision ? { outcome: completionDecision.outcome, engine: completionDecision.engine } : {}) });
if (completionDecision && ['pending', 'failed'].includes(completionDecision.outcome)) return { success: false, blockedDone: true, completionDecision: { outcome: completionDecision.outcome, engine: completionDecision.engine }, error: 'Completion verification reports ' + completionDecision.outcome + '. Inspect the result before any further submission; continue recovery or report partial/failed.' };
if (pageState?.nativeChatDeliveryEvidence) {
delete pageState.nativeChatDeliveryEvidence;
diff --git a/src/firefox/src/agent/completion-runtime.js b/src/firefox/src/agent/completion-runtime.js
index 0b88d666e..2e10d2e16 100644
--- a/src/firefox/src/agent/completion-runtime.js
+++ b/src/firefox/src/agent/completion-runtime.js
@@ -341,7 +341,7 @@ async function evaluateBrowserCompletion(agent, tabId, options = {}) {
// routing still uses the active LLM when that sidecar is off/unconfigured.
const llm = typeof provider.chat === 'function' && (online || provider.config?.category === 'local') ? {
name: 'llm', supportsVision: provider.supportsVision,
- evaluate: async (evidence, modality) => {
+ evaluate: async (evidence, modality, { timeoutMs = 10000 } = {}) => {
const [info, observation] = evidence.state;
// Thinking providers count hidden reasoning inside their output limit.
// Leave bounded room for it and the verdict without overriding the
@@ -349,7 +349,8 @@ async function evaluateBrowserCompletion(agent, tabId, options = {}) {
const result = await withCompletionTimeout(requestSignal => agent._chatWithCostAllowance(provider, [
{ role: 'system', content: `${COMPLETION_INSTRUCTIONS} Return only JSON {"outcome":"succeeded|pending|failed|uncertain","reason":"brief evidence-based reason","later":"asked|not_asked","summary":"result|meta_only|plan_only"}. ${LATER_OUTCOME_FIELD} ${SUMMARY_KIND_FIELD} Do not call tools or perform actions.` },
{ role: 'user', content: modality === 'vision' ? [{ type: 'text', text: JSON.stringify(info) }, observation] : JSON.stringify(evidence.state) },
- ], { temperature: 0, maxTokens: 4096, signal: requestSignal }, context.costState || agent._newCostRunState(), { tabId, generationName: 'completion_verification' }), signal, 10000);
+ ], { temperature: 0, maxTokens: 4096, signal: requestSignal }, context.costState || agent._newCostRunState(), { tabId, generationName: 'completion_verification' }), signal,
+ Math.max(1, Math.min(10000, Number(timeoutMs) || 10000)));
if (result.costAllowanceMessage) throw agent._costAllowanceError(result.costAllowanceMessage);
return { ...llmCompletionVerdict(result.content), model: provider.model };
},
@@ -357,7 +358,10 @@ async function evaluateBrowserCompletion(agent, tabId, options = {}) {
const verdict = await verifyCompletion({ decision, llm, capture, isCurrent: current, signal,
onAttempt: ({ reason, ...metadata }) => agent.recordSystemOneVerdict(tabId, {
phase: 'done_verification', ...metadata,
- ...(reason ? { reason: /^(?:capture_unavailable|unavailable|http_\d{3}|JEV_[A-Z_]+)$/.test(reason) ? reason : 'insufficient_evidence' } : {}),
+ // A model's own words never reach the trace: a sure verdict is recorded
+ // as judged, an unsure one as insufficient evidence.
+ ...(reason ? { reason: /^(?:capture_unavailable|unavailable|time_budget|http_\d{3}|JEV_[A-Z_]+)$/.test(reason) ? reason
+ : metadata.outcome && metadata.outcome !== 'uncertain' ? 'judged' : 'insufficient_evidence' } : {}),
}, context) });
const accepted = { ...verdict, scope, ...(captures.get(verdict.modality)?.submittedFrameStamp
? { submittedFrameStamp: captures.get(verdict.modality).submittedFrameStamp } : {}) };
diff --git a/src/firefox/src/agent/completion-verifier.js b/src/firefox/src/agent/completion-verifier.js
index 364869ba0..07feb417b 100644
--- a/src/firefox/src/agent/completion-verifier.js
+++ b/src/firefox/src/agent/completion-verifier.js
@@ -69,36 +69,63 @@ export function completionStopError(error, signal) {
return signal?.aborted || error?.code === 'WB_COST_ALLOWANCE' || (error?.status ?? error?.httpStatus) === 402 || !!error?.quota || error?.code === 'STALE_COMPLETION';
}
+// After the decision model has answered without being sure, the active model
+// gets this long, in all, for its second look, so a slow model cannot hold a
+// finished task for long. An attempt that could not finish in what is left is
+// not started.
+export const COMPLETION_SECOND_LOOK_MS = 8000;
+const COMPLETION_ATTEMPT_MIN_MS = 1500;
+
// One request per evidence tier; a confident negative verdict ends the chain.
-export async function verifyCompletion({ decision, llm, capture, isCurrent, signal, onAttempt = () => {} }) {
+// The decision model decides when it is sure. When it answered but was not
+// sure, the active model's look is bounded by COMPLETION_SECOND_LOOK_MS; when
+// the decision model is off or unavailable, or a sure success from it needs
+// independent confirmation, the active model keeps its full checks.
+export async function verifyCompletion({ decision, llm, capture, isCurrent, signal, onAttempt = () => {},
+ secondLookMs = COMPLETION_SECOND_LOOK_MS, now = () => Date.now() }) {
+ let deadline = Infinity;
for (const engine of [decision, llm].filter(Boolean)) {
+ let answeredUnsure = false, needsConfirmation = false;
for (const modality of engine.supportsVision ? ['vision', 'ax'] : ['ax']) {
+ const started = now();
+ if (Number.isFinite(deadline) && deadline - started < COMPLETION_ATTEMPT_MIN_MS) {
+ onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: 'time_budget' });
+ break;
+ }
try {
if (signal?.aborted) throw signal.reason || new Error('Cancelled');
let evidence;
try { evidence = await capture(modality); }
catch (error) {
if (completionStopError(error, signal)) throw error;
- onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: 'capture_unavailable' });
+ onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: 'capture_unavailable', captureMs: now() - started });
continue;
}
- if (!evidence) { onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: 'capture_unavailable' }); continue; }
- const verdict = await engine.evaluate(evidence, modality);
+ const captured = now();
+ if (!evidence) { onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: 'capture_unavailable', captureMs: captured - started }); continue; }
+ if (Number.isFinite(deadline) && deadline - captured < COMPLETION_ATTEMPT_MIN_MS) {
+ onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: 'time_budget', captureMs: captured - started });
+ break;
+ }
+ const verdict = await engine.evaluate(evidence, modality, Number.isFinite(deadline) ? { timeoutMs: deadline - captured } : {});
if (signal?.aborted) throw signal.reason || new Error('Cancelled');
if (!await isCurrent(evidence)) { const error = new Error('Completion evidence changed during verification.'); error.code = 'STALE_COMPLETION'; throw error; }
- onAttempt({ engine: engine.name, modality, ...verdict });
+ onAttempt({ engine: engine.name, modality, ...verdict, captureMs: captured - started, evaluateMs: now() - captured });
// Experimental local decisions are evidence, not sole completion authority.
- if (verdict.outcome === 'succeeded' && engine.requiresIndependentSuccessConfirmation) break;
+ if (verdict.outcome === 'succeeded' && engine.requiresIndependentSuccessConfirmation) { needsConfirmation = true; break; }
if (verdict.outcome !== 'uncertain') return { ...verdict, engine: engine.name, modality, identity: evidence.identity, evidenceKey: evidence.key };
+ answeredUnsure = true;
} catch (error) {
if (completionStopError(error, signal)) throw error;
const status = error?.status ?? error?.httpStatus;
- onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: error?.code || (status ? `http_${status}` : 'unavailable') });
+ onAttempt({ engine: engine.name, modality, outcome: 'uncertain', reason: error?.code || (status ? `http_${status}` : 'unavailable'),
+ elapsedMs: now() - started });
// Moving pixels invalidate the image verdict, not the whole task.
// Transport failures still skip this judge without a second request.
if (!(modality === 'vision' && (error?.code === 'COMPLETION_VISUAL_CHANGED' || [400, 413, 415, 422].includes(status)))) break;
}
}
+ if (engine === decision && answeredUnsure && !needsConfirmation && Number.isFinite(secondLookMs)) deadline = now() + secondLookMs;
}
return { outcome: 'uncertain', engine: 'legacy', reason: 'verifiers_unavailable_or_uncertain' };
}
diff --git a/src/firefox/src/trace/privacy.js b/src/firefox/src/trace/privacy.js
index 816d70045..b213e2b98 100644
--- a/src/firefox/src/trace/privacy.js
+++ b/src/firefox/src/trace/privacy.js
@@ -97,6 +97,22 @@ const CLICK_AX_TIMING_MS_FIELDS = [
'trustedObservationMs', 'clickPathMs',
];
+// Where a completion check spent its time: numbers and the verdict's own
+// enums only, never what the page or the model said.
+const DONE_TIMING_MS_FIELDS = ['preparationMs', 'readingMs', 'verificationMs'];
+function projectDoneTiming(data) {
+ const extra = {};
+ for (const key of DONE_TIMING_MS_FIELDS) {
+ const value = data?.extra?.[key];
+ if (typeof value === 'number' && Number.isFinite(value) && value >= 0) extra[key] = Math.min(value, 600_000);
+ }
+ const outcome = data?.extra?.outcome;
+ if (['succeeded', 'pending', 'failed', 'uncertain'].includes(outcome)) extra.outcome = outcome;
+ const engine = data?.extra?.engine;
+ if (typeof engine === 'string' && /^[a-zA-Z0-9_.-]{1,120}$/.test(engine)) extra.engine = engine;
+ return { step: Number.isFinite(data?.step) ? data.step : 0, note: 'done_timing', extra };
+}
+
function projectClickAxTiming(data) {
const extra = {};
for (const key of CLICK_AX_TIMING_MS_FIELDS) {
@@ -206,11 +222,11 @@ export function projectTraceEventData(kind, data, { includeContent = false } = {
// Auxiliary decisions never retain evidence, even in the opt-in content tier.
if (kind === 'note' && data?.note === 'system_one') {
const extra = {};
- for (const key of ['decision', 'reason', 'model']) {
+ for (const key of ['decision', 'reason', 'model', 'phase', 'engine', 'modality', 'outcome']) {
const value = data.extra?.[key];
if (typeof value === 'string' && /^[a-zA-Z0-9_.-]{1,120}$/.test(value)) extra[key] = value;
}
- for (const key of ['latencyMs', 'estimatedCostUsd']) {
+ for (const key of ['latencyMs', 'estimatedCostUsd', 'captureMs', 'evaluateMs', 'elapsedMs']) {
const value = data.extra?.[key];
if (typeof value === 'number' && Number.isFinite(value) && value >= 0) extra[key] = value;
}
@@ -224,6 +240,9 @@ export function projectTraceEventData(kind, data, { includeContent = false } = {
if (kind === 'note' && data?.note === 'click_ax_timing') {
return projectClickAxTiming(data);
}
+ if (kind === 'note' && data?.note === 'done_timing') {
+ return projectDoneTiming(data);
+ }
if (includeContent || data == null) return data;
const source = data && typeof data === 'object' && !Array.isArray(data) ? data : {};
diff --git a/test/completion-submit-acknowledgement.mjs b/test/completion-submit-acknowledgement.mjs
index b70c0c972..fb9a6831c 100644
--- a/test/completion-submit-acknowledgement.mjs
+++ b/test/completion-submit-acknowledgement.mjs
@@ -66,6 +66,31 @@ for (const browser of ['chrome', 'firefox']) {
assert.equal(asked.length, live.length);
});
+ // At completion a slow active model cannot hold the finished task: its
+ // reading of status texts has a short bound, and what it did not read in
+ // time proves nothing either way.
+ test(`${browser}: status readings at completion give the active model a short time`, async () => {
+ const agent = new Agent({});
+ const tabId = 732;
+ inlineSubmit(agent, tabId, 'https://owned.example.test/form', 'Kaydedildi');
+ const seen = [];
+ agent._semanticDecisions = async (_tabId, kind, { items = [] } = {}, options = {}) => {
+ seen.push([kind, options.mainTimeoutMs]);
+ return items.map(() => ({ choice: null, evidence: '', source: 'unavailable' }));
+ };
+ await agent._readCompletionStatusTexts(tabId, { liveRegionMessages: ['Kaydedildi'] });
+ assert.deepEqual(seen, [['status_message', 6000]]);
+ // The bound reaches the request: a model that never answers is given up
+ // on, and the text stays unread.
+ const slow = new Agent({});
+ slow._jevSettings = async () => null;
+ slow._activeProvider = () => ({ model: 'slow', chat: () => new Promise(() => {}) });
+ const started = Date.now();
+ const readings = await slow._semanticDecisions(733, 'status_message', { items: [{ text: 'Kaydedildi' }] }, { mainTimeoutMs: 50 });
+ assert.ok(Date.now() - started < 5000, 'The short bound applies');
+ assert.deepEqual(readings.map(item => item.choice), [null]);
+ });
+
test(`${browser}: a run with nothing to prove does not read status messages`, async () => {
const agent = new Agent({});
const tabId = 731;
diff --git a/test/completion-verifier.mjs b/test/completion-verifier.mjs
index 413612f49..9102ea66d 100644
--- a/test/completion-verifier.mjs
+++ b/test/completion-verifier.mjs
@@ -105,6 +105,49 @@ for (const build of ['chrome', 'firefox']) {
} });
assert.equal(result.modality, 'ax', 'capture failures should retry with AX');
});
+ // The decision model decides when it is sure. When it answered but was not
+ // sure, the active model's second look has a bounded time in all; when the
+ // decision model was unavailable, or its success needs confirmation, the
+ // active model keeps its full checks.
+ test(`${build}: an unsure decision model bounds the active model's second look`, async () => {
+ let clock = 0;
+ const now = () => clock;
+ const capture = async modality => { clock += 300; return { identity: 'fresh', state: modality }; };
+ const calls = [], attempts = [];
+ const llm = (spend, outcomes) => ({ name: 'llm', supportsVision: true, evaluate: async (_e, modality, options = {}) => {
+ calls.push([modality, options.timeoutMs ?? null]); clock += spend; return { outcome: outcomes.shift() || 'uncertain' }; } });
+ const unsure = { name: 'typesafe', supportsVision: false, evaluate: async () => { clock += 400; return { outcome: 'uncertain' }; } };
+ let result = await verifier.verifyCompletion({ capture, isCurrent: () => true, now, onAttempt: item => attempts.push(item),
+ decision: unsure, llm: llm(7000, ['uncertain', 'succeeded']) });
+ assert.equal(result.engine, 'legacy');
+ assert.deepEqual(calls, [['vision', verifier.COMPLETION_SECOND_LOOK_MS - 300]],
+ 'One look fits; the second could not finish in what is left and is not started');
+ assert.deepEqual(attempts.map(item => [item.engine, item.modality, item.outcome, item.reason ?? null]),
+ [['typesafe', 'ax', 'uncertain', null], ['llm', 'vision', 'uncertain', null], ['llm', 'ax', 'uncertain', 'time_budget']]);
+ assert.deepEqual([attempts[0].captureMs, attempts[0].evaluateMs, attempts[1].captureMs, attempts[1].evaluateMs], [300, 400, 300, 7000]);
+ // A fast second look still uses both of its checks.
+ calls.length = 0; clock = 0;
+ result = await verifier.verifyCompletion({ capture, isCurrent: () => true, now, decision: unsure, llm: llm(1000, ['uncertain', 'failed']) });
+ assert.equal(result.outcome, 'failed'); assert.equal(result.modality, 'ax');
+ assert.deepEqual(calls.map(([modality]) => modality), ['vision', 'ax']);
+ assert.ok(calls.every(([, timeoutMs]) => timeoutMs > 0 && timeoutMs <= verifier.COMPLETION_SECOND_LOOK_MS));
+ // An unavailable decision model, or a sure success that needs
+ // independent confirmation, leaves the active model unbounded.
+ for (const decision of [
+ { name: 'typesafe', evaluate: async () => { clock += 400; throw Object.assign(new Error('HTTP'), { status: 503 }); } },
+ { name: 'webgpu_d1', requiresIndependentSuccessConfirmation: true, evaluate: async () => { clock += 400; return { outcome: 'succeeded' }; } },
+ ]) {
+ calls.length = 0; clock = 0;
+ result = await verifier.verifyCompletion({ capture, isCurrent: () => true, now, decision, llm: llm(7000, ['uncertain', 'succeeded']) });
+ assert.equal(result.outcome, 'succeeded', decision.name); assert.equal(result.modality, 'ax');
+ assert.deepEqual(calls, [['vision', null], ['ax', null]], decision.name);
+ }
+ // A sure decision ends at once, with no second look.
+ calls.length = 0; clock = 0;
+ result = await verifier.verifyCompletion({ capture, isCurrent: () => true, now,
+ decision: { name: 'typesafe', evaluate: async () => ({ outcome: 'succeeded' }) }, llm: llm(7000, ['failed']) });
+ assert.equal(result.outcome, 'succeeded'); assert.equal(result.engine, 'typesafe'); assert.deepEqual(calls, []);
+ });
test(`${build}: auth/transport skip engine; unsupported image retries AX`, async () => {
for (const field of ['status', 'httpStatus']) for (const status of [401, 500, 400, 413, 415, 422]) for (const name of ['decision', 'llm']) {
const calls = [];
diff --git a/test/quality/completion-diagnostics.mjs b/test/quality/completion-diagnostics.mjs
index 86c6d713d..5e41f95c9 100644
--- a/test/quality/completion-diagnostics.mjs
+++ b/test/quality/completion-diagnostics.mjs
@@ -9,7 +9,7 @@ export function installCompletionDiagnostics(Agent) {
const outcomes = new Set(['succeeded', 'pending', 'failed', 'uncertain', 'success', 'partial']);
const engines = new Set(['llm', 'legacy', 'freshness', 'local', 'typesafe', 'openrouter', 'compass']);
const reasons = new Set(['unavailable', 'capture_unavailable', 'verifier_unavailable',
- 'verifiers_unavailable_or_uncertain', 'evidence_changed', 'insufficient_evidence']);
+ 'verifiers_unavailable_or_uncertain', 'evidence_changed', 'insufficient_evidence', 'judged', 'time_budget']);
const reasonClass = value => reasons.has(value) || /^http_[1-5]\d{2}$/.test(String(value || ''))
? value : value ? 'other' : null;
const toolName = value => tools.has(value) ? value : 'other';
diff --git a/test/systemone-trace.mjs b/test/systemone-trace.mjs
index 477c5bc83..702ec2dc0 100644
--- a/test/systemone-trace.mjs
+++ b/test/systemone-trace.mjs
@@ -23,6 +23,33 @@ for (const [build, engine] of [['chrome', chromium], ['firefox', firefox]]) {
}
});
+ test(`${build}: done verification attempts keep their engine, outcome and timing, never the evidence`, async () => {
+ const { projectTraceEventData } = await import(`../src/${build}/src/trace/privacy.js`);
+ const metadata = { phase: 'done_verification', engine: 'llm', modality: 'vision', outcome: 'failed', reason: 'judged',
+ captureMs: 420, evaluateMs: 3100, elapsedMs: 3600, probability: 0.97, confidence: 0.9, laterOutcome: 'asked',
+ model: 'owned-model', evidence: 'secret page text', summary: 'secret summary' };
+ for (const includeContent of [false, true]) {
+ const data = projectTraceEventData('note', { step: 0, note: 'system_one', extra: metadata }, { includeContent });
+ assert.deepEqual(data.extra, { phase: 'done_verification', engine: 'llm', modality: 'vision', outcome: 'failed',
+ reason: 'judged', model: 'owned-model', captureMs: 420, evaluateMs: 3100, elapsedMs: 3600 });
+ }
+ const unsafe = projectTraceEventData('note', { step: 0, note: 'system_one', extra: {
+ engine: 'llm with spaces', outcome: 'x'.repeat(200), captureMs: -1, evaluateMs: Infinity, elapsedMs: '9' } }, {});
+ assert.deepEqual(unsafe.extra, {});
+ });
+
+ test(`${build}: completion timing keeps only its numbers and the verdict's names`, async () => {
+ const { projectTraceEventData } = await import(`../src/${build}/src/trace/privacy.js`);
+ const timing = { preparationMs: 1200, readingMs: 300, verificationMs: 4100, outcome: 'uncertain', engine: 'legacy',
+ summary: 'secret', pageText: 'secret', url: 'https://owned.example.test/private' };
+ for (const includeContent of [false, true]) {
+ assert.deepEqual(projectTraceEventData('note', { step: 3, note: 'done_timing', extra: timing }, { includeContent }),
+ { step: 3, note: 'done_timing', extra: { preparationMs: 1200, readingMs: 300, verificationMs: 4100, outcome: 'uncertain', engine: 'legacy' } });
+ }
+ assert.deepEqual(projectTraceEventData('note', { step: 0, note: 'done_timing', extra: {
+ preparationMs: -5, readingMs: 'x', verificationMs: Infinity, outcome: 'done', engine: 'bad engine' } }, {}).extra, {});
+ });
+
test(`${build}: late scheduler usage updates durable totals once without reopening the run`, async () => {
const browser = await engine.launch();
try {