Skip to content

Open-source Python library for AI browser agents, plus Browser Use Cloud: stealth browsers and a hosted agent API.

Record latest real_v8 eval gates

Magnus Müllerbb33f4cb917d
diff --git a/RUST_AGENT_INTEGRATION_FEATURES.md b/RUST_AGENT_INTEGRATION_FEATURES.md
index 99526c6d9..b2c1b9b7a 100644
--- a/RUST_AGENT_INTEGRATION_FEATURES.md
+++ b/RUST_AGENT_INTEGRATION_FEATURES.md
@@ -1605,6 +1605,20 @@ Terminal core branch: `magnus/browser-use-rust-main-integration` at terminal mai
## Not Verified Yet
+- 2026-06-04 post-dataset-timebox five-task real_v8 Agent SDK gate:
+ - Run `kh783as9s2c1s1cf6j7rzgd77h8815a6` used `BU_RUNTIME=brust`, eval-internal commit `6e75e49`, terminal commit `ce498f7`, browser-use commit `590234b7`, `--browser browser-use-cloud`, `BROWSER_USE_CLOUD_PROXY_COUNTRY_CODE=us`, `BU_BROWSER_SCRIPT_INITIAL_WAIT_MS=7000`, `AGENT_SDK_JUDGE_TIMEOUT_SECONDS=25`, `AGENT_SDK_JUDGE_FALLBACK_ATTEMPTS=3`, `AGENT_SDK_JUDGE_CONCURRENCY=16`, `LLM_BROWSER_CAPTURE_FPS=0`, `BU_DISABLE_FALLBACK_CAPTURE_GIF=1`, `BU_BROWSER_SCRIPT_SESSION_OUTPUTS=1`, `--use-vision false`, `--judge-type agent_sdk`, `--model claude-sonnet-4-6`, `--eval-model claude-sonnet-4-6`, `--parallel-runs 5`, and `--max-steps 100` for real_v8 tasks 6-10.
+ - Browser Use cloud startup was healthy: all five sessions were created around 2026-06-04T12:35:14Z, all five tasks entered Rust-core agent execution, and no CAPTCHA/local-browser misconfiguration appeared.
+ - Saved results before the 10-minute cap: task 9 score `0.5` in 252.21s agent time/302.09s total pipeline and task 10 score `1.0` in 549.82s/586.58s. Result IDs were `kn7292v9fqnspg0pkjs7rr2355881gpm` and `kn74ct6r2trssf3sbh15fa34hh880smb`.
+ - Tasks 6, 7, and 8 were still in agent-stage execution when the cap fired. Their remaining Browser Use cloud sessions were explicitly stopped after the cap, and no eval/terminal subprocesses remained.
+ - The prior Agent SDK `success`/max-turn exception did not recur for completed tasks: both saved tasks logged `Agent SDK judge trajectory-only mode enabled; skipping tool-enabled verification attempt` and saved normally. One Laminar Claude proxy health-check restart and Convex progress-save read timeouts appeared but did not break judging.
+ - This gate does not justify scaling to 50: saved-task mean score was `0.75`, but missing-task-as-zero mean score was `0.30` with only 2/5 tasks saved within the iteration budget. Prompt-only timebox guidance did not force long research/document tasks to finalize before the external cap.
+- 2026-06-04 post-text-heavy-screenshot-restraint five-task real_v8 Agent SDK gate:
+ - Run `kh72amba59xvvhw18x2pgj3vfd880xp5` used `BU_RUNTIME=brust`, eval-internal commit `6e75e49`, terminal commit `3453591`, browser-use commit `ae62bea2`, `--browser browser-use-cloud`, `BROWSER_USE_CLOUD_PROXY_COUNTRY_CODE=us`, `BU_BROWSER_SCRIPT_INITIAL_WAIT_MS=7000`, `AGENT_SDK_JUDGE_TIMEOUT_SECONDS=25`, `AGENT_SDK_JUDGE_FALLBACK_ATTEMPTS=3`, `AGENT_SDK_JUDGE_CONCURRENCY=16`, `LLM_BROWSER_CAPTURE_FPS=0`, `BU_DISABLE_FALLBACK_CAPTURE_GIF=1`, `BU_BROWSER_SCRIPT_SESSION_OUTPUTS=1`, `--use-vision false`, `--judge-type agent_sdk`, `--model claude-sonnet-4-6`, `--eval-model claude-sonnet-4-6`, `--parallel-runs 5`, and `--max-steps 100` for real_v8 tasks 6-10.
+ - Browser Use cloud startup was healthy: all five sessions were created around 2026-06-04T12:22:23Z, all five tasks entered Rust-core agent execution, and no CAPTCHA/local-browser misconfiguration appeared.
+ - Saved results before the 10-minute cap: task 9 score `1.0` in 202.27s agent time/228.65s total pipeline and task 10 score `1.0` in 410.01s/450.59s. Result IDs were `kn77qash6gyqzagq6ts0qawdpd8818cq` and `kn76ghc82n0agbjtcv325ccc8x880hvj`.
+ - Tasks 6, 7, and 8 were still in agent-stage execution when the cap fired. Their remaining Browser Use cloud sessions were explicitly stopped after the cap, and no eval/terminal subprocesses remained.
+ - The prior Agent SDK `success`/max-turn exception did not recur for completed tasks: both saved tasks logged `Agent SDK judge trajectory-only mode enabled; skipping tool-enabled verification attempt` and saved normally.
+ - This gate does not justify scaling to 50: saved-task mean score was `1.0`, but missing-task-as-zero mean score was `0.40` with only 2/5 tasks saved within the iteration budget. Text-heavy screenshot restraint reduced some non-vision artifact volume, but long-tail agent execution still dominated tasks 6-8.
- 2026-06-04 post-Agent-SDK-trajectory-judge five-task real_v8 Agent SDK gate:
- Run `kh7ahf1zhwn895e84ftn1n59x9881hhe` used `BU_RUNTIME=brust`, eval-internal commit `6e75e49`, terminal commit `8af682f`, `--browser browser-use-cloud`, `BROWSER_USE_CLOUD_PROXY_COUNTRY_CODE=us`, `BU_BROWSER_SCRIPT_INITIAL_WAIT_MS=7000`, `AGENT_SDK_JUDGE_TIMEOUT_SECONDS=25`, `AGENT_SDK_JUDGE_FALLBACK_ATTEMPTS=3`, fallback timeout default floor `45`, `AGENT_SDK_JUDGE_CONCURRENCY=16`, `LLM_BROWSER_CAPTURE_FPS=0`, `BU_DISABLE_FALLBACK_CAPTURE_GIF=1`, `BU_BROWSER_SCRIPT_SESSION_OUTPUTS=1`, `--use-vision false`, `--judge-type agent_sdk`, `--model claude-sonnet-4-6`, `--eval-model claude-sonnet-4-6`, `--parallel-runs 5`, and `--max-steps 100` for real_v8 tasks 6-10.
- The shell explicitly unset `AGENT_SDK_JUDGE_TRAJECTORY_ONLY`; eval service still enabled trajectory-only mode by default for `agent_sdk`, proving the new service default.