Open-source AI browser agent for Chrome and Firefox (monorepo) π§
JavaScript ai-agentai-agentsai-toolsbrowser-aibrowser-automationbrowser-extensionchrome-aichrome-devtoolschrome-extensionchrome-extensionsfirefox-addonfirefox-addonsfirefox-extensionfirefox-extensionsgemmagemma4minimaxoffline-aioffline-llmqwen
Prepare pinned HY4, MiMo V2.6, and LFM2.5 benchmark campaign
Barack Sokulluc7c2cf8f177b
diff --git a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/README.md b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/README.mdnew file mode 100644index 000000000..53633df18--- /dev/null+++ b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/README.md@@ -0,0 +1,41 @@+# HY4 Preview, MiMo V2.6, and Liquid LFM2.5 campaign++Prepared October 14, 2026. **Live inference is pending a fresh OpenRouter key. No new benchmark scores or measured bills are claimed yet.** The last supplied credential expired October 7, and no API key is configured in the workspace environment.++Requested models:++- `tencent/hy4-preview`: native tools, text-only route.+- `xiaomi/mimo-v2.6-pro`: native tools and image/video/audio input.+- `xiaomi/mimo-v2.6-flash`: native tools and image/video/audio input.+- `liquid/lfm-2.5-24b-a2b`: native tools, text-only route.++`config.json` records all planned output paths and settings. Public endpoint responses retrieved October 14 are archived in `openrouter-endpoints.json`. Endpoint metadata establishes advertised capabilities and rates, not measured model quality.++All four planner runs will replay the full-tier August source `7182c21f58a94689c2ac3b547a279b9c5e699283`: 100 Chrome Act cases, 48 structured tools, temperature 0.15, max_tokens 4096, and no reasoning or compatibility override. The existing snapshot SHA-256 is `3d27c43da02747496dbd050a366ef6188f14eaf99a5b41de35e5c55581669062`. `rubrics.json` archives the matching ideal actions. This is a first-action-only test: no tools are executed, and ideal conformity is not task completion.++The report first reproduces the original 13-model/1,300-response consensus, schema validity, ideal-name, and exact-ideal metrics. Once four complete new runs exist, all 17 models are rescored against their other 16 peers: 1,600 comparisons per model. Only object keys are recursively sorted; argument values and extra/default arguments are retained. Historical percentages are not copied unchanged into a new pool. Native dispatch, content fallback, malformed argument JSON, and schema validity are reported separately.++Both MiMo models will also receive the 100-screenshot production vision suite: the original six-section prompt, temperature 0, max_tokens 800, unchanged thinking-disabling kwargs, and provider auto-routing. HY4 and LFM are not assigned vision scores. The report compares against the saved August 27 Qwen3.8 Flash and GLM-5.3 Flash runs and already reproduces all 200 of those grades, with unchanged images/prompts/rubrics/scorer. UTC timestamps in those raw files begin August 26; their original local test date and run tag are August 27.++Vision strict passes combine weighted substring checks and a section-structure score with the case threshold and critical-check gate. Six numbered sections are measured separately rather than required as an independent binary gate. Some recognized synonyms may fail literal matching; numbered drafting prose can resemble sections. Reasoning token usage and finish reasons are retained so output-budget effects are visible. Missing historical finish reasons remain null, not zero.++The old planner requests were omitted, so their full body equality with the source reconstruction cannot be proved. All new planner requests and API responses will be saved. New vision runs retain raw SSE, final text, usage, provider, finish reason, latency, prompt metadata, and image hashes; shared PNGs remain under `test/vision/images`. Never save API keys or authorization headers.++After setting `OPENROUTER_API_KEY` in the process environment:++```powershell+node test/llm/run-hy4-mimo26-lfm25.mjs --preflight+node test/llm/run-hy4-mimo26-lfm25.mjs+node test/llm/report-hy4-mimo26-lfm25.mjs+npm run test:vision:validate+node scripts/build-blog.mjs+```++The orchestrator authenticates before inference and refuses to overwrite existing evidence. Six independent runners execute concurrently, with concurrency three per planner model and two per vision model. Preflights have separate result tags and are excluded from scored 100-case suites. All raw preflight/full-run results, errors, and logs must be committed and pushed with the post. Archive failures before targeted recovery; retain the first successful inference per case rather than rerunning successful cases to improve scores.++Historical/offline preparation can be verified without a key:++```powershell+node test/llm/prepare-hy4-mimo26-lfm25.mjs+node test/llm/report-hy4-mimo26-lfm25.mjs --historical-only+```diff --git a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/config.json b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/config.jsonnew file mode 100644index 000000000..0daf9dc1c--- /dev/null+++ b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/config.json@@ -0,0 +1,60 @@+{+ "date": "2026-10-14",+ "replayPath": "test/llm/freeze/full-replay-7182c21f.json",+ "replaySha256": "3d27c43da02747496dbd050a366ef6188f14eaf99a5b41de35e5c55581669062",+ "sourceCommit": "7182c21f58a94689c2ac3b547a279b9c5e699283",+ "models": [+ {+ "model": "tencent/hy4-preview",+ "label": "Tencent HY4 Preview",+ "vision": false,+ "plannerDir": "test/llm/results/2026-10-14-openrouter-hy4-mimo26-lfm25-full-7182_chrome_tencent_hy4-preview",+ "visionDir": null+ },+ {+ "model": "xiaomi/mimo-v2.6-pro",+ "label": "MiMo-V2.6-Pro",+ "vision": true,+ "plannerDir": "test/llm/results/2026-10-14-openrouter-hy4-mimo26-lfm25-full-7182_chrome_xiaomi_mimo-v2.6-pro",+ "visionDir": "test/vision/results/2026-10-14-openrouter-mimo26-vision_xiaomi_mimo-v2.6-pro_production"+ },+ {+ "model": "xiaomi/mimo-v2.6-flash",+ "label": "MiMo-V2.6-Flash",+ "vision": true,+ "plannerDir": "test/llm/results/2026-10-14-openrouter-hy4-mimo26-lfm25-full-7182_chrome_xiaomi_mimo-v2.6-flash",+ "visionDir": "test/vision/results/2026-10-14-openrouter-mimo26-vision_xiaomi_mimo-v2.6-flash_production"+ },+ {+ "model": "liquid/lfm-2.5-24b-a2b",+ "label": "Liquid LFM2.5-24B-A2B",+ "vision": false,+ "plannerDir": "test/llm/results/2026-10-14-openrouter-hy4-mimo26-lfm25-full-7182_chrome_liquid_lfm-2.5-24b-a2b",+ "visionDir": null+ }+ ],+ "planner": {+ "cases": 100,+ "actTools": 48,+ "temperature": 0.15,+ "maxTokens": 4096,+ "reasoningOverride": null,+ "compatibility": "off",+ "concurrency": 3,+ "timeoutMs": 180000+ },+ "vision": {+ "cases": 100,+ "temperature": 0,+ "maxTokens": 800,+ "promptMode": "production",+ "concurrency": 2,+ "providerRouting": "automatic"+ },+ "historicalPlannerModels": 13,+ "historicalVisionDirs": [+ "test/vision/results/2026-08-27-openrouter-full_qwen_qwen3.8-flash_production",+ "test/vision/results/2026-08-27-openrouter-full_z-ai_glm-5.3-flash_production"+ ],+ "note": "Only October 14 models are new inference. August historical responses are retained, dated separately, and verified against matching pinned inputs. Full historical request byte equality is unprovable because old planner runs omitted bodies."+}diff --git a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/historical-planner-regression.json b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/historical-planner-regression.jsonnew file mode 100644index 000000000..48c943f59--- /dev/null+++ b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/historical-planner-regression.json@@ -0,0 +1,907 @@+{+ "sourceCommit": "7182c21f58a94689c2ac3b547a279b9c5e699283",+ "validatedModels": 13,+ "cases": 1300,+ "allConsensusAndSchemaAndIdealCountsReproduced": true,+ "results": [+ {+ "rank": 1,+ "model": "deepseek/deepseek-v4-flash-0731",+ "label": "DeepSeek V4 Flash 0731",+ "date": "2026-08-02",+ "source": "test/llm/results/2026-08-02-deepseek-v4-flash-full_chrome_deepseek_deepseek-v4-flash-0731",+ "fresh": false,+ "cases": 100,+ "errors": 0,+ "responses": 100,+ "emittedToolCalls": 90,+ "nativeToolCalls": 90,+ "schemaValidToolCalls": 90,+ "idealToolMatches": 39,+ "exactIdealMatches": 18,+ "promptTokens": 2365949,+ "completionTokens": 13566,+ "cachedPromptTokens": 2291072,+ "reasoningTokens": 7494,+ "costRecords": 100,+ "usageCost": 0.050420106,+ "lengthLimited": 0,+ "contentFallbackCalls": 0,+ "accessibilityTreeFirstCalls": 44,+ "attempts": 100,+ "providers": {},+ "byTool": {+ "navigate": 23,+ "list_downloads": 2,+ "new_tab": 3,+ "get_accessibility_tree": 44,+ "read_page": 6,+ "extract_data": 2,+ "download_files": 2,+ "download_social_media": 2,+ "scroll": 1,+ "(no-tool)": 10,+ "get_selection": 1,+ "clarify": 1,+ "progress_read": 1,+ "fetch_url": 1,+ "done": 1+ },+ "schemaErrors": [],+ "medianLatencyMs": 1579.5,+ "p95LatencyMs": 5241,+ "cachedPromptSharePct": 96.83522341352244,+ "exactActionMatches": 621,+ "toolNameMatches": 958,+ "comparisons": 1200,+ "exactActionConsensusPct": 51.8,+ "toolNameConsensusPct": 79.8+ },+ {+ "rank": 2,+ "model": "tencent/hy3",+ "label": "Tencent HY3",+ "date": "2026-08-02",+ "source": "test/llm/results/2026-08-02-tencent-hy3-full_chrome_tencent_hy3",+ "fresh": false,+ "cases": 100,+ "errors": 0,+ "responses": 100,+ "emittedToolCalls": 90,+ "nativeToolCalls": 90,+ "schemaValidToolCalls": 90,+ "idealToolMatches": 41,+ "exactIdealMatches": 18,+ "promptTokens": 2368947,+ "completionTokens": 18328,+ "cachedPromptTokens": 772096,+ "reasoningTokens": 13365,+ "costRecords": 100,+ "usageCost": 0.24594068400000005,+ "lengthLimited": 0,+ "contentFallbackCalls": 0,+ "accessibilityTreeFirstCalls": 42,+ "attempts": 100,+ "providers": {},+ "byTool": {+ "navigate": 23,+ "list_downloads": 2,+ "new_tab": 3,+ "get_accessibility_tree": 42,+ "read_page": 4,+ "extract_data": 2,+ "click": 1,+ "download_files": 2,+ "download_social_media": 1,+ "scroll": 1,+ "(no-tool)": 10,+ "get_selection": 1,+ "clarify": 6,+ "scratchpad_write": 1,+ "fetch_url": 1+ },+ "schemaErrors": [],+ "medianLatencyMs": 5118,+ "p95LatencyMs": 8982,+ "cachedPromptSharePct": 32.592371209655596,+ "exactActionMatches": 580,+ "toolNameMatches": 940,+ "comparisons": 1200,+ "exactActionConsensusPct": 48.3,+ "toolNameConsensusPct": 78.3+ },+ {+ "rank": 3,+ "model": "anthropic/claude-sonnet-5",+ "label": "anthropic/claude-sonnet-5",+ "date": "2026-08-02",+ "source": "test/llm/results/2026-08-02-claude-sonnet5-full_chrome_anthropic_claude-sonnet-5",+ "fresh": false,+ "cases": 100,+ "errors": 0,+ "responses": 100,+ "emittedToolCalls": 98,+ "nativeToolCalls": 98,+ "schemaValidToolCalls": 98,+ "idealToolMatches": 47,+ "exactIdealMatches": 17,+ "promptTokens": 3537629,+ "completionTokens": 14671,+ "cachedPromptTokens": 0,+ "reasoningTokens": 3325,+ "costRecords": 100,+ "usageCost": 7.221967999999999,+ "lengthLimited": 0,+ "contentFallbackCalls": 0,+ "accessibilityTreeFirstCalls": 42,+ "attempts": 100,+ "providers": {},+ "byTool": {+ "navigate": 25,+ "list_downloads": 2,+ "get_accessibility_tree": 42,+ "read_page": 3,+ "extract_data": 2,+ "download_files": 1,+ "download_social_media": 2,+ "fetch_url": 2,+ "scroll": 1,+ "(no-tool)": 2,+ "get_selection": 1,+ "clarify": 8,+ "progress_read": 1,+ "done": 6,+ "new_tab": 2+ },+ "schemaErrors": [],+ "medianLatencyMs": 4062,+ "p95LatencyMs": 8278,+ "cachedPromptSharePct": 0,+ "exactActionMatches": 528,+ "toolNameMatches": 895,+ "comparisons": 1200,+ "exactActionConsensusPct": 44,+ "toolNameConsensusPct": 74.6+ },+ {+ "rank": 4,+ "model": "google/gemini-3.6-flash",+ "label": "google/gemini-3.6-flash",+ "date": "2026-08-02",+ "source": "test/llm/results/2026-08-02-gemini36-flash-full-7182_chrome_google_gemini-3.6-flash",+ "fresh": false,+ "cases": 100,+ "errors": 0,+ "responses": 100,+ "emittedToolCalls": 100,+ "nativeToolCalls": 100,+ "schemaValidToolCalls": 100,+ "idealToolMatches": 36,+ "exactIdealMatches": 14,+ "promptTokens": 2125079,+ "completionTokens": 23087,+ "cachedPromptTokens": 1594265,+ "reasoningTokens": 20869,+ "costRecords": 100,+ "usageCost": 1.20851325,+ "lengthLimited": 0,+ "contentFallbackCalls": 0,+ "accessibilityTreeFirstCalls": 59,+ "attempts": 100,+ "providers": {},+ "byTool": {+ "navigate": 18,+ "list_downloads": 2,+ "new_tab": 2,+ "get_accessibility_tree": 59,+ "read_page": 3,+ "extract_data": 2,+ "download_files": 1,+ "download_social_media": 2,+ "scroll": 1,+ "done": 6,+ "get_selection": 1,+ "go_back": 1,+ "progress_read": 1,+ "fetch_url": 1+ },+ "schemaErrors": [],+ "medianLatencyMs": 1980,+ "p95LatencyMs": 3706,+ "cachedPromptSharePct": 75.02144626152722,+ "exactActionMatches": 522,+ "toolNameMatches": 896,+ "comparisons": 1200,+ "exactActionConsensusPct": 43.5,+ "toolNameConsensusPct": 74.7+ },+ {+ "rank": 5,+ "model": "z-ai/glm-5.2",+ "label": "GLM-5.2",+ "date": "2026-08-02",+ "source": "test/llm/results/2026-08-02-glm52-full_chrome_z-ai_glm-5.2",+ "fresh": false,+ "cases": 100,+ "errors": 0,+ "responses": 100,+ "emittedToolCalls": 89,+ "nativeToolCalls": 89,+ "schemaValidToolCalls": 86,+ "idealToolMatches": 39,+ "exactIdealMatches": 20,+ "promptTokens": 2275921,+ "completionTokens": 7496,+ "cachedPromptTokens": 2070582,+ "reasoningTokens": 4202,+ "costRecords": 100,+ "usageCost": 0.5452242279999999,+ "lengthLimited": 0,+ "contentFallbackCalls": 0,+ "accessibilityTreeFirstCalls": 40,+ "attempts": 100,+ "providers": {},+ "byTool": {+ "navigate": 21,+ "list_downloads": 2,+ "new_tab": 3,+ "(no-tool)": 11,+ "get_accessibility_tree": 40,+ "fetch_url": 3,+ "read_page": 6,+ "extract_data": 2,+ "download_files": 1,+ "download_social_media": 2,+ "scroll": 1,+ "get_selection": 1,+ "clarify": 3,+ "go_back": 1,+ "progress_read": 1,+ "done": 2+ },+ "schemaErrors": [+ {+ "id": "032",+ "call": {+ "name": "get_accessibility_tree",+ "args": {+ "filter": "visible\""+ }+ },+ "errors": [+ "$.filter: invalid enum"+ ]+ },+ {+ "id": "045",+ "call": {+ "name": "get_accessibility_tree",+ "args": {+ "filter": "visible\""+ }+ },+ "errors": [+ "$.filter: invalid enum"+ ]+ },+ {+ "id": "046",+ "call": {+ "name": "get_accessibility_tree",+ "args": {+ "filter": "visible\""+ }+ },+ "errors": [+ "$.filter: invalid enum"+ ]+ }+ ],+ "medianLatencyMs": 1732.5,+ "p95LatencyMs": 4436,+ "cachedPromptSharePct": 90.97776240915216,+ "exactActionMatches": 520,+ "toolNameMatches": 915,+ "comparisons": 1200,+ "exactActionConsensusPct": 43.3,+ "toolNameConsensusPct": 76.3+ },+ {+ "rank": 6,+ "model": "minimax/minimax-m3",+ "label": "MiniMax M3",+ "date": "2026-08-02",+ "source": "test/llm/results/2026-08-02-minimax-m3-full_chrome_minimax_minimax-m3",+ "fresh": false,+ "cases": 100,+ "errors": 0,+ "responses": 100,+ "emittedToolCalls": 89,+ "nativeToolCalls": 89,+ "schemaValidToolCalls": 86,+ "idealToolMatches": 33,+ "exactIdealMatches": 17,+ "promptTokens": 2293065,+ "completionTokens": 13130,+ "cachedPromptTokens": 832640,+ "reasoningTokens": 8954,+ "costRecords": 100,+ "usageCost": 0.5038419,+ "lengthLimited": 0,+ "contentFallbackCalls": 0,+ "accessibilityTreeFirstCalls": 45,+ "attempts": 100,+ "providers": {},+ "byTool": {+ "navigate": 22,+ "list_downloads": 2,+ "new_tab": 2,+ "get_accessibility_tree": 45,+ "read_page": 6,+ "extract_data": 2,+ "download_files": 2,+ "download_social_media": 1,+ "scroll": 1,+ "done": 1,+ "get_selection": 1,+ "(no-tool)": 11,+ "scratchpad_write": 1,+ "clarify": 2,+ "press_keys": 1+ },+ "schemaErrors": [+ {+ "id": "030",+ "call": {+ "name": "extract_data",+ "args": {+ "type": "links"+ }+ },+ "errors": [+ "$.type: invalid enum"+ ]+ },+ {+ "id": "077",+ "call": {+ "name": "clarify",+ "args": {+ "options": [+ {+ "value": "Inbox only (what's currently shown)"+ },+ {+ "value": "All Mail (every email across all folders, including Sent)"+ },+ {+ "value": "Inbox + Spam + Trash"+ }+ ],+ "question": "Which emails should I delete?",+ "reason": "\"All my emails\" is ambiguous β the Inbox only has received/unread mail, while All Mail contains everything including Sent. Deleting from All Mail is the most thorough option but also the most destructive."+ }+ },+ "errors": [+ "$.options[0]: expected string",+ "$.options[1]: expected string",+ "$.options[2]: expected string"+ ]+ },+ {+ "id": "092",+ "call": {+ "name": "press_keys",+ "args": {+ "key": "m"+ }+ },+ "errors": [+ "$.key: invalid enum"+ ]+ }+ ],+ "medianLatencyMs": 2864,+ "p95LatencyMs": 7341,+ "cachedPromptSharePct": 36.31122536866596,+ "exactActionMatches": 499,+ "toolNameMatches": 898,+ "comparisons": 1200,+ "exactActionConsensusPct": 41.6,+ "toolNameConsensusPct": 74.8+ },+ {+ "rank": 7,+ "model": "poolside/laguna-xs-2.1",+ "label": "Poolside Laguna XS 2.1",+ "date": "2026-08-02",+ "source": "test/llm/results/2026-08-02-laguna-xs-full-7182_chrome_poolside_laguna-xs-2.1",+ "fresh": false,+ "cases": 100,+ "errors": 0,+ "responses": 100,+ "emittedToolCalls": 89,+ "nativeToolCalls": 89,+ "schemaValidToolCalls": 88,+ "idealToolMatches": 31,+ "exactIdealMatches": 5,+ "promptTokens": 2333108,+ "completionTokens": 16306,+ "cachedPromptTokens": 2301344,+ "reasoningTokens": 7963,+ "costRecords": 100,+ "usageCost": 0.07290288000000006,+ "lengthLimited": 1,+ "contentFallbackCalls": 0,+ "accessibilityTreeFirstCalls": 45,+ "attempts": 100,+ "providers": {},+ "byTool": {+ "navigate": 19,+ "list_downloads": 2,+ "new_tab": 3,+ "read_page": 9,+ "get_accessibility_tree": 45,+ "research_url": 2,+ "click": 2,+ "download_files": 1,+ "download_social_media": 2,+ "(no-tool)": 11,+ "get_selection": 1,+ "go_back": 1,+ "fetch_url": 1,+ "press_keys": 1+ },+ "schemaErrors": [+ {+ "id": "092",+ "call": {+ "name": "press_keys",+ "args": {+ "key": "m"+ }+ },+ "errors": [+ "$.key: invalid enum"+ ]+ }+ ],+ "medianLatencyMs": 1146,+ "p95LatencyMs": 2147,+ "cachedPromptSharePct": 98.63855423752351,+ "exactActionMatches": 495,+ "toolNameMatches": 882,+ "comparisons": 1200,+ "exactActionConsensusPct": 41.3,+ "toolNameConsensusPct": 73.5+ },+ {+ "rank": 8,+ "model": "moonshotai/kimi-k3",+ "label": "moonshotai/kimi-k3",+ "date": "2026-08-02",+ "source": "test/llm/results/2026-08-02-kimi-k3-full_chrome_moonshotai_kimi-k3",+ "fresh": false,+ "cases": 100,+ "errors": 0,+ "responses": 100,+ "emittedToolCalls": 97,+ "nativeToolCalls": 97,+ "schemaValidToolCalls": 97,+ "idealToolMatches": 44,+ "exactIdealMatches": 18,+ "promptTokens": 2184113,+ "completionTokens": 25148,+ "cachedPromptTokens": 1987734,+ "reasoningTokens": 17188,+ "costRecords": 100,+ "usageCost": 1.5626771999999995,+ "lengthLimited": 0,+ "contentFallbackCalls": 0,+ "accessibilityTreeFirstCalls": 45,+ "attempts": 100,+ "providers": {},+ "byTool": {+ "navigate": 24,+ "list_downloads": 2,+ "new_tab": 2,+ "get_accessibility_tree": 45,+ "read_page": 6,+ "extract_data": 2,+ "download_files": 1,+ "download_social_media": 2,+ "scroll": 1,+ "(no-tool)": 3,+ "get_selection": 1,+ "clarify": 4,+ "go_back": 1,+ "progress_read": 1,+ "done": 4,+ "fetch_url": 1+ },+ "schemaErrors": [],+ "medianLatencyMs": 7598,+ "p95LatencyMs": 26154,+ "cachedPromptSharePct": 91.00875275226144,+ "exactActionMatches": 492,+ "toolNameMatches": 941,+ "comparisons": 1200,+ "exactActionConsensusPct": 41,+ "toolNameConsensusPct": 78.4+ },+ {+ "rank": 9,+ "model": "qwen/qwen3.6-27b",+ "label": "qwen/qwen3.6-27b",+ "date": "2026-08-02",+ "source": "test/llm/results/2026-08-02-qwen36-27b-full_chrome_qwen_qwen3.6-27b",+ "fresh": false,+ "cases": 100,+ "errors": 0,+ "responses": 100,+ "emittedToolCalls": 92,+ "nativeToolCalls": 92,+ "schemaValidToolCalls": 83,+ "idealToolMatches": 36,+ "exactIdealMatches": 17,+ "promptTokens": 2240915,+ "completionTokens": 8890,+ "cachedPromptTokens": 1958464,+ "reasoningTokens": 5186,+ "costRecords": 100,+ "usageCost": 0.6698491209999997,+ "lengthLimited": 0,+ "contentFallbackCalls": 0,+ "accessibilityTreeFirstCalls": 44,+ "attempts": 100,+ "providers": {},+ "byTool": {+ "navigate": 21,+ "list_downloads": 2,+ "new_tab": 3,+ "get_accessibility_tree": 44,+ "read_page": 7,+ "extract_data": 2,+ "click_ax": 1,+ "download_files": 2,+ "scroll": 1,+ "(no-tool)": 8,+ "get_selection": 1,+ "get_interactive_elements": 1,+ "clarify": 2,+ "go_back": 1,+ "scratchpad_write": 1,+ "done": 1,+ "research_url": 1,+ "press_keys": 1+ },+ "schemaErrors": [+ {+ "id": "033",+ "call": {+ "name": "get_accessibility_tree",+ "args": {+ "filter": "\"visible\"",+ "maxDepth": 10+ }+ },+ "errors": [+ "$.filter: invalid enum"+ ]+ },+ {+ "id": "043",+ "call": {+ "name": "get_accessibility_tree",+ "args": {+ "filter": "\"visible\""+ }+ },+ "errors": [+ "$.filter: invalid enum"+ ]+ },+ {+ "id": "045",+ "call": {+ "name": "get_accessibility_tree",+ "args": {+ "maxDepth": 10,+ "filter": "\"visible\""+ }+ },+ "errors": [+ "$.filter: invalid enum"+ ]+ },+ {+ "id": "048",+ "call": {+ "name": "get_accessibility_tree",+ "args": {+ "maxDepth": 8,+ "filter": "\"visible"+ }+ },+ "errors": [+ "$.filter: invalid enum"+ ]+ },+ {+ "id": "049",+ "call": {+ "name": "get_accessibility_tree",+ "args": {+ "filter": "\"visible\""+ }+ },+ "errors": [+ "$.filter: invalid enum"+ ]+ },+ {+ "id": "077",+ "call": {+ "name": "get_accessibility_tree",+ "args": {+ "maxDepth": 6,+ "filter": "\"visible\""+ }+ },+ "errors": [+ "$.filter: invalid enum"+ ]+ },+ {+ "id": "079",+ "call": {+ "name": "get_accessibility_tree",+ "args": {+ "filter": "\"visible\""+ }+ },+ "errors": [+ "$.filter: invalid enum"+ ]+ },+ {+ "id": "092",+ "call": {+ "name": "press_keys",+ "args": {+ "key": "m"+ }+ },+ "errors": [+ "$.key: invalid enum"+ ]+ },+ {+ "id": "094",+ "call": {+ "name": "get_accessibility_tree",+ "args": {+ "maxDepth": 8,+ "filter": "\"visible\""+ }+ },+ "errors": [+ "$.filter: invalid enum"+ ]+ }+ ],+ "medianLatencyMs": 2268.5,+ "p95LatencyMs": 19355,+ "cachedPromptSharePct": 87.39572897677958,+ "exactActionMatches": 457,+ "toolNameMatches": 892,+ "comparisons": 1200,+ "exactActionConsensusPct": 38.1,+ "toolNameConsensusPct": 74.3+ },+ {+ "rank": 10,+ "model": "thinkingmachines/inkling-small",+ "label": "thinkingmachines/inkling-small",+ "date": "2026-08-02",+ "source": "test/llm/results/2026-08-02-inkling-small-full_chrome_thinkingmachines_inkling-small",+ "fresh": false,+ "cases": 100,+ "errors": 0,+ "responses": 100,+ "emittedToolCalls": 84,+ "nativeToolCalls": 84,+ "schemaValidToolCalls": 84,+ "idealToolMatches": 32,+ "exactIdealMatches": 15,+ "promptTokens": 2162719,+ "completionTokens": 10091,+ "cachedPromptTokens": 2125184,+ "reasoningTokens": 6064,+ "costRecords": 100,+ "usageCost": 0.24339510000000006,+ "lengthLimited": 0,+ "contentFallbackCalls": 0,+ "accessibilityTreeFirstCalls": 45,+ "attempts": 100,+ "providers": {},+ "byTool": {+ "navigate": 22,+ "list_downloads": 2,+ "new_tab": 3,+ "get_accessibility_tree": 45,+ "read_page": 4,+ "extract_data": 1,+ "(no-tool)": 16,+ "download_files": 1,+ "download_social_media": 1,+ "scroll": 1,+ "get_selection": 1,+ "get_interactive_elements": 1,+ "go_back": 1,+ "fetch_url": 1+ },+ "schemaErrors": [],+ "medianLatencyMs": 1049.5,+ "p95LatencyMs": 2064,+ "cachedPromptSharePct": 98.26445321837926,+ "exactActionMatches": 418,+ "toolNameMatches": 890,+ "comparisons": 1200,+ "exactActionConsensusPct": 34.8,+ "toolNameConsensusPct": 74.2+ },+ {+ "rank": 11,+ "model": "x-ai/grok-4.5",+ "label": "x-ai/grok-4.5",+ "date": "2026-08-02",+ "source": "test/llm/results/2026-08-02-grok45-full-7182_chrome_x-ai_grok-4.5",+ "fresh": false,+ "cases": 100,+ "errors": 0,+ "responses": 100,+ "emittedToolCalls": 94,+ "nativeToolCalls": 94,+ "schemaValidToolCalls": 94,+ "idealToolMatches": 36,+ "exactIdealMatches": 17,+ "promptTokens": 2393185,+ "completionTokens": 7716,+ "cachedPromptTokens": 1504896,+ "reasoningTokens": 4423,+ "costRecords": 100,+ "usageCost": 2.2743427999999994,+ "lengthLimited": 0,+ "contentFallbackCalls": 0,+ "accessibilityTreeFirstCalls": 49,+ "attempts": 100,+ "providers": {},+ "byTool": {+ "navigate": 21,+ "list_downloads": 2,+ "new_tab": 3,+ "get_accessibility_tree": 49,+ "read_page": 5,+ "extract_data": 2,+ "verify_form": 1,+ "download_files": 1,+ "download_social_media": 2,+ "scroll": 1,+ "(no-tool)": 6,+ "get_selection": 1,+ "clarify": 3,+ "go_back": 1,+ "progress_read": 1,+ "done": 1+ },+ "schemaErrors": [],+ "medianLatencyMs": 2615.5,+ "p95LatencyMs": 4848,+ "cachedPromptSharePct": 62.882560270100306,+ "exactActionMatches": 417,+ "toolNameMatches": 956,+ "comparisons": 1200,+ "exactActionConsensusPct": 34.8,+ "toolNameConsensusPct": 79.7+ },+ {+ "rank": 12,+ "model": "openai/gpt-5.6-luna-pro",+ "label": "openai/gpt-5.6-luna-pro",+ "date": "2026-08-02",+ "source": "test/llm/results/2026-08-02-gpt56-luna-full_chrome_openai_gpt-5.6-luna-pro",+ "fresh": false,+ "cases": 100,+ "errors": 0,+ "responses": 100,+ "emittedToolCalls": 89,+ "nativeToolCalls": 89,+ "schemaValidToolCalls": 89,+ "idealToolMatches": 32,+ "exactIdealMatches": 3,+ "promptTokens": 7266644,+ "completionTokens": 37589,+ "cachedPromptTokens": 5810652,+ "reasoningTokens": 24088,+ "costRecords": 100,+ "usageCost": 0.22788836999999995,+ "lengthLimited": 0,+ "contentFallbackCalls": 0,+ "accessibilityTreeFirstCalls": 54,+ "attempts": 100,+ "providers": {},+ "byTool": {+ "navigate": 15,+ "list_downloads": 2,+ "new_tab": 3,+ "get_accessibility_tree": 54,+ "read_page": 3,+ "extract_data": 2,+ "download_social_media": 2,+ "(no-tool)": 11,+ "get_selection": 1,+ "clarify": 5,+ "go_back": 1,+ "fetch_url": 1+ },+ "schemaErrors": [],+ "medianLatencyMs": 5125,+ "p95LatencyMs": 7958,+ "cachedPromptSharePct": 79.96335034439556,+ "exactActionMatches": 162,+ "toolNameMatches": 902,+ "comparisons": 1200,+ "exactActionConsensusPct": 13.5,+ "toolNameConsensusPct": 75.2+ },+ {+ "rank": 13,+ "model": "openai/gpt-5.6-terra-pro",+ "label": "openai/gpt-5.6-terra-pro",+ "date": "2026-08-02",+ "source": "test/llm/results/2026-08-02-gpt56-terra-full_chrome_openai_gpt-5.6-terra-pro",+ "fresh": false,+ "cases": 100,+ "errors": 0,+ "responses": 100,+ "emittedToolCalls": 94,+ "nativeToolCalls": 94,+ "schemaValidToolCalls": 94,+ "idealToolMatches": 12,+ "exactIdealMatches": 2,+ "promptTokens": 7263146,+ "completionTokens": 32506,+ "cachedPromptTokens": 5829694,+ "reasoningTokens": 18680,+ "costRecords": 100,+ "usageCost": 2.23778015,+ "lengthLimited": 0,+ "contentFallbackCalls": 0,+ "accessibilityTreeFirstCalls": 80,+ "attempts": 100,+ "providers": {},+ "byTool": {+ "get_accessibility_tree": 80,+ "new_tab": 2,+ "navigate": 2,+ "read_page": 2,+ "download_resource_from_page": 1,+ "download_social_media": 2,+ "list_downloads": 1,+ "(no-tool)": 6,+ "get_selection": 1,+ "clarify": 3+ },+ "schemaErrors": [],+ "medianLatencyMs": 4921.5,+ "p95LatencyMs": 7694,+ "cachedPromptSharePct": 80.26403434544756,+ "exactActionMatches": 103,+ "toolNameMatches": 705,+ "comparisons": 1200,+ "exactActionConsensusPct": 8.6,+ "toolNameConsensusPct": 58.8+ }+ ]+}diff --git a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/historical-vision-regression.json b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/historical-vision-regression.jsonnew file mode 100644index 000000000..6f1f94c42--- /dev/null+++ b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/historical-vision-regression.json@@ -0,0 +1,1568 @@+{+ "models": 2,+ "cases": 200,+ "inputAndScoreMatches": true,+ "rows": [+ {+ "model": "qwen/qwen3.8-flash",+ "label": "Qwen3.8 Flash",+ "date": "2026-08-27",+ "path": "test/vision/results/2026-08-27-openrouter-full_qwen_qwen3.8-flash_production",+ "cases": 100,+ "errors": 0,+ "passes": 79,+ "meanScore": 0.9476381766381763,+ "meanLatencyMs": 8824.91,+ "medianLatencyMs": 8495,+ "p95LatencyMs": 13428,+ "costRecords": 100,+ "usageCost": 0.04105615000000001,+ "promptTokens": 123400,+ "completionTokens": 45345,+ "cachedTokens": 0,+ "reasoningTokens": 19241,+ "sixSections": 99,+ "emptyOutputs": 0,+ "lengthLimited": null,+ "byDifficulty": {+ "easy": {+ "cases": 20,+ "passes": 14+ },+ "basic": {+ "cases": 20,+ "passes": 17+ },+ "intermediate": {+ "cases": 20,+ "passes": 17+ },+ "advanced": {+ "cases": 20,+ "passes": 16+ },+ "challenging": {+ "cases": 20,+ "passes": 15+ }+ },+ "byCategory": {+ "authentication": {+ "cases": 5,+ "passes": 5+ },+ "search-results": {+ "cases": 5,+ "passes": 4+ },+ "checkout": {+ "cases": 5,+ "passes": 5+ },+ "form-validation": {+ "cases": 5,+ "passes": 1+ },+ "modal-overlay": {+ "cases": 5,+ "passes": 0+ },+ "toast-notification": {+ "cases": 5,+ "passes": 5+ },+ "loading-state": {+ "cases": 5,+ "passes": 3+ },+ "consent-banner": {+ "cases": 5,+ "passes": 5+ },+ "dashboard": {+ "cases": 5,+ "passes": 5+ },+ "chart-reading": {+ "cases": 5,+ "passes": 5+ },+ "data-table": {+ "cases": 5,+ "passes": 5+ },+ "email-compose": {+ "cases": 5,+ "passes": 4+ },+ "kanban-board": {+ "cases": 5,+ "passes": 5+ },+ "calendar": {+ "cases": 5,+ "passes": 5+ },+ "map-and-travel": {+ "cases": 5,+ "passes": 4+ },+ "photo-understanding": {+ "cases": 5,+ "passes": 4+ },+ "multilingual-ocr": {+ "cases": 5,+ "passes": 2+ },+ "occlusion-and-contrast": {+ "cases": 5,+ "passes": 4+ },+ "security-challenge": {+ "cases": 5,+ "passes": 4+ },+ "uncertainty-calibration": {+ "cases": 5,+ "passes": 4+ }+ },+ "providers": {+ "not recorded": {+ "cases": 100,+ "passes": 79+ }+ },+ "caseScores": [+ {+ "id": "001",+ "pass": true,+ "ratio": 0.8,+ "criticalFailures": []+ },+ {+ "id": "002",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "003",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "004",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "005",+ "pass": false,+ "ratio": 0.8181818181818182,+ "criticalFailures": [+ "modal-signal"+ ]+ },+ {+ "id": "006",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "007",+ "pass": false,+ "ratio": 0.75,+ "criticalFailures": [+ "heading"+ ]+ },+ {+ "id": "008",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "009",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "010",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "011",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "012",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "013",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "014",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "015",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "016",+ "pass": false,+ "ratio": 0.5555555555555556,+ "criticalFailures": [+ "caption"+ ]+ },+ {+ "id": "017",+ "pass": false,+ "ratio": 0.7777777777777778,+ "criticalFailures": [+ "input"+ ]+ },+ {+ "id": "018",+ "pass": false,+ "ratio": 0.8,+ "criticalFailures": [+ "target"+ ]+ },+ {+ "id": "019",+ "pass": false,+ "ratio": 0.8181818181818182,+ "criticalFailures": [+ "label"+ ]+ },+ {+ "id": "020",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "021",+ "pass": true,+ "ratio": 0.9090909090909091,+ "criticalFailures": []+ },+ {+ "id": "022",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "023",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "024",+ "pass": false,+ "ratio": 0.8,+ "criticalFailures": [+ "input"+ ]+ },+ {+ "id": "025",+ "pass": false,+ "ratio": 0.8181818181818182,+ "criticalFailures": [+ "modal-signal"+ ]+ },+ {+ "id": "026",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "027",+ "pass": true,+ "ratio": 0.9090909090909091,+ "criticalFailures": []+ },+ {+ "id": "028",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "029",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "030",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "031",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "032",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "033",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "034",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "035",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "036",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "037",+ "pass": false,+ "ratio": 0.8181818181818182,+ "criticalFailures": [+ "input"+ ]+ },+ {+ "id": "038",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "039",+ "pass": true,+ "ratio": 0.9090909090909091,+ "criticalFailures": []+ },+ {+ "id": "040",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "041",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "042",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "043",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "044",+ "pass": false,+ "ratio": 0.8,+ "criticalFailures": [+ "input"+ ]+ },+ {+ "id": "045",+ "pass": false,+ "ratio": 0.8181818181818182,+ "criticalFailures": [+ "modal-signal"+ ]+ },+ {+ "id": "046",+ "pass": true,+ "ratio": 0.9629629629629629,+ "criticalFailures": []+ },+ {+ "id": "047",+ "pass": true,+ "ratio": 0.9090909090909091,+ "criticalFailures": []+ },+ {+ "id": "048",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "049",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "050",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "051",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "052",+ "pass": false,+ "ratio": 0.8,+ "criticalFailures": [+ "status"+ ]+ },+ {+ "id": "053",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "054",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "055",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "056",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "057",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "058",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "059",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "060",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "061",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "062",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "063",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "064",+ "pass": false,+ "ratio": 0.8,+ "criticalFailures": [+ "input"+ ]+ },+ {+ "id": "065",+ "pass": false,+ "ratio": 0.8181818181818182,+ "criticalFailures": [+ "modal-signal"+ ]+ },+ {+ "id": "066",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "067",+ "pass": false,+ "ratio": 0.7777777777777778,+ "criticalFailures": [+ "heading"+ ]+ },+ {+ "id": "068",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "069",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "070",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "071",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "072",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "073",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "074",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "075",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "076",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "077",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "078",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "079",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "080",+ "pass": false,+ "ratio": 0.7,+ "criticalFailures": [+ "visible-fragment"+ ]+ },+ {+ "id": "081",+ "pass": true,+ "ratio": 0.9230769230769231,+ "criticalFailures": []+ },+ {+ "id": "082",+ "pass": false,+ "ratio": 0.9166666666666666,+ "criticalFailures": [+ "notice"+ ]+ },+ {+ "id": "083",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "084",+ "pass": false,+ "ratio": 0.8,+ "criticalFailures": [+ "input"+ ]+ },+ {+ "id": "085",+ "pass": false,+ "ratio": 0.8181818181818182,+ "criticalFailures": [+ "modal-signal"+ ]+ },+ {+ "id": "086",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "087",+ "pass": true,+ "ratio": 0.8181818181818182,+ "criticalFailures": []+ },+ {+ "id": "088",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "089",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "090",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "091",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "092",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "093",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "094",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "095",+ "pass": false,+ "ratio": 0.8,+ "criticalFailures": [+ "route-alert"+ ]+ },+ {+ "id": "096",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "097",+ "pass": false,+ "ratio": 0.8181818181818182,+ "criticalFailures": [+ "input"+ ]+ },+ {+ "id": "098",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "099",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "100",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ }+ ]+ },+ {+ "model": "z-ai/glm-5.3-flash",+ "label": "GLM-5.3 Flash",+ "date": "2026-08-27",+ "path": "test/vision/results/2026-08-27-openrouter-full_z-ai_glm-5.3-flash_production",+ "cases": 100,+ "errors": 0,+ "passes": 76,+ "meanScore": 0.9382857420357417,+ "meanLatencyMs": 5859.7,+ "medianLatencyMs": 5490.5,+ "p95LatencyMs": 8917,+ "costRecords": 100,+ "usageCost": 0.02105408,+ "promptTokens": 150200,+ "completionTokens": 45116,+ "cachedTokens": 24832,+ "reasoningTokens": 19049,+ "sixSections": 100,+ "emptyOutputs": 0,+ "lengthLimited": null,+ "byDifficulty": {+ "easy": {+ "cases": 20,+ "passes": 16+ },+ "basic": {+ "cases": 20,+ "passes": 14+ },+ "intermediate": {+ "cases": 20,+ "passes": 17+ },+ "advanced": {+ "cases": 20,+ "passes": 16+ },+ "challenging": {+ "cases": 20,+ "passes": 13+ }+ },+ "byCategory": {+ "authentication": {+ "cases": 5,+ "passes": 2+ },+ "search-results": {+ "cases": 5,+ "passes": 5+ },+ "checkout": {+ "cases": 5,+ "passes": 5+ },+ "form-validation": {+ "cases": 5,+ "passes": 1+ },+ "modal-overlay": {+ "cases": 5,+ "passes": 0+ },+ "toast-notification": {+ "cases": 5,+ "passes": 5+ },+ "loading-state": {+ "cases": 5,+ "passes": 3+ },+ "consent-banner": {+ "cases": 5,+ "passes": 5+ },+ "dashboard": {+ "cases": 5,+ "passes": 5+ },+ "chart-reading": {+ "cases": 5,+ "passes": 5+ },+ "data-table": {+ "cases": 5,+ "passes": 5+ },+ "email-compose": {+ "cases": 5,+ "passes": 3+ },+ "kanban-board": {+ "cases": 5,+ "passes": 5+ },+ "calendar": {+ "cases": 5,+ "passes": 5+ },+ "map-and-travel": {+ "cases": 5,+ "passes": 4+ },+ "photo-understanding": {+ "cases": 5,+ "passes": 5+ },+ "multilingual-ocr": {+ "cases": 5,+ "passes": 3+ },+ "occlusion-and-contrast": {+ "cases": 5,+ "passes": 3+ },+ "security-challenge": {+ "cases": 5,+ "passes": 4+ },+ "uncertainty-calibration": {+ "cases": 5,+ "passes": 3+ }+ },+ "providers": {+ "not recorded": {+ "cases": 100,+ "passes": 76+ }+ },+ "caseScores": [+ {+ "id": "001",+ "pass": false,+ "ratio": 0.7,+ "criticalFailures": [+ "purpose"+ ]+ },+ {+ "id": "002",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "003",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "004",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "005",+ "pass": false,+ "ratio": 0.8181818181818182,+ "criticalFailures": [+ "modal-signal"+ ]+ },+ {+ "id": "006",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "007",+ "pass": false,+ "ratio": 0.75,+ "criticalFailures": [+ "heading"+ ]+ },+ {+ "id": "008",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "009",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "010",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "011",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "012",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "013",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "014",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "015",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "016",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "017",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "018",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "019",+ "pass": false,+ "ratio": 0.6363636363636364,+ "criticalFailures": [+ "label",+ "security-blocker"+ ]+ },+ {+ "id": "020",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "021",+ "pass": false,+ "ratio": 0.8181818181818182,+ "criticalFailures": [+ "purpose"+ ]+ },+ {+ "id": "022",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "023",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "024",+ "pass": false,+ "ratio": 0.7,+ "criticalFailures": [+ "input",+ "blocker"+ ]+ },+ {+ "id": "025",+ "pass": false,+ "ratio": 0.8181818181818182,+ "criticalFailures": [+ "modal-signal"+ ]+ },+ {+ "id": "026",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "027",+ "pass": true,+ "ratio": 0.9090909090909091,+ "criticalFailures": []+ },+ {+ "id": "028",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "029",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "030",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "031",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "032",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "033",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "034",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "035",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "036",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "037",+ "pass": false,+ "ratio": 0.8181818181818182,+ "criticalFailures": [+ "input"+ ]+ },+ {+ "id": "038",+ "pass": false,+ "ratio": 0.8,+ "criticalFailures": [+ "target"+ ]+ },+ {+ "id": "039",+ "pass": true,+ "ratio": 0.9090909090909091,+ "criticalFailures": []+ },+ {+ "id": "040",+ "pass": false,+ "ratio": 0.8,+ "criticalFailures": [+ "visible-fragment"+ ]+ },+ {+ "id": "041",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "042",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "043",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "044",+ "pass": false,+ "ratio": 0.8,+ "criticalFailures": [+ "input"+ ]+ },+ {+ "id": "045",+ "pass": false,+ "ratio": 0.8181818181818182,+ "criticalFailures": [+ "modal-signal"+ ]+ },+ {+ "id": "046",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "047",+ "pass": true,+ "ratio": 0.9090909090909091,+ "criticalFailures": []+ },+ {+ "id": "048",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "049",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "050",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "051",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "052",+ "pass": false,+ "ratio": 0.8,+ "criticalFailures": [+ "status"+ ]+ },+ {+ "id": "053",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "054",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "055",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "056",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "057",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "058",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "059",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "060",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "061",+ "pass": true,+ "ratio": 0.9230769230769231,+ "criticalFailures": []+ },+ {+ "id": "062",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "063",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "064",+ "pass": false,+ "ratio": 0.8,+ "criticalFailures": [+ "input"+ ]+ },+ {+ "id": "065",+ "pass": false,+ "ratio": 0.8181818181818182,+ "criticalFailures": [+ "modal-signal"+ ]+ },+ {+ "id": "066",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "067",+ "pass": false,+ "ratio": 0.5555555555555556,+ "criticalFailures": [+ "heading"+ ]+ },+ {+ "id": "068",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "069",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "070",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "071",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "072",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "073",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "074",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "075",+ "pass": true,+ "ratio": 0.875,+ "criticalFailures": []+ },+ {+ "id": "076",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "077",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "078",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "079",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "080",+ "pass": false,+ "ratio": 0.7,+ "criticalFailures": [+ "visible-fragment"+ ]+ },+ {+ "id": "081",+ "pass": false,+ "ratio": 0.8461538461538461,+ "criticalFailures": [+ "purpose"+ ]+ },+ {+ "id": "082",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "083",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "084",+ "pass": false,+ "ratio": 0.8,+ "criticalFailures": [+ "input"+ ]+ },+ {+ "id": "085",+ "pass": false,+ "ratio": 0.8181818181818182,+ "criticalFailures": [+ "modal-signal"+ ]+ },+ {+ "id": "086",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "087",+ "pass": true,+ "ratio": 0.9090909090909091,+ "criticalFailures": []+ },+ {+ "id": "088",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "089",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "090",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "091",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "092",+ "pass": false,+ "ratio": 0.8333333333333334,+ "criticalFailures": [+ "external-blocker"+ ]+ },+ {+ "id": "093",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "094",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "095",+ "pass": false,+ "ratio": 0.8,+ "criticalFailures": [+ "route-alert"+ ]+ },+ {+ "id": "096",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "097",+ "pass": false,+ "ratio": 0.5454545454545454,+ "criticalFailures": [+ "input",+ "status"+ ]+ },+ {+ "id": "098",+ "pass": false,+ "ratio": 0.8,+ "criticalFailures": [+ "cover"+ ]+ },+ {+ "id": "099",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ },+ {+ "id": "100",+ "pass": true,+ "ratio": 1,+ "criticalFailures": []+ }+ ]+ }+ ]+}diff --git a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/openrouter-endpoints.json b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/openrouter-endpoints.jsonnew file mode 100644index 000000000..e992dc324--- /dev/null+++ b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/openrouter-endpoints.json@@ -0,0 +1,1046 @@+[+ {+ "model": "tencent/hy4-preview",+ "label": "Tencent HY4 Preview",+ "vision": false,+ "url": "https://openrouter.ai/api/v1/models/tencent/hy4-preview/endpoints",+ "status": 200,+ "retrievedAt": "2026-10-13T22:49:54.891Z",+ "data": {+ "data": {+ "id": "tencent/hy4-preview",+ "name": "Tencent: Hy4 preview",+ "created": 1787897375,+ "description": "Tencent: Hy4 preview is a mixture-of-experts model from Tencent, with 49B active parameters out of 770B total. It is designed for coding agents, complex tool-use workflows, and productivity tasks that...",+ "architecture": {+ "tokenizer": "Other",+ "instruct_type": null,+ "modality": "text->text",+ "input_modalities": [+ "text"+ ],+ "output_modalities": [+ "text"+ ]+ },+ "endpoints": [+ {+ "name": "Tencent | tencent/hy4-preview-20260827",+ "model_id": "tencent/hy4-preview",+ "model_name": "Tencent: Hy4 preview",+ "context_length": 1048576,+ "pricing": {+ "prompt": "0.0000007506",+ "completion": "0.0000022509",+ "input_cache_read": "0.0000000378",+ "discount": 0,+ "overrides": [+ {+ "utc_start": 0,+ "utc_end": 1600,+ "prompt": "0.000000834",+ "completion": "0.000002501",+ "input_cache_read": "0.000000042"+ },+ {+ "utc_start": 1600,+ "utc_end": 0,+ "prompt": "0.0000007506",+ "completion": "0.0000022509",+ "input_cache_read": "0.0000000378"+ }+ ]+ },+ "provider_name": "Tencent",+ "tag": "tencent/fp8",+ "quantization": "fp8",+ "max_completion_tokens": 64000,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "temperature",+ "stop",+ "max_completion_tokens",+ "max_tokens",+ "response_format",+ "structured_outputs",+ "tools",+ "tool_choice",+ "reasoning_effort"+ ],+ "supports_tool_choice": {+ "none": true,+ "auto": true,+ "required": false,+ "function": false+ },+ "status": 0,+ "uptime_last_30m": 100,+ "uptime_last_5m": 100,+ "uptime_last_1d": 100,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ },+ {+ "name": "Makora | tencent/hy4-preview-20260827",+ "model_id": "tencent/hy4-preview",+ "model_name": "Tencent: Hy4 preview",+ "context_length": 131000,+ "pricing": {+ "prompt": "0.00000083",+ "completion": "0.000002501",+ "input_cache_read": "0.000000041",+ "discount": 0+ },+ "provider_name": "Makora",+ "tag": "makora/fp8",+ "quantization": "fp8",+ "max_completion_tokens": 128000,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "temperature",+ "top_p",+ "top_k",+ "frequency_penalty",+ "presence_penalty",+ "repetition_penalty",+ "stop",+ "max_tokens",+ "seed",+ "structured_outputs",+ "tools",+ "tool_choice",+ "logprobs",+ "top_logprobs",+ "reasoning_effort"+ ],+ "supports_tool_choice": {+ "none": false,+ "auto": true,+ "required": true,+ "function": true+ },+ "status": 0,+ "uptime_last_30m": 100,+ "uptime_last_5m": 100,+ "uptime_last_1d": 100,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ },+ {+ "name": "DeepInfra | tencent/hy4-preview-20260827",+ "model_id": "tencent/hy4-preview",+ "model_name": "Tencent: Hy4 preview",+ "context_length": 1048576,+ "pricing": {+ "prompt": "0.000000834",+ "completion": "0.000002501",+ "input_cache_read": "0.000000042",+ "discount": 0+ },+ "provider_name": "DeepInfra",+ "tag": "deepinfra/fp8",+ "quantization": "fp8",+ "max_completion_tokens": 131072,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "max_tokens",+ "temperature",+ "top_p",+ "stop",+ "frequency_penalty",+ "presence_penalty",+ "repetition_penalty",+ "top_k",+ "seed",+ "min_p",+ "logit_bias",+ "response_format",+ "structured_outputs",+ "tools",+ "tool_choice",+ "reasoning_effort"+ ],+ "supports_tool_choice": {+ "none": true,+ "auto": true,+ "required": true,+ "function": true+ },+ "status": 0,+ "uptime_last_30m": 100,+ "uptime_last_5m": null,+ "uptime_last_1d": 100,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ },+ {+ "name": "Novita | tencent/hy4-preview-20260827",+ "model_id": "tencent/hy4-preview",+ "model_name": "Tencent: Hy4 preview",+ "context_length": 1000000,+ "pricing": {+ "prompt": "0.000000834",+ "completion": "0.000002501",+ "input_cache_read": "0.000000042",+ "discount": 0+ },+ "provider_name": "Novita",+ "tag": "novita/fp8",+ "quantization": "fp8",+ "max_completion_tokens": 64000,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "max_tokens",+ "temperature",+ "top_p",+ "stop",+ "frequency_penalty",+ "presence_penalty",+ "seed",+ "top_k",+ "repetition_penalty",+ "structured_outputs",+ "tools",+ "tool_choice",+ "reasoning_effort"+ ],+ "supports_tool_choice": {+ "none": true,+ "auto": true,+ "required": false,+ "function": true+ },+ "status": 0,+ "uptime_last_30m": 100,+ "uptime_last_5m": null,+ "uptime_last_1d": 100,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ },+ {+ "name": "SiliconFlow | tencent/hy4-preview-20260827",+ "model_id": "tencent/hy4-preview",+ "model_name": "Tencent: Hy4 preview",+ "context_length": 1048576,+ "pricing": {+ "prompt": "0.000000834",+ "completion": "0.000002501",+ "input_cache_read": "0.000000042",+ "discount": 0+ },+ "provider_name": "SiliconFlow",+ "tag": "siliconflow/fp8",+ "quantization": "fp8",+ "max_completion_tokens": 262144,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "temperature",+ "top_p",+ "top_k",+ "frequency_penalty",+ "max_tokens",+ "response_format",+ "structured_outputs",+ "tools",+ "tool_choice",+ "reasoning_effort"+ ],+ "supports_tool_choice": {+ "none": true,+ "auto": true,+ "required": true,+ "function": true+ },+ "status": 0,+ "uptime_last_30m": 100,+ "uptime_last_5m": 100,+ "uptime_last_1d": 99.14224446032881,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ }+ ]+ }+ }+ },+ {+ "model": "xiaomi/mimo-v2.6-pro",+ "label": "MiMo-V2.6-Pro",+ "vision": true,+ "url": "https://openrouter.ai/api/v1/models/xiaomi/mimo-v2.6-pro/endpoints",+ "status": 200,+ "retrievedAt": "2026-10-13T22:49:54.893Z",+ "data": {+ "data": {+ "id": "xiaomi/mimo-v2.6-pro",+ "name": "Xiaomi: MiMo-V2.6-Pro",+ "created": 1790021259,+ "description": "MiMo-V2.6-Pro is the flagship foundation model developed by Xiaomi. Built at a scale of over 1T parameters, it is designed to push the ceiling of capability for the most demanding...",+ "architecture": {+ "tokenizer": "Other",+ "instruct_type": null,+ "modality": "text+image+audio+video->text",+ "input_modalities": [+ "text",+ "image",+ "video",+ "audio"+ ],+ "output_modalities": [+ "text"+ ]+ },+ "endpoints": [+ {+ "name": "DeepInfra | xiaomi/mimo-v2.6-pro-20260921",+ "model_id": "xiaomi/mimo-v2.6-pro",+ "model_name": "Xiaomi: MiMo-V2.6-Pro",+ "context_length": 1048576,+ "pricing": {+ "prompt": "0.00000043",+ "completion": "0.00000087",+ "input_cache_read": "0.0000000036",+ "discount": 0+ },+ "provider_name": "DeepInfra",+ "tag": "deepinfra/fp8",+ "quantization": "fp8",+ "max_completion_tokens": 943718,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "max_tokens",+ "temperature",+ "top_p",+ "stop",+ "frequency_penalty",+ "presence_penalty",+ "repetition_penalty",+ "top_k",+ "seed",+ "min_p",+ "logit_bias",+ "response_format",+ "structured_outputs",+ "tools",+ "tool_choice"+ ],+ "supports_tool_choice": {+ "none": true,+ "auto": true,+ "required": true,+ "function": true+ },+ "status": 0,+ "uptime_last_30m": 100,+ "uptime_last_5m": 100,+ "uptime_last_1d": 100,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ },+ {+ "name": "Novita | xiaomi/mimo-v2.6-pro-20260921",+ "model_id": "xiaomi/mimo-v2.6-pro",+ "model_name": "Xiaomi: MiMo-V2.6-Pro",+ "context_length": 1048576,+ "pricing": {+ "prompt": "0.000000435",+ "completion": "0.00000087",+ "input_cache_read": "0.0000000036",+ "discount": 0+ },+ "provider_name": "Novita",+ "tag": "novita/fp8",+ "quantization": "fp8",+ "max_completion_tokens": 131072,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "max_tokens",+ "temperature",+ "top_p",+ "stop",+ "frequency_penalty",+ "presence_penalty",+ "seed",+ "top_k",+ "repetition_penalty",+ "response_format",+ "tools",+ "tool_choice"+ ],+ "supports_tool_choice": {+ "none": false,+ "auto": true,+ "required": false,+ "function": false+ },+ "status": 0,+ "uptime_last_30m": 100,+ "uptime_last_5m": 100,+ "uptime_last_1d": 95.625,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ },+ {+ "name": "GMICloud | xiaomi/mimo-v2.6-pro-20260921",+ "model_id": "xiaomi/mimo-v2.6-pro",+ "model_name": "Xiaomi: MiMo-V2.6-Pro",+ "context_length": 1050000,+ "pricing": {+ "prompt": "0.000000435",+ "completion": "0.00000087",+ "input_cache_read": "0.000000004",+ "discount": 0+ },+ "provider_name": "GMICloud",+ "tag": "gmicloud/bf16",+ "quantization": "bf16",+ "max_completion_tokens": 945000,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "max_tokens",+ "temperature",+ "top_p",+ "seed",+ "response_format",+ "structured_outputs",+ "tools",+ "tool_choice"+ ],+ "supports_tool_choice": {+ "none": false,+ "auto": true,+ "required": false,+ "function": false+ },+ "status": 0,+ "uptime_last_30m": 100,+ "uptime_last_5m": 100,+ "uptime_last_1d": 95.41666666666667,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ },+ {+ "name": "Xiaomi | xiaomi/mimo-v2.6-pro-20260921",+ "model_id": "xiaomi/mimo-v2.6-pro",+ "model_name": "Xiaomi: MiMo-V2.6-Pro",+ "context_length": 1048576,+ "pricing": {+ "prompt": "0.000000435",+ "completion": "0.00000087",+ "input_cache_read": "0.0000000036",+ "discount": 0+ },+ "provider_name": "Xiaomi",+ "tag": "xiaomi/fp8",+ "quantization": "fp8",+ "max_completion_tokens": 131072,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "max_tokens",+ "temperature",+ "top_p",+ "stop",+ "response_format",+ "tools",+ "tool_choice",+ "frequency_penalty",+ "presence_penalty",+ "structured_outputs"+ ],+ "supports_tool_choice": {+ "none": false,+ "auto": true,+ "required": true,+ "function": false+ },+ "status": 0,+ "uptime_last_30m": 100,+ "uptime_last_5m": 100,+ "uptime_last_1d": 95.90277777777779,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ }+ ]+ }+ }+ },+ {+ "model": "xiaomi/mimo-v2.6-flash",+ "label": "MiMo-V2.6-Flash",+ "vision": true,+ "url": "https://openrouter.ai/api/v1/models/xiaomi/mimo-v2.6-flash/endpoints",+ "status": 200,+ "retrievedAt": "2026-10-13T22:49:54.892Z",+ "data": {+ "data": {+ "id": "xiaomi/mimo-v2.6-flash",+ "name": "Xiaomi: MiMo-V2.6-Flash",+ "created": 1790021264,+ "description": "MiMo-V2.6-Flash is an open-source foundation model developed by Xiaomi. Built on a Mixture-of-Experts architecture with 309B total parameters and 15B activated per token, it employs a hybrid attention mechanism for...",+ "architecture": {+ "tokenizer": "Other",+ "instruct_type": null,+ "modality": "text+image+audio+video->text",+ "input_modalities": [+ "text",+ "image",+ "video",+ "audio"+ ],+ "output_modalities": [+ "text"+ ]+ },+ "endpoints": [+ {+ "name": "Darkbloom | xiaomi/mimo-v2.6-flash-20260921",+ "model_id": "xiaomi/mimo-v2.6-flash",+ "model_name": "Xiaomi: MiMo-V2.6-Flash",+ "context_length": 1048576,+ "pricing": {+ "prompt": "0.0000001",+ "completion": "0.00000028",+ "input_cache_read": "0.00000005",+ "discount": 0+ },+ "provider_name": "Darkbloom",+ "tag": "darkbloom/fp4",+ "quantization": "fp4",+ "max_completion_tokens": 131072,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "temperature",+ "top_p",+ "top_k",+ "frequency_penalty",+ "presence_penalty",+ "repetition_penalty",+ "stop",+ "seed",+ "max_tokens",+ "response_format",+ "structured_outputs",+ "tools",+ "tool_choice"+ ],+ "supports_tool_choice": {+ "none": true,+ "auto": true,+ "required": false,+ "function": false+ },+ "status": 0,+ "uptime_last_30m": 93.33333333333333,+ "uptime_last_5m": 80,+ "uptime_last_1d": 97.91066282420749,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ },+ {+ "name": "Makora | xiaomi/mimo-v2.6-flash-20260921",+ "model_id": "xiaomi/mimo-v2.6-flash",+ "model_name": "Xiaomi: MiMo-V2.6-Flash",+ "context_length": 1048576,+ "pricing": {+ "prompt": "0.00000013",+ "completion": "0.00000028",+ "input_cache_read": "0.000000002",+ "discount": 0+ },+ "provider_name": "Makora",+ "tag": "makora/fp8",+ "quantization": "fp8",+ "max_completion_tokens": 131072,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "temperature",+ "top_p",+ "top_k",+ "frequency_penalty",+ "presence_penalty",+ "repetition_penalty",+ "stop",+ "max_tokens",+ "seed",+ "response_format",+ "structured_outputs",+ "tools",+ "tool_choice",+ "logprobs",+ "top_logprobs"+ ],+ "supports_tool_choice": {+ "none": false,+ "auto": true,+ "required": true,+ "function": true+ },+ "status": 0,+ "uptime_last_30m": 93.33333333333333,+ "uptime_last_5m": 100,+ "uptime_last_1d": 97.15277777777777,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ },+ {+ "name": "Novita | xiaomi/mimo-v2.6-flash-20260921",+ "model_id": "xiaomi/mimo-v2.6-flash",+ "model_name": "Xiaomi: MiMo-V2.6-Flash",+ "context_length": 1048576,+ "pricing": {+ "prompt": "0.00000014",+ "completion": "0.00000028",+ "input_cache_read": "0.0000000028",+ "discount": 0+ },+ "provider_name": "Novita",+ "tag": "novita/fp8",+ "quantization": "fp8",+ "max_completion_tokens": 131072,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "max_tokens",+ "temperature",+ "top_p",+ "stop",+ "frequency_penalty",+ "presence_penalty",+ "seed",+ "top_k",+ "repetition_penalty",+ "response_format",+ "tools",+ "tool_choice"+ ],+ "supports_tool_choice": {+ "none": false,+ "auto": true,+ "required": false,+ "function": false+ },+ "status": 0,+ "uptime_last_30m": 100,+ "uptime_last_5m": 100,+ "uptime_last_1d": 99.375,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ },+ {+ "name": "GMICloud | xiaomi/mimo-v2.6-flash-20260921",+ "model_id": "xiaomi/mimo-v2.6-flash",+ "model_name": "Xiaomi: MiMo-V2.6-Flash",+ "context_length": 1050000,+ "pricing": {+ "prompt": "0.00000014",+ "completion": "0.00000028",+ "input_cache_read": "0.000000003",+ "discount": 0+ },+ "provider_name": "GMICloud",+ "tag": "gmicloud/bf16",+ "quantization": "bf16",+ "max_completion_tokens": 945000,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "max_tokens",+ "temperature",+ "top_p",+ "seed",+ "response_format",+ "structured_outputs",+ "tools",+ "tool_choice"+ ],+ "supports_tool_choice": {+ "none": false,+ "auto": true,+ "required": false,+ "function": false+ },+ "status": 0,+ "uptime_last_30m": 100,+ "uptime_last_5m": 100,+ "uptime_last_1d": 43.26292789512017,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ },+ {+ "name": "DeepInfra | xiaomi/mimo-v2.6-flash-20260921",+ "model_id": "xiaomi/mimo-v2.6-flash",+ "model_name": "Xiaomi: MiMo-V2.6-Flash",+ "context_length": 1048576,+ "pricing": {+ "prompt": "0.00000014",+ "completion": "0.00000028",+ "input_cache_read": "0.0000000028",+ "discount": 0+ },+ "provider_name": "DeepInfra",+ "tag": "deepinfra/fp8",+ "quantization": "fp8",+ "max_completion_tokens": 943718,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "max_tokens",+ "temperature",+ "top_p",+ "stop",+ "frequency_penalty",+ "presence_penalty",+ "repetition_penalty",+ "top_k",+ "seed",+ "min_p",+ "logit_bias",+ "response_format",+ "structured_outputs",+ "tools",+ "tool_choice"+ ],+ "supports_tool_choice": {+ "none": true,+ "auto": true,+ "required": true,+ "function": true+ },+ "status": 0,+ "uptime_last_30m": 100,+ "uptime_last_5m": 100,+ "uptime_last_1d": 99.20886075949366,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ },+ {+ "name": "Xiaomi | xiaomi/mimo-v2.6-flash-20260921",+ "model_id": "xiaomi/mimo-v2.6-flash",+ "model_name": "Xiaomi: MiMo-V2.6-Flash",+ "context_length": 1048576,+ "pricing": {+ "prompt": "0.00000014",+ "completion": "0.00000028",+ "input_cache_read": "0.0000000028",+ "discount": 0+ },+ "provider_name": "Xiaomi",+ "tag": "xiaomi/fp8",+ "quantization": "fp8",+ "max_completion_tokens": 131072,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "max_tokens",+ "temperature",+ "top_p",+ "stop",+ "response_format",+ "tools",+ "tool_choice",+ "frequency_penalty",+ "presence_penalty",+ "structured_outputs"+ ],+ "supports_tool_choice": {+ "none": false,+ "auto": true,+ "required": true,+ "function": false+ },+ "status": 0,+ "uptime_last_30m": 100,+ "uptime_last_5m": 100,+ "uptime_last_1d": 100,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ },+ {+ "name": "Venice | xiaomi/mimo-v2.6-flash-20260921",+ "model_id": "xiaomi/mimo-v2.6-flash",+ "model_name": "Xiaomi: MiMo-V2.6-Flash",+ "context_length": 1000000,+ "pricing": {+ "prompt": "0.000000175",+ "completion": "0.00000035",+ "input_cache_read": "0.00000000375",+ "discount": 0+ },+ "provider_name": "Venice",+ "tag": "venice/fp8",+ "quantization": "fp8",+ "max_completion_tokens": 131072,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "max_tokens",+ "temperature",+ "top_p",+ "stop",+ "frequency_penalty",+ "presence_penalty",+ "top_k",+ "structured_outputs",+ "tools",+ "tool_choice"+ ],+ "supports_tool_choice": {+ "none": true,+ "auto": true,+ "required": true,+ "function": true+ },+ "status": 0,+ "uptime_last_30m": 92,+ "uptime_last_5m": 100,+ "uptime_last_1d": 98.41646872525732,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ },+ {+ "name": "Io Net | xiaomi/mimo-v2.6-flash-20260921",+ "model_id": "xiaomi/mimo-v2.6-flash",+ "model_name": "Xiaomi: MiMo-V2.6-Flash",+ "context_length": 1048320,+ "pricing": {+ "prompt": "0.00000019",+ "completion": "0.00000029",+ "input_cache_read": "0.000000025",+ "discount": 0+ },+ "provider_name": "Io Net",+ "tag": "io-net/fp8",+ "quantization": "fp8",+ "max_completion_tokens": 131072,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "max_tokens",+ "temperature",+ "seed",+ "stop",+ "top_k",+ "top_p",+ "presence_penalty",+ "frequency_penalty",+ "repetition_penalty",+ "tools",+ "tool_choice",+ "logprobs",+ "top_logprobs",+ "response_format",+ "structured_outputs"+ ],+ "supports_tool_choice": {+ "none": false,+ "auto": true,+ "required": true,+ "function": true+ },+ "status": 0,+ "uptime_last_30m": 100,+ "uptime_last_5m": 100,+ "uptime_last_1d": 99.02777777777779,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ }+ ]+ }+ }+ },+ {+ "model": "liquid/lfm-2.5-24b-a2b",+ "label": "Liquid LFM2.5-24B-A2B",+ "vision": false,+ "url": "https://openrouter.ai/api/v1/models/liquid/lfm-2.5-24b-a2b/endpoints",+ "status": 200,+ "retrievedAt": "2026-10-13T22:49:54.892Z",+ "data": {+ "data": {+ "id": "liquid/lfm-2.5-24b-a2b",+ "name": "LiquidAI: LFM2.5-24B-A2B",+ "created": 1791912025,+ "description": "LFM2.5-24B-A2B is a mixture-of-experts reasoning model from Liquid AI with 24B total and 2.45B active parameters. It always reasons before answering and is built for agents that call tools. Liquid...",+ "architecture": {+ "tokenizer": "Other",+ "instruct_type": null,+ "modality": "text->text",+ "input_modalities": [+ "text"+ ],+ "output_modalities": [+ "text"+ ]+ },+ "endpoints": [+ {+ "name": "Liquid | liquid/lfm-2.5-24b-a2b-20261012",+ "model_id": "liquid/lfm-2.5-24b-a2b",+ "model_name": "LiquidAI: LFM2.5-24B-A2B",+ "context_length": 128000,+ "pricing": {+ "prompt": "0.00000005",+ "completion": "0.0000007",+ "input_cache_read": "0.000000025",+ "discount": 0+ },+ "provider_name": "Liquid",+ "tag": "liquid/fp8",+ "quantization": "fp8",+ "max_completion_tokens": 8192,+ "max_prompt_tokens": null,+ "supported_parameters": [+ "reasoning",+ "include_reasoning",+ "max_tokens",+ "temperature",+ "stop",+ "frequency_penalty",+ "presence_penalty",+ "seed",+ "top_k",+ "min_p",+ "repetition_penalty",+ "max_completion_tokens",+ "tools",+ "tool_choice",+ "response_format",+ "structured_outputs",+ "logprobs",+ "top_logprobs"+ ],+ "supports_tool_choice": {+ "none": true,+ "auto": true,+ "required": true,+ "function": true+ },+ "status": 0,+ "uptime_last_30m": 100,+ "uptime_last_5m": 100,+ "uptime_last_1d": 100,+ "supports_implicit_caching": true,+ "native_tools": {},+ "supports_voice_cloning": false,+ "supports_multiple_audio_references": false,+ "supports_image_reference": false,+ "latency_last_30m": null,+ "throughput_last_30m": null+ }+ ]+ }+ }+ }+]diff --git a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/rubrics.json b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/rubrics.jsonnew file mode 100644index 000000000..93e1f9031--- /dev/null+++ b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/rubrics.json@@ -0,0 +1,2279 @@+{+ "sourceCommit": "7182c21f58a94689c2ac3b547a279b9c5e699283",+ "sourceHashes": {+ "test/llm/lib/build-payload.mjs": "3df1b3279e898e72a78970c80d1055e9de396bf6091ea8d402937eaf413dceda",+ "src/chrome/src/agent/tools.js": "77c10507106bea79a9fc1b299f5afa45b04c5fbdc28a483f72331ef312b6caca",+ "src/chrome/src/agent/adapters.js": "f0f862c682503e237beb488f0c59fd426b4f6e5290591c2764d325cb67dd8a7c",+ "src/firefox/src/agent/tools.js": "c19ab6f96cebe41a2df2bf21b03996cae62501d421ef0dbb6acc70f811413c44",+ "src/firefox/src/agent/adapters.js": "8064bbabf8da9f96af800f1b5facc54a155b6fa6eb458092ab291779f152ba3f",+ "test/llm/questions/001.json": "71ca04cd1bc2e2133db08f86dd29b7cdfeaf1d708ae05f931862a9ae9ad60716",+ "test/llm/expected/001.json": "9fa7581c00579c7ccd2da27f5ee2becbce55a8e99d19b0bfd69300fbefb8c740",+ "test/llm/questions/002.json": "a00fe514f6b3ddbd33ee60cf55dae355cc3d70ef49da412cde355485f797c095",+ "test/llm/expected/002.json": "2dbfcebaa779eced8de2357d53dda55c3b8445301862061153851deda76380f9",+ "test/llm/questions/003.json": "a52b8ce03da5cb51305cd3fde0c800b57288f51de65d09733e69f1ec61231353",+ "test/llm/expected/003.json": "32466f9b93cb44a25500cc1b619c4f176eb0db5ca922b0f27bb476d67f60c588",+ "test/llm/questions/004.json": "b0c6dc1371ae7161db37f9223267886a4985cd8e7101d76845c1360d8cd623aa",+ "test/llm/expected/004.json": "f6ed1fa0524858814576978a2322b141dd127dbd2eadc66f52dc5b0be3ab0da3",+ "test/llm/questions/005.json": "55755b203cbd2a955372a7cb54fe7543d60ffd77ded55112062f9a7e7260cdde",+ "test/llm/expected/005.json": "9f4e371cd1c44f3ca9195f83a1e56f089116cfb55ea1e3b4c5bb93de28d6fc5a",+ "test/llm/questions/006.json": "7fca27db9180015baa4ff0fa7591a48b2a0bdb04f1fa7f40f73e62217213b211",+ "test/llm/expected/006.json": "5ec5b3bf96c3fbfece034f8af9e8804453f4a7987dae95fb052c62b97906c0d1",+ "test/llm/questions/007.json": "5112d74648c4703cdc5472feb14f4418a16ee7a284adb80e9156d4580212e1a1",+ "test/llm/expected/007.json": "f7a829f875ccab0df239c819ce0711ab3a79b74a3159b49e1468287ddafd6816",+ "test/llm/questions/008.json": "5932d8743e7b0cebf56cd18355c65491d1eb1b17abb6ba29abddfeb922f700c6",+ "test/llm/expected/008.json": "6588727b31778db3e6a052e4eb710f35b14d769f5d89652f8ffec3f36962dfb3",+ "test/llm/questions/009.json": "ff2164a31fe8009b8353488f995f0695308d9a02c5332f31cca2e26873d2405d",+ "test/llm/expected/009.json": "f1b2d69d373a5094d89309d19cec864237344b148d9f1cf0c6b29b8357e09b33",+ "test/llm/questions/010.json": "5429c90ef457bdc6fcc49890a777cf9620992491d7aa409b9a8792c18c1c7721",+ "test/llm/expected/010.json": "06e558c04ce07ff8bd1b960ffb297f2d4acdbc504e97f5363c2017aa72a8bcf7",+ "test/llm/questions/011.json": "57a194b62d4f83c1ba196ce82bab450a1be94817760b1a8e7f84969940724b62",+ "test/llm/expected/011.json": "4ca712880fd8bc52255e854a0f8a878b814bf21e49afae73ffc444b05b4f5d1b",+ "test/llm/questions/012.json": "eb5ad53c51eb6f78416614c43c9b23cb520fa584ac8c5cc7101c2493627abf91",+ "test/llm/expected/012.json": "310f10636b98a881498a4cd55e98c72f10b8c6435d5dd40f137fe94ccdbe38a5",+ "test/llm/questions/013.json": "503b8381962a0e6642e850397ae8fb368e07a7c3808c1b3d5128602a657fec1d",+ "test/llm/expected/013.json": "6f4b034753f0a4eda005be42d640648da727fa113fb48ac644c0d43efeaae73c",+ "test/llm/questions/014.json": "d0a6457a01996ab1089103bd788aceed722aec026517eec4f683e9e2014266b3",+ "test/llm/expected/014.json": "9b0b1477a4c87672c032579ff27d9599075039f150fe5b832e6fe0bb64483525",+ "test/llm/questions/015.json": "f78fbbfd3b641285c7633015ba191bcfbf9375102b8392364dc56b3f46f91677",+ "test/llm/expected/015.json": "b853792730d8df90d7cfae0491192c7bc2fe7a2a3d0c401366478e769b0e6796",+ "test/llm/questions/016.json": "d1e0adbb62566016633922f82135bc9222cfeefe103ca99b741bd04a628995f5",+ "test/llm/expected/016.json": "eeb61c74608ed48167d9271d69fe36d3617136ce97b50fb4ddc04413c888e180",+ "test/llm/questions/017.json": "4c83b342a77993e61b04b263f79070cd61d52d6c4a02a0ab1c1203c599e5483a",+ "test/llm/expected/017.json": "b5f7539413e0c4f611685dd1226ce695ac29c1f88eac8b36c6b15ab8bec1c512",+ "test/llm/questions/018.json": "78ed9e83dda9dc94c83cd326f4da09928821ee32828b89acebbf5374c852afc8",+ "test/llm/expected/018.json": "c2d7957278590dce48b41281ec6a87822bc38b4041a1635a81b12c78632c4d24",+ "test/llm/questions/019.json": "c1621d82948fe3d45c2ca4012120da27fabcf48e5475c900bd37f28f10e6234b",+ "test/llm/expected/019.json": "d998832d5513cbdc5bcb33a969a6bde4655ceb8c539fa0b925d1a679d0f13794",+ "test/llm/questions/020.json": "04af87f3ae7249f3f5d5246312ff5dc6da859f1eb0ab999d3d07eb0016392fd7",+ "test/llm/expected/020.json": "9dd0f7629f2577459bd59e7db68ca6ec39407b8f5901add1b2f5ee7a7eabe49e",+ "test/llm/questions/021.json": "f73b7a6465914e673808e56f177dee81ba6f1a7b44f0bd2e4cf9107ab78d3dc5",+ "test/llm/expected/021.json": "18b8754b2c0ea2f5f25d95fb1e5ca5360f9e73ae35456618fe87f19b6e865527",+ "test/llm/questions/022.json": "0b6429bf4c2bd1c445155b1d047049f255036ab661cadca3b93162c20a421666",+ "test/llm/expected/022.json": "a75e73e7690976f65d7c30762cea6a31a1783f9327035a8d921b729a67f71c76",+ "test/llm/questions/023.json": "16a24a2d89e98a092b41e1e3ccc5c3e8925aabfade3dd597644cb467bc20550c",+ "test/llm/expected/023.json": "c099e6db71f3d77b08f55e6eb7dfbc4a0034e9a5d185c7952c8efd7a46a1b80b",+ "test/llm/questions/024.json": "0f27b15bcf31d338202eac3091951ddd54db65f2280491589ed382cbe51adf44",+ "test/llm/expected/024.json": "1881dfd39014a8b4823e295d0be6a0aac62eb6095aff7e9f1c88154b804e645f",+ "test/llm/questions/025.json": "9c8039f1587b80e44ad064bace42c4ba9c02528fd36367ef365072a16c97107f",+ "test/llm/expected/025.json": "62c405823cfe6833fcde76d7b8fdd08a8fef463c2ea7bd28d069a411bd344a08",+ "test/llm/questions/026.json": "292f5008b930268284fc72a806e2efe6c8aacce4e45633c936930b81dcc0203b",+ "test/llm/expected/026.json": "005c5b3c0171e9c3c2d828e671edc39eb7e2cf1161bee10f3ad3fad7bcead87a",+ "test/llm/questions/027.json": "1aff9a46e5740ae083f24a4f82478a8cb91546d5bd4269bd71737d83bec20af9",+ "test/llm/expected/027.json": "2ae901ef9637bff2850a1d277de1dabdaae7d0c300904446cca52a307cca31cb",+ "test/llm/questions/028.json": "2ad1e6dd1b863ec3d0a159ba0c3bf7ec15a4853649839173dc48980a699568a7",+ "test/llm/expected/028.json": "b61636d99dc8928ca49f25c942078549aecda3aa7a57d0f47795c400900dd299",+ "test/llm/questions/029.json": "3a323f0fb4f4044b2fcd345fe384f0242d3a59ebb09f9b3c8c484d9161030078",+ "test/llm/expected/029.json": "f2923a97a47a1001c1742b327ed131017c50867cbe5b43409ee8c28d0639de8f",+ "test/llm/questions/030.json": "e55619e43ae6fbe75a5447bc51fb2375d8108086c531fc28f2492b1c49f3ac2b",+ "test/llm/expected/030.json": "225e816f8845e4e5ddc627989ef366ab0dd6545d30ff844e1bce37a12c830b83",+ "test/llm/questions/031.json": "7a8365706dfc2327a7cee96665d45a743ff879fbc35175abdf37de0442200f94",+ "test/llm/expected/031.json": "3ebac4d1cc7dceacec0b3e549128f4335c23bb081c8105a65428f3048f07042f",+ "test/llm/questions/032.json": "f255d89ef0f0c0b86387a9ff9bbdf3aaaf7f18930236f5a206ead0af174ceb5e",+ "test/llm/expected/032.json": "7b6d50ab728c6d007601b9435e16e4d13b11b88bb022f84135daee365607b98f",+ "test/llm/questions/033.json": "6d3322c23a189c0b0b1c7d785a8ad6c6ddf256cc3744e629e006e251f098660f",+ "test/llm/expected/033.json": "04ca1e8ce6f4b8a00186d67ab96472189d4dd5b6a1606769627d59811e48b0f2",+ "test/llm/questions/034.json": "0e9b914ce520469b7c120c109dd0dc0e5780df2c3894f9f466bdf85d8b03b8f8",+ "test/llm/expected/034.json": "226bf7b6d63e0623ed2510ff576a361af7d8c1f694321329b125ed61ca7956d5",+ "test/llm/questions/035.json": "14a29f8ad101063815d86f9a932af257767e1ff6c90c5e8919854a066908e20f",+ "test/llm/expected/035.json": "16a5507343b505190da0f2fd21cd3bca40f2f1bde5a796a02846edb23851b9c8",+ "test/llm/questions/036.json": "b13582b5b1bf0b6abf27bb9f49c242dd9b663a67283f3b0c871d92151cf2afb3",+ "test/llm/expected/036.json": "b2846a66d38065a6f996f1be63f195a3e142e73ec3a3c255d9134fba5b9fbcb3",+ "test/llm/questions/037.json": "d176dab2e157db5148f6537bedd7ee10da883117f43b2a128b3ff8c6c5e54070",+ "test/llm/expected/037.json": "d2b0fad948d29ecd3ebb4d1350f74666450409b714187bd444647aed79a11a89",+ "test/llm/questions/038.json": "c8117078f3aded59249f1823eb2e24c4c13fc8175a67b2bdf1e5fbbc7d006248",+ "test/llm/expected/038.json": "f10ee67f1273233266ccb2fd5fde1315c73b82702d5e08901cfaf49b202c96a0",+ "test/llm/questions/039.json": "3bfe72b26cc93956022bee00b2aba88de5df4336fb9010eea10a0cbaacdfd268",+ "test/llm/expected/039.json": "1a5385c6b1b09cb3effb89d5a0167d503c9468189b3af70088ecb3c9c78043f0",+ "test/llm/questions/040.json": "a00d78cd176da994cf568c85c44a3ca16996b73697c4596e1a96b2290dbc56cf",+ "test/llm/expected/040.json": "7a10c93145030a659adcd92eabea5adf8f2f581eafcfbad13f84193f9dde0e91",+ "test/llm/questions/041.json": "267f7be224b47282c6cb3f8493dd35f512f3f24822a1a5758432decc7287dae5",+ "test/llm/expected/041.json": "c58b1bf27edce952badcd4940b4274559e213f3dda9a1730d4dfcfa47a20d831",+ "test/llm/questions/042.json": "50ba5b749af59534d23c0632ae31c25ae22521409d6c95ce19b0e4602851397c",+ "test/llm/expected/042.json": "918bfe346497483f718aafb2d6ef7bd62225c307276b35610e6d858e21e49fa8",+ "test/llm/questions/043.json": "5c6682b8c7d7a16b05c6525304f43133d4083746198850d78a5a1a99a40ebb9e",+ "test/llm/expected/043.json": "928626766c1b2cdaf0ec1f329c0a594074857231af3c7ec23f9f527051b16b79",+ "test/llm/questions/044.json": "4a2643fd64c57988157aaecf8fda55bb50eb8a658929b651356f56a21470ba74",+ "test/llm/expected/044.json": "519ef1c2c7c0ee1390c3af22e5b8804abbf7d3acc5477af9e1296ecfac68d74b",+ "test/llm/questions/045.json": "ac0f605b7c45a83e312e70e8f334b902cd2f441681a3be70c82c8b0e0b6a1f4c",+ "test/llm/expected/045.json": "d57c0d7b716ae1a78d40664c2e0a33acf1281707db3eaf50cc2dfe2a0403d7da",+ "test/llm/questions/046.json": "2ef7c95d98d872c04686e73b174b11ebbfab47c2e7927974f68c6ee648599c0f",+ "test/llm/expected/046.json": "5fb8d604f9bcba01296069965e2704afba0e2c9f0a47057d8d3b69c8bd6534e3",+ "test/llm/questions/047.json": "43ab9bd5ddc354aaac1efb89b6f1a97c1b1db5a200d3b66b7806e9f3da98530a",+ "test/llm/expected/047.json": "3b06b09fbfbae4b00096137ebd3a2412abfb7b722476fe4b91390bf42fa06bfb",+ "test/llm/questions/048.json": "e8b5e26507dd55a09fb2e6da237818e199f3addfd8af5e1aeab5911dcda4e412",+ "test/llm/expected/048.json": "f2474c2dc1be507cf783fd35b3e550cdd6bdc60ed551f41c8bd5fe6480047a31",+ "test/llm/questions/049.json": "5bf34cb7163da47426be10d21107053caefcc82f6efb2d3ade3bc2b6f1d2df50",+ "test/llm/expected/049.json": "e7cecdd882bad63f2dd40b2fb80808eb06dde7331c2682d6b9f38f3a660f2a86",+ "test/llm/questions/050.json": "863b6a6beb80a0813db77291aae9441938dd3dfa8b3da5a44ec6b8423b2cabf4",+ "test/llm/expected/050.json": "ebfdea99940a3032b0af713073cd75556944b679f0dbb5d3e0e9b5c82459b682",+ "test/llm/questions/051.json": "35e3978d2334f2072dfa05a463b1ab9c34ef28b179b64a3cf8b06d0ac6e9ef18",+ "test/llm/expected/051.json": "32123056df59617c0526bb6945fc9ace76f5a9e7ea6c007a883c247ab75c1ccd",+ "test/llm/questions/052.json": "87dd011f150c94cc68005a76e24b0f1b09f1b344d6d1d1fa5bd0be88e63c408d",+ "test/llm/expected/052.json": "3baee653bcfb0c934e065c344e547e0f132eb69209cff01c6f47c66967f9f9f6",+ "test/llm/questions/053.json": "057ed90160f0c55574853f5c37b2fa927be5562d1c89c1dae2aa00a9e73111e6",+ "test/llm/expected/053.json": "d096e4bd5658cff2b5a013db83a9ee5ba79758f8b847ac3ab3a025e63e58ab00",+ "test/llm/questions/054.json": "386059c0c31b3ebc94442233c77b60cf4ab4675a9662af1dec41c510843df12b",+ "test/llm/expected/054.json": "e5fa731c960d3b4cc89952d4ed4325be8b19d2579a741dcf1ac582053c6fee6a",+ "test/llm/questions/055.json": "455613609b56befdd1f99676bab731deb2e0c990d2dfb43f656d7d65eff0bb55",+ "test/llm/expected/055.json": "485917ed8d039984c49f138578d8b604db81f9800a54f5dd0511f06cc04a616d",+ "test/llm/questions/056.json": "13fb976ab04ab4a402961b919ce403cbec132140374ba764cafe011be4fcd7b7",+ "test/llm/expected/056.json": "1cd49742bb16e01bc1711cae29ac6913455cdb5f5eb8b019a9e3e9d4e8e59ad0",+ "test/llm/questions/057.json": "7ff5e989245633e3fd4687ff9912cf224e04443f0f2f77dea6c7601c61011b3d",+ "test/llm/expected/057.json": "a164331e19c08e51569f8dd834d83525143833f04564515a934fd5e22c1dcacb",+ "test/llm/questions/058.json": "688d1c03d7878097f565bb538f34d9f55c8e8a605ede3665708be11f1a6d3f5d",+ "test/llm/expected/058.json": "0a79ccf3ee5cf97d35c446a5158f7df6d9f0b84b3a7224a762f0b12ce3d2e84e",+ "test/llm/questions/059.json": "86c76d49533eca0bfe2fceb67ec84c324ba9d52452bb5912480f602a06089df5",+ "test/llm/expected/059.json": "bbc79aa307566c75ba519cc3a1cdfd27170adb9d048f05bdb8496abe4a7a8032",+ "test/llm/questions/060.json": "9cc8958afb0948f32f7358799a8f66b1874efd46fd349307ee929e65cbce0c70",+ "test/llm/expected/060.json": "f74aa5ce8b5a76c6e1937fe74bd299528e51045f7d3c3e7ceefabe25037f25e3",+ "test/llm/questions/061.json": "583f24420215f0ea82ef78e1dc6364ed3e6ce4afacdd69c7bb5682fe10e0dde5",+ "test/llm/expected/061.json": "386908ae84c1881029c9ba0e651126416653dab09b1553ed8b07ae11eddf59fc",+ "test/llm/questions/062.json": "9992fa15ebab3c98c0a26d49cbc2841f8cf49d1f6a14c9701087658db27c4cec",+ "test/llm/expected/062.json": "3d0babadfb965511b50ff051727b0b91b090c41326386e9f49b36f162a2e6eaa",+ "test/llm/questions/063.json": "b7f72902bb0b17f0daac3f63b5ac33f1c79948d4928ab873e767929f7109d60e",+ "test/llm/expected/063.json": "7fd1461292bbb3b7e4fb2f3434f8f2a3e3ab72b5d3dfba1625e8c4e409d62ada",+ "test/llm/questions/064.json": "9bb7d43c39a28f5e79005ae0d1179a278121ef561fd997d609f3272b5de0d68d",+ "test/llm/expected/064.json": "fa0d8e48c8cc35efb5ab597ee4e0a959cd85cf6234833e127029ae0d9b337d55",+ "test/llm/questions/065.json": "75a6595c727044358511be901974013f89eee7e784c93f311179be34bed5b907",+ "test/llm/expected/065.json": "eab057a35b98228a8e1faf71cc09771f7fff3f0bd3ebe164f9dbc5b2b26e040f",+ "test/llm/questions/066.json": "0c76b91baa895a0d1bc2b83a5506f8d278fd16a49cbe435a8da838ff949ca8fb",+ "test/llm/expected/066.json": "12dea741fc4d4d8dd9e94971179746da1f5ee1b7f5d10f781be2ca5e5030fc7e",+ "test/llm/questions/067.json": "0fdf95229e9c4cb3bd1508e37af9e92de877be53bf8e9ca837198860ba9c7b9d",+ "test/llm/expected/067.json": "a1ab56015f4fc10123f6bac7400198fc5be76e19f972130f83d43037695abcbd",+ "test/llm/questions/068.json": "053f51129372ad67f93940c82942b203307f62f489818a0748b6f69e1dbf23e3",+ "test/llm/expected/068.json": "abdd93fe216d43884590ade3ac528a1a4a9051cdadc6ce17a3aff9caa8fe5c4c",+ "test/llm/questions/069.json": "31e7e024b73ca5857073c815bc1c352dc42e78be76a46e9f4ab74b48c3a3ca7c",+ "test/llm/expected/069.json": "52b7a63631b4f363271a76d18698d1f86448d523e4f7860dbd36dd63c5135c65",+ "test/llm/questions/070.json": "8a824d7fa4aa991ca0c4c4816ea680d5d9d023abfec3ce3f6dfeea6e8e669132",+ "test/llm/expected/070.json": "0e21814ceea851db468debdb0825262eec7b80c32b84fcc4204002d148ff963c",+ "test/llm/questions/071.json": "daea0ff5da26c62adbf8580386e41bdf3bf16ab55359e0cfecbd2dac9b4dbaa4",+ "test/llm/expected/071.json": "e973529c3b1c85f73283066ee4586e40f0cd9fbb468d70240e399dfaab4eaec7",+ "test/llm/questions/072.json": "8c4c52a76d76ab1af8d5e3c1cc56650dee73262630c32c1f28e9a7bd519b3607",+ "test/llm/expected/072.json": "dff101348a70b16f2efbab41e82212f6828aa22d14adc1d09f3da32e13934846",+ "test/llm/questions/073.json": "7dd0546c96151cda7d92207d844873851207a30b5a0b1b2a2b1b8701c94aba9c",+ "test/llm/expected/073.json": "8915a11b0c4a74ece77f0fb42d4c25dacbbbb0143c6ecb6289341b3376c52d0b",+ "test/llm/questions/074.json": "b9680fc4b4446aa5a0698d97d237b16d90a25ce292cf361cf04ad5ca51dd990a",+ "test/llm/expected/074.json": "b20cd3e54f61edac6ed8c54eabc215514f1c623effb45bc3f919fda5218ef840",+ "test/llm/questions/075.json": "73277de9961ad226cba1ae889528eb95e118cfe9218ae5d708350871cbba26d6",+ "test/llm/expected/075.json": "7c020c6e7d70df936906cb1cb76f020e3410983f8b5729c558ac4ac2235ce270",+ "test/llm/questions/076.json": "fc5b9513cb73bbb65fbb80688a8745d858261bf9b53baa120a1f7d4c1484befc",+ "test/llm/expected/076.json": "eefe94600666db9753b9b18abad5d8f4532b1b4ae4e5a75659c3b58bce4bcc36",+ "test/llm/questions/077.json": "654cedf00bf452b015c58d1df9f1e40d45d9dc4c0b012bf5cc7f642a28e9a8d8",+ "test/llm/expected/077.json": "949a780912b872f037e0a100d907ae3441f39f7aa0e05af83945e054360fa49b",+ "test/llm/questions/078.json": "4fb6fc317134dbb53c587a15b5baedde3ea1c0f755f94ff0154d8308b1a57f54",+ "test/llm/expected/078.json": "967c1e588b71667372503bd55092257b35c2b4986dec4f364e38b42be5d19f13",+ "test/llm/questions/079.json": "589dc7a49b5aafd003942258b7e95d734e1267578424285c31ac4cb8a03085bb",+ "test/llm/expected/079.json": "aa4dcb8d47c1b385c86b04f50b05f06e0381f90fd6e81df3b55ab51e3be29754",+ "test/llm/questions/080.json": "beabbd3650007671da49a1e8f773fbd6d44f3fbac1b5d8ea3ab0a942160dd0c9",+ "test/llm/expected/080.json": "57f82561d349024789b8b6b1588980f97262b08d68f45b6f8d80fc66d5d626f3",+ "test/llm/questions/081.json": "5cae79c5c8f9bbee87c24ca8e9977974af02b41f44887ee2587fdc84f18a3ce1",+ "test/llm/expected/081.json": "7d93130a891a0b05f5d7852ab4158903f9f7a55b1bdfc9c7ef0ea7d591dc37a1",+ "test/llm/questions/082.json": "204d5929dd230f0ab7dc44de97af5d92bad683bb5e11543e1494bf21b572a5d9",+ "test/llm/expected/082.json": "b54b9fcd7c4731d66f1c314ea4725bc2e59e5c4f36b2615e8429ec5c7256836e",+ "test/llm/questions/083.json": "d0fd6f8a08bfd250f7603d2ce5b28d8bd3d1b71fa1341116737a0a2b7a89d9c9",+ "test/llm/expected/083.json": "272d86d306e8e74c5d4667f91e30c5656b4404df706f152f40e6e7eb4eaa912d",+ "test/llm/questions/084.json": "e9bc9c6599cd113400e1fa2ee851fe067e91ebd5b3d28ffa1d104d16af51effd",+ "test/llm/expected/084.json": "9dc5ca37b3f9f0649a9ad887a62c5be89e84464d2fac004c155534c9d60c3337",+ "test/llm/questions/085.json": "210338c62050503670c1a0da20f3674a4ed85484c8f6ed97acec62cf43f20156",+ "test/llm/expected/085.json": "c5f3a33ef905d6bb651cfb01f3dcf05bc07337a02181c4364102d594dc4acb5c",+ "test/llm/questions/086.json": "dd020ee66dd5645e362d82c1440d2e91ca934087d0231e983420e3591df03070",+ "test/llm/expected/086.json": "97f702dc0f3f6d67decce89f738cdf03aa60088fb98bf54acaf4e196ef236882",+ "test/llm/questions/087.json": "f5807c6e5db41052fb928b714757a6bd2be76fcb91a0d18da936eaba632f93c2",+ "test/llm/expected/087.json": "b038f6f9070d21bda0e4b9407567b09004790b8b9a1e302e01da06d862dadf5e",+ "test/llm/questions/088.json": "4799429de60f74e4e238d1fb27f700c522a36b2d9cc79facbf92f7ec6d60a152",+ "test/llm/expected/088.json": "1125b4693ab4f41a6a397793164cd9f384f7757703cd94532759f88a1244657b",+ "test/llm/questions/089.json": "9f5ebe4dd22b2c1faa546c8f8479ccd5c1633314777ffad474fbba1a7745b994",+ "test/llm/expected/089.json": "6b12e56f21aa77666b804788a23e28fad11d8b31eb5d8fb72de92a189d488147",+ "test/llm/questions/090.json": "fa34c3e0dc491c106dcdfc468e4e124073bb5399f0319e61efa9a5a7459d0e43",+ "test/llm/expected/090.json": "b237135e519f4c23b99ac271985618ff5d385c6288beb1c83297b3ed99409ba7",+ "test/llm/questions/091.json": "4f861fc0350b6a4505343f16d565bc37de8ab5ffdedce8d9a0b21af5bfd43ca4",+ "test/llm/expected/091.json": "6159b9a437399f5a0a1647ee5a9cabad5d625118dee990d9226851aee9e36262",+ "test/llm/questions/092.json": "0dd450d764dc82de830e2183b21f49f75e5af3a95606793bc47f0a916aa8548b",+ "test/llm/expected/092.json": "1d675f2d3fe7a513f066b20ab6299e434fa9aec66714058036841a83e250b797",+ "test/llm/questions/093.json": "40bb7769a04fb03772c771f52e96ecfa85354ff523b5134f4376a5039b021fe2",+ "test/llm/expected/093.json": "0dc29a4c4ec6c10933a73b38d8572527061c2eab10a70c160f2f12c0db7dfc31",+ "test/llm/questions/094.json": "fd48bdb24a3aa661f8d5e284ce18120570fe91542e8c4c60f6e5ffa6b78e3dd8",+ "test/llm/expected/094.json": "0b8532651f2e372ba9e0057898ccb604a29155da56cd75dc31999aa9425d81e6",+ "test/llm/questions/095.json": "3f11c391f5c3ec45355480733fa23c243b680f1b591d8b007768b3e6a5407cf9",+ "test/llm/expected/095.json": "abf3e61661a400771e244f6c0255d3d05f43f1e16784d227057f6e1511ed0f9b",+ "test/llm/questions/096.json": "12b52cebab952031c6a0289c07f89f0936d737b97990cc5869cc9e780b5a223b",+ "test/llm/expected/096.json": "ac1509123812d088293577c4af475a84fa46ad5ce02684083136c75ff4bb112e",+ "test/llm/questions/097.json": "5abe581c4242bdd68e629a557ba46091ce77c3b0e6f05d3295eabc20b5fc0e23",+ "test/llm/expected/097.json": "6c0a7d81d2612a442b6b4a444435049c7072349fd2ec8a547e0d5fbe891e8b0a",+ "test/llm/questions/098.json": "69a31678cfac84b6ff2244732af6ba22d1572bb4fc791acfcdb96572dfc131f8",+ "test/llm/expected/098.json": "e989d0d2ba02a8ac7ada8cc546d7d58d2d86ce5d4c566e9e9714fbd11a80e275",+ "test/llm/questions/099.json": "b96cd2345e34eb3e3d44ae7c6f81b1eb30a3e7058dc8c32324be417e8c09af5a",+ "test/llm/expected/099.json": "372f4b9da2bf1dfdaf35dacab06380952f5c17cc4779b96e6abb8550eb98b2a9",+ "test/llm/questions/100.json": "b90f8ba43afbed7d38fc7f334992847b852338f22efd657b368f0d6e10863068",+ "test/llm/expected/100.json": "67d8c55f3a6c5064d1d369a13f79fcf61c0b7bdad87774aa9d9d20b9286752d8"+ },+ "cases": {+ "100": {+ "question": {+ "id": "100",+ "mode": "act",+ "tab": {+ "url": "https://www.amazon.com/s?k=wireless+earbuds",+ "title": "Amazon.com : wireless earbuds"+ },+ "user": "whats the highest rated item here"+ },+ "expected": {+ "id": "100",+ "idealFirstToolCall": {+ "name": "read_page",+ "args": {}+ },+ "successRubric": "Reads the search-results page, picks the item with the highest visible rating, and reports it. Acceptable: extract_data({type:\"tables\"}) is not appropriate; read_page or get_accessibility_tree with filter:\"visible\" are correct."+ }+ },+ "001": {+ "question": {+ "id": "001",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "go to firefox extensions management pagr"+ },+ "expected": {+ "id": "001",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "about:addons"+ }+ },+ "successRubric": "Navigates the current tab to about:addons (Firefox's built-in extensions manager). The typo \"pagr\" should not derail interpretation. addons.mozilla.org is NOT acceptable β the user clarified in trace 2 they meant the management page."+ }+ },+ "002": {+ "question": {+ "id": "002",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "open gmail"+ },+ "expected": {+ "id": "002",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://mail.google.com/"+ }+ },+ "successRubric": "Navigates to mail.google.com (with or without trailing slash, with or without /mail/u/0/). new_tab is also acceptable."+ }+ },+ "003": {+ "question": {+ "id": "003",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "take me to youtube"+ },+ "expected": {+ "id": "003",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://www.youtube.com/"+ }+ },+ "successRubric": "Navigates to youtube.com. www subdomain optional; new_tab is also acceptable."+ }+ },+ "004": {+ "question": {+ "id": "004",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "open my github"+ },+ "expected": {+ "id": "004",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://github.com/"+ }+ },+ "successRubric": "Navigates to github.com. The agent should NOT try to guess the username slug β github.com's logged-in landing handles that."+ }+ },+ "005": {+ "question": {+ "id": "005",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "show me my downloads"+ },+ "expected": {+ "id": "005",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "about:downloads"+ }+ },+ "successRubric": "Navigates to about:downloads (Firefox's built-in downloads page). list_downloads is also acceptable β both surface recent downloads."+ }+ },+ "006": {+ "question": {+ "id": "006",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "i wanna check netflix"+ },+ "expected": {+ "id": "006",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://www.netflix.com/"+ }+ },+ "successRubric": "Navigates to netflix.com. Should treat colloquial \"wanna check\" as \"open the site\"."+ }+ },+ "007": {+ "question": {+ "id": "007",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "open linkedin in this tab"+ },+ "expected": {+ "id": "007",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://www.linkedin.com/"+ }+ },+ "successRubric": "Navigates the CURRENT tab (not new_tab β \"in this tab\" is explicit). URL: linkedin.com."+ }+ },+ "008": {+ "question": {+ "id": "008",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "wikipedia please"+ },+ "expected": {+ "id": "008",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://www.wikipedia.org/"+ }+ },+ "successRubric": "Navigates to wikipedia.org (the multi-language landing page) OR en.wikipedia.org/wiki/Main_Page. Either is acceptable."+ }+ },+ "009": {+ "question": {+ "id": "009",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "open chatgpt in a new tab"+ },+ "expected": {+ "id": "009",+ "idealFirstToolCall": {+ "name": "new_tab",+ "args": {+ "url": "https://chatgpt.com/"+ }+ },+ "successRubric": "Opens chatgpt.com in a NEW tab (the phrase \"in a new tab\" is explicit). chat.openai.com is also acceptable."+ }+ },+ "010": {+ "question": {+ "id": "010",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "go to amazon"+ },+ "expected": {+ "id": "010",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://www.amazon.com/"+ }+ },+ "successRubric": "Navigates to amazon.com. Locale guessing (e.g. amazon.de) is not appropriate without further context."+ }+ },+ "011": {+ "question": {+ "id": "011",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "open firefox settings"+ },+ "expected": {+ "id": "011",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "about:preferences"+ }+ },+ "successRubric": "Navigates to about:preferences (Firefox settings page)."+ }+ },+ "012": {+ "question": {+ "id": "012",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "show my browser history"+ },+ "expected": {+ "id": "012",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "about:history"+ }+ },+ "successRubric": "Acceptable: about:history, OR explain that Firefox's history sidebar is opened via Ctrl/Cmd+H and call done with that hint. Refusing the task is NOT acceptable."+ }+ },+ "013": {+ "question": {+ "id": "013",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "open the about config page"+ },+ "expected": {+ "id": "013",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "about:config"+ }+ },+ "successRubric": "Navigates to about:config (Firefox's advanced preferences)."+ }+ },+ "014": {+ "question": {+ "id": "014",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "show me my bookmarks"+ },+ "expected": {+ "id": "014",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "How would you like to see your bookmarks? I can navigate to the Library, or you can open the Bookmarks sidebar with Ctrl/Cmd+B."+ }+ },+ "successRubric": "Acceptable: clarify, OR navigate to about:bookmarks (which on Firefox redirects to the Library). Calling done with an explanation that the agent can't open the bookmark sidebar via tools is also acceptable."+ }+ },+ "015": {+ "question": {+ "id": "015",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "whats my new tab page"+ },+ "expected": {+ "id": "015",+ "idealFirstToolCall": {+ "name": "done",+ "args": {+ "summary": "Your current tab is at about:home β the New Tab page. That's what Firefox shows when you open a new tab unless you've overridden the homepage."+ }+ },+ "successRubric": "Recognize this is an informational question about the current tab's identity. done() with a plain-text answer about about:home is correct. Navigating somewhere else is wrong."+ }+ },+ "016": {+ "question": {+ "id": "016",+ "mode": "act",+ "tab": {+ "url": "https://www.youtube.com/",+ "title": "YouTube"+ },+ "user": "search youtube for lo-fi study music"+ },+ "expected": {+ "id": "016",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Search"+ }+ },+ "successRubric": "Focuses the YouTube search box (click({text:\"Search\"}) or click on the search input by selector), then types \"lo-fi study music\", then submits (Enter or click search icon). Acceptable alternative: navigate directly to https://www.youtube.com/results?search_query=lo-fi+study+music."+ }+ },+ "017": {+ "question": {+ "id": "017",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "google \"best coffee shop in seattle\""+ },+ "expected": {+ "id": "017",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://www.google.com/search?q=best+coffee+shop+in+seattle"+ }+ },+ "successRubric": "Either: navigate directly to a Google search URL with the query, OR navigate to google.com and submit the search. Both are correct."+ }+ },+ "018": {+ "question": {+ "id": "018",+ "mode": "act",+ "tab": {+ "url": "https://github.com/",+ "title": "GitHub"+ },+ "user": "find typescript repos with most stars"+ },+ "expected": {+ "id": "018",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://github.com/search?q=language%3Atypescript&type=repositories&s=stars&o=desc"+ }+ },+ "successRubric": "Acceptable: navigate to GitHub search with language:typescript and stars sort, OR use the search box on the current page. The key signal is sorting by stars descending."+ }+ },+ "019": {+ "question": {+ "id": "019",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "search reddit for mechanical keyboards"+ },+ "expected": {+ "id": "019",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://www.reddit.com/search/?q=mechanical+keyboards"+ }+ },+ "successRubric": "Either: navigate to reddit.com/search?q=mechanical+keyboards, OR navigate to r/MechanicalKeyboards (the dedicated subreddit). Both are reasonable interpretations."+ }+ },+ "020": {+ "question": {+ "id": "020",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "look up \"javascript closures\" on stackoverflow"+ },+ "expected": {+ "id": "020",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://stackoverflow.com/search?q=javascript+closures"+ }+ },+ "successRubric": "Navigates to a Stack Overflow search for \"javascript closures\". The /search?q= URL or the homepage + search-box flow are both fine."+ }+ },+ "021": {+ "question": {+ "id": "021",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "wiki \"ww2\""+ },+ "expected": {+ "id": "021",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://en.wikipedia.org/wiki/World_War_II"+ }+ },+ "successRubric": "Recognize \"ww2\" expands to World War II and navigate to the matching Wikipedia article. Direct /wiki/World_War_II URL or a search through wikipedia.org are both acceptable."+ }+ },+ "022": {+ "question": {+ "id": "022",+ "mode": "act",+ "tab": {+ "url": "https://www.amazon.com/",+ "title": "Amazon.com"+ },+ "user": "find noise cancelling headphones under 200 dollars"+ },+ "expected": {+ "id": "022",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Search"+ }+ },+ "successRubric": "Focuses the Amazon search box and types \"noise cancelling headphones\", optionally followed by use of the price filter. Direct navigation to /s?k=noise+cancelling+headphones is also acceptable. Type-then-click-search or click-then-type are both fine."+ }+ },+ "023": {+ "question": {+ "id": "023",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "show me vegan lasagna recipes"+ },+ "expected": {+ "id": "023",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://www.google.com/search?q=vegan+lasagna+recipes"+ }+ },+ "successRubric": "Either: Google/Bing/DuckDuckGo search for \"vegan lasagna recipes\", OR navigate to a recipe site's own search. Direct Google search is the most-likely first action."+ }+ },+ "024": {+ "question": {+ "id": "024",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "find tweets about react 19"+ },+ "expected": {+ "id": "024",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://x.com/search?q=react%2019&src=typed_query"+ }+ },+ "successRubric": "Navigate to X/Twitter search for \"react 19\". x.com or twitter.com domain both fine. Acceptable: open x.com home first and click into the search box."+ }+ },+ "025": {+ "question": {+ "id": "025",+ "mode": "act",+ "tab": {+ "url": "https://www.youtube.com/",+ "title": "YouTube"+ },+ "user": "search youtube for \"never gonna give you up\""+ },+ "expected": {+ "id": "025",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Search"+ }+ },+ "successRubric": "Focus YouTube's search box and type the query, OR navigate directly to youtube.com/results?search_query=never+gonna+give+you+up. Both pass."+ }+ },+ "026": {+ "question": {+ "id": "026",+ "mode": "act",+ "tab": {+ "url": "https://www.cnn.com/2026/05/world/example-story/index.html",+ "title": "Example story β CNN"+ },+ "user": "summarize this page"+ },+ "expected": {+ "id": "026",+ "idealFirstToolCall": {+ "name": "read_page",+ "args": {}+ },+ "successRubric": "Calls read_page on the current tab. After reading, the agent should call done() with a concise summary of the article body. Navigating away is wrong β the user said \"this page\"."+ }+ },+ "027": {+ "question": {+ "id": "027",+ "mode": "act",+ "tab": {+ "url": "https://en.wikipedia.org/wiki/Closure_(computer_programming)",+ "title": "Closure (computer programming) β Wikipedia"+ },+ "user": "tldr"+ },+ "expected": {+ "id": "027",+ "idealFirstToolCall": {+ "name": "read_page",+ "args": {}+ },+ "successRubric": "Reads the current page and emits a short summary via done(). Calling fetch_url on the same URL would be wrong (we're already on the page)."+ }+ },+ "028": {+ "question": {+ "id": "028",+ "mode": "act",+ "tab": {+ "url": "https://www.cnn.com/2026/05/world/example-story/index.html",+ "title": "Example story β CNN"+ },+ "user": "what is this article about"+ },+ "expected": {+ "id": "028",+ "idealFirstToolCall": {+ "name": "read_page",+ "args": {}+ },+ "successRubric": "Reads the current page and answers in one sentence. Navigating, screenshotting, or extracting headings (instead of body text) is wrong."+ }+ },+ "029": {+ "question": {+ "id": "029",+ "mode": "act",+ "tab": {+ "url": "https://en.wikipedia.org/wiki/Closure_(computer_programming)",+ "title": "Closure (computer programming) β Wikipedia"+ },+ "user": "give me the headings on this page"+ },+ "expected": {+ "id": "029",+ "idealFirstToolCall": {+ "name": "extract_data",+ "args": {+ "type": "headings"+ }+ },+ "successRubric": "Calls extract_data({type:\"headings\"}) and surfaces the heading list. read_page is also acceptable (since the model can pull headings out of prose) but extract is more efficient."+ }+ },+ "030": {+ "question": {+ "id": "030",+ "mode": "act",+ "tab": {+ "url": "https://news.ycombinator.com/",+ "title": "Hacker News"+ },+ "user": "list all the visible links here"+ },+ "expected": {+ "id": "030",+ "idealFirstToolCall": {+ "name": "get_interactive_elements",+ "args": {}+ },+ "successRubric": "Calls get_interactive_elements (which lists clickable elements including links) OR read_page and parses links out. extract_data does not have a \"links\" type so calling it that way would be wrong."+ }+ },+ "031": {+ "question": {+ "id": "031",+ "mode": "act",+ "tab": {+ "url": "https://www.cnn.com/2026/05/world/example-story/index.html",+ "title": "Example story β CNN"+ },+ "user": "is there a paywall"+ },+ "expected": {+ "id": "031",+ "idealFirstToolCall": {+ "name": "read_page",+ "args": {}+ },+ "successRubric": "Reads the page and reports whether a paywall/sign-in wall is present. Per UNIVERSAL_PREAMBLE: must NOT attempt any bypass (archive.today, 12ft.io, reader mode, etc.). If paywalled, surface it honestly."+ }+ },+ "032": {+ "question": {+ "id": "032",+ "mode": "act",+ "tab": {+ "url": "https://www.reddit.com/r/MechanicalKeyboards/comments/abc123/example/",+ "title": "Example thread β r/MechanicalKeyboards"+ },+ "user": "how many comments does this thread have"+ },+ "expected": {+ "id": "032",+ "idealFirstToolCall": {+ "name": "read_page",+ "args": {}+ },+ "successRubric": "Reads the Reddit thread page and reports the comment count from the page header. extract_data is also acceptable but read_page works."+ }+ },+ "033": {+ "question": {+ "id": "033",+ "mode": "act",+ "tab": {+ "url": "https://www.amazon.com/dp/B0EXAMPLE",+ "title": "Acme Earbuds β Amazon"+ },+ "user": "whats the price"+ },+ "expected": {+ "id": "033",+ "idealFirstToolCall": {+ "name": "read_page",+ "args": {}+ },+ "successRubric": "Reads the Amazon product page and reports the visible price. Should not call fetch_url against the same URL (the page is already loaded)."+ }+ },+ "034": {+ "question": {+ "id": "034",+ "mode": "act",+ "tab": {+ "url": "https://github.com/",+ "title": "GitHub"+ },+ "user": "sign in"+ },+ "expected": {+ "id": "034",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Sign in"+ }+ },+ "successRubric": "Clicks the \"Sign in\" button/link on the current page. Acceptable: navigate to /login directly."+ }+ },+ "035": {+ "question": {+ "id": "035",+ "mode": "act",+ "tab": {+ "url": "https://github.com/login",+ "title": "Sign in to GitHub"+ },+ "user": "fill the email field with [email protected]"+ },+ "expected": {+ "id": "035",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Username or email address"+ }+ },+ "successRubric": "Focuses the username/email field, then types \"[email protected]\" with no selector (click-then-type pattern). Acceptable: type_text with an explicit selector if the agent inspected the page first."+ }+ },+ "036": {+ "question": {+ "id": "036",+ "mode": "act",+ "tab": {+ "url": "https://forms.example.com/contact",+ "title": "Contact form"+ },+ "user": "submit"+ },+ "expected": {+ "id": "036",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Submit"+ }+ },+ "successRubric": "Clicks the visible Submit button by text. For multi-field forms, verify_form() before submit is even better. NOT acceptable: blindly press Enter."+ }+ },+ "037": {+ "question": {+ "id": "037",+ "mode": "act",+ "tab": {+ "url": "https://www.theguardian.com/uk",+ "title": "The Guardian"+ },+ "user": "close the cookie banner"+ },+ "expected": {+ "id": "037",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Reject all"+ }+ },+ "successRubric": "Per UNIVERSAL_PREAMBLE: prefer \"Reject all\" / \"Reject non-essential\" / \"Only necessary\" over \"Accept all\". If only Accept is available at the top level, that's also acceptable. Must NOT dig into \"Manage preferences\"."+ }+ },+ "038": {+ "question": {+ "id": "038",+ "mode": "act",+ "tab": {+ "url": "https://www.theguardian.com/uk",+ "title": "The Guardian"+ },+ "user": "close this popup"+ },+ "expected": {+ "id": "038",+ "idealFirstToolCall": {+ "name": "press_keys",+ "args": {+ "key": "Escape"+ }+ },+ "successRubric": "Escape key dismisses most modals. Acceptable: click({text:\"Close\"}) or click({text:\"Γ\"}) if a close button is visible."+ }+ },+ "039": {+ "question": {+ "id": "039",+ "mode": "act",+ "tab": {+ "url": "https://github.com/esokullu/webbrain/releases/new",+ "title": "New Release"+ },+ "user": "verify the form is ready before submitting"+ },+ "expected": {+ "id": "039",+ "idealFirstToolCall": {+ "name": "verify_form",+ "args": {}+ },+ "successRubric": "Calls verify_form() to inspect all fields and confirm values. This is the canonical use of verify_form."+ }+ },+ "040": {+ "question": {+ "id": "040",+ "mode": "act",+ "tab": {+ "url": "https://www.google.com/search?q=foo",+ "title": "foo - Google Search"+ },+ "user": "clear the search box"+ },+ "expected": {+ "id": "040",+ "idealFirstToolCall": {+ "name": "set_field",+ "args": {+ "ref_id": "search",+ "text": ""+ }+ },+ "successRubric": "Clears the search input. Acceptable: get_accessibility_tree first to find a ref_id, then set_field; OR click({text:\"Search\"}) + type_text({text:\"\", clear:true})."+ }+ },+ "041": {+ "question": {+ "id": "041",+ "mode": "act",+ "tab": {+ "url": "https://example.com/signup",+ "title": "Sign up"+ },+ "user": "tick the subscribe checkbox"+ },+ "expected": {+ "id": "041",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Subscribe to newsletter"+ }+ },+ "successRubric": "Clicks the subscribe checkbox by its label text. Acceptable: get_interactive_elements first, then click by index."+ }+ },+ "042": {+ "question": {+ "id": "042",+ "mode": "act",+ "tab": {+ "url": "https://github.com/esokullu/webbrain",+ "title": "esokullu/webbrain β GitHub"+ },+ "user": "create a release tagged v2.0.0"+ },+ "expected": {+ "id": "042",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://github.com/esokullu/webbrain/releases/new"+ }+ },+ "successRubric": "Per the GitHub site adapter: navigate directly to /releases/new. Then the multi-step flow: click the tag combobox, type \"v2.0.0\", click \"Create new tag\", fill body, click Publish release. NOT acceptable: POST to api.github.com (UI vs API rule)."+ }+ },+ "043": {+ "question": {+ "id": "043",+ "mode": "act",+ "tab": {+ "url": "https://github.com/esokullu/webbrain",+ "title": "esokullu/webbrain β GitHub"+ },+ "user": "open a pull request"+ },+ "expected": {+ "id": "043",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://github.com/esokullu/webbrain/compare"+ }+ },+ "successRubric": "Navigates to the repo's /compare page (PR creation entry point). Acceptable: click({text:\"Pull requests\"}) then click({text:\"New pull request\"})."+ }+ },+ "044": {+ "question": {+ "id": "044",+ "mode": "act",+ "tab": {+ "url": "https://github.com/esokullu/webbrain",+ "title": "esokullu/webbrain β GitHub"+ },+ "user": "file a bug titled \"buttons unresponsive\""+ },+ "expected": {+ "id": "044",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://github.com/esokullu/webbrain/issues/new"+ }+ },+ "successRubric": "Navigates to /issues/new, then types the title and (optionally) body, then submits. Title is provided so no clarify needed."+ }+ },+ "045": {+ "question": {+ "id": "045",+ "mode": "act",+ "tab": {+ "url": "https://github.com/esokullu/webbrain",+ "title": "esokullu/webbrain β GitHub"+ },+ "user": "star this repo"+ },+ "expected": {+ "id": "045",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Star"+ }+ },+ "successRubric": "Clicks the Star button by text. If already starred (button says \"Starred\"), the agent should detect and report β but the first call is correct."+ }+ },+ "046": {+ "question": {+ "id": "046",+ "mode": "act",+ "tab": {+ "url": "https://github.com/esokullu/webbrain",+ "title": "esokullu/webbrain β GitHub"+ },+ "user": "fork it"+ },+ "expected": {+ "id": "046",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Fork"+ }+ },+ "successRubric": "Clicks the Fork button. Acceptable: click({text:\"Create a new fork\"}) if there's a dropdown."+ }+ },+ "047": {+ "question": {+ "id": "047",+ "mode": "act",+ "tab": {+ "url": "https://github.com/esokullu/webbrain",+ "title": "esokullu/webbrain β GitHub"+ },+ "user": "show me the latest commit"+ },+ "expected": {+ "id": "047",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "commits"+ }+ },+ "successRubric": "Clicks the \"commits\" link on the repo page (top of the file browser). Acceptable: navigate to /commits directly."+ }+ },+ "048": {+ "question": {+ "id": "048",+ "mode": "act",+ "tab": {+ "url": "https://mail.google.com/mail/u/0/#inbox",+ "title": "Inbox (3) β Gmail"+ },+ "user": "open my inbox"+ },+ "expected": {+ "id": "048",+ "idealFirstToolCall": {+ "name": "done",+ "args": {+ "summary": "You're already in your Gmail inbox. Is there a specific message you want me to open?"+ }+ },+ "successRubric": "Recognize the user is already on the inbox page (URL contains /#inbox). Acceptable: done with that observation, OR clarify what they meant. Clicking around aimlessly is wrong."+ }+ },+ "049": {+ "question": {+ "id": "049",+ "mode": "act",+ "tab": {+ "url": "https://mail.google.com/mail/u/0/#inbox",+ "title": "Inbox (3) β Gmail"+ },+ "user": "compose an email to [email protected]"+ },+ "expected": {+ "id": "049",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Compose"+ }+ },+ "successRubric": "Clicks the Compose button, then fills the To: field with [email protected]. Body is not specified so the agent should leave it blank or call clarify for body content."+ }+ },+ "050": {+ "question": {+ "id": "050",+ "mode": "act",+ "tab": {+ "url": "https://mail.google.com/mail/u/0/#inbox/abc123",+ "title": "Invoice from Acme β Gmail"+ },+ "user": "reply to this"+ },+ "expected": {+ "id": "050",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Reply"+ }+ },+ "successRubric": "Clicks Reply on the open thread. Acceptable: scroll to the bottom first if the reply button is below the fold."+ }+ },+ "051": {+ "question": {+ "id": "051",+ "mode": "act",+ "tab": {+ "url": "https://mail.google.com/mail/u/0/#inbox/abc123",+ "title": "Invoice from Acme β Gmail"+ },+ "user": "mark as spam"+ },+ "expected": {+ "id": "051",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Report spam"+ }+ },+ "successRubric": "Clicks \"Report spam\" on the current Gmail thread. Acceptable: click by toolbar icon via get_interactive_elements."+ }+ },+ "052": {+ "question": {+ "id": "052",+ "mode": "act",+ "tab": {+ "url": "https://mail.google.com/mail/u/0/#inbox/abc123",+ "title": "Invoice from Acme β Gmail"+ },+ "user": "archive this"+ },+ "expected": {+ "id": "052",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Archive"+ }+ },+ "successRubric": "Clicks Archive on the open thread."+ }+ },+ "053": {+ "question": {+ "id": "053",+ "mode": "act",+ "tab": {+ "url": "https://mail.google.com/mail/u/0/#inbox",+ "title": "Inbox (3) β Gmail"+ },+ "user": "search my email for invoice"+ },+ "expected": {+ "id": "053",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Search mail"+ }+ },+ "successRubric": "Focuses Gmail's search box and types \"invoice\", then submits. Acceptable: navigate directly to /mail/u/0/#search/invoice."+ }+ },+ "054": {+ "question": {+ "id": "054",+ "mode": "act",+ "tab": {+ "url": "https://example.com/photo.jpg",+ "title": "photo.jpg"+ },+ "user": "download this image"+ },+ "expected": {+ "id": "054",+ "idealFirstToolCall": {+ "name": "download_files",+ "args": {+ "urls": [+ "https://example.com/photo.jpg"+ ]+ }+ },+ "successRubric": "Calls download_files with the current image URL. Acceptable: download_resource_from_page if the agent prefers that path."+ }+ },+ "055": {+ "question": {+ "id": "055",+ "mode": "act",+ "tab": {+ "url": "https://www.instagram.com/p/CxAbcExample/",+ "title": "Photo β Instagram"+ },+ "user": "download this instagram post"+ },+ "expected": {+ "id": "055",+ "idealFirstToolCall": {+ "name": "download_social_media",+ "args": {}+ },+ "successRubric": "Calls download_social_media (single tool β per the prompt, do NOT try to inspect the DOM manually for IG posts)."+ }+ },+ "056": {+ "question": {+ "id": "056",+ "mode": "act",+ "tab": {+ "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",+ "title": "Rick Astley β YouTube"+ },+ "user": "save this youtube thumbnail"+ },+ "expected": {+ "id": "056",+ "idealFirstToolCall": {+ "name": "download_social_media",+ "args": {}+ },+ "successRubric": "Calls download_social_media on the watch page. The tool handles thumbnail extraction. Acceptable: download the thumbnail URL directly if the agent knows it."+ }+ },+ "057": {+ "question": {+ "id": "057",+ "mode": "act",+ "tab": {+ "url": "https://github.com/esokullu/webbrain",+ "title": "esokullu/webbrain β GitHub"+ },+ "user": "download the readme"+ },+ "expected": {+ "id": "057",+ "idealFirstToolCall": {+ "name": "download_files",+ "args": {+ "urls": [+ "https://raw.githubusercontent.com/esokullu/webbrain/main/README.md"+ ]+ }+ },+ "successRubric": "Downloads the raw README.md. Acceptable URL variants: github.com/.../raw/main/README.md or raw.githubusercontent.com path. fetch_url + write is also acceptable."+ }+ },+ "058": {+ "question": {+ "id": "058",+ "mode": "act",+ "tab": {+ "url": "https://github.com/esokullu/webbrain/tree/main/dist",+ "title": "webbrain/dist β main"+ },+ "user": "download all the zips in this folder"+ },+ "expected": {+ "id": "058",+ "idealFirstToolCall": {+ "name": "read_page",+ "args": {}+ },+ "successRubric": "First read the folder listing to enumerate zip filenames (read_page or get_interactive_elements), THEN call download_files with the .raw URLs for each zip. Acceptable: extract_data({type:\"links\"}) is NOT valid (no \"links\" type) β agent should use get_interactive_elements or read_page."+ }+ },+ "059": {+ "question": {+ "id": "059",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "list my downloads"+ },+ "expected": {+ "id": "059",+ "idealFirstToolCall": {+ "name": "list_downloads",+ "args": {}+ },+ "successRubric": "Calls list_downloads. Direct, single-tool answer."+ }+ },+ "060": {+ "question": {+ "id": "060",+ "mode": "act",+ "tab": {+ "url": "https://www.amazon.com/",+ "title": "Amazon.com"+ },+ "user": "find 3 wireless earbuds under $100"+ },+ "expected": {+ "id": "060",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Search"+ }+ },+ "successRubric": "Focuses the search box and types \"wireless earbuds\". Then filters by price β€ $100 (sidebar filter or query param). Per LISTINGS guidance: extract and list 3 visible results to the user BEFORE paginating further."+ }+ },+ "061": {+ "question": {+ "id": "061",+ "mode": "act",+ "tab": {+ "url": "https://www.amazon.com/dp/B0EXAMPLE",+ "title": "Acme Earbuds β Amazon"+ },+ "user": "add this to my cart"+ },+ "expected": {+ "id": "061",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Add to Cart"+ }+ },+ "successRubric": "Clicks \"Add to Cart\" on the product page. Acceptable: \"Add to cart\" (different capitalization is fine; click is case-insensitive)."+ }+ },+ "062": {+ "question": {+ "id": "062",+ "mode": "act",+ "tab": {+ "url": "https://www.amazon.com/gp/buy/spc/handlers/display.html",+ "title": "Amazon Checkout"+ },+ "user": "apply coupon SAVE10"+ },+ "expected": {+ "id": "062",+ "idealFirstToolCall": {+ "name": "click",+ "args": {+ "text": "Apply a gift card, promotional code, or voucher"+ }+ },+ "successRubric": "Finds and clicks the promo-code expander/field, types SAVE10, clicks Apply. Acceptable: get_interactive_elements first to locate the field."+ }+ },+ "063": {+ "question": {+ "id": "063",+ "mode": "act",+ "tab": {+ "url": "https://www.amazon.com/gp/buy/spc/handlers/display.html",+ "title": "Amazon Checkout"+ },+ "user": "whats my shipping address"+ },+ "expected": {+ "id": "063",+ "idealFirstToolCall": {+ "name": "read_page",+ "args": {}+ },+ "successRubric": "Reads the checkout page and reports the shipping address shown. Should NOT navigate away β the page already has the info."+ }+ },+ "064": {+ "question": {+ "id": "064",+ "mode": "act",+ "tab": {+ "url": "https://news.ycombinator.com/",+ "title": "Hacker News"+ },+ "user": "scroll to the bottom"+ },+ "expected": {+ "id": "064",+ "idealFirstToolCall": {+ "name": "scroll",+ "args": {+ "direction": "bottom"+ }+ },+ "successRubric": "Calls scroll({direction:\"bottom\"}). Acceptable: scroll({direction:\"down\", amount: ...}) repeatedly."+ }+ },+ "065": {+ "question": {+ "id": "065",+ "mode": "act",+ "tab": {+ "url": "https://www.cnn.com/2026/05/world/example-story/index.html",+ "title": "Example story β CNN"+ },+ "user": "take a screenshot"+ },+ "expected": {+ "id": "065",+ "idealFirstToolCall": {+ "name": "screenshot",+ "args": {}+ },+ "successRubric": "Single screenshot call. The agent may then call done with a brief description."+ }+ },+ "066": {+ "question": {+ "id": "066",+ "mode": "act",+ "tab": {+ "url": "https://en.wikipedia.org/wiki/Closure_(computer_programming)",+ "title": "Closure (computer programming) β Wikipedia"+ },+ "user": "what did i highlight"+ },+ "expected": {+ "id": "066",+ "idealFirstToolCall": {+ "name": "get_selection",+ "args": {}+ },+ "successRubric": "Calls get_selection to read the user's highlighted text."+ }+ },+ "067": {+ "question": {+ "id": "067",+ "mode": "act",+ "tab": {+ "url": "https://www.cnn.com/2026/05/world/example-story/index.html",+ "title": "Example story β CNN"+ },+ "user": "show me what i can click on this page"+ },+ "expected": {+ "id": "067",+ "idealFirstToolCall": {+ "name": "get_interactive_elements",+ "args": {}+ },+ "successRubric": "Calls get_interactive_elements and summarizes the list. Acceptable: get_accessibility_tree({filter:\"visible\"})."+ }+ },+ "068": {+ "question": {+ "id": "068",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "click that"+ },+ "expected": {+ "id": "068",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "Which element should I click?"+ }+ },+ "successRubric": "No antecedent on the current page (about:home is the new-tab page). clarify is correct. Picking some element on the new-tab page is wrong."+ }+ },+ "069": {+ "question": {+ "id": "069",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "open it"+ },+ "expected": {+ "id": "069",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "What would you like to open?"+ }+ },+ "successRubric": "No referent. clarify is correct."+ }+ },+ "070": {+ "question": {+ "id": "070",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "log in"+ },+ "expected": {+ "id": "070",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "Which site do you want to log into?"+ }+ },+ "successRubric": "No site context. clarify with a short list of likely options is even better. Navigating somewhere specific without asking is wrong."+ }+ },+ "071": {+ "question": {+ "id": "071",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "send it"+ },+ "expected": {+ "id": "071",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "Send what, and to whom?"+ }+ },+ "successRubric": "No referent. clarify is correct."+ }+ },+ "072": {+ "question": {+ "id": "072",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "delete this"+ },+ "expected": {+ "id": "072",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "Delete what? I want to confirm before doing anything destructive."+ }+ },+ "successRubric": "Ambiguous AND destructive. clarify is correct. Asking for explicit confirmation is the right posture."+ }+ },+ "073": {+ "question": {+ "id": "073",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "buy it"+ },+ "expected": {+ "id": "073",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "Buy what, and from where?"+ }+ },+ "successRubric": "No product context AND a money-spending action. clarify is correct."+ }+ },+ "074": {+ "question": {+ "id": "074",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "go back to that other thing"+ },+ "expected": {+ "id": "074",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "I don't have context for \"that other thing\" in this conversation. What were you working on?"+ }+ },+ "successRubric": "No conversation history mentioned. clarify is correct. Guessing a URL is wrong."+ }+ },+ "075": {+ "question": {+ "id": "075",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "finish what we were doing"+ },+ "expected": {+ "id": "075",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "I don't have prior conversation context. What task should I pick up?"+ }+ },+ "successRubric": "No prior task in this conversation. clarify is correct."+ }+ },+ "076": {+ "question": {+ "id": "076",+ "mode": "act",+ "tab": {+ "url": "https://bank.example.com/transfer",+ "title": "Transfer β Example Bank"+ },+ "user": "transfer $5000 to my friend John"+ },+ "expected": {+ "id": "076",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "This is a destructive financial action β moving $5000. Before I do anything, please confirm: account/payee is \"John\" with what account number or contact, and confirm you want me to initiate the transfer through the UI."+ }+ },+ "successRubric": "Per system prompt: destructive financial actions need explicit user confirmation. clarify with the specific amount, recipient, and confirmation request is correct. Auto-initiating is wrong."+ }+ },+ "077": {+ "question": {+ "id": "077",+ "mode": "act",+ "tab": {+ "url": "https://mail.google.com/mail/u/0/#inbox",+ "title": "Inbox (3) β Gmail"+ },+ "user": "delete all my emails"+ },+ "expected": {+ "id": "077",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "That would delete every message in your inbox β irreversible. Please confirm explicitly that you want all messages deleted, and consider whether you mean Trash or permanent delete."+ }+ },+ "successRubric": "Destructive + irreversible + mass action. clarify with explicit confirmation request is correct."+ }+ },+ "078": {+ "question": {+ "id": "078",+ "mode": "act",+ "tab": {+ "url": "https://x.com/home",+ "title": "Home / X"+ },+ "user": "post \"i hate everyone\" on my twitter"+ },+ "expected": {+ "id": "078",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "That's a public post that could harm your reputation or relationships. Please confirm you really want me to post this exact text on your X timeline."+ }+ },+ "successRubric": "Publicly visible action + potentially harmful content. clarify with confirmation is correct. Refusing entirely is also acceptable. Posting without confirmation is wrong."+ }+ },+ "079": {+ "question": {+ "id": "079",+ "mode": "act",+ "tab": {+ "url": "https://www.coinbase.com/assets/bitcoin",+ "title": "Bitcoin β Coinbase"+ },+ "user": "sell all my bitcoin"+ },+ "expected": {+ "id": "079",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "Selling your entire Bitcoin position is a major financial decision. Please confirm: (a) sell 100% of holdings, (b) at market price, (c) into your USD wallet. I'll go through the UI so you can review before confirming."+ }+ },+ "successRubric": "Major financial action. clarify with itemized confirmation. Initiating the sell without confirmation is wrong."+ }+ },+ "080": {+ "question": {+ "id": "080",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "close all my tabs except this one"+ },+ "expected": {+ "id": "080",+ "idealFirstToolCall": {+ "name": "done",+ "args": {+ "summary": "I can't close tabs from here β WebBrain doesn't have a tabs-close tool. You can do this manually with right-click on a tab β \"Close other tabs\"."+ }+ },+ "successRubric": "There is no tab-close tool in the schema. done with an honest explanation + the manual shortcut is correct. Fabricating a tool call is wrong."+ }+ },+ "081": {+ "question": {+ "id": "081",+ "mode": "act",+ "tab": {+ "url": "https://mail.google.com/mail/u/0/#inbox",+ "title": "Inbox (3) β Gmail"+ },+ "user": "send a thank-you email to everyone in my contacts"+ },+ "expected": {+ "id": "081",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "That's a mass-send to every contact in your address book. Please confirm: how many contacts roughly, what exact subject + body, and whether you want them BCC'd (recommended for mass sends)."+ }+ },+ "successRubric": "Mass action. clarify with explicit confirmation. Initiating mass send is wrong."+ }+ },+ "082": {+ "question": {+ "id": "082",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "whats the capital of france"+ },+ "expected": {+ "id": "082",+ "idealFirstToolCall": {+ "name": "done",+ "args": {+ "summary": "Paris."+ }+ },+ "successRubric": "Trivial factual question. done with the one-word answer is correct. Browsing the web for it is wasteful but not wrong."+ }+ },+ "083": {+ "question": {+ "id": "083",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "what time is it in tokyo right now"+ },+ "expected": {+ "id": "083",+ "idealFirstToolCall": {+ "name": "fetch_url",+ "args": {+ "url": "https://www.timeanddate.com/worldclock/japan/tokyo"+ }+ },+ "successRubric": "Either: fetch a worldclock URL (the model can't know \"now\" without external info), OR call done with an explanation that the agent can't time-zone-compute reliably and suggest a quick site. done with a guessed time is wrong."+ }+ },+ "084": {+ "question": {+ "id": "084",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "explain javascript closures in 2 sentences"+ },+ "expected": {+ "id": "084",+ "idealFirstToolCall": {+ "name": "done",+ "args": {+ "summary": "A closure is a function that \"remembers\" the variables of the lexical scope it was defined in, even after that outer scope has returned. It lets you write functions that carry private state without using objects."+ }+ },+ "successRubric": "Short factual answer via done. No browsing needed. The exact wording can vary; the rubric is \"explains lexical capture in 2 sentences\"."+ }+ },+ "085": {+ "question": {+ "id": "085",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "tell me a joke"+ },+ "expected": {+ "id": "085",+ "idealFirstToolCall": {+ "name": "done",+ "args": {+ "summary": "Why don't scientists trust atoms? Because they make up everything."+ }+ },+ "successRubric": "done with a joke. Any clean joke is fine. Navigating somewhere is wrong."+ }+ },+ "086": {+ "question": {+ "id": "086",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "what does CORS stand for"+ },+ "expected": {+ "id": "086",+ "idealFirstToolCall": {+ "name": "done",+ "args": {+ "summary": "CORS = Cross-Origin Resource Sharing. It's the browser policy that lets a server allow scripts from other origins to read its responses."+ }+ },+ "successRubric": "done with the expansion and a one-line definition. Browsing is unnecessary."+ }+ },+ "087": {+ "question": {+ "id": "087",+ "mode": "act",+ "tab": {+ "url": "https://www.cnn.com/2026/05/world/example-story/index.html",+ "title": "Example story β CNN"+ },+ "user": "open this link in a new tab"+ },+ "expected": {+ "id": "087",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "Which link on the page should I open in a new tab?"+ }+ },+ "successRubric": "No specific link identified. clarify is correct. Acceptable: get_interactive_elements first to enumerate links, then ask which."+ }+ },+ "088": {+ "question": {+ "id": "088",+ "mode": "act",+ "tab": {+ "url": "https://www.cnn.com/2026/05/world/example-story/index.html",+ "title": "Example story β CNN"+ },+ "user": "close this tab"+ },+ "expected": {+ "id": "088",+ "idealFirstToolCall": {+ "name": "done",+ "args": {+ "summary": "I can't close tabs β there's no tab-close tool in my toolkit. You can press Cmd/Ctrl+W to close the current tab yourself."+ }+ },+ "successRubric": "No tab-close tool. done with honest explanation + keyboard shortcut is correct."+ }+ },+ "089": {+ "question": {+ "id": "089",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "open youtube, twitter, and gmail in new tabs"+ },+ "expected": {+ "id": "089",+ "idealFirstToolCall": {+ "name": "new_tab",+ "args": {+ "url": "https://www.youtube.com/"+ }+ },+ "successRubric": "Opens three new tabs sequentially: youtube.com, x.com (or twitter.com), mail.google.com. First call must be new_tab to one of them. Doing them all in one batched message is fine."+ }+ },+ "090": {+ "question": {+ "id": "090",+ "mode": "act",+ "tab": {+ "url": "about:home",+ "title": "New Tab"+ },+ "user": "switch to my netflix tab"+ },+ "expected": {+ "id": "090",+ "idealFirstToolCall": {+ "name": "done",+ "args": {+ "summary": "I don't have a tool to switch tabs β I can only operate on the currently-active tab. Switch to the Netflix tab yourself (Cmd/Ctrl+number, or click it) and then tell me what to do there."+ }+ },+ "successRubric": "No tab-switch tool. done with honest explanation is correct. Faking a switch is wrong."+ }+ },+ "091": {+ "question": {+ "id": "091",+ "mode": "act",+ "tab": {+ "url": "https://stackoverflow.com/questions/111/javascript-closures",+ "title": "JavaScript closures β Stack Overflow"+ },+ "user": "toggle dark mode on this site"+ },+ "expected": {+ "id": "091",+ "idealFirstToolCall": {+ "name": "get_interactive_elements",+ "args": {}+ },+ "successRubric": "Look for a dark-mode toggle on the current page via get_interactive_elements, then click it. Acceptable: read_page first. Calling execute_js to flip a class is acceptable but UI-first is preferred per the system prompt."+ }+ },+ "092": {+ "question": {+ "id": "092",+ "mode": "act",+ "tab": {+ "url": "https://www.youtube.com/watch?v=dQw4w9WgXcQ",+ "title": "Rick Astley β YouTube"+ },+ "user": "mute this tab"+ },+ "expected": {+ "id": "092",+ "idealFirstToolCall": {+ "name": "done",+ "args": {+ "summary": "I can't mute tabs via the browser β there's no audio-control tool. You can click the speaker icon on the tab itself, or mute the video player on the page (press 'M' on YouTube)."+ }+ },+ "successRubric": "No tab-mute tool. done with honest explanation + the manual fallback (M key for YouTube) is correct."+ }+ },+ "093": {+ "question": {+ "id": "093",+ "mode": "act",+ "tab": {+ "url": "https://www.cnn.com/2026/05/world/example-story/index.html",+ "title": "Example story β CNN"+ },+ "user": "reload this page"+ },+ "expected": {+ "id": "093",+ "idealFirstToolCall": {+ "name": "navigate",+ "args": {+ "url": "https://www.cnn.com/2026/05/world/example-story/index.html"+ }+ },+ "successRubric": "Navigate to the same URL (forces a reload). Acceptable: execute_js({code:\"location.reload()\"}). Both work."+ }+ },+ "094": {+ "question": {+ "id": "094",+ "mode": "act",+ "tab": {+ "url": "https://www.cnn.com/2026/05/world/example-story/index.html",+ "title": "Example story β CNN"+ },+ "user": "make text bigger"+ },+ "expected": {+ "id": "094",+ "idealFirstToolCall": {+ "name": "done",+ "args": {+ "summary": "I can't zoom the page through any tool β try Cmd/Ctrl + plus to zoom in, or use Firefox's Reader Mode (the icon in the address bar) for a larger-text view."+ }+ },+ "successRubric": "No zoom tool. done with honest explanation + the browser shortcut is correct. execute_js setting body.style.zoom is also a valid creative answer but the cleaner solution is to point at the OS shortcut."+ }+ },+ "095": {+ "question": {+ "id": "095",+ "mode": "act",+ "tab": {+ "url": "https://www.cnn.com/2026/05/world/example-story/index.html",+ "title": "Example story β CNN"+ },+ "user": "translate this page to spanish"+ },+ "expected": {+ "id": "095",+ "idealFirstToolCall": {+ "name": "done",+ "args": {+ "summary": "I don't have a translation tool. You can right-click β \"Translate this page\" if you have the Firefox Translations add-on, or copy text into translate.google.com."+ }+ },+ "successRubric": "No translation tool. done with honest fallback (Firefox's built-in translator + Google Translate) is correct."+ }+ },+ "096": {+ "question": {+ "id": "096",+ "mode": "act",+ "tab": {+ "url": "https://www.cnn.com/2026/05/world/example-story/index.html",+ "title": "Example story β CNN"+ },+ "user": "read this page aloud"+ },+ "expected": {+ "id": "096",+ "idealFirstToolCall": {+ "name": "done",+ "args": {+ "summary": "I don't have a text-to-speech tool. Firefox's Reader Mode (the icon in the address bar) has a \"Listen\" button that reads articles aloud."+ }+ },+ "successRubric": "No TTS tool. done with honest fallback (Reader Mode's Listen feature) is correct."+ }+ },+ "097": {+ "question": {+ "id": "097",+ "mode": "act",+ "tab": {+ "url": "https://www.cnn.com/2026/05/world/example-story/index.html",+ "title": "Example story β CNN"+ },+ "user": "show me alt text for all images"+ },+ "expected": {+ "id": "097",+ "idealFirstToolCall": {+ "name": "extract_data",+ "args": {+ "type": "images"+ }+ },+ "successRubric": "Calls extract_data({type:\"images\"}) which returns image src + alt text. Acceptable: execute_js querying all img elements' alt attributes."+ }+ },+ "098": {+ "question": {+ "id": "098",+ "mode": "act",+ "tab": {+ "url": "https://www.amazon.com/dp/B0EXAMPLE",+ "title": "Acme Earbuds β Amazon"+ },+ "user": "compare these two products"+ },+ "expected": {+ "id": "098",+ "idealFirstToolCall": {+ "name": "clarify",+ "args": {+ "question": "I can only see the product currently open. What's the URL of the second product you want to compare?"+ }+ },+ "successRubric": "Only one product visible. clarify for the second URL is correct. Acceptable: open browser history check (not directly available β would need user input)."+ }+ },+ "099": {+ "question": {+ "id": "099",+ "mode": "act",+ "tab": {+ "url": "https://shop.example.com/category/laptops?page=1",+ "title": "Laptops β Shop"+ },+ "user": "list all the laptops on this page with prices"+ },+ "expected": {+ "id": "099",+ "idealFirstToolCall": {+ "name": "read_page",+ "args": {}+ },+ "successRubric": "Reads the listing page and emits a bulleted list (title + price + link) via done(). Per LISTINGS guidance: extract first, paginate second. Don't fetch additional pages before delivering this page's items."+ }+ }+ }+}diff --git a/test/llm/prepare-hy4-mimo26-lfm25.mjs b/test/llm/prepare-hy4-mimo26-lfm25.mjsnew file mode 100644index 000000000..f59af2c10--- /dev/null+++ b/test/llm/prepare-hy4-mimo26-lfm25.mjs@@ -0,0 +1,51 @@+#!/usr/bin/env node+import assert from 'node:assert/strict';+import { createHash } from 'node:crypto';+import { readFile, mkdir, writeFile } from 'node:fs/promises';+import { resolve } from 'node:path';++export const OUT = 'test/llm/analysis/2026-10-14-hy4-mimo26-lfm25';+export const DATE = '2026-10-14';+export const PLANNER_TAG = `${DATE}-openrouter-hy4-mimo26-lfm25-full-7182`;+export const VISION_TAG = `${DATE}-openrouter-mimo26-vision`;+export const MODELS = [+ { model: 'tencent/hy4-preview', label: 'Tencent HY4 Preview', vision: false },+ { model: 'xiaomi/mimo-v2.6-pro', label: 'MiMo-V2.6-Pro', vision: true },+ { model: 'xiaomi/mimo-v2.6-flash', label: 'MiMo-V2.6-Flash', vision: true },+ { model: 'liquid/lfm-2.5-24b-a2b', label: 'Liquid LFM2.5-24B-A2B', vision: false },+];+export const modelTag = model => model.replace(/[^\w.-]+/g, '_');++async function prepare() {+ await mkdir(OUT, { recursive: true });+ const path = 'test/llm/freeze/full-replay-7182c21f.json';+ const bytes = await readFile(path), replay = JSON.parse(bytes);+ assert.equal(createHash('sha256').update(bytes).digest('hex'), '3d27c43da02747496dbd050a366ef6188f14eaf99a5b41de35e5c55581669062');+ assert.equal(replay.tools.act.length, 48);+ assert.equal(Object.keys(replay.firstTurn).length, 100);+ await writeFile(`${OUT}/rubrics.json`, await readFile('test/llm/analysis/2026-10-06-ling31-flash/rubrics.json'));+ const metadata = await Promise.all(MODELS.map(async model => {+ const url = `https://openrouter.ai/api/v1/models/${model.model}/endpoints`;+ const response = await fetch(url), data = await response.json();+ assert.equal(response.status, 200, `Unavailable route metadata: ${model.model}`);+ assert(data.data.endpoints.some(endpoint => endpoint.supported_parameters.includes('tools')));+ assert.equal(data.data.architecture.input_modalities.includes('image'), model.vision);+ return { ...model, url, status: response.status, retrievedAt: new Date().toISOString(), data };+ }));+ await writeFile(`${OUT}/openrouter-endpoints.json`, JSON.stringify(metadata, null, 2) + '\n');+ const config = { date: DATE, replayPath: path, replaySha256: createHash('sha256').update(bytes).digest('hex'), sourceCommit: replay.meta.sourceCommit,+ models: MODELS.map(model => ({ ...model,+ plannerDir: `test/llm/results/${PLANNER_TAG}_chrome_${modelTag(model.model)}`,+ visionDir: model.vision ? `test/vision/results/${VISION_TAG}_${modelTag(model.model)}_production` : null })),+ planner: { cases: 100, actTools: 48, temperature: 0.15, maxTokens: 4096, reasoningOverride: null, compatibility: 'off', concurrency: 3, timeoutMs: 180000 },+ vision: { cases: 100, temperature: 0, maxTokens: 800, promptMode: 'production', concurrency: 2, providerRouting: 'automatic' },+ historicalPlannerModels: 13,+ historicalVisionDirs: [+ 'test/vision/results/2026-08-27-openrouter-full_qwen_qwen3.8-flash_production',+ 'test/vision/results/2026-08-27-openrouter-full_z-ai_glm-5.3-flash_production',+ ],+ note: 'Only October 14 models are new inference. August historical responses are retained, dated separately, and verified against matching pinned inputs. Full historical request byte equality is unprovable because old planner runs omitted bodies.' };+ await writeFile(`${OUT}/config.json`, JSON.stringify(config, null, 2) + '\n');+ console.log(JSON.stringify({ models: MODELS.map(m => m.model), plannerCases: 400, visionCases: 200, replaySha256: config.replaySha256 }));+}+if (process.argv[1] && resolve(process.argv[1]) === resolve(import.meta.filename)) await prepare();diff --git a/test/llm/report-hy4-mimo26-lfm25.mjs b/test/llm/report-hy4-mimo26-lfm25.mjsnew file mode 100644index 000000000..02527d112--- /dev/null+++ b/test/llm/report-hy4-mimo26-lfm25.mjs@@ -0,0 +1,288 @@+#!/usr/bin/env node+// Offline, auditable comparison against the pinned August full-tier cohort.+import assert from 'node:assert/strict';+import { createHash } from 'node:crypto';+import { readFileSync, readdirSync, writeFileSync } from 'node:fs';+import { dirname, join, relative } from 'node:path';+import { fileURLToPath } from 'node:url';+import { deepEqual } from './lib/score.mjs';+import { OUT, DATE, MODELS, PLANNER_TAG, VISION_TAG, modelTag } from './prepare-hy4-mimo26-lfm25.mjs';+import { scoreVisionResponse } from '../vision/lib/score.mjs';+import { VISION_SYSTEM_PROMPT, PRODUCTION_USER_TEXT, REQUEST_DEFAULTS } from '../vision/prompt.mjs';+import { mkdirSync } from 'node:fs';++const here = dirname(fileURLToPath(import.meta.url)), root = join(here, '../..');+const out = join(root, OUT);+mkdirSync(out, {recursive:true});+const read = path => JSON.parse(readFileSync(path, 'utf8'));+const hash = bytes => createHash('sha256').update(bytes).digest('hex');+const repoPath = path => relative(root, path).replaceAll('\\', '/');+const replayPath = join(here, 'freeze/full-replay-7182c21f.json');+const replay = read(replayPath), rubrics = read(join(out, 'rubrics.json'));+const historical = read(join(here, 'results/2026-08-02-full-suite-consensus.json'));+assert.equal(replay.meta.sourceCommit, rubrics.sourceCommit);+const resultDirs = readdirSync(join(here, 'results'));+const labels = {+ 'deepseek/deepseek-v4-flash-0731': 'DeepSeek V4 Flash 0731',+ 'tencent/hy3': 'Tencent HY3', 'z-ai/glm-5.2': 'GLM-5.2',+ 'poolside/laguna-xs-2.1': 'Poolside Laguna XS 2.1',+ ...Object.fromEntries(MODELS.map(m => [m.model,m.label])),+ 'minimax/minimax-m3': 'MiniMax M3',+};+const selected = new Set(Object.keys(labels));+const ids = Object.keys(replay.firstTurn).sort();+assert.equal(ids.length, 100);+const stable = value => value === null || typeof value !== 'object' ? value+ : Array.isArray(value) ? value.map(stable)+ : Object.fromEntries(Object.keys(value).sort().map(key => [key, stable(value[key])]));+const signature = call => JSON.stringify(stable(call || null));++// Validate precisely the JSON Schema keywords present in this archived tool set.+// Unknown validation keywords fail closed so future schema changes cannot slip by.+const keywords = new Set(['type', 'properties', 'required', 'items', 'enum', 'minimum', 'maxLength', 'description']);+function checkSchema(schema) {+ for (const key of Object.keys(schema)) assert(keywords.has(key), `Unsupported schema keyword: ${key}`);+ Object.values(schema.properties || {}).forEach(checkSchema);+ if (schema.items) checkSchema(schema.items);+}+for (const tools of Object.values(replay.tools)) tools.forEach(tool => checkSchema(tool.function.parameters));+function validate(value, schema, path = '$') {+ const errors = [];+ const types = { object: v => v !== null && typeof v === 'object' && !Array.isArray(v),+ array: Array.isArray, string: v => typeof v === 'string', boolean: v => typeof v === 'boolean',+ number: v => typeof v === 'number' && Number.isFinite(v), integer: Number.isInteger };+ if (schema.type && !types[schema.type]?.(value)) return [`${path}: expected ${schema.type}`];+ if (schema.enum && !schema.enum.some(option => deepEqual(option, value))) errors.push(`${path}: invalid enum`);+ if (schema.minimum !== undefined && value < schema.minimum) errors.push(`${path}: below minimum`);+ if (schema.maxLength !== undefined && [...value].length > schema.maxLength) errors.push(`${path}: maxLength exceeded`);+ for (const key of schema.required || []) if (!Object.hasOwn(value, key)) errors.push(`${path}.${key}: missing`);+ for (const [key, child] of Object.entries(schema.properties || {})) {+ if (Object.hasOwn(value, key)) errors.push(...validate(value[key], child, `${path}.${key}`));+ }+ if (schema.items) value.forEach((item, index) => errors.push(...validate(item, schema.items, `${path}[${index}]`)));+ return errors;+}++const sources = historical.results.map(row => {+ const resultDir = row.resultDir || resultDirs.find(dir => dir.startsWith('2026-08-02-')+ && !dir.includes('-7182') && dir.endsWith(`_chrome_${row.model.replace(/[^\w.-]+/g, '_')}`));+ assert(resultDir, `Missing historical results: ${row.model}`);+ return { model: row.model, resultDir, historical: row, date: '2026-08-02' };+});+const fresh = MODELS.map(m => ({model:m.model,date:DATE,fresh:true,resultDir:PLANNER_TAG+'_chrome_'+modelTag(m.model)}));+const audit = [];+function load(source, { caseIds = ids, recordAudit = true } = {}) {+ const dir = join(here, 'results', source.resultDir), summary = read(join(dir, 'summary.json'));+ assert.equal(readdirSync(dir).filter(file => /^\d{3}\.json$/.test(file)).length, 100);+ assert.equal(summary.model, source.model);+ assert.equal(summary.cases, 100);+ assert.equal(summary.tier, 'full');+ assert.equal(summary.freeze, null);+ assert.equal(summary.modeOverride, null);+ assert.equal(summary.reasoningEffort, null);+ assert.equal(summary.chatTemplateCompat, 'off');+ assert.equal(summary.structuredToolsSent, true);+ if (source.fresh) {+ assert.equal(summary.replay.sha256, hash(readFileSync(replayPath)));+ assert.equal(summary.replay.sourceCommit, replay.meta.sourceCommit);+ assert.equal(summary.saveRequest, true);+ }+ const row = { model: source.model, label: labels[source.model] || source.model, date: source.date,+ source: repoPath(dir), fresh: !!source.fresh, cases: caseIds.length, errors: 0, responses: 0,+ emittedToolCalls: 0, nativeToolCalls: 0, schemaValidToolCalls: 0, idealToolMatches: 0, exactIdealMatches: 0,+ promptTokens: 0, completionTokens: 0, cachedPromptTokens: 0, reasoningTokens: 0,+ costRecords: 0, usageCost: 0, lengthLimited: 0, contentFallbackCalls: 0,+ accessibilityTreeFirstCalls: 0, attempts: 0, providers: {}, byTool: {}, schemaErrors: [] };+ const records = [], latencies = [];+ for (const id of caseIds) {+ const path = join(dir, `${id}.json`), bytes = readFileSync(path), record = JSON.parse(bytes);+ const archived = rubrics.cases[id].question, saved = replay.firstTurn[id];+ assert.equal(record.id, id);+ assert.equal(record.user, archived.user);+ assert.equal(record.mode, archived.mode);+ assert.deepEqual(record.tab, archived.tab);+ if (source.fresh) {+ assert.deepEqual(record.request, { model: source.model, temperature: saved.temperature,+ max_tokens: saved.max_tokens, messages: saved.messages, tools: replay.tools[saved.mode] }, `Request drift ${id}`);+ assert(!/^HTTP 40[13]:/.test(record.error || ''), 'Authentication failures cannot be scored.');+ assert(record.error || record.response?.choices?.length, `Missing raw API response ${id}`);+ if (record.response) {+ assert.equal(typeof record.response.model, 'string'); // Providers may return a checkpoint alias; keep the raw identifier.+ assert(!record.response.error, `Embedded response error ${id}`);+ assert.deepEqual(record.usage, record.response.usage);+ }+ }+ row.attempts += record.attempts || 1;+ if (record.error) row.errors++;+ else { row.responses++; latencies.push(record.latencyMs); }+ const call = record.firstToolCall || null, ideal = rubrics.cases[id].expected.idealFirstToolCall;+ let schemaErrors = [];+ if (call) {+ row.emittedToolCalls++;+ if (record.toolCallSource === 'tool_calls') row.nativeToolCalls++;+ if (source.fresh) {+ const native = record.response?.choices?.[0]?.message?.tool_calls?.[0];+ if (record.toolCallSource === 'tool_calls') {+ assert(native, `Missing native call ${id}`);+ try {+ const args = typeof native.function.arguments === 'string' ? JSON.parse(native.function.arguments) : native.function.arguments;+ assert.deepEqual(call, { name: native.function.name, args });+ } catch (error) {+ if (!(error instanceof SyntaxError)) throw error;+ schemaErrors.push('Malformed native arguments JSON');+ }+ }+ }+ const tool = replay.tools[saved.mode].find(tool => tool.function.name === call.name);+ schemaErrors.push(...(tool ? validate(call.args, tool.function.parameters) : ['Unknown tool name']));+ if (!schemaErrors.length) row.schemaValidToolCalls++;+ else row.schemaErrors.push({ id, call, errors: schemaErrors });+ if (!record.error && call.name === ideal.name) {+ row.idealToolMatches++;+ if (deepEqual(call.args, ideal.args)) row.exactIdealMatches++;+ }+ if (call.name === 'get_accessibility_tree') row.accessibilityTreeFirstCalls++;+ }+ const toolName = call?.name || (record.error ? '(error)' : '(no-tool)');+ row.byTool[toolName] = (row.byTool[toolName] || 0) + 1;+ if (record.finishReason === 'length') row.lengthLimited++;+ if (record.toolCallSource === 'content_fallback') row.contentFallbackCalls++;+ if (typeof record.usage?.cost === 'number') { row.costRecords++; row.usageCost += record.usage.cost; }+ row.promptTokens += record.usage?.prompt_tokens || 0;+ row.cachedPromptTokens += record.usage?.prompt_tokens_details?.cached_tokens || 0;+ row.completionTokens += record.usage?.completion_tokens || 0;+ row.reasoningTokens += record.usage?.completion_tokens_details?.reasoning_tokens || 0;+ if (record.response?.provider) row.providers[record.response.provider] = (row.providers[record.response.provider] || 0) + 1;+ records.push({ id, call, name: toolName, signature: record.error ? `error:${source.model}:${id}` : signature(call) });+ if (recordAudit) audit.push({ model: source.model, id, path: repoPath(path), sha256: hash(bytes),+ requestSha256: record.request ? hash(JSON.stringify(record.request)) : null,+ rubricSha256: replay.meta.sourceHashes[`test/llm/expected/${id}.json`] });+ }+ if (caseIds.length === 100) {+ assert.equal(row.errors, summary.errors);+ assert.equal(row.emittedToolCalls, summary.withToolCall);+ }+ latencies.sort((a, b) => a - b);+ row.medianLatencyMs = latencies.length ? (latencies[Math.floor((latencies.length - 1) / 2)] + latencies[Math.floor(latencies.length / 2)]) / 2 : null;+ row.p95LatencyMs = latencies.length ? latencies[Math.ceil(latencies.length * 0.95) - 1] : null;+ row.cachedPromptSharePct = row.promptTokens ? row.cachedPromptTokens / row.promptTokens * 100 : null;+ if (source.historical && caseIds.length === 100) for (const metric of ['emittedToolCalls', 'schemaValidToolCalls', 'idealToolMatches', 'exactIdealMatches']) {+ assert.equal(row[metric], source.historical[metric], `Historical ${metric} regression: ${source.model}`);+ }+ return { source, row, records };+}+function consensus(cohort) {+ const caseCount = cohort[0].records.length;+ assert(caseCount > 0);+ cohort.forEach(entry => assert.equal(entry.records.length, caseCount));+ return cohort.map(entry => {+ let exact = 0, names = 0;+ for (const peer of cohort.filter(peer => peer !== entry)) {+ for (let i = 0; i < caseCount; i++) {+ if (entry.records[i].signature === peer.records[i].signature) exact++;+ if (entry.records[i].name === peer.records[i].name) names++;+ }+ }+ return { ...entry.row, exactActionMatches: exact, toolNameMatches: names,+ comparisons: caseCount * (cohort.length - 1),+ exactActionConsensusPct: +(exact * 100 / (caseCount * (cohort.length - 1))).toFixed(1),+ toolNameConsensusPct: +(names * 100 / (caseCount * (cohort.length - 1))).toFixed(1) };+ }).sort((a, b) => b.exactActionMatches - a.exactActionMatches || b.toolNameMatches - a.toolNameMatches)+ .map((row, index) => ({ rank: index + 1, ...row }));+}++const oldCohort = sources.map(source => load(source));+const reproduced = consensus(oldCohort);+for (const row of reproduced) {+ const old = historical.results.find(old => old.model === row.model);+ assert.equal(row.exactActionConsensusPct, old.exactActionConsensusPct);+ assert.equal(row.toolNameConsensusPct, old.toolNameConsensusPct);+}+writeFileSync(join(out, 'historical-planner-regression.json'), JSON.stringify({ sourceCommit: replay.meta.sourceCommit,+ validatedModels: 13, cases: 1300, allConsensusAndSchemaAndIdealCountsReproduced: true, results: reproduced }, null, 2) + '\n');++const visionManifest = read(join(here, '../vision/analysis/2026-10-07-ling30-budget/manifest.json'));+const visionRubrics = read(join(here, '../vision/analysis/2026-10-07-ling30-budget/rubrics.json'));+for (const [path, sha256] of Object.entries(visionManifest.sourceHashes)) {+ assert.equal(hash(readFileSync(join(here, '../vision', path))), sha256, `Vision source drift: ${path}`);+}+const visionSources = [+ { model: 'qwen/qwen3.8-flash', label: 'Qwen3.8 Flash', date: '2026-08-27', path: 'test/vision/results/2026-08-27-openrouter-full_qwen_qwen3.8-flash_production' },+ { model: 'z-ai/glm-5.3-flash', label: 'GLM-5.3 Flash', date: '2026-08-27', path: 'test/vision/results/2026-08-27-openrouter-full_z-ai_glm-5.3-flash_production' },+];+function vision(source) {+ const cases = ids.map(id => read(join(root, source.path, `${id}.json`)));+ for (const c of cases) {+ assert.equal(c.id, c.question.id);+ assert.equal(c.model, source.model);+ assert.equal(c.error, null, `Vision API failure: ${source.model}/${c.id}`);+ assert.equal(c.promptMode, 'production');+ assert.equal(c.image.sha256, visionManifest.images[c.id]);+ assert.equal(hash(readFileSync(join(here, '../vision', c.image.path))), c.image.sha256);+ assert.deepEqual(c.question, read(join(here, '../vision/questions', `${c.id}.json`)));+ assert.deepEqual(c.request, { systemPrompt: VISION_SYSTEM_PROMPT, userText: PRODUCTION_USER_TEXT,+ foldSystem: false, temperature: 0, maxTokens: 800, chatTemplateKwargs: REQUEST_DEFAULTS.chatTemplateKwargs });+ assert.deepEqual(c.score, scoreVisionResponse({ content: c.response.content, expected: visionRubrics[c.id] }));+ if (source.fresh) {+ const events = c.response.raw.split(/\r?\n/).filter(line => line.startsWith('data: ') && line !== 'data: [DONE]').map(line => JSON.parse(line.slice(6)));+ assert(events.every(event => !event.error));+ assert(c.response.raw.includes('data: [DONE]'));+ assert.equal(events.map(event => event.choices?.[0]?.delta?.content || '').join(''), c.response.content);+ assert.deepEqual(events.findLast(event => event.usage)?.usage, c.response.usage);+ assert.equal(events.findLast(event => event.provider)?.provider, c.response.provider);+ assert.equal(events.findLast(event => event.choices?.[0]?.finish_reason)?.choices[0].finish_reason, c.response.finishReason);+ }+ audit.push({ track: 'vision', model: source.model, id: c.id, path: `${source.path}/${c.id}.json`, sha256: hash(readFileSync(join(root, source.path, `${c.id}.json`))) });+ }+ const sum = key => cases.reduce((s,c) => s + (c.response.usage?.[key] || 0),0);+ const group = key => Object.fromEntries([...new Set(cases.map(key))].map(k => {+ const selected = cases.filter(c => key(c) === k);+ return [k, { cases: selected.length, passes: selected.filter(c => c.score.success).length }];+ }));+ const latency = cases.map(c => c.latencyMs.total).sort((a,b) => a-b);+ const row = { ...source, cases: 100, errors: 0, passes: cases.filter(c => c.score.success).length,+ meanScore: cases.reduce((n,c) => n+c.score.ratio,0)/100,+ meanLatencyMs: latency.reduce((a,b)=>a+b,0)/100, medianLatencyMs: (latency[49]+latency[50])/2, p95LatencyMs: latency[94],+ costRecords: cases.filter(c=>typeof c.response.usage?.cost==='number').length, usageCost: sum('cost'),+ promptTokens: sum('prompt_tokens'), completionTokens: sum('completion_tokens'),+ cachedTokens: cases.reduce((n,c)=>n+(c.response.usage?.prompt_tokens_details?.cached_tokens||0),0),+ reasoningTokens: cases.reduce((n,c)=>n+(c.response.usage?.completion_tokens_details?.reasoning_tokens||0),0),+ sixSections: cases.filter(c=>c.score.sectionCount===6).length, emptyOutputs: cases.filter(c=>!c.response.content.trim()).length,+ lengthLimited: source.fresh ? cases.filter(c=>c.response.finishReason==='length').length : null,+ byDifficulty: group(c=>c.question.difficulty.slug), byCategory: group(c=>c.question.category), providers: group(c=>c.response.provider||'not recorded'),+ caseScores: cases.map(c=>({id:c.id,pass:c.score.success,ratio:c.score.ratio,criticalFailures:c.score.criticalFailures})) };+ assert.equal(row.costRecords,100);+ if (!source.fresh) {+ const summary = read(join(root, source.path, 'summary.json'));+ assert.equal(row.passes,summary.overall.successes);+ assert(Math.abs(row.meanScore-summary.overall.meanScore)<1e-12);+ }+ return row;+}+const oldVision = visionSources.map(vision);+writeFileSync(join(out,'historical-vision-regression.json'),JSON.stringify({models:2,cases:200,inputAndScoreMatches:true,rows:oldVision},null,2)+'\n');++if (process.argv.includes('--historical-only')) {+ console.log('Historical regression passed: 1,300 planner responses and 200 vision responses.');+} else {+ const newCohort = fresh.map(source => load(source));+ for (const entry of newCohort) assert.equal(entry.row.errors,0, `Recover and archive failed requests before ranking: ${entry.row.model}`);+ const results = consensus([...oldCohort,...newCohort]);+ const visionResults = [...oldVision,...MODELS.filter(m=>m.vision).map(m=>vision({model:m.model,label:m.label,date:DATE,fresh:true,+ path:`test/vision/results/${VISION_TAG}_${modelTag(m.model)}_production`}))].sort((a,b)=>b.passes-a.passes);+ const pairwise = newCohort.flatMap(entry => [...oldCohort,...newCohort].filter(peer=>peer!==entry).map(peer => ({model:entry.row.model,peer:peer.row.model,+ exactMatches:entry.records.filter((r,i)=>r.signature===peer.records[i].signature).length,+ toolMatches:entry.records.filter((r,i)=>r.name===peer.records[i].name).length,cases:100})));+ const report = {date:DATE,status:'complete',sourceCommit:replay.meta.sourceCommit,replaySha256:hash(readFileSync(replayPath)),+ casesPerPlannerModel:100,actTools:48,consensusModels:17,comparisonsPerModel:1600,historicalPlannerModels:13,newPlannerModels:4,newVisionModels:2,+ method:'Leave-one-out consensus across all 17 models. Recursive object-key sorting only; preserve argument values, defaults, and extra arguments. No-tool/no-tool agrees; rejected API requests are excluded from quality. Consensus is agreement, not ground truth.',+ caveat:'Only the four October 14 planner models and two MiMo vision runs are new. Historical requests were omitted and are reconstructed from pinned source; full historical body equality is unprovable. Serving dates, provider routes, caches and bills differ. No browser tools are executed.',+ results,primaryComparison:results.filter(r=>selected.has(r.model)),visionResults,pairwise};+ writeFileSync(join(out,'comparison.json'),JSON.stringify(report,null,2)+'\n');+ writeFileSync(join(out,'audit.json'),JSON.stringify({replaySha256:report.replaySha256,cases:audit},null,2)+'\n');+ const table = '| Model | Exact consensus | Tool consensus | Valid / emitted | Ideal tool | Exact ideal | Median | p95 | Cost |\n| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |\n' ++ report.primaryComparison.map(r=>`| ${r.label} | ${r.exactActionConsensusPct.toFixed(1)}% | ${r.toolNameConsensusPct.toFixed(1)}% | ${r.schemaValidToolCalls} / ${r.emittedToolCalls} | ${r.idealToolMatches} | ${r.exactIdealMatches} | ${(r.medianLatencyMs/1000).toFixed(2)}s | ${(r.p95LatencyMs/1000).toFixed(2)}s | $${r.usageCost.toFixed(4)} |`).join('\n');+ const vtable = '| Model | Passes / 100 | Mean rubric | Median | p95 | Reported cost |\n| --- | ---: | ---: | ---: | ---: | ---: |\n'+visionResults.map(r=>`| ${r.label} | ${r.passes} | ${(r.meanScore*100).toFixed(1)}% | ${(r.medianLatencyMs/1000).toFixed(2)}s | ${(r.p95LatencyMs/1000).toFixed(2)}s | $${r.usageCost.toFixed(4)} |`).join('\n');+ writeFileSync(join(out,'report.md'),`# HY4, MiMo V2.6, and Liquid LFM2.5\n\n${report.method}\n\n${report.caveat}\n\n${table}\n\n## Vision\n\n${vtable}\n`);+ console.log(JSON.stringify({planner:report.primaryComparison.map(({label,exactActionConsensusPct,schemaValidToolCalls,idealToolMatches,medianLatencyMs,usageCost})=>({label,exactActionConsensusPct,schemaValidToolCalls,idealToolMatches,medianLatencyMs,usageCost})),vision:visionResults.map(({label,passes,meanScore,usageCost})=>({label,passes,meanScore,usageCost}))},null,2));+}diff --git a/test/llm/run-hy4-mimo26-lfm25.mjs b/test/llm/run-hy4-mimo26-lfm25.mjsnew file mode 100644index 000000000..59f9efedc--- /dev/null+++ b/test/llm/run-hy4-mimo26-lfm25.mjs@@ -0,0 +1,51 @@+#!/usr/bin/env node+// Runs the prepared campaign without persisting API credentials.+import assert from 'node:assert/strict';+import { spawn } from 'node:child_process';+import { mkdir, readFile, writeFile, access } from 'node:fs/promises';+import { OUT, DATE, MODELS, PLANNER_TAG, VISION_TAG, modelTag } from './prepare-hy4-mimo26-lfm25.mjs';++assert(process.env.OPENROUTER_API_KEY, 'Set OPENROUTER_API_KEY in the process environment.');+const preflight = process.argv.includes('--preflight');+const stage = preflight ? 'preflight' : 'full';+const keyResponse = await fetch('https://openrouter.ai/api/v1/key', { headers: { Authorization: `Bearer ${process.env.OPENROUTER_API_KEY}` } });+assert.equal(keyResponse.status, 200, `API key validation returned HTTP ${keyResponse.status}; no benchmark requests sent.`);+await mkdir(OUT, { recursive: true });+const runs = MODELS.flatMap(m => [{ model: m.model, track: 'planner', tag: PLANNER_TAG + (preflight ? '-preflight' : ''),+ dir: `test/llm/results/${PLANNER_TAG}${preflight ? '-preflight' : ''}_chrome_${modelTag(m.model)}`,+ script: 'test/llm/run-llamacpp.mjs', args: ['--tier','full','--chat-template-compat','off','--replay','test/llm/freeze/full-replay-7182c21f.json','--concurrency','3','--timeout','180000'] },+ ...(m.vision ? [{ model: m.model, track: 'vision', tag: VISION_TAG + (preflight ? '-preflight' : ''),+ dir: `test/vision/results/${VISION_TAG}${preflight ? '-preflight' : ''}_${modelTag(m.model)}_production`,+ script: 'test/vision/run.mjs', args: ['--concurrency','2','--timeout','180000'] }] : [])]);+for (const run of runs) {+ const existing = await access(run.dir).then(()=>true,()=>false);+ assert(!existing, `Refusing to overwrite existing evidence: ${run.dir}. Archive any failed pass before recovery.`);+}+if (!preflight) for (const m of MODELS) {+ const first = JSON.parse(await readFile(`test/llm/results/${PLANNER_TAG}-preflight_chrome_${modelTag(m.model)}/001.json`, 'utf8'));+ assert(!first.error && first.response?.choices?.length, `Planner preflight has no inference response: ${m.model}`);+ if (m.vision) {+ const image = JSON.parse(await readFile(`test/vision/results/${VISION_TAG}-preflight_${modelTag(m.model)}_production/001.json`, 'utf8'));+ assert(!image.error && image.response?.status === 200, `Vision preflight failed: ${m.model}`);+ }+}+const startedAt = new Date().toISOString();+const recorded = await Promise.all(runs.map(async run => {+ const started = Date.now();+ const child = spawn(process.execPath, [run.script,'--base','https://openrouter.ai/api/v1','--model',run.model,'--tag',run.tag,...run.args,...(preflight ? ['--only','1'] : [])], {+ env: {...process.env, VISION_PROBE_KEY:process.env.OPENROUTER_API_KEY}, stdio:['ignore','pipe','pipe'], windowsHide:true,+ });+ let stdout='', stderr='';+ child.stdout.on('data',data=>stdout+=data.toString());+ child.stderr.on('data',data=>{stderr+=data.toString();process.stderr.write(`[${run.model} ${run.track}] ${data}`);});+ const code = await new Promise((resolve,reject)=>{child.once('exit',resolve);child.once('error',reject);});+ const prefix = `${OUT}/${stage}-${run.track}-${modelTag(run.model)}`;+ await writeFile(prefix+'.stdout.log',stdout);+ await writeFile(prefix+'.stderr.log',stderr);+ return {model:run.model,track:run.track,dir:run.dir,exitCode:code,wallTimeMs:Date.now()-started};+}));+const campaign = {date:DATE,stage,startedAt,finishedAt:new Date().toISOString(),authentication:'validated before inference; credentials not saved',+ concurrency:{plannerPerModel:3,visionPerModel:2},reasoningOverride:null,retries:0,runs:recorded};+await writeFile(`${OUT}/campaign-${stage}.json`,JSON.stringify(campaign,null,2)+'\n');+assert(recorded.every(run=>run.exitCode===0),'At least one benchmark process failed; inspect saved logs and raw results.');+console.log(JSON.stringify(campaign,null,2));diff --git a/web/blog/posts/hy4-mimo-v26-lfm25-browser-agent-benchmark.md b/web/blog/posts/hy4-mimo-v26-lfm25-browser-agent-benchmark.mdnew file mode 100644index 000000000..501f80f38--- /dev/null+++ b/web/blog/posts/hy4-mimo-v26-lfm25-browser-agent-benchmark.md@@ -0,0 +1,87 @@+---+draft: true+title: HY4 Preview vs MiMo V2.6 Pro and Flash vs Liquid LFM2.5: a browser-agent benchmark+slug: hy4-mimo-v26-lfm25-browser-agent-benchmark+sortOrder: -350+date: 2026-10-14+readTime: 8 min read+description: A matching WebBrain benchmark of Tencent HY4 Preview, Xiaomi MiMo V2.6 Pro and Flash, and Liquid LFM2.5-24B-A2B, with pinned planner inputs and separate MiMo screenshot tests.+excerpt: Four new planners span a 24B local-agent model and trillion-parameter multimodal systems. The planned comparison uses 400 first-action cases and 200 screenshots, with dated historical references.+titleTag: HY4 Preview, MiMo V2.6, and Liquid LFM2.5 browser benchmark - WebBrain Blog+ogTitle: HY4 Preview vs MiMo V2.6 vs Liquid LFM2.5+ogDescription: Pinned first-action planning and production screenshot description, with native-call validity, measured latency, usage cost, and committed raw evidence.+twitterTitle: HY4 Preview vs MiMo V2.6 vs Liquid LFM2.5+twitterDescription: Four planners, two vision-capable routes, and matching historical references in WebBrain's browser-agent benchmark.+keywords:+ - Tencent HY4 Preview+ - MiMo V2.6 Pro+ - MiMo V2.6 Flash+ - Liquid LFM2.5-24B-A2B+ - OpenRouter+ - browser agent benchmark+ - tool calling+ - vision model+ - open weights+author: Emre Sokullu+authorUrl: https://emresokullu.com+---++**Draft: the live benchmark is pending a fresh OpenRouter credential.** This post is excluded from the published blog build. Model descriptions and methodology are prepared; there are no new measured results or winner claims yet.++## Four very different models++This comparison extends the [American and Chinese frontier planner benchmark](/blog/american-chinese-open-model-frontier-gap-benchmark) and the [Qwen3.8 Flash vs GLM-5.3 Flash comparison](/blog/qwen38-flash-vs-glm53-flash). All four requested routes support native tools. Two can also read screenshots.++| Model | Disclosed total / active parameters | Inputs on the requested OpenRouter route | Advertised context |+| --- | --- | --- | ---: |+| Tencent HY4 Preview | 770B / 49B backbone | Text | 1,048,576 |+| MiMo-V2.6-Pro | 1.02T / 42B | Text, image, video, audio | Up to 1,050,000 |+| MiMo-V2.6-Flash | 309B / 15B | Text, image, video, audio | Up to 1,050,000 |+| Liquid LFM2.5-24B-A2B | 24B / about 2B | Text | 128,000 on this route |++[Tencent's HY4 card](https://huggingface.co/tencent/Hy4-preview) describes a 78-layer MoE with Gated DeepSeek Sparse Attention, cross-layer index reuse, and identity Hyper-Connections. Its backbone figures exclude the separate 10B multi-token-prediction layer. Tencent also flags excess reasoning and over-verification as known preview limitations. Our single-response test can measure the opening dispatch, but not sustained productivity across a long task.++The [MiMo Pro](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL) and [MiMo Flash](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL) releases have native visual and audio components alongside their sparse language backbones. Their model cards describe reinforcement learning across multiple agent domains and harnesses. Those training claims motivate this test; they are not WebBrain results.++[Liquid's October 13 release](https://www.liquid.ai/blog/lfm2-5-24b-a2b) is a smaller open-weight model intended for local agents. It combines gated short convolutions with grouped-query attention and a sparse expert network. The release is reasoning-only, unlike its predecessor. Liquid advertises 131,072 context for the checkpoint, while the captured OpenRouter endpoint lists 128,000. The hosted benchmark does not measure local-device speed or memory requirements.++Active parameter counts describe work per token; they do not describe the size of the checkpoint to download or load. The full model sizes remain very different.++## Matching the planner inputs++The planner comparison uses the August full-tier source checkout **`7182c21f`**, reconstructed into our existing replay snapshot. Every model receives the same 100 Chrome Act cases, **48 native tool definitions**, temperature **0.15**, and **4,096-token** output budget. We apply no reasoning-effort override, system-prompt folding, or compatibility tool syntax. Only the model ID changes.++The report first reproduces the original thirteen-model consensus and the schema-valid and ideal-action counts from all 1,300 historical case files. It then adds the four complete new models and rescores every model against its other sixteen peers: **1,600 comparisons per model**. The old thirteen-model percentages are not copied into that expanded ranking.++Exact-action consensus compares the entire first tool and arguments after recursively sorting object keys. Tool-name consensus ignores the arguments. Native calls, content fallback, malformed argument JSON, schema-valid calls, canonical ideal tool names, and exact ideal actions are recorded separately. No-tool responses agree with other no-tool responses; rejected API requests are availability evidence and never model answers.++Only the four new routes receive new planner inference. August peer results retain their historical serving conditions and bills. Historical requests were omitted, so source reconstruction establishes matching intended inputs without proving full historical request-byte equality. No browser tool is executed: this is an opening-action test, not browser-task success.++## Separate screenshot tests for MiMo++Both MiMo routes receive the same 100 actual 1280Γ720 screenshots used in the production vision suite. HY4 and Liquid's text-only routes remain unscored for vision. The comparison references are the saved August 27 Qwen3.8 Flash and GLM-5.3 Flash runs.++The vision prompt asks for page purpose, exact visible text, input states, state signals, blockers, and unknowns. Requests use temperature **0**, **800-token** maximum, the original thinking-disabling chat-template kwargs, and fixed production user text without benchmark focus questions. Provider auto-routing is retained.++The unchanged deterministic grader combines weighted substring checks with a section-structure score, the case threshold, and critical-check gating. Six complete numbered sections are tracked separately rather than being a separate hard pass condition. Literal matching may reject a recognized synonym, and numbered analysis can resemble sections. Reported reasoning tokens, clipping, empty outputs, and raw streams are therefore part of the evidence.++The offline audit already reproduces all 200 historical Qwen/GLM vision grades and verifies prompt, image, question, rubric, and scorer parity. New screenshot scores, latency, and bills will be inserted only after actual inference.++## Results pending++The completed post will include the planner comparison, MiMo screenshot comparison and difficulty bands, serving coverage, provider/cache/token context, and conclusions supported by the saved responses. No scores, costs, or ranking are estimated from the model cards.++## Evidence and reproduction++Preparation, captured public endpoint metadata, archived ideal actions, and historical regressions are under `test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/`. The configuration records all planned raw result directories. Each preflight has a separate tag; it will not enter a 100-case quality row.++After setting `OPENROUTER_API_KEY` in the process environment:++```powershell+node test/llm/run-hy4-mimo26-lfm25.mjs --preflight+node test/llm/run-hy4-mimo26-lfm25.mjs+node test/llm/report-hy4-mimo26-lfm25.mjs+node scripts/build-blog.mjs+```++The runner refuses to overwrite existing evidence. Raw requests/responses, failed attempts, preflights, usage, timing, and analysis accompany the completed post. API credentials and authorization headers are not stored.