Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Add Ling and Nex historical planner panel and prepare matched vision replay

Barack Sokullu25db6b3d587e
diff --git a/test/llm/analysis/2026-10-07-ling31-historical-panel/README.md b/test/llm/analysis/2026-10-07-ling31-historical-panel/README.md
new file mode 100644
index 000000000..65114602e
--- /dev/null
+++ b/test/llm/analysis/2026-10-07-ling31-historical-panel/README.md
@@ -0,0 +1,7 @@
+# October 7 historical panel update
+
+The Ling 3.1 post now includes the four models from the Ling 3.0/Nex article in a separate May-frozen table. These rows do not change the August fourteen-model pool or its scores. The two MiniMax rows refer to different runs.
+
+`comparison.json` extracts the displayed metrics and records the SHA-256 of the original October 5 report. Original requests and responses remain in the linked source result directories. No new planner inference was performed for this update.
+
+The rebuilt page was checked in Playwright at desktop and 390px mobile widths. All four historical rows matched their source report, alongside the original ranking, pairwise, and availability tables. Screenshots and `blog-verification.json` record the updated page. No browser errors or page-level horizontal overflow occurred.
diff --git a/test/llm/analysis/2026-10-07-ling31-historical-panel/blog-comparison-table.png b/test/llm/analysis/2026-10-07-ling31-historical-panel/blog-comparison-table.png
new file mode 100644
index 000000000..eeef8dfcf
Binary files /dev/null and b/test/llm/analysis/2026-10-07-ling31-historical-panel/blog-comparison-table.png differ
diff --git a/test/llm/analysis/2026-10-07-ling31-historical-panel/blog-desktop.png b/test/llm/analysis/2026-10-07-ling31-historical-panel/blog-desktop.png
new file mode 100644
index 000000000..d449b9e6d
Binary files /dev/null and b/test/llm/analysis/2026-10-07-ling31-historical-panel/blog-desktop.png differ
diff --git a/test/llm/analysis/2026-10-07-ling31-historical-panel/blog-mobile.png b/test/llm/analysis/2026-10-07-ling31-historical-panel/blog-mobile.png
new file mode 100644
index 000000000..2a578dc7c
Binary files /dev/null and b/test/llm/analysis/2026-10-07-ling31-historical-panel/blog-mobile.png differ
diff --git a/test/llm/analysis/2026-10-07-ling31-historical-panel/blog-verification.json b/test/llm/analysis/2026-10-07-ling31-historical-panel/blog-verification.json
new file mode 100644
index 000000000..b62858584
--- /dev/null
+++ b/test/llm/analysis/2026-10-07-ling31-historical-panel/blog-verification.json
@@ -0,0 +1,46 @@
+{
+ "date": "2026-10-07",
+ "tableRows": [
+ 6,
+ 5,
+ 2,
+ 4
+ ],
+ "comparisonMatchesReport": true,
+ "pairwiseMatchesReport": true,
+ "availabilityMatchesReport": true,
+ "tokenCountsMatchReport": true,
+ "canonical": "https://webbrain.one/blog/ling31-flash-vs-deepseek-hy3-glm52-laguna-xs",
+ "jsonLdParsed": 1,
+ "internalLinksChecked": 4,
+ "mobile": {
+ "viewport": 390,
+ "page": 390,
+ "body": 390,
+ "tables": [
+ {
+ "width": 350,
+ "content": 686,
+ "overflow": "auto"
+ },
+ {
+ "width": 350,
+ "content": 560,
+ "overflow": "auto"
+ },
+ {
+ "width": 350,
+ "content": 560,
+ "overflow": "auto"
+ },
+ {
+ "width": 350,
+ "content": 560,
+ "overflow": "auto"
+ }
+ ]
+ },
+ "indexCardCount": 1,
+ "indexSearchPassed": true,
+ "pageErrors": []
+}
diff --git a/test/llm/analysis/2026-10-07-ling31-historical-panel/comparison.json b/test/llm/analysis/2026-10-07-ling31-historical-panel/comparison.json
new file mode 100644
index 000000000..0ab172ad3
--- /dev/null
+++ b/test/llm/analysis/2026-10-07-ling31-historical-panel/comparison.json
@@ -0,0 +1,86 @@
+{
+ "date": "2026-10-07",
+ "source": "test/llm/analysis/2026-10-05-ling30-flash-vl/comparison.json",
+ "sha256": "52d5d274ee15bb3d339f56ef55cde2059301cca7359e5a24db97a67aa05435c2",
+ "notes": [
+ "First response only, text inputs only, no browser tool execution. These scores are not task completion or vision quality.",
+ "All rows use the May 23 frozen system prompt and 41 tools, plus May 23 ideal goldens archived in the prior Nex report.",
+ "Exact ideal requires deep-equal arguments; ideal name ignores arguments. Neither credits terminal prose. Sonnet alignment measures tool-name agreement including no-tool agreement.",
+ "Historical MiniMax requests were not saved, so per-case message byte equality is unverified. New requests are checked against the payload builder; historical Nex requests retain the frozen system and tools.",
+ "Latency and usage.cost are measured on different dates and provider routes. Provider auto-routing is retained; advertised promotional prices do not guarantee every request uses that provider.",
+ "Nex mini remains unscored: its saved 100-case run and authenticated October 5 route recheck reject structured tools before inference.",
+ "Expired-key attempts were quarantined before the authenticated run and are excluded from these metrics."
+ ],
+ "rows": [
+ {
+ "label": "Ling 3.0 Flash VL",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "date": "2026-10-05",
+ "source": "test/llm/results/2026-10-05-openrouter-ling30-flash-vl_chrome_inclusionai_ling-3.0-flash-vl_frozen",
+ "cases": 100,
+ "errors": 0,
+ "responses": 100,
+ "nativeCalls": 90,
+ "exactIdeal": 22,
+ "idealToolName": 35,
+ "sonnetMatches": 72,
+ "medianLatencyMs": 1732.5,
+ "reportedCostUsd": 0.009769365200000003,
+ "malformedNativeArgumentJson": [
+ {
+ "id": "075",
+ "name": "scratchpad_write",
+ "arguments": "{\"replace\": , \"text\": \"Checking scratchpad for prior context\"}"
+ }
+ ]
+ },
+ {
+ "label": "Nex-N2.5-Pro",
+ "model": "nex-agi/nex-n2.5-pro",
+ "date": "2026-10-03",
+ "source": "test/llm/results/2026-10-03-openrouter-nex-n25-pro_chrome_nex-agi_nex-n2.5-pro_frozen",
+ "cases": 100,
+ "errors": 0,
+ "responses": 100,
+ "nativeCalls": 95,
+ "exactIdeal": 11,
+ "idealToolName": 27,
+ "sonnetMatches": 67,
+ "medianLatencyMs": 7225.5,
+ "reportedCostUsd": 0.1499900600000001,
+ "malformedNativeArgumentJson": null
+ },
+ {
+ "label": "Nex-N2.5-mini",
+ "model": "nex-agi/nex-n2.5-mini",
+ "date": "2026-10-03",
+ "source": "test/llm/results/2026-10-03-openrouter-nex-n25-mini_chrome_nex-agi_nex-n2.5-mini_frozen",
+ "cases": 100,
+ "errors": 100,
+ "responses": 0,
+ "nativeCalls": 0,
+ "exactIdeal": null,
+ "idealToolName": null,
+ "sonnetMatches": 0,
+ "medianLatencyMs": null,
+ "reportedCostUsd": null,
+ "malformedNativeArgumentJson": null
+ },
+ {
+ "label": "MiniMax M3",
+ "model": "minimax/minimax-m3",
+ "date": "2026-06-21",
+ "source": "test/llm/results/2026-06-21-openrouter-minimax-m3_chrome_minimax_minimax-m3_frozen",
+ "cases": 100,
+ "errors": 0,
+ "responses": 100,
+ "nativeCalls": 85,
+ "exactIdeal": 17,
+ "idealToolName": 32,
+ "sonnetMatches": 75,
+ "medianLatencyMs": 3101.5,
+ "reportedCostUsd": 1.0562093999999997,
+ "malformedNativeArgumentJson": null
+ }
+ ]
+}
diff --git a/test/vision/analysis/2026-10-07-ling30-budget/README.md b/test/vision/analysis/2026-10-07-ling30-budget/README.md
new file mode 100644
index 000000000..e77cd3765
--- /dev/null
+++ b/test/vision/analysis/2026-10-07-ling30-budget/README.md
@@ -0,0 +1,23 @@
+# Ling 3.0 Flash VL versus budget vision models
+
+Preparation date: October 7, 2026. **The live Ling vision run is pending a replacement API credential.** The previous credential returned HTTP 401, `API key expired`, from the key-validation endpoint before any vision inference. This directory currently contains six historical Qwen rows only; no Ling vision score is claimed.
+
+`report-ling30-budget.mjs --historical-only` reproduces all 600 August 22 responses using the unchanged production prompt and grader. It verifies the byte hashes of the prompt, scorer, 100 questions, and 100 rubrics against revision `d25be94f4981c2fe5ff1d220f8b698ee72e88296`, compares all screenshot hashes against all six historical runs, and exactly reproduces every saved case score. `manifest.json`, `rubrics.json`, and `regression.json` archive that audit.
+
+The six production-mode runs use identical 1280×720 image fixtures, separate system/user messages, temperature 0, max_tokens 800, streaming usage, and the original thinking-disabling chat-template kwargs. No focus question, reasoning override, tools, or provider pin is added. Scoring uses final text only: weighted literal/semantic checks plus a proportional six-section structure component, subject to critical-check failures and each case's threshold. Six complete sections are recorded separately and are not an independent hard pass gate in the original grader.
+
+Costs are summed from all 100 numeric `response.usage.cost` records for each model, not estimated from advertised prices. In particular the saved Qwen3.5-35B-A3B bill is $0.07214; the older article's $0.099 is not reproduced by these raw records. The six saved runs total $0.506781112. Historical cost and latency reflect August serving conditions and must be dated separately from a new Ling run.
+
+`openrouter-endpoints.json` captures the public October 7 endpoint metadata. Ling accepts images. The Novita rates are promotional and may change. Credentials and authorization headers are never saved.
+
+After supplying `VISION_PROBE_KEY` in the process environment:
+
+```powershell
+node test/vision/run.mjs --base https://openrouter.ai/api/v1 --model inclusionai/ling-3.0-flash-vl --only 1 --tag 2026-10-07-openrouter-ling30-budget-preflight
+node test/vision/run.mjs --base https://openrouter.ai/api/v1 --model inclusionai/ling-3.0-flash-vl --concurrency 2 --tag 2026-10-07-openrouter-ling30-budget
+node test/vision/report-ling30-budget.mjs
+```
+
+Keep the preflight and any failed full passes as separate archived results. Do not overwrite failures during recovery. The runner now preserves raw SSE and records provider/finish reason for new successful requests, and treats an error embedded in an HTTP 200 stream as an API failure. Historical runner fields remain unchanged. Original `reasoningChars` do not measure OpenRouter's `reasoning` stream field; token usage and new raw SSE retain the relevant evidence.
+
+The new blog is to be published only after the real screenshot run and comparison are complete. Its intended title is “Ling 3.0 Flash VL compared with budget vision models.” This is a visual screenshot-description benchmark, distinct from the previously published text-only tool-selection tests.
diff --git a/test/vision/analysis/2026-10-07-ling30-budget/comparison.json b/test/vision/analysis/2026-10-07-ling30-budget/comparison.json
new file mode 100644
index 000000000..3e51af8a5
--- /dev/null
+++ b/test/vision/analysis/2026-10-07-ling30-budget/comparison.json
@@ -0,0 +1,6252 @@
+{
+ "date": "2026-10-07",
+ "protocol": "production screenshot description, temperature 0, max_tokens 800, unchanged August scorer",
+ "regression": {
+ "verifiedHistoricalModels": 6,
+ "verifiedHistoricalResponses": 600,
+ "sourceRevision": "d25be94f4981c2fe5ff1d220f8b698ee72e88296",
+ "promptImageAndQuestionMatches": true,
+ "savedScoresExactlyReproduced": true
+ },
+ "rows": [
+ {
+ "model": "qwen/qwen3-vl-30b-a3b-instruct",
+ "source": "test/vision/results/2026-08-22-openrouter-full_qwen_qwen3-vl-30b-a3b-instruct_production",
+ "date": "2026-08-22",
+ "cases": 100,
+ "errors": 0,
+ "passes": 68,
+ "meanScore": 0.9129671439671436,
+ "meanLatencyMs": 2225.78,
+ "medianLatencyMs": 2193,
+ "p95LatencyMs": 2866,
+ "costRecords": 100,
+ "costUsd": 0.024314160000000005,
+ "promptTokens": 118600,
+ "completionTokens": 17108,
+ "cachedTokens": 0,
+ "reasoningTokens": 0,
+ "sixSections": 100,
+ "emptyOutputs": 0,
+ "lengthLimited": 0,
+ "providers": {
+ "not recorded": {
+ "cases": 100,
+ "passes": 68,
+ "meanScore": 0.9129671439671436
+ }
+ },
+ "byDifficulty": {
+ "easy": {
+ "cases": 20,
+ "passes": 14,
+ "meanScore": 0.9141161616161618
+ },
+ "basic": {
+ "cases": 20,
+ "passes": 16,
+ "meanScore": 0.9477272727272729
+ },
+ "intermediate": {
+ "cases": 20,
+ "passes": 14,
+ "meanScore": 0.9231060606060606
+ },
+ "advanced": {
+ "cases": 20,
+ "passes": 12,
+ "meanScore": 0.8767840492840493
+ },
+ "challenging": {
+ "cases": 20,
+ "passes": 12,
+ "meanScore": 0.9031021756021756
+ }
+ },
+ "byCategory": {
+ "authentication": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.8295104895104896
+ },
+ "search-results": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.95
+ },
+ "checkout": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9384615384615385
+ },
+ "form-validation": {
+ "cases": 5,
+ "passes": 1,
+ "meanScore": 0.8400000000000001
+ },
+ "modal-overlay": {
+ "cases": 5,
+ "passes": 0,
+ "meanScore": 0.8181818181818181
+ },
+ "toast-notification": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "loading-state": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.8469696969696969
+ },
+ "consent-banner": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 0.96
+ },
+ "dashboard": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "chart-reading": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "data-table": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "email-compose": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.8866666666666665
+ },
+ "kanban-board": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9142857142857143
+ },
+ "calendar": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 0.9714285714285715
+ },
+ "map-and-travel": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "photo-understanding": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9555555555555555
+ },
+ "multilingual-ocr": {
+ "cases": 5,
+ "passes": 0,
+ "meanScore": 0.7191919191919192
+ },
+ "occlusion-and-contrast": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.8400000000000001
+ },
+ "security-challenge": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.909090909090909
+ },
+ "uncertainty-calibration": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.8800000000000001
+ }
+ }
+ },
+ {
+ "model": "qwen/qwen3-vl-30b-a3b-thinking",
+ "source": "test/vision/results/2026-08-22-openrouter-full_qwen_qwen3-vl-30b-a3b-thinking_production",
+ "date": "2026-08-22",
+ "cases": 100,
+ "errors": 0,
+ "passes": 62,
+ "meanScore": 0.8891347818847817,
+ "meanLatencyMs": 6333.01,
+ "medianLatencyMs": 5620.5,
+ "p95LatencyMs": 13548,
+ "costRecords": 100,
+ "costUsd": 0.18816959999999996,
+ "promptTokens": 118800,
+ "completionTokens": 68504,
+ "cachedTokens": 0,
+ "reasoningTokens": 54984,
+ "sixSections": 100,
+ "emptyOutputs": 0,
+ "lengthLimited": 0,
+ "providers": {
+ "not recorded": {
+ "cases": 100,
+ "passes": 62,
+ "meanScore": 0.8891347818847817
+ }
+ },
+ "byDifficulty": {
+ "easy": {
+ "cases": 20,
+ "passes": 13,
+ "meanScore": 0.8957070707070708
+ },
+ "basic": {
+ "cases": 20,
+ "passes": 16,
+ "meanScore": 0.9441258741258742
+ },
+ "intermediate": {
+ "cases": 20,
+ "passes": 13,
+ "meanScore": 0.9062229437229437
+ },
+ "advanced": {
+ "cases": 20,
+ "passes": 10,
+ "meanScore": 0.8051871739371739
+ },
+ "challenging": {
+ "cases": 20,
+ "passes": 10,
+ "meanScore": 0.8944308469308468
+ }
+ },
+ "byCategory": {
+ "authentication": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.814125874125874
+ },
+ "search-results": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.95
+ },
+ "checkout": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.9384615384615385
+ },
+ "form-validation": {
+ "cases": 5,
+ "passes": 0,
+ "meanScore": 0.78
+ },
+ "modal-overlay": {
+ "cases": 5,
+ "passes": 1,
+ "meanScore": 0.8545454545454545
+ },
+ "toast-notification": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "loading-state": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.8429292929292929
+ },
+ "consent-banner": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 0.8800000000000001
+ },
+ "dashboard": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.96
+ },
+ "chart-reading": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.95
+ },
+ "data-table": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 0.9714285714285715
+ },
+ "email-compose": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.8166666666666667
+ },
+ "kanban-board": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9142857142857143
+ },
+ "calendar": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.7999999999999999
+ },
+ "map-and-travel": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.925
+ },
+ "photo-understanding": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9333333333333333
+ },
+ "multilingual-ocr": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.8828282828282829
+ },
+ "occlusion-and-contrast": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.8400000000000001
+ },
+ "security-challenge": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.9090909090909092
+ },
+ "uncertainty-calibration": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.8200000000000001
+ }
+ }
+ },
+ {
+ "model": "qwen/qwen3-vl-32b-instruct",
+ "source": "test/vision/results/2026-08-22-openrouter-full_qwen_qwen3-vl-32b-instruct_production",
+ "date": "2026-08-22",
+ "cases": 100,
+ "errors": 0,
+ "passes": 69,
+ "meanScore": 0.9192472804972802,
+ "meanLatencyMs": 4605,
+ "medianLatencyMs": 4240.5,
+ "p95LatencyMs": 7075,
+ "costRecords": 100,
+ "costUsd": 0.021568352000000002,
+ "promptTokens": 118600,
+ "completionTokens": 22197,
+ "cachedTokens": 0,
+ "reasoningTokens": 0,
+ "sixSections": 100,
+ "emptyOutputs": 0,
+ "lengthLimited": 0,
+ "providers": {
+ "not recorded": {
+ "cases": 100,
+ "passes": 69,
+ "meanScore": 0.9192472804972802
+ }
+ },
+ "byDifficulty": {
+ "easy": {
+ "cases": 20,
+ "passes": 14,
+ "meanScore": 0.9332070707070708
+ },
+ "basic": {
+ "cases": 20,
+ "passes": 15,
+ "meanScore": 0.935034965034965
+ },
+ "intermediate": {
+ "cases": 20,
+ "passes": 16,
+ "meanScore": 0.9481060606060605
+ },
+ "advanced": {
+ "cases": 20,
+ "passes": 13,
+ "meanScore": 0.8841792929292931
+ },
+ "challenging": {
+ "cases": 20,
+ "passes": 11,
+ "meanScore": 0.8957090132090133
+ }
+ },
+ "byCategory": {
+ "authentication": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.9282517482517483
+ },
+ "search-results": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.95
+ },
+ "checkout": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.9384615384615385
+ },
+ "form-validation": {
+ "cases": 5,
+ "passes": 0,
+ "meanScore": 0.74
+ },
+ "modal-overlay": {
+ "cases": 5,
+ "passes": 0,
+ "meanScore": 0.8181818181818181
+ },
+ "toast-notification": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "loading-state": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.7883838383838384
+ },
+ "consent-banner": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 0.9399999999999998
+ },
+ "dashboard": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.9099999999999999
+ },
+ "chart-reading": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "data-table": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "email-compose": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.96
+ },
+ "kanban-board": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "calendar": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9555555555555555
+ },
+ "map-and-travel": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.885
+ },
+ "photo-understanding": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9555555555555555
+ },
+ "multilingual-ocr": {
+ "cases": 5,
+ "passes": 1,
+ "meanScore": 0.8101010101010102
+ },
+ "occlusion-and-contrast": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 0.9800000000000001
+ },
+ "security-challenge": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9454545454545455
+ },
+ "uncertainty-calibration": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.8800000000000001
+ }
+ }
+ },
+ {
+ "model": "qwen/qwen3-vl-8b-instruct",
+ "source": "test/vision/results/2026-08-22-openrouter-full_qwen_qwen3-vl-8b-instruct_production",
+ "date": "2026-08-22",
+ "cases": 100,
+ "errors": 0,
+ "passes": 48,
+ "meanScore": 0.839557997557997,
+ "meanLatencyMs": 2118.3,
+ "medianLatencyMs": 1990.5,
+ "p95LatencyMs": 3215,
+ "costRecords": 100,
+ "costUsd": 0.020646599999999998,
+ "promptTokens": 118600,
+ "completionTokens": 14880,
+ "cachedTokens": 0,
+ "reasoningTokens": 0,
+ "sixSections": 100,
+ "emptyOutputs": 0,
+ "lengthLimited": 0,
+ "providers": {
+ "not recorded": {
+ "cases": 100,
+ "passes": 48,
+ "meanScore": 0.839557997557997
+ }
+ },
+ "byDifficulty": {
+ "easy": {
+ "cases": 20,
+ "passes": 12,
+ "meanScore": 0.8539375901875902
+ },
+ "basic": {
+ "cases": 20,
+ "passes": 13,
+ "meanScore": 0.8845154845154847
+ },
+ "intermediate": {
+ "cases": 20,
+ "passes": 10,
+ "meanScore": 0.858595571095571
+ },
+ "advanced": {
+ "cases": 20,
+ "passes": 7,
+ "meanScore": 0.784263098013098
+ },
+ "challenging": {
+ "cases": 20,
+ "passes": 6,
+ "meanScore": 0.8164782439782442
+ }
+ },
+ "byCategory": {
+ "authentication": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.8741258741258742
+ },
+ "search-results": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.95
+ },
+ "checkout": {
+ "cases": 5,
+ "passes": 1,
+ "meanScore": 0.8405594405594407
+ },
+ "form-validation": {
+ "cases": 5,
+ "passes": 0,
+ "meanScore": 0.6
+ },
+ "modal-overlay": {
+ "cases": 5,
+ "passes": 0,
+ "meanScore": 0.8181818181818181
+ },
+ "toast-notification": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "loading-state": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.6684343434343434
+ },
+ "consent-banner": {
+ "cases": 5,
+ "passes": 0,
+ "meanScore": 0.68
+ },
+ "dashboard": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.9099999999999999
+ },
+ "chart-reading": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "data-table": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "email-compose": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.8766666666666666
+ },
+ "kanban-board": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9142857142857143
+ },
+ "calendar": {
+ "cases": 5,
+ "passes": 1,
+ "meanScore": 0.6317460317460318
+ },
+ "map-and-travel": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.885
+ },
+ "photo-understanding": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9555555555555555
+ },
+ "multilingual-ocr": {
+ "cases": 5,
+ "passes": 0,
+ "meanScore": 0.7373737373737373
+ },
+ "occlusion-and-contrast": {
+ "cases": 5,
+ "passes": 1,
+ "meanScore": 0.8
+ },
+ "security-challenge": {
+ "cases": 5,
+ "passes": 1,
+ "meanScore": 0.7692307692307693
+ },
+ "uncertainty-calibration": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.8800000000000001
+ }
+ }
+ },
+ {
+ "model": "qwen/qwen3-vl-8b-thinking",
+ "source": "test/vision/results/2026-08-22-openrouter-full_qwen_qwen3-vl-8b-thinking_production",
+ "date": "2026-08-22",
+ "cases": 100,
+ "errors": 0,
+ "passes": 61,
+ "meanScore": 0.8670892163392158,
+ "meanLatencyMs": 6368.96,
+ "medianLatencyMs": 5292.5,
+ "p95LatencyMs": 12937,
+ "costRecords": 100,
+ "costUsd": 0.17994239999999997,
+ "promptTokens": 118800,
+ "completionTokens": 75504,
+ "cachedTokens": 0,
+ "reasoningTokens": 62237,
+ "sixSections": 99,
+ "emptyOutputs": 0,
+ "lengthLimited": 0,
+ "providers": {
+ "not recorded": {
+ "cases": 100,
+ "passes": 61,
+ "meanScore": 0.8670892163392158
+ }
+ },
+ "byDifficulty": {
+ "easy": {
+ "cases": 20,
+ "passes": 13,
+ "meanScore": 0.8190512265512264
+ },
+ "basic": {
+ "cases": 20,
+ "passes": 14,
+ "meanScore": 0.9213986013986014
+ },
+ "intermediate": {
+ "cases": 20,
+ "passes": 13,
+ "meanScore": 0.8771320346320346
+ },
+ "advanced": {
+ "cases": 20,
+ "passes": 11,
+ "meanScore": 0.8320697358197359
+ },
+ "challenging": {
+ "cases": 20,
+ "passes": 10,
+ "meanScore": 0.8857944832944835
+ }
+ },
+ "byCategory": {
+ "authentication": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.8205594405594405
+ },
+ "search-results": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9666666666666666
+ },
+ "checkout": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.9020979020979022
+ },
+ "form-validation": {
+ "cases": 5,
+ "passes": 0,
+ "meanScore": 0.8
+ },
+ "modal-overlay": {
+ "cases": 5,
+ "passes": 1,
+ "meanScore": 0.8181818181818181
+ },
+ "toast-notification": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "loading-state": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.6202020202020202
+ },
+ "consent-banner": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 0.8200000000000001
+ },
+ "dashboard": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.9099999999999999
+ },
+ "chart-reading": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "data-table": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "email-compose": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.8766666666666666
+ },
+ "kanban-board": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9142857142857143
+ },
+ "calendar": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.7142857142857142
+ },
+ "map-and-travel": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9349999999999999
+ },
+ "photo-understanding": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9555555555555555
+ },
+ "multilingual-ocr": {
+ "cases": 5,
+ "passes": 0,
+ "meanScore": 0.7555555555555556
+ },
+ "occlusion-and-contrast": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.8400000000000001
+ },
+ "security-challenge": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.8727272727272727
+ },
+ "uncertainty-calibration": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.82
+ }
+ }
+ },
+ {
+ "model": "qwen/qwen3.5-35b-a3b",
+ "source": "test/vision/results/2026-08-22-openrouter-full_qwen_qwen3.5-35b-a3b_production",
+ "date": "2026-08-22",
+ "cases": 100,
+ "errors": 0,
+ "passes": 67,
+ "meanScore": 0.9075665630665627,
+ "meanLatencyMs": 5212.18,
+ "medianLatencyMs": 5119.5,
+ "p95LatencyMs": 6964,
+ "costRecords": 100,
+ "costUsd": 0.07214,
+ "promptTokens": 119700,
+ "completionTokens": 55382,
+ "cachedTokens": 0,
+ "reasoningTokens": 36857,
+ "sixSections": 93,
+ "emptyOutputs": 0,
+ "lengthLimited": 0,
+ "providers": {
+ "not recorded": {
+ "cases": 100,
+ "passes": 67,
+ "meanScore": 0.9075665630665627
+ }
+ },
+ "byDifficulty": {
+ "easy": {
+ "cases": 20,
+ "passes": 16,
+ "meanScore": 0.9263744588744588
+ },
+ "basic": {
+ "cases": 20,
+ "passes": 14,
+ "meanScore": 0.9099483849483849
+ },
+ "intermediate": {
+ "cases": 20,
+ "passes": 12,
+ "meanScore": 0.8942521367521369
+ },
+ "advanced": {
+ "cases": 20,
+ "passes": 14,
+ "meanScore": 0.9121484071484071
+ },
+ "challenging": {
+ "cases": 20,
+ "passes": 11,
+ "meanScore": 0.8951094276094276
+ }
+ },
+ "byCategory": {
+ "authentication": {
+ "cases": 5,
+ "passes": 1,
+ "meanScore": 0.8241491841491841
+ },
+ "search-results": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.9111111111111111
+ },
+ "checkout": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.9076923076923077
+ },
+ "form-validation": {
+ "cases": 5,
+ "passes": 1,
+ "meanScore": 0.8
+ },
+ "modal-overlay": {
+ "cases": 5,
+ "passes": 0,
+ "meanScore": 0.8181818181818181
+ },
+ "toast-notification": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "loading-state": {
+ "cases": 5,
+ "passes": 3,
+ "meanScore": 0.7156565656565658
+ },
+ "consent-banner": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 0.96
+ },
+ "dashboard": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "chart-reading": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "data-table": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 0.9714285714285715
+ },
+ "email-compose": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.8799999999999999
+ },
+ "kanban-board": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 1
+ },
+ "calendar": {
+ "cases": 5,
+ "passes": 5,
+ "meanScore": 0.9470899470899472
+ },
+ "map-and-travel": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.96
+ },
+ "photo-understanding": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9555555555555555
+ },
+ "multilingual-ocr": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.8363636363636363
+ },
+ "occlusion-and-contrast": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.96
+ },
+ "security-challenge": {
+ "cases": 5,
+ "passes": 2,
+ "meanScore": 0.764102564102564
+ },
+ "uncertainty-calibration": {
+ "cases": 5,
+ "passes": 4,
+ "meanScore": 0.9400000000000001
+ }
+ }
+ }
+ ],
+ "casesByModel": {
+ "qwen/qwen3-vl-30b-a3b-instruct": [
+ {
+ "id": "001",
+ "category": "authentication",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": []
+ },
+ {
+ "id": "002",
+ "category": "search-results",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "003",
+ "category": "checkout",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "004",
+ "category": "form-validation",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "005",
+ "category": "modal-overlay",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "006",
+ "category": "toast-notification",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "007",
+ "category": "loading-state",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.75,
+ "criticalFailures": [
+ "heading"
+ ]
+ },
+ {
+ "id": "008",
+ "category": "consent-banner",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "009",
+ "category": "dashboard",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "010",
+ "category": "chart-reading",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "011",
+ "category": "data-table",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "012",
+ "category": "email-compose",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "013",
+ "category": "kanban-board",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "014",
+ "category": "calendar",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "015",
+ "category": "map-and-travel",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "016",
+ "category": "photo-understanding",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "017",
+ "category": "multilingual-ocr",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.7777777777777778,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "018",
+ "category": "occlusion-and-contrast",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.6,
+ "criticalFailures": [
+ "target",
+ "cover"
+ ]
+ },
+ {
+ "id": "019",
+ "category": "security-challenge",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.6363636363636364,
+ "criticalFailures": [
+ "label",
+ "captcha-signal"
+ ]
+ },
+ {
+ "id": "020",
+ "category": "uncertainty-calibration",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "021",
+ "category": "authentication",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "022",
+ "category": "search-results",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "023",
+ "category": "checkout",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "024",
+ "category": "form-validation",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "025",
+ "category": "modal-overlay",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "026",
+ "category": "toast-notification",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "027",
+ "category": "loading-state",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "028",
+ "category": "consent-banner",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "029",
+ "category": "dashboard",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "030",
+ "category": "chart-reading",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "031",
+ "category": "data-table",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "032",
+ "category": "email-compose",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "033",
+ "category": "kanban-board",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "034",
+ "category": "calendar",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "035",
+ "category": "map-and-travel",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "036",
+ "category": "photo-understanding",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "037",
+ "category": "multilingual-ocr",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "038",
+ "category": "occlusion-and-contrast",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "cover"
+ ]
+ },
+ {
+ "id": "039",
+ "category": "security-challenge",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "040",
+ "category": "uncertainty-calibration",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "041",
+ "category": "authentication",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "042",
+ "category": "search-results",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": []
+ },
+ {
+ "id": "043",
+ "category": "checkout",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "044",
+ "category": "form-validation",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "045",
+ "category": "modal-overlay",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "046",
+ "category": "toast-notification",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "047",
+ "category": "loading-state",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "048",
+ "category": "consent-banner",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "049",
+ "category": "dashboard",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "050",
+ "category": "chart-reading",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "051",
+ "category": "data-table",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "052",
+ "category": "email-compose",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "status"
+ ]
+ },
+ {
+ "id": "053",
+ "category": "kanban-board",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "054",
+ "category": "calendar",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "055",
+ "category": "map-and-travel",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "056",
+ "category": "photo-understanding",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "057",
+ "category": "multilingual-ocr",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "058",
+ "category": "occlusion-and-contrast",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "cover"
+ ]
+ },
+ {
+ "id": "059",
+ "category": "security-challenge",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "060",
+ "category": "uncertainty-calibration",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "visible-fragment"
+ ]
+ },
+ {
+ "id": "061",
+ "category": "authentication",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.7692307692307693,
+ "criticalFailures": [
+ "blocker"
+ ]
+ },
+ {
+ "id": "062",
+ "category": "search-results",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": []
+ },
+ {
+ "id": "063",
+ "category": "checkout",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "064",
+ "category": "form-validation",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "065",
+ "category": "modal-overlay",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "066",
+ "category": "toast-notification",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "067",
+ "category": "loading-state",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.6666666666666666,
+ "criticalFailures": [
+ "heading"
+ ]
+ },
+ {
+ "id": "068",
+ "category": "consent-banner",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "069",
+ "category": "dashboard",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "070",
+ "category": "chart-reading",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "071",
+ "category": "data-table",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "072",
+ "category": "email-compose",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "status"
+ ]
+ },
+ {
+ "id": "073",
+ "category": "kanban-board",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.5714285714285714,
+ "criticalFailures": [
+ "card-column"
+ ]
+ },
+ {
+ "id": "074",
+ "category": "calendar",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 0.8571428571428571,
+ "criticalFailures": []
+ },
+ {
+ "id": "075",
+ "category": "map-and-travel",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "076",
+ "category": "photo-understanding",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "077",
+ "category": "multilingual-ocr",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.6363636363636364,
+ "criticalFailures": [
+ "input",
+ "status"
+ ]
+ },
+ {
+ "id": "078",
+ "category": "occlusion-and-contrast",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "079",
+ "category": "security-challenge",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "080",
+ "category": "uncertainty-calibration",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "visible-fragment"
+ ]
+ },
+ {
+ "id": "081",
+ "category": "authentication",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.7692307692307693,
+ "criticalFailures": [
+ "blocker"
+ ]
+ },
+ {
+ "id": "082",
+ "category": "search-results",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": [
+ "notice"
+ ]
+ },
+ {
+ "id": "083",
+ "category": "checkout",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.6923076923076923,
+ "criticalFailures": [
+ "banner",
+ "blocked"
+ ]
+ },
+ {
+ "id": "084",
+ "category": "form-validation",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "085",
+ "category": "modal-overlay",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "086",
+ "category": "toast-notification",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "087",
+ "category": "loading-state",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "088",
+ "category": "consent-banner",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "089",
+ "category": "dashboard",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "090",
+ "category": "chart-reading",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "091",
+ "category": "data-table",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "092",
+ "category": "email-compose",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8333333333333334,
+ "criticalFailures": [
+ "external-blocker"
+ ]
+ },
+ {
+ "id": "093",
+ "category": "kanban-board",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "094",
+ "category": "calendar",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "095",
+ "category": "map-and-travel",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "096",
+ "category": "photo-understanding",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.7777777777777778,
+ "criticalFailures": [
+ "visual-content"
+ ]
+ },
+ {
+ "id": "097",
+ "category": "multilingual-ocr",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.5454545454545454,
+ "criticalFailures": [
+ "input",
+ "status"
+ ]
+ },
+ {
+ "id": "098",
+ "category": "occlusion-and-contrast",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "099",
+ "category": "security-challenge",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "100",
+ "category": "uncertainty-calibration",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ }
+ ],
+ "qwen/qwen3-vl-30b-a3b-thinking": [
+ {
+ "id": "001",
+ "category": "authentication",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": []
+ },
+ {
+ "id": "002",
+ "category": "search-results",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "003",
+ "category": "checkout",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "004",
+ "category": "form-validation",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "005",
+ "category": "modal-overlay",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "006",
+ "category": "toast-notification",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "007",
+ "category": "loading-state",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.75,
+ "criticalFailures": [
+ "heading"
+ ]
+ },
+ {
+ "id": "008",
+ "category": "consent-banner",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "009",
+ "category": "dashboard",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "010",
+ "category": "chart-reading",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "011",
+ "category": "data-table",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "012",
+ "category": "email-compose",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 0.75,
+ "criticalFailures": []
+ },
+ {
+ "id": "013",
+ "category": "kanban-board",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "014",
+ "category": "calendar",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "015",
+ "category": "map-and-travel",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "016",
+ "category": "photo-understanding",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "017",
+ "category": "multilingual-ocr",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.7777777777777778,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "018",
+ "category": "occlusion-and-contrast",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.6,
+ "criticalFailures": [
+ "target",
+ "cover"
+ ]
+ },
+ {
+ "id": "019",
+ "category": "security-challenge",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "label"
+ ]
+ },
+ {
+ "id": "020",
+ "category": "uncertainty-calibration",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "021",
+ "category": "authentication",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "022",
+ "category": "search-results",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "023",
+ "category": "checkout",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "banner"
+ ]
+ },
+ {
+ "id": "024",
+ "category": "form-validation",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "025",
+ "category": "modal-overlay",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "026",
+ "category": "toast-notification",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "027",
+ "category": "loading-state",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "028",
+ "category": "consent-banner",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.8,
+ "criticalFailures": []
+ },
+ {
+ "id": "029",
+ "category": "dashboard",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "030",
+ "category": "chart-reading",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "031",
+ "category": "data-table",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "032",
+ "category": "email-compose",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "033",
+ "category": "kanban-board",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "034",
+ "category": "calendar",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "035",
+ "category": "map-and-travel",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "036",
+ "category": "photo-understanding",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "037",
+ "category": "multilingual-ocr",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "038",
+ "category": "occlusion-and-contrast",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "target"
+ ]
+ },
+ {
+ "id": "039",
+ "category": "security-challenge",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "040",
+ "category": "uncertainty-calibration",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "041",
+ "category": "authentication",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "042",
+ "category": "search-results",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": []
+ },
+ {
+ "id": "043",
+ "category": "checkout",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "044",
+ "category": "form-validation",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "045",
+ "category": "modal-overlay",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "046",
+ "category": "toast-notification",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "047",
+ "category": "loading-state",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "048",
+ "category": "consent-banner",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "049",
+ "category": "dashboard",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "050",
+ "category": "chart-reading",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "051",
+ "category": "data-table",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "052",
+ "category": "email-compose",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "status"
+ ]
+ },
+ {
+ "id": "053",
+ "category": "kanban-board",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "054",
+ "category": "calendar",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.5714285714285714,
+ "criticalFailures": [
+ "selected-event"
+ ]
+ },
+ {
+ "id": "055",
+ "category": "map-and-travel",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "056",
+ "category": "photo-understanding",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "057",
+ "category": "multilingual-ocr",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "058",
+ "category": "occlusion-and-contrast",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "cover"
+ ]
+ },
+ {
+ "id": "059",
+ "category": "security-challenge",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "060",
+ "category": "uncertainty-calibration",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "visible-fragment"
+ ]
+ },
+ {
+ "id": "061",
+ "category": "authentication",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.6923076923076923,
+ "criticalFailures": [
+ "purpose",
+ "blocker"
+ ]
+ },
+ {
+ "id": "062",
+ "category": "search-results",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": []
+ },
+ {
+ "id": "063",
+ "category": "checkout",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "064",
+ "category": "form-validation",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "065",
+ "category": "modal-overlay",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "066",
+ "category": "toast-notification",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "067",
+ "category": "loading-state",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.5555555555555556,
+ "criticalFailures": [
+ "purpose",
+ "heading"
+ ]
+ },
+ {
+ "id": "068",
+ "category": "consent-banner",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "069",
+ "category": "dashboard",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "070",
+ "category": "chart-reading",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.75,
+ "criticalFailures": [
+ "highest-label"
+ ]
+ },
+ {
+ "id": "071",
+ "category": "data-table",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 0.8571428571428571,
+ "criticalFailures": []
+ },
+ {
+ "id": "072",
+ "category": "email-compose",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "status"
+ ]
+ },
+ {
+ "id": "073",
+ "category": "kanban-board",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.5714285714285714,
+ "criticalFailures": [
+ "card-column"
+ ]
+ },
+ {
+ "id": "074",
+ "category": "calendar",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.42857142857142855,
+ "criticalFailures": [
+ "selected-event"
+ ]
+ },
+ {
+ "id": "075",
+ "category": "map-and-travel",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.625,
+ "criticalFailures": [
+ "selection"
+ ]
+ },
+ {
+ "id": "076",
+ "category": "photo-understanding",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 0.8888888888888888,
+ "criticalFailures": []
+ },
+ {
+ "id": "077",
+ "category": "multilingual-ocr",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "078",
+ "category": "occlusion-and-contrast",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "079",
+ "category": "security-challenge",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "security-blocker"
+ ]
+ },
+ {
+ "id": "080",
+ "category": "uncertainty-calibration",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "visible-fragment"
+ ]
+ },
+ {
+ "id": "081",
+ "category": "authentication",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.7692307692307693,
+ "criticalFailures": [
+ "blocker"
+ ]
+ },
+ {
+ "id": "082",
+ "category": "search-results",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": [
+ "notice"
+ ]
+ },
+ {
+ "id": "083",
+ "category": "checkout",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "blocked"
+ ]
+ },
+ {
+ "id": "084",
+ "category": "form-validation",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "085",
+ "category": "modal-overlay",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "086",
+ "category": "toast-notification",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "087",
+ "category": "loading-state",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "088",
+ "category": "consent-banner",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "089",
+ "category": "dashboard",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "unknown"
+ ]
+ },
+ {
+ "id": "090",
+ "category": "chart-reading",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "091",
+ "category": "data-table",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "092",
+ "category": "email-compose",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8333333333333334,
+ "criticalFailures": [
+ "external-blocker"
+ ]
+ },
+ {
+ "id": "093",
+ "category": "kanban-board",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "094",
+ "category": "calendar",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "095",
+ "category": "map-and-travel",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "096",
+ "category": "photo-understanding",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.7777777777777778,
+ "criticalFailures": [
+ "visual-content"
+ ]
+ },
+ {
+ "id": "097",
+ "category": "multilingual-ocr",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "098",
+ "category": "occlusion-and-contrast",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "099",
+ "category": "security-challenge",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "100",
+ "category": "uncertainty-calibration",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "visible-fragment"
+ ]
+ }
+ ],
+ "qwen/qwen3-vl-32b-instruct": [
+ {
+ "id": "001",
+ "category": "authentication",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.9,
+ "criticalFailures": [
+ "purpose"
+ ]
+ },
+ {
+ "id": "002",
+ "category": "search-results",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "003",
+ "category": "checkout",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "004",
+ "category": "form-validation",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "error",
+ "blocker"
+ ]
+ },
+ {
+ "id": "005",
+ "category": "modal-overlay",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "006",
+ "category": "toast-notification",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "007",
+ "category": "loading-state",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.75,
+ "criticalFailures": [
+ "heading"
+ ]
+ },
+ {
+ "id": "008",
+ "category": "consent-banner",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "009",
+ "category": "dashboard",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "010",
+ "category": "chart-reading",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "011",
+ "category": "data-table",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "012",
+ "category": "email-compose",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "013",
+ "category": "kanban-board",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "014",
+ "category": "calendar",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "015",
+ "category": "map-and-travel",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "016",
+ "category": "photo-understanding",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "017",
+ "category": "multilingual-ocr",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.7777777777777778,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "018",
+ "category": "occlusion-and-contrast",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "019",
+ "category": "security-challenge",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "label"
+ ]
+ },
+ {
+ "id": "020",
+ "category": "uncertainty-calibration",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "021",
+ "category": "authentication",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "purpose"
+ ]
+ },
+ {
+ "id": "022",
+ "category": "search-results",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "023",
+ "category": "checkout",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "banner"
+ ]
+ },
+ {
+ "id": "024",
+ "category": "form-validation",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "025",
+ "category": "modal-overlay",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "026",
+ "category": "toast-notification",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "027",
+ "category": "loading-state",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "028",
+ "category": "consent-banner",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "029",
+ "category": "dashboard",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "030",
+ "category": "chart-reading",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "031",
+ "category": "data-table",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "032",
+ "category": "email-compose",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "033",
+ "category": "kanban-board",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "034",
+ "category": "calendar",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "035",
+ "category": "map-and-travel",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "036",
+ "category": "photo-understanding",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "037",
+ "category": "multilingual-ocr",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "038",
+ "category": "occlusion-and-contrast",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "039",
+ "category": "security-challenge",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "040",
+ "category": "uncertainty-calibration",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "visible-fragment"
+ ]
+ },
+ {
+ "id": "041",
+ "category": "authentication",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "042",
+ "category": "search-results",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": []
+ },
+ {
+ "id": "043",
+ "category": "checkout",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "044",
+ "category": "form-validation",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "045",
+ "category": "modal-overlay",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "046",
+ "category": "toast-notification",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "047",
+ "category": "loading-state",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "048",
+ "category": "consent-banner",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "049",
+ "category": "dashboard",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "050",
+ "category": "chart-reading",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "051",
+ "category": "data-table",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "052",
+ "category": "email-compose",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "status"
+ ]
+ },
+ {
+ "id": "053",
+ "category": "kanban-board",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "054",
+ "category": "calendar",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "055",
+ "category": "map-and-travel",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "056",
+ "category": "photo-understanding",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "057",
+ "category": "multilingual-ocr",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "058",
+ "category": "occlusion-and-contrast",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "059",
+ "category": "security-challenge",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "060",
+ "category": "uncertainty-calibration",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "061",
+ "category": "authentication",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "062",
+ "category": "search-results",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": []
+ },
+ {
+ "id": "063",
+ "category": "checkout",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "064",
+ "category": "form-validation",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.6,
+ "criticalFailures": [
+ "input",
+ "error"
+ ]
+ },
+ {
+ "id": "065",
+ "category": "modal-overlay",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "066",
+ "category": "toast-notification",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "067",
+ "category": "loading-state",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.5555555555555556,
+ "criticalFailures": [
+ "heading"
+ ]
+ },
+ {
+ "id": "068",
+ "category": "consent-banner",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "069",
+ "category": "dashboard",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.75,
+ "criticalFailures": [
+ "alert"
+ ]
+ },
+ {
+ "id": "070",
+ "category": "chart-reading",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "071",
+ "category": "data-table",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "072",
+ "category": "email-compose",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "073",
+ "category": "kanban-board",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "074",
+ "category": "calendar",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "075",
+ "category": "map-and-travel",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.625,
+ "criticalFailures": [
+ "selection"
+ ]
+ },
+ {
+ "id": "076",
+ "category": "photo-understanding",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "077",
+ "category": "multilingual-ocr",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "078",
+ "category": "occlusion-and-contrast",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "079",
+ "category": "security-challenge",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "080",
+ "category": "uncertainty-calibration",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "visible-fragment"
+ ]
+ },
+ {
+ "id": "081",
+ "category": "authentication",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 0.9230769230769231,
+ "criticalFailures": []
+ },
+ {
+ "id": "082",
+ "category": "search-results",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": [
+ "notice"
+ ]
+ },
+ {
+ "id": "083",
+ "category": "checkout",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "blocked"
+ ]
+ },
+ {
+ "id": "084",
+ "category": "form-validation",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "085",
+ "category": "modal-overlay",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "086",
+ "category": "toast-notification",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "087",
+ "category": "loading-state",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": []
+ },
+ {
+ "id": "088",
+ "category": "consent-banner",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "089",
+ "category": "dashboard",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "alert"
+ ]
+ },
+ {
+ "id": "090",
+ "category": "chart-reading",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "091",
+ "category": "data-table",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "092",
+ "category": "email-compose",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "093",
+ "category": "kanban-board",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "094",
+ "category": "calendar",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.7777777777777778,
+ "criticalFailures": [
+ "conflict"
+ ]
+ },
+ {
+ "id": "095",
+ "category": "map-and-travel",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "route-alert"
+ ]
+ },
+ {
+ "id": "096",
+ "category": "photo-understanding",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.7777777777777778,
+ "criticalFailures": [
+ "visual-content"
+ ]
+ },
+ {
+ "id": "097",
+ "category": "multilingual-ocr",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.6363636363636364,
+ "criticalFailures": [
+ "input",
+ "status"
+ ]
+ },
+ {
+ "id": "098",
+ "category": "occlusion-and-contrast",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "099",
+ "category": "security-challenge",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "100",
+ "category": "uncertainty-calibration",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ }
+ ],
+ "qwen/qwen3-vl-8b-instruct": [
+ {
+ "id": "001",
+ "category": "authentication",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "002",
+ "category": "search-results",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "003",
+ "category": "checkout",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "004",
+ "category": "form-validation",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.6,
+ "criticalFailures": [
+ "input",
+ "error"
+ ]
+ },
+ {
+ "id": "005",
+ "category": "modal-overlay",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "006",
+ "category": "toast-notification",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "007",
+ "category": "loading-state",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.625,
+ "criticalFailures": [
+ "heading"
+ ]
+ },
+ {
+ "id": "008",
+ "category": "consent-banner",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "consent-signal"
+ ]
+ },
+ {
+ "id": "009",
+ "category": "dashboard",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "010",
+ "category": "chart-reading",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "011",
+ "category": "data-table",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "012",
+ "category": "email-compose",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 0.75,
+ "criticalFailures": []
+ },
+ {
+ "id": "013",
+ "category": "kanban-board",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "014",
+ "category": "calendar",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.5714285714285714,
+ "criticalFailures": [
+ "selected-event"
+ ]
+ },
+ {
+ "id": "015",
+ "category": "map-and-travel",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "016",
+ "category": "photo-understanding",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "017",
+ "category": "multilingual-ocr",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.7777777777777778,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "018",
+ "category": "occlusion-and-contrast",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.6,
+ "criticalFailures": [
+ "target",
+ "cover"
+ ]
+ },
+ {
+ "id": "019",
+ "category": "security-challenge",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.6363636363636364,
+ "criticalFailures": [
+ "label",
+ "captcha-signal"
+ ]
+ },
+ {
+ "id": "020",
+ "category": "uncertainty-calibration",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "021",
+ "category": "authentication",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "022",
+ "category": "search-results",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "023",
+ "category": "checkout",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "banner"
+ ]
+ },
+ {
+ "id": "024",
+ "category": "form-validation",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.6,
+ "criticalFailures": [
+ "input",
+ "error"
+ ]
+ },
+ {
+ "id": "025",
+ "category": "modal-overlay",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "026",
+ "category": "toast-notification",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "027",
+ "category": "loading-state",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": []
+ },
+ {
+ "id": "028",
+ "category": "consent-banner",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.6,
+ "criticalFailures": [
+ "consent-signal"
+ ]
+ },
+ {
+ "id": "029",
+ "category": "dashboard",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "030",
+ "category": "chart-reading",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "031",
+ "category": "data-table",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "032",
+ "category": "email-compose",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "033",
+ "category": "kanban-board",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "034",
+ "category": "calendar",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.5714285714285714,
+ "criticalFailures": [
+ "selected-event"
+ ]
+ },
+ {
+ "id": "035",
+ "category": "map-and-travel",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "036",
+ "category": "photo-understanding",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "037",
+ "category": "multilingual-ocr",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "038",
+ "category": "occlusion-and-contrast",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "cover"
+ ]
+ },
+ {
+ "id": "039",
+ "category": "security-challenge",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "040",
+ "category": "uncertainty-calibration",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "041",
+ "category": "authentication",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "042",
+ "category": "search-results",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": []
+ },
+ {
+ "id": "043",
+ "category": "checkout",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "banner"
+ ]
+ },
+ {
+ "id": "044",
+ "category": "form-validation",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.6,
+ "criticalFailures": [
+ "input",
+ "error"
+ ]
+ },
+ {
+ "id": "045",
+ "category": "modal-overlay",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "046",
+ "category": "toast-notification",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "047",
+ "category": "loading-state",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.6363636363636364,
+ "criticalFailures": []
+ },
+ {
+ "id": "048",
+ "category": "consent-banner",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.6,
+ "criticalFailures": [
+ "consent-signal"
+ ]
+ },
+ {
+ "id": "049",
+ "category": "dashboard",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "050",
+ "category": "chart-reading",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "051",
+ "category": "data-table",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "052",
+ "category": "email-compose",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "status"
+ ]
+ },
+ {
+ "id": "053",
+ "category": "kanban-board",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "054",
+ "category": "calendar",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "055",
+ "category": "map-and-travel",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "056",
+ "category": "photo-understanding",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "057",
+ "category": "multilingual-ocr",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.6363636363636364,
+ "criticalFailures": [
+ "input",
+ "status"
+ ]
+ },
+ {
+ "id": "058",
+ "category": "occlusion-and-contrast",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "cover"
+ ]
+ },
+ {
+ "id": "059",
+ "category": "security-challenge",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "captcha-signal"
+ ]
+ },
+ {
+ "id": "060",
+ "category": "uncertainty-calibration",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "visible-fragment"
+ ]
+ },
+ {
+ "id": "061",
+ "category": "authentication",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.6153846153846154,
+ "criticalFailures": [
+ "blocker",
+ "status"
+ ]
+ },
+ {
+ "id": "062",
+ "category": "search-results",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": []
+ },
+ {
+ "id": "063",
+ "category": "checkout",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "banner"
+ ]
+ },
+ {
+ "id": "064",
+ "category": "form-validation",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.6,
+ "criticalFailures": [
+ "input",
+ "error"
+ ]
+ },
+ {
+ "id": "065",
+ "category": "modal-overlay",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "066",
+ "category": "toast-notification",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "067",
+ "category": "loading-state",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.4444444444444444,
+ "criticalFailures": [
+ "purpose",
+ "heading"
+ ]
+ },
+ {
+ "id": "068",
+ "category": "consent-banner",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "consent-signal"
+ ]
+ },
+ {
+ "id": "069",
+ "category": "dashboard",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.75,
+ "criticalFailures": [
+ "alert"
+ ]
+ },
+ {
+ "id": "070",
+ "category": "chart-reading",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "071",
+ "category": "data-table",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "072",
+ "category": "email-compose",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "073",
+ "category": "kanban-board",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.5714285714285714,
+ "criticalFailures": [
+ "card-column"
+ ]
+ },
+ {
+ "id": "074",
+ "category": "calendar",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.5714285714285714,
+ "criticalFailures": [
+ "selected-event"
+ ]
+ },
+ {
+ "id": "075",
+ "category": "map-and-travel",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.625,
+ "criticalFailures": [
+ "selection"
+ ]
+ },
+ {
+ "id": "076",
+ "category": "photo-understanding",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "077",
+ "category": "multilingual-ocr",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "status"
+ ]
+ },
+ {
+ "id": "078",
+ "category": "occlusion-and-contrast",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "079",
+ "category": "security-challenge",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.6363636363636364,
+ "criticalFailures": [
+ "captcha-signal",
+ "security-blocker"
+ ]
+ },
+ {
+ "id": "080",
+ "category": "uncertainty-calibration",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "visible-fragment"
+ ]
+ },
+ {
+ "id": "081",
+ "category": "authentication",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "blocker"
+ ]
+ },
+ {
+ "id": "082",
+ "category": "search-results",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": [
+ "notice"
+ ]
+ },
+ {
+ "id": "083",
+ "category": "checkout",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.6923076923076923,
+ "criticalFailures": [
+ "banner",
+ "blocked"
+ ]
+ },
+ {
+ "id": "084",
+ "category": "form-validation",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.6,
+ "criticalFailures": [
+ "input",
+ "error"
+ ]
+ },
+ {
+ "id": "085",
+ "category": "modal-overlay",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "086",
+ "category": "toast-notification",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "087",
+ "category": "loading-state",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": []
+ },
+ {
+ "id": "088",
+ "category": "consent-banner",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "consent-signal"
+ ]
+ },
+ {
+ "id": "089",
+ "category": "dashboard",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "alert"
+ ]
+ },
+ {
+ "id": "090",
+ "category": "chart-reading",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "091",
+ "category": "data-table",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "092",
+ "category": "email-compose",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8333333333333334,
+ "criticalFailures": [
+ "status"
+ ]
+ },
+ {
+ "id": "093",
+ "category": "kanban-board",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "094",
+ "category": "calendar",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.4444444444444444,
+ "criticalFailures": [
+ "selected-event",
+ "conflict"
+ ]
+ },
+ {
+ "id": "095",
+ "category": "map-and-travel",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "route-alert"
+ ]
+ },
+ {
+ "id": "096",
+ "category": "photo-understanding",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.7777777777777778,
+ "criticalFailures": [
+ "visual-content"
+ ]
+ },
+ {
+ "id": "097",
+ "category": "multilingual-ocr",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.6363636363636364,
+ "criticalFailures": [
+ "input",
+ "status"
+ ]
+ },
+ {
+ "id": "098",
+ "category": "occlusion-and-contrast",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "cover"
+ ]
+ },
+ {
+ "id": "099",
+ "category": "security-challenge",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "captcha-signal"
+ ]
+ },
+ {
+ "id": "100",
+ "category": "uncertainty-calibration",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ }
+ ],
+ "qwen/qwen3-vl-8b-thinking": [
+ {
+ "id": "001",
+ "category": "authentication",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "002",
+ "category": "search-results",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "003",
+ "category": "checkout",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "004",
+ "category": "form-validation",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "005",
+ "category": "modal-overlay",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.7272727272727273,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "006",
+ "category": "toast-notification",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "007",
+ "category": "loading-state",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0,
+ "criticalFailures": [
+ "purpose",
+ "heading",
+ "spinner"
+ ]
+ },
+ {
+ "id": "008",
+ "category": "consent-banner",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 0.8,
+ "criticalFailures": []
+ },
+ {
+ "id": "009",
+ "category": "dashboard",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "010",
+ "category": "chart-reading",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "011",
+ "category": "data-table",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "012",
+ "category": "email-compose",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 0.75,
+ "criticalFailures": []
+ },
+ {
+ "id": "013",
+ "category": "kanban-board",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "014",
+ "category": "calendar",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.5714285714285714,
+ "criticalFailures": [
+ "selected-event"
+ ]
+ },
+ {
+ "id": "015",
+ "category": "map-and-travel",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "016",
+ "category": "photo-understanding",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "017",
+ "category": "multilingual-ocr",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.7777777777777778,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "018",
+ "category": "occlusion-and-contrast",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.6,
+ "criticalFailures": [
+ "target",
+ "cover"
+ ]
+ },
+ {
+ "id": "019",
+ "category": "security-challenge",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.45454545454545453,
+ "criticalFailures": [
+ "label",
+ "captcha-signal",
+ "security-blocker"
+ ]
+ },
+ {
+ "id": "020",
+ "category": "uncertainty-calibration",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "021",
+ "category": "authentication",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "purpose"
+ ]
+ },
+ {
+ "id": "022",
+ "category": "search-results",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "023",
+ "category": "checkout",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "banner"
+ ]
+ },
+ {
+ "id": "024",
+ "category": "form-validation",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "025",
+ "category": "modal-overlay",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "026",
+ "category": "toast-notification",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "027",
+ "category": "loading-state",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": []
+ },
+ {
+ "id": "028",
+ "category": "consent-banner",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.8,
+ "criticalFailures": []
+ },
+ {
+ "id": "029",
+ "category": "dashboard",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "030",
+ "category": "chart-reading",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "031",
+ "category": "data-table",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "032",
+ "category": "email-compose",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "033",
+ "category": "kanban-board",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "034",
+ "category": "calendar",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "035",
+ "category": "map-and-travel",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "036",
+ "category": "photo-understanding",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "037",
+ "category": "multilingual-ocr",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "038",
+ "category": "occlusion-and-contrast",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "cover"
+ ]
+ },
+ {
+ "id": "039",
+ "category": "security-challenge",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "040",
+ "category": "uncertainty-calibration",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "041",
+ "category": "authentication",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "042",
+ "category": "search-results",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": []
+ },
+ {
+ "id": "043",
+ "category": "checkout",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "044",
+ "category": "form-validation",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "045",
+ "category": "modal-overlay",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "046",
+ "category": "toast-notification",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "047",
+ "category": "loading-state",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": []
+ },
+ {
+ "id": "048",
+ "category": "consent-banner",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 0.8,
+ "criticalFailures": []
+ },
+ {
+ "id": "049",
+ "category": "dashboard",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "050",
+ "category": "chart-reading",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "051",
+ "category": "data-table",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "052",
+ "category": "email-compose",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "status"
+ ]
+ },
+ {
+ "id": "053",
+ "category": "kanban-board",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "054",
+ "category": "calendar",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.5714285714285714,
+ "criticalFailures": [
+ "selected-event"
+ ]
+ },
+ {
+ "id": "055",
+ "category": "map-and-travel",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "056",
+ "category": "photo-understanding",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "057",
+ "category": "multilingual-ocr",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "058",
+ "category": "occlusion-and-contrast",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "cover"
+ ]
+ },
+ {
+ "id": "059",
+ "category": "security-challenge",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "060",
+ "category": "uncertainty-calibration",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.4,
+ "criticalFailures": [
+ "visible-fragment",
+ "unknown"
+ ]
+ },
+ {
+ "id": "061",
+ "category": "authentication",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.6153846153846154,
+ "criticalFailures": [
+ "blocker",
+ "status"
+ ]
+ },
+ {
+ "id": "062",
+ "category": "search-results",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "063",
+ "category": "checkout",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "total"
+ ]
+ },
+ {
+ "id": "064",
+ "category": "form-validation",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "065",
+ "category": "modal-overlay",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "066",
+ "category": "toast-notification",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "067",
+ "category": "loading-state",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.5555555555555556,
+ "criticalFailures": [
+ "purpose",
+ "heading"
+ ]
+ },
+ {
+ "id": "068",
+ "category": "consent-banner",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 0.8,
+ "criticalFailures": []
+ },
+ {
+ "id": "069",
+ "category": "dashboard",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.75,
+ "criticalFailures": [
+ "alert"
+ ]
+ },
+ {
+ "id": "070",
+ "category": "chart-reading",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "071",
+ "category": "data-table",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "072",
+ "category": "email-compose",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "073",
+ "category": "kanban-board",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.5714285714285714,
+ "criticalFailures": [
+ "card-column"
+ ]
+ },
+ {
+ "id": "074",
+ "category": "calendar",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.42857142857142855,
+ "criticalFailures": [
+ "selected-event"
+ ]
+ },
+ {
+ "id": "075",
+ "category": "map-and-travel",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 0.875,
+ "criticalFailures": []
+ },
+ {
+ "id": "076",
+ "category": "photo-understanding",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "077",
+ "category": "multilingual-ocr",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "078",
+ "category": "occlusion-and-contrast",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "079",
+ "category": "security-challenge",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "080",
+ "category": "uncertainty-calibration",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "visible-fragment"
+ ]
+ },
+ {
+ "id": "081",
+ "category": "authentication",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.7692307692307693,
+ "criticalFailures": [
+ "blocker"
+ ]
+ },
+ {
+ "id": "082",
+ "category": "search-results",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": [
+ "notice"
+ ]
+ },
+ {
+ "id": "083",
+ "category": "checkout",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "blocked"
+ ]
+ },
+ {
+ "id": "084",
+ "category": "form-validation",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "085",
+ "category": "modal-overlay",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "086",
+ "category": "toast-notification",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "087",
+ "category": "loading-state",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "088",
+ "category": "consent-banner",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "089",
+ "category": "dashboard",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "alert"
+ ]
+ },
+ {
+ "id": "090",
+ "category": "chart-reading",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "091",
+ "category": "data-table",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "092",
+ "category": "email-compose",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8333333333333334,
+ "criticalFailures": [
+ "external-blocker"
+ ]
+ },
+ {
+ "id": "093",
+ "category": "kanban-board",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "094",
+ "category": "calendar",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "095",
+ "category": "map-and-travel",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "route-alert"
+ ]
+ },
+ {
+ "id": "096",
+ "category": "photo-understanding",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.7777777777777778,
+ "criticalFailures": [
+ "visual-content"
+ ]
+ },
+ {
+ "id": "097",
+ "category": "multilingual-ocr",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.5454545454545454,
+ "criticalFailures": [
+ "input",
+ "status"
+ ]
+ },
+ {
+ "id": "098",
+ "category": "occlusion-and-contrast",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "099",
+ "category": "security-challenge",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "100",
+ "category": "uncertainty-calibration",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ }
+ ],
+ "qwen/qwen3.5-35b-a3b": [
+ {
+ "id": "001",
+ "category": "authentication",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": []
+ },
+ {
+ "id": "002",
+ "category": "search-results",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "003",
+ "category": "checkout",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "004",
+ "category": "form-validation",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "005",
+ "category": "modal-overlay",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "006",
+ "category": "toast-notification",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "007",
+ "category": "loading-state",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.75,
+ "criticalFailures": [
+ "heading"
+ ]
+ },
+ {
+ "id": "008",
+ "category": "consent-banner",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "009",
+ "category": "dashboard",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "010",
+ "category": "chart-reading",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "011",
+ "category": "data-table",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "012",
+ "category": "email-compose",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "013",
+ "category": "kanban-board",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "014",
+ "category": "calendar",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 0.9047619047619048,
+ "criticalFailures": []
+ },
+ {
+ "id": "015",
+ "category": "map-and-travel",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "016",
+ "category": "photo-understanding",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "017",
+ "category": "multilingual-ocr",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "018",
+ "category": "occlusion-and-contrast",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "019",
+ "category": "security-challenge",
+ "difficulty": "easy",
+ "pass": false,
+ "ratio": 0.45454545454545453,
+ "criticalFailures": [
+ "label",
+ "captcha-signal",
+ "security-blocker"
+ ]
+ },
+ {
+ "id": "020",
+ "category": "uncertainty-calibration",
+ "difficulty": "easy",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "021",
+ "category": "authentication",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "blocker"
+ ]
+ },
+ {
+ "id": "022",
+ "category": "search-results",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "023",
+ "category": "checkout",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "banner"
+ ]
+ },
+ {
+ "id": "024",
+ "category": "form-validation",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "input",
+ "blocker"
+ ]
+ },
+ {
+ "id": "025",
+ "category": "modal-overlay",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "026",
+ "category": "toast-notification",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "027",
+ "category": "loading-state",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.7272727272727273,
+ "criticalFailures": []
+ },
+ {
+ "id": "028",
+ "category": "consent-banner",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "029",
+ "category": "dashboard",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "030",
+ "category": "chart-reading",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "031",
+ "category": "data-table",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.8571428571428571,
+ "criticalFailures": []
+ },
+ {
+ "id": "032",
+ "category": "email-compose",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "033",
+ "category": "kanban-board",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "034",
+ "category": "calendar",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.9047619047619048,
+ "criticalFailures": []
+ },
+ {
+ "id": "035",
+ "category": "map-and-travel",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "036",
+ "category": "photo-understanding",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "037",
+ "category": "multilingual-ocr",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "038",
+ "category": "occlusion-and-contrast",
+ "difficulty": "basic",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "target"
+ ]
+ },
+ {
+ "id": "039",
+ "category": "security-challenge",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "040",
+ "category": "uncertainty-calibration",
+ "difficulty": "basic",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "041",
+ "category": "authentication",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8333333333333334,
+ "criticalFailures": [
+ "blocker"
+ ]
+ },
+ {
+ "id": "042",
+ "category": "search-results",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.638888888888889,
+ "criticalFailures": [
+ "query"
+ ]
+ },
+ {
+ "id": "043",
+ "category": "checkout",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "blocked"
+ ]
+ },
+ {
+ "id": "044",
+ "category": "form-validation",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "045",
+ "category": "modal-overlay",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "046",
+ "category": "toast-notification",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "047",
+ "category": "loading-state",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 0.7272727272727273,
+ "criticalFailures": []
+ },
+ {
+ "id": "048",
+ "category": "consent-banner",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "049",
+ "category": "dashboard",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "050",
+ "category": "chart-reading",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "051",
+ "category": "data-table",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "052",
+ "category": "email-compose",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.7666666666666667,
+ "criticalFailures": [
+ "status"
+ ]
+ },
+ {
+ "id": "053",
+ "category": "kanban-board",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "054",
+ "category": "calendar",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "055",
+ "category": "map-and-travel",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "056",
+ "category": "photo-understanding",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "057",
+ "category": "multilingual-ocr",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "058",
+ "category": "occlusion-and-contrast",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "059",
+ "category": "security-challenge",
+ "difficulty": "intermediate",
+ "pass": false,
+ "ratio": 0.6363636363636364,
+ "criticalFailures": [
+ "captcha-signal",
+ "security-blocker"
+ ]
+ },
+ {
+ "id": "060",
+ "category": "uncertainty-calibration",
+ "difficulty": "intermediate",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "061",
+ "category": "authentication",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.7692307692307693,
+ "criticalFailures": [
+ "blocker"
+ ]
+ },
+ {
+ "id": "062",
+ "category": "search-results",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "063",
+ "category": "checkout",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "064",
+ "category": "form-validation",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "input",
+ "blocker"
+ ]
+ },
+ {
+ "id": "065",
+ "category": "modal-overlay",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "066",
+ "category": "toast-notification",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "067",
+ "category": "loading-state",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.5555555555555556,
+ "criticalFailures": [
+ "purpose",
+ "heading"
+ ]
+ },
+ {
+ "id": "068",
+ "category": "consent-banner",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "069",
+ "category": "dashboard",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "070",
+ "category": "chart-reading",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "071",
+ "category": "data-table",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "072",
+ "category": "email-compose",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "status"
+ ]
+ },
+ {
+ "id": "073",
+ "category": "kanban-board",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "074",
+ "category": "calendar",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "075",
+ "category": "map-and-travel",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "076",
+ "category": "photo-understanding",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "077",
+ "category": "multilingual-ocr",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "078",
+ "category": "occlusion-and-contrast",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "079",
+ "category": "security-challenge",
+ "difficulty": "advanced",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "080",
+ "category": "uncertainty-calibration",
+ "difficulty": "advanced",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "visible-fragment"
+ ]
+ },
+ {
+ "id": "081",
+ "category": "authentication",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "082",
+ "category": "search-results",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": [
+ "notice"
+ ]
+ },
+ {
+ "id": "083",
+ "category": "checkout",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "blocked"
+ ]
+ },
+ {
+ "id": "084",
+ "category": "form-validation",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "085",
+ "category": "modal-overlay",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "086",
+ "category": "toast-notification",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "087",
+ "category": "loading-state",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": []
+ },
+ {
+ "id": "088",
+ "category": "consent-banner",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "089",
+ "category": "dashboard",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "090",
+ "category": "chart-reading",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "091",
+ "category": "data-table",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "092",
+ "category": "email-compose",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8333333333333334,
+ "criticalFailures": [
+ "external-blocker"
+ ]
+ },
+ {
+ "id": "093",
+ "category": "kanban-board",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "094",
+ "category": "calendar",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 0.925925925925926,
+ "criticalFailures": []
+ },
+ {
+ "id": "095",
+ "category": "map-and-travel",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "route-alert"
+ ]
+ },
+ {
+ "id": "096",
+ "category": "photo-understanding",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.7777777777777778,
+ "criticalFailures": [
+ "visual-content"
+ ]
+ },
+ {
+ "id": "097",
+ "category": "multilingual-ocr",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.5454545454545454,
+ "criticalFailures": [
+ "input",
+ "status"
+ ]
+ },
+ {
+ "id": "098",
+ "category": "occlusion-and-contrast",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "099",
+ "category": "security-challenge",
+ "difficulty": "challenging",
+ "pass": false,
+ "ratio": 0.8205128205128205,
+ "criticalFailures": [
+ "unknown-code"
+ ]
+ },
+ {
+ "id": "100",
+ "category": "uncertainty-calibration",
+ "difficulty": "challenging",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ }
+ ]
+ }
+}
diff --git a/test/vision/analysis/2026-10-07-ling30-budget/manifest.json b/test/vision/analysis/2026-10-07-ling30-budget/manifest.json
new file mode 100644
index 000000000..08819f057
--- /dev/null
+++ b/test/vision/analysis/2026-10-07-ling30-budget/manifest.json
@@ -0,0 +1,309 @@
+{
+ "sourceRevision": "d25be94f4981c2fe5ff1d220f8b698ee72e88296",
+ "sourceHashes": {
+ "prompt.mjs": "881cb5b8284d3685e2eba3a238222ce480810449e137743c1cd3a12c1da4f07c",
+ "lib/score.mjs": "e16865c877edac2403fb476d3b62faf8eb5e23ff602b8db3cd0e8e2c25a61662",
+ "questions/001.json": "d701621199db6efc5be90be4f31aef3a981b0c3792ddf8e00298f451abd86aaf",
+ "expected/001.json": "7e8cef0d8463bb022f165b5c1596142a0aa9aaf48e842283fcad39ddfc4b4966",
+ "questions/002.json": "12d770b503bba388e0f1e95248d4344642fa043d50b7b14b53666e65df61d1d0",
+ "expected/002.json": "2f1daba6019bb9004bd218408daea93effdc836c54b4485f450505fd7d70ada9",
+ "questions/003.json": "94cb92872f64fba6d1e688b5148cec5f21b5815703dd9dc43a2f00e7428554b8",
+ "expected/003.json": "39577717218d2fcf3244ee003777664b7518d8446cb624c77b0a60b8ba43786f",
+ "questions/004.json": "85dfb926ec6d35ad5e8972f5c7fdb87a12d31d7e890f66a252988411ed095688",
+ "expected/004.json": "16fe809b1dfe7c95e52e0fffbcddabb92cc76d977b9ca9138e3257e29b7b30bf",
+ "questions/005.json": "d728e895168a7c01b722c59d2efac6dda299bcb5ecbdca2c2383ef866d6abfe0",
+ "expected/005.json": "78f9f43834972a88106658c6ce8d8356f02e31acaa6a01195a236451e8226339",
+ "questions/006.json": "0c541c297c539ea39a87f71833cdb7a23bfcf0788c57e68cc1abc0066284277b",
+ "expected/006.json": "99efb22a1a398e809aa4d93a1c4393e24b8ebc33b458a2fe2b89e6b5af84a203",
+ "questions/007.json": "1ae749421da7d8c81afbc0465fa1d7bb022fd90361529882428372802e2a93f5",
+ "expected/007.json": "4fc48bdbdfb2b077ca4fd42af0f2f69a89dd624feaa0102c57273dfc40714a3c",
+ "questions/008.json": "73467f344b83a97c71ef9c79681d1880ecf576c1f45a288bfad34d9286816c80",
+ "expected/008.json": "77d232d16a6c665fc03b2bb700dbfeb8906cd214207e624d65f736a99510447f",
+ "questions/009.json": "1aa06a87359529312f98647a25db6139f1fad44ef28b56323b4e6eee2d48e17d",
+ "expected/009.json": "b73687d1cd3b61368d95d81fc0ab58201a5acd79e7c8dd14a12c2195b0a5fb23",
+ "questions/010.json": "90c24444a5bd89e6faf7f264130cefce8e3b2c9f549e0b0bd04d43bfb61c7b0c",
+ "expected/010.json": "8aff2d76a6dce17b5325bd28a4ed9d96d6615d2c0f6cd98b0053a32c2f62060c",
+ "questions/011.json": "ae3b172dffc0a90f5003ec7506634ac0b75b6b71037727fec97c19564900c5e2",
+ "expected/011.json": "a10f5593cf090cdcfa0da781fe6685ea027f5be659a2b0cb7d8eaa08dd9f0984",
+ "questions/012.json": "7aed47f557ba73417f131373a56a1cf53f8eef7a87219c23cf47fa4aca87d307",
+ "expected/012.json": "790c9df5fe77bb1d51a48566d9b2a299e3b4017fab55979343d44e4efab00d08",
+ "questions/013.json": "edae3dd3208a4408b439dc7c1f4fdc0ccb1f145ebf5307daa777a67aa292fdfb",
+ "expected/013.json": "62dfcab86bd20cdf98a0398a75e0e2a83512a2d183f1e71ca86468eaeab8c97a",
+ "questions/014.json": "f1199d1af32341b945f6185b7e692dc2115863d66f9e8a373a89d73331431253",
+ "expected/014.json": "19f81c82c76ce5c1311bca10a877d5074bd203966f2b83f2256828366cc0a5c9",
+ "questions/015.json": "a28193b801419418784bc586a0e9cd6452b5c076ae15bb3f261a453ccdf2a808",
+ "expected/015.json": "995b804ac97ef450d180dd9e8b0a15027e72581b2b5ed1ee331a132962302a91",
+ "questions/016.json": "7166e74faed7eaa3416441c2423cb67b0ef922f9f282c4481d96b038b730e0d3",
+ "expected/016.json": "2b13c6fdc2d34f4387a3bbbbf123ea820c3d6f2a3b231bff7cfca9e3ce0da0a3",
+ "questions/017.json": "bb3f61c729863be67c2b99b3519ae24776f6c994f48c4ce1c152acdb7bf9fc63",
+ "expected/017.json": "77fbb25215c2789b6cd05ca21a4348f788030948571755cc82b516da6a7612df",
+ "questions/018.json": "4f4ad5e1d47c79d92c044dd8a8cf35679edc69e8e98cad792cd143209c723ff9",
+ "expected/018.json": "83326f21117df5ea9d697c4bf868a60d6c63f0efb30b38134639132496888e94",
+ "questions/019.json": "0e8ea1c5adee1ab995f9b473dcef72276ba276e5bc9f1a8f186792beedaad032",
+ "expected/019.json": "572ad205745632f6281b82f0f96ea77a6078f091b99a484742da7ed13ee1b671",
+ "questions/020.json": "b66dd9e176b253344cb2b703c96be248c084326183244f0409a42ca54df030b4",
+ "expected/020.json": "fad84c150d2df0a18d8e5d9948315e932a0154af1fc196b37bf4ed8befd957bb",
+ "questions/021.json": "205215ee849e8b611147852c523cdcf8e832ba32369ba42abfb79a82918fc2c7",
+ "expected/021.json": "5f574901e21eb214103dc982b95e21a32379cf1afcda18eb4a38e41d75d4af77",
+ "questions/022.json": "3a8c600604b611a47ea7ca0a393f0b831784c7d55912629ce98b8f41a6a95210",
+ "expected/022.json": "3bd162a86e130d62bce971292bf4958601f32acdb7085a3b349135d9bb63b5c4",
+ "questions/023.json": "b9b73667f399201a8844c52bc73f896e2831063f4bcc79f0f3cc83930f476875",
+ "expected/023.json": "b09c09ac6c41e4dc03b6d9dc6e2deed7b869ac70587e0c75d8f39a847c7c18c2",
+ "questions/024.json": "44db4e58186ff8a03a5a9930482c6e6bb47fc260cf4d54c539c1f6ff676a10fe",
+ "expected/024.json": "e4c622ccdd9a1d9750d125d6cab4ff28dd43ead9ced2c997f7dbb62b8b00a1aa",
+ "questions/025.json": "4d58cc20f92de07a9ffde0294ae029c77a67b936effd51bfde6654da4c36b600",
+ "expected/025.json": "d6f26eb9bc626200200d2cf86dc498764e1d855ca4ab1a44dc99baafb90896d4",
+ "questions/026.json": "46e9b30afa90ff83130fc3a198b4631df0b128dec51d82873cf049d7caa56168",
+ "expected/026.json": "845f176c7348d747766eb37e189471c3ece5e20bcb0a069827c12f0ee49f9322",
+ "questions/027.json": "4a60de9a258d9e06fb08346053013b0a2b0451c36b5946ad6c67046064cee9dd",
+ "expected/027.json": "293f8c86d66a4bc1dd1745bf816206837c4a155cd84a52484d0e94266ff7ebf6",
+ "questions/028.json": "cdcc1ded3c811c4223fa476c330b90853a8a83026b9fd44b6fe7f21d66b24ec0",
+ "expected/028.json": "1adb10ef2da28509abc4aef026ce6a20886093e08c80122125169e6e40b810e0",
+ "questions/029.json": "b118a183c382f5fe807996357c9b324bfec0ceae3f9e794e928add48dce0b4a8",
+ "expected/029.json": "19afaa0c49ac6cf6f3123df1d681b9fb064189a6c65fbc51af91f62b387025da",
+ "questions/030.json": "516a9628491ff18d4b88ae287934038ea5ef35fcfc504aacfd8f12cb70ab621a",
+ "expected/030.json": "1e3cf317a945cdd4994d5f0236ecc4df1fd87f296a1045a88969cd4a9e92aad0",
+ "questions/031.json": "65d2746ff2e4659575c6aa42ec0831a4ad1cb771fb558e7c698f390d28907162",
+ "expected/031.json": "9cfbda127c0948c0947c85c2504335c4f4c23b8ff0a75aa889d476185c9b017a",
+ "questions/032.json": "e4471ed2f25c7decad2db94e400a0dea2cfa605fc3f87e1de8780ab4c3f356f2",
+ "expected/032.json": "7b812e01a4dd9d6173ce0038e49a346d04ca7e706b8eaacf5538959ab23e6a01",
+ "questions/033.json": "92f04d4fdd5d9a4fdf4e2c2796243b6117f2338ec259a504bd0024d2ee2b3802",
+ "expected/033.json": "08171cf6a9345dcade56f7f6c1292c09ac76058417842ddcaafb2757444d6ba0",
+ "questions/034.json": "a1ff7f4b7ce33473ec4c07baf691858a9d4a1bd5de3909e186f76f63332020d2",
+ "expected/034.json": "37196d2ecc909915b09afc73cedf526fdbd106f09a64874923f4fb3ce5ee321c",
+ "questions/035.json": "798dfd217d7af0b679005b0b25bc15a6f214d1345772f5e12b975548534e0b77",
+ "expected/035.json": "0247924bbaab2d1b1d831adad483eab9b36e236501042e92fed863dbf2b77e5d",
+ "questions/036.json": "da1db42ed78bd7d9030e567f719dc89877c9f5cc3a9401b418c45a2a86a4b76b",
+ "expected/036.json": "95f24db8c8727f4f6377e5887b8ba39cd495e73957e1547b9a85bfd7adfe27e9",
+ "questions/037.json": "a0f6b637134d4cb3b591ff8e7e4e5d7a73ae9aa0557ffdc77ac75a47a77f85ea",
+ "expected/037.json": "5444f686803d4bbd103231fb81f870115f45e5f75f271fb71ef093ec89ddcb5c",
+ "questions/038.json": "08ba0ab3515a344a78b3476aa81610a796c9c34daefa22fbfd1efd740e3c5b3f",
+ "expected/038.json": "dc6af82d39442f9d03c2e6c001904e31004ebdf5a2376ac8250f49fd0e4419c5",
+ "questions/039.json": "69e5ab421e85b62868e813f465b0785232f6eabf682ef20d7511b53e29ac67ea",
+ "expected/039.json": "8e3f768f3622f8a25133a272d711e22bddbba8b759e7b2adff8387ae4701a6f5",
+ "questions/040.json": "a785d7c27f64677a40c6d40199cfc607c9f62992fd73ceabdc0011da08387c82",
+ "expected/040.json": "8ada7c6f09262d703e0bb91bfe4f2b6c04ac60ffb7d61daf499092881bbb7a73",
+ "questions/041.json": "849a88e77cb21673faeafe0f3593a25e504289349f1c0bbd5d7962ec63c75ef0",
+ "expected/041.json": "86dd02a7812c0e5e20a702c290a87282b5c4b0669d2873d12eb48b8d9624b36d",
+ "questions/042.json": "f289e24219ea631aa9515123d8f89e7c535cdde6398ee1bc4de6120f4a2f3bd0",
+ "expected/042.json": "8f3ad39dad2c97b28b1fc81550d8a63697c0cf7b2516719e624d171399cd5abb",
+ "questions/043.json": "7934e20c0bd3e9dd8047f610643d4e305a9d0e93ac025b93cc8e12183fb2904a",
+ "expected/043.json": "6e38feaa96a6fd55685db60deddb04ed9d5ed607ce74316f72277248787da6ad",
+ "questions/044.json": "86e8611e2b368d845f03424c95c4f90e0a9a01744e2e9afe8b87ce67be1c14d7",
+ "expected/044.json": "13554f5cf58fbf63fcaae084a3fbe5d2b96e592776aedbf2f53bda1c74659822",
+ "questions/045.json": "eaa81da434c8c6955be84574beed19546229423b5a24e393dbd08410b7e8dca0",
+ "expected/045.json": "aea83782089a5827aff2b67cdb7a8cf3d421528b310df572a5139b4a9584b207",
+ "questions/046.json": "9ef5f5fbea5dc14d1e03e42f7ed680353f9b0a834604da098f6ede0bc0a68258",
+ "expected/046.json": "04c1c82c48bcfcb2ffed84c136a4bdba726fbbafbf3024d65503dc4f364eeffc",
+ "questions/047.json": "8ca5c86b77768fd8367392006b66af931b2df38bfd34c639e7a4b5a3e64d3207",
+ "expected/047.json": "f70f66501dd018b544ca38d97f4dbec95da3e1501f23705bc8066f908b81bb88",
+ "questions/048.json": "caa1499f6d7350e2bf92ef6d0c7a5fd2810fa6de10e8d3ffa9c3016ab3b1331e",
+ "expected/048.json": "6937665052238286488e23b88f9b498933ebba181804f8e854c233b01a547323",
+ "questions/049.json": "29bbdb28699f7882aaf55834ad4bdec8f0c524f25632fd30d0188b5add95e1b4",
+ "expected/049.json": "5b395dcc76c03c77a7dfbe9b0728b5f31e561a050d39d879bfe2dde4639546d7",
+ "questions/050.json": "437c319597ee810822178284878d62e78218f1182d4a1539a99687441faf9dd0",
+ "expected/050.json": "71687fb8113f1a786244f6badf423eb35f88ffcff50d9a9723d63bd801b58d49",
+ "questions/051.json": "81eb7105b20c3f73a5afc4cfb01b947ad08ebe9e3cfc684cc288e6be8ddd79ae",
+ "expected/051.json": "d1b38cde3d5fd558e1511af2a25aaeaeac7b04694c65fe312a0e872c34b5a390",
+ "questions/052.json": "c839564a68f9e19135f84ae157e7dd11ea8a3c89b907dcc3d403c7fd7ef0c4e8",
+ "expected/052.json": "c500612863b4a0f0116aee9865fca19f4ad4782ae2a2aaddd79d81dc2759a7a8",
+ "questions/053.json": "63db37501389409c9273104942fd10882a84cd3fd573171b56b7899f141d52d2",
+ "expected/053.json": "45b34c6d9b764318fe120bbe82e139a7c0a9d8b0eb8ea59f8b766e8994d7714b",
+ "questions/054.json": "870ae9a2f4cf304c982ce7fc5a6a225f87e651701efdcc90f48d8c5611476347",
+ "expected/054.json": "9a9a23f504e029f854cf7d42db60d4953ac3ea5670150f41b18576af4e3f62b4",
+ "questions/055.json": "eb1f68d44cebd913c51d4622da50d6f3789c852aa1cac4d60b795b9155d13453",
+ "expected/055.json": "1d68b89e84327f369f2846252a0cced41a1dc8b3d2ac45d945839f38520ae258",
+ "questions/056.json": "cddb3082d2d5154ebdebb241fc58c6558df7fe0186cbdbcee3f94df41b76679b",
+ "expected/056.json": "b15a2ed56baaa484fa7a331487aaa74bf19454d0f841944fb0549e95f3755f55",
+ "questions/057.json": "8a32df6c3e93a51dc189a9eecc6db273e2ce622024d6edc57b536a4945d13a2b",
+ "expected/057.json": "342137a85cac0a0d3ba037d4ea66ef250035e5611001987050b4effd056e4012",
+ "questions/058.json": "8dd3d683f8d529ac6c733ab70f92a85b454f707e1865807838d40c90015ee27b",
+ "expected/058.json": "ee4ebcd582218b9f0450e184954dedd6ac806b64915b9fa1aa17c72627e26258",
+ "questions/059.json": "14ad51b8f1cdab8221f3fd2e2e624e1420d9d677ed045de60559465094c1a973",
+ "expected/059.json": "74339909208523c6a7172737feaf56a63ba401e49d7175f2a114058f99d835fd",
+ "questions/060.json": "8969f0a10e6c85126fcee7c23d2cf7fa08aad6b1d448a1424cb4a4b1eed1183f",
+ "expected/060.json": "f58343cade6054d667ac12a68982f4e58015db606432042f39f06a368e806de3",
+ "questions/061.json": "9e459506324f84fbf8d84f8cbaa2d98477e077905bcdc541995cdbfba09ba231",
+ "expected/061.json": "167450232fa30bb47a83dc25bbd16906f0bc77f5e1b65083fa5debcb5d6e019d",
+ "questions/062.json": "0347f2a89b7a0e8945c6b8164075f37d38a28e5a2da3c0cedef831b1546f4d68",
+ "expected/062.json": "07d3254da1ea3b5e3efc561f0ec0a499115639f6f1d8007d0ee5fc49a27ecfab",
+ "questions/063.json": "5dbdaaaffcf5f91835ff0988befc91de808c8fa2ba11ba6e4556430128c29cd9",
+ "expected/063.json": "e3ea76e72514bb29cb6bbcab32155e382661ad4fcfbf767daa6499c9ee574d47",
+ "questions/064.json": "9c00b6f20985b14cadc7dce1986f74b2eb285a27f1fd3e56dec9734694f32425",
+ "expected/064.json": "b43bbc015c5f2473c442574801aab8c96ebe4f946ab96bc1d8b46a14e4031c22",
+ "questions/065.json": "57368ecafd0103fbbd1c216e770f08c0ffb17955a85c2350ad0916d55a1a55f5",
+ "expected/065.json": "187e9cf599d39ba374b9ac9c5192e30f02f63b31a40861a996ef6a15201218c2",
+ "questions/066.json": "660d191d1bfb36c6d2557cb7edbcc75a0b71696a385463f15eeb716b56c40e08",
+ "expected/066.json": "cbeda9b753aedfcf778bb88b2a4be656c9adcd7010de9fc5b9f3f1221c93d464",
+ "questions/067.json": "b754e24c33dfbb9e74c214b7549dce44f907ed61bd9f3cf145f6d3a9b55cf168",
+ "expected/067.json": "28f521456af0e12335cdf2fa278e6e2d05638cbf2e7d4101c19fd5b55d0adda9",
+ "questions/068.json": "93347b9aff0f8e4c4675f145d06fe78a838dc0db794af3b66785a822a8b4d03e",
+ "expected/068.json": "af51339a0b181fe551f36a4c78bb978fa3dc40edff5461f188939333f0a06673",
+ "questions/069.json": "65b07ef920ff8100e6f002d3e40a4cd1dd40ad104c64c209cbe17070c140d443",
+ "expected/069.json": "b6beb5b87e265221a36f9bb2a01a1965c02ce0bf2943bca992148f02e77d349d",
+ "questions/070.json": "cfb18aee745f147593bd9b3526fd705d3b3e81fda9ce6f266fb902cc4bb99d4e",
+ "expected/070.json": "13944e9d6f578372fc51c1860a8944b8dc9d83b3a655129f152e7b4c41af27e5",
+ "questions/071.json": "62e946691f296e01b2ac2cd401930c260000fffe1cc1d6536eb1292b484b5e76",
+ "expected/071.json": "ebdba33d34fb7208648096a1281ad385ddeb36e8ae34877f54164f3129cdc44d",
+ "questions/072.json": "70c70ad566978ae92e3e1a2a138027bf3ce4d9a7b6bc3b9429d1124903f5e00c",
+ "expected/072.json": "3a65200fc1cb2bfbb00a0c72691eb1f937a15f4546e25bbe856d1ad4dc329b27",
+ "questions/073.json": "551eedb005f614231ac02b69aa08c4c3094e9707937792c5a7f922b034b49294",
+ "expected/073.json": "47a0592f362c76ed15b2cf63a35563a7649c13209205996042bdc6a9aa6184c3",
+ "questions/074.json": "dd2ad27a51948327aab8151e141d053015343d6c51ecad44fb09054676f1dd06",
+ "expected/074.json": "fcf99c22abb8934ccb5ab30ffebb4d630425b6e913ef076a0be5f1ea71653895",
+ "questions/075.json": "34fdbeceff93053594285594231f6f4e58a42538ad25c32b540c0a9e2e69ee22",
+ "expected/075.json": "3fd1052efc11ad09c4d54278c8823b503fc91c7eef6f02b02444e02c816c73b4",
+ "questions/076.json": "66666a99d434b65f754b623080b89b69d1b96357378aba03d459f80acbc555d0",
+ "expected/076.json": "74ed8e22f33f6897cc5d11bc54b596cff2373a612b9885e6f958d383ecd8ef29",
+ "questions/077.json": "62660debbdc65cc05a9c65dad8315829e29daaac658d327ecb69e5cf770b87d5",
+ "expected/077.json": "c3b8c2b3fe50ca2176908961cc2364c2f5280e5b449f6d977a98c158697f560e",
+ "questions/078.json": "b3df79007461e355192e3ba4f1998b283090e170560a6fc0b34562f417484dca",
+ "expected/078.json": "6965c6eab2231ea625acbebd440ebf6e746903c1544693215d7b52d916f958d7",
+ "questions/079.json": "73de045ea8641409b15fa8c632196de3d6def92be779cd41ddfd54927d8f9e37",
+ "expected/079.json": "0d7e484a714b607e935a2dee6b5cc900abd36e9409151999799159b177f717ea",
+ "questions/080.json": "7afd8e1c039bdb33889ee0e97eac7b1e9b58d9f56fd093c5ec5ada6fc9af9208",
+ "expected/080.json": "8f1d0db80c7608e39eba744cf48c17b9ead0c064c93edd877025e5ead32c319d",
+ "questions/081.json": "451684c50e9cd7dcb6ad5b63aabf2b2ad9b1344b99d2d0d2ab06e69e7e6dd7d5",
+ "expected/081.json": "6f0bc64183e1015affaf72e9f8ecce6613677d858180c4fbacb8fb2a35c904e6",
+ "questions/082.json": "d089e0e08db9b5d6cea77dc908938a2ab5ca0301280592bb2cd80390b06022f7",
+ "expected/082.json": "7711ef94c44d7940bc0c484e3212d5197fc84d0bc506aa2b025b288411810520",
+ "questions/083.json": "7fee637b1529b37bb389ff9f84a4542688e643609187f539e7f9b01f831d3580",
+ "expected/083.json": "27d5a38de74026b4b2b2cec5e5b7d15ceaca610bfbcc2d801fe64affeb7c8eb1",
+ "questions/084.json": "3705e8f62be4e46dfa5969c4fb927a728e3c0d2ee0c3dc05d1fa05df23320d46",
+ "expected/084.json": "ab2f43652bf9f1f515c93cad781dd6d041f7854904249dd264b2ac5230ef789b",
+ "questions/085.json": "485e935e92cf3726e37e9045332411d587b1f6fa7766ced145156101bd2a5c0e",
+ "expected/085.json": "7e9ec32d8d05e5c86eef163bbdaf1ebfacea0d8073747ca4b11a495b0bbd339a",
+ "questions/086.json": "e83f5cc8360ae1328fe670453b18cce0f6dcfe0b0fd6beeac21ab7f272e4b20f",
+ "expected/086.json": "d01eeeac8baefdddda91224dfeb28ae64832c26dd88f787125ab838382ac9b4d",
+ "questions/087.json": "caaa134497a5f82e5e17036f5cd69f3fbd843d3475688248e8c68a10489b5e38",
+ "expected/087.json": "39d9d1b9c5d3e2ed67136a70beef9f939378838daae591155f73ac5977f10123",
+ "questions/088.json": "9e9df605424ffab081a77a2da8a8bd4124133efb0b0261c4769d832ee9fbc148",
+ "expected/088.json": "28a26b0cc8df0c0a8bb0ae44dbe85b8536d51e503c6333b3ab4af3a6f9658626",
+ "questions/089.json": "6bdad3ba3d7d52e5fe4cf4c06845f2032c87f37b7e954a17db58cb3adcc21bd6",
+ "expected/089.json": "a0128c152b426f0a16a4e50e4146f058aa1a6dc3f72b08f5d0757fa65171d7b2",
+ "questions/090.json": "507fa31b59eb373778a9d30eaf9850c1726dff59964eab14877968468c55839c",
+ "expected/090.json": "40e27120989ddd6b7f2f85b9cf90f21943d5a1dbc55396d671251e50567b1526",
+ "questions/091.json": "03e01ab5299e8a9f5d23790df93efee39bb2dfc5db0131e29d5bf6b267af3046",
+ "expected/091.json": "44ff5ee6359e1f22b8588b4a9268c6be8ff79602b0edea537581666789dc1032",
+ "questions/092.json": "6fe88238ebcc44a2ab4fa9dfc4f348d0e71ffaeeec812924446519906b74d15c",
+ "expected/092.json": "5aa36bbb38e2cd05b039d07aef76534b3f8e3619ae97fc5b4bacab3204af6036",
+ "questions/093.json": "572a6af2f1140f2093fafc8e4be9159805fa9857d69499afbe4e5a36867619a8",
+ "expected/093.json": "20e12fe53fa458d2aeeae475b07b0322810b37e064e2bb9c944b19cd8a9122c2",
+ "questions/094.json": "1fae6e91bbf5a9d9ba9cac1759a976b6eabb27216a579d4567af9e0ec32c0516",
+ "expected/094.json": "a9ff4ae5b3763985ca6f05ade58f0e167d64eeac74818f5dab04ef3c38ce348f",
+ "questions/095.json": "60b070c93701b84686bf9e011409ff6f11df1526fba3f7313588888bcc64afad",
+ "expected/095.json": "9d161b2b97a48a1494553fd44ea386b3ac0bda8b797cf6615c74fe3e67b95344",
+ "questions/096.json": "425259e1234d60bcce3b57725ef295760d5caf9aad999b5ff339345c845680b3",
+ "expected/096.json": "7d230abbf52e2ba218c5408be29f5c53648e2ebc735f176bcb202bb376470b16",
+ "questions/097.json": "c5a2a6e3ab7365ff1e050e070a2ddced8bfd96bfca5e0312c2ffd68826860941",
+ "expected/097.json": "240148e313df3529e5d01fd7237d4ef1c72f13a0d2f911652eb98e3269497c53",
+ "questions/098.json": "747120a51acf43b7163429be2fa781bfab9e31577444d881788a4104d52a0b8e",
+ "expected/098.json": "298abc9a909c6d7629ed1e0cac6586f096c360f5ad4914ce0ba92ddb5fff984b",
+ "questions/099.json": "3a306e8c23d72cae80cb1427ae040b767ee329e207577297ad9cb819e0bb8097",
+ "expected/099.json": "415c982ea1d5cda94fbffd39e3e9454f2ca30f39f2f25789bc2819f423f09292",
+ "questions/100.json": "7b04ff3e60807a48fd54b3d81da59831e0c100699d1d237169b80c2bce3b7f7e",
+ "expected/100.json": "dcf73ac90fac11881ddc78bd0a4d2b9ba00c14f3cacaa503b4d81bc1ac0a42ed"
+ },
+ "images": {
+ "100": "b54d44e9c1c39985328649781ecd3bc1368253691f5fad7b9bb997f9ab638fae",
+ "001": "ebc7460f2b041e1b39c4fe671bb79744cb6dae04910c3fa142f6441980a08723",
+ "002": "ecc661fdc93f850a81815cf17e70bda06bbec4c75c2ca1065353364f070139c4",
+ "003": "a128709a7c464f92a94600744a642b7dfad41036c016495f16b5850ccb50b137",
+ "004": "849907433218133ac3bac8c8a2d33a37c024551912358f479120c747747445ce",
+ "005": "f22777d9159874730a381c04fcac2b315f5402e51d8290ec6b4d86c24d8a577f",
+ "006": "d6dc7841cd258bc41f69a729651c87cb3d791c4f4763b71ebaa87517c03bcdd0",
+ "007": "789ef9270b96c55fba9322624d0d78cc0286558f64990c63b18212c788dd6447",
+ "008": "5680008d1d69c4e8f9179e6e575e9be17501d6da4f816df2cb75e8f980323522",
+ "009": "7cff830423d6170fb604f728628bcd93d8b68e14f93cd091e9a6b11883be7c86",
+ "010": "33df6b6abc9840699fadaee8db4307dbf26b1f5f01dc931dbb9bb55a0ac368d9",
+ "011": "69eab4961caab45e18a67c74cc3c2a5d271092fdadef255808f90ceb483e6a25",
+ "012": "c835537324789da9c5ebc2a3366e935d947aea186fdb5ed6b7bc3e280042f946",
+ "013": "5b282dab1fc06d5b1519769705511c379fb2bc746c7d8b947dcee787ac1d57eb",
+ "014": "99dbb57d690a06263dc5f5a52782da2dd3aa1ec6e783c2cac2c71f980546f9cd",
+ "015": "18a9a726b9cf133e8c716795366bb5b45ece8f504576b8379a0b54b49f4a8c9e",
+ "016": "8d7fbd02cf443521bb1fcdd45ac0e0933051fa8e580fd3f01f85423e5d51e714",
+ "017": "255f1a0ab22bdba9962fe61bc78eea896c1c1b450078d0e9ceec3abb7685888d",
+ "018": "a742f78ae4483c10421ef88543e50bfbc815bec94f3f0a02bf1535a8f791dfdb",
+ "019": "7104b602de2052296cd05692e2ebd07ad0aeece65414bff305d9a036dca57ada",
+ "020": "5411249f484706dee156972fd1c20288fb4e38cfe79042c0b99442ac167f6843",
+ "021": "8ccaa8d7e5a9824abcd3ced72262009867f7340c45868b39f7c67f48e3783df9",
+ "022": "90fb777a8ac9b18808807d605aefe26dec5e95bb00475df4e0d2a4540f5c9048",
+ "023": "84afef3f467792f7256a21762aec83caa40f070a44cd7573367620f855faad10",
+ "024": "b36fa493114875bc8650465cd9e25d272ea14258e3406cf21ed63b67b4fbc885",
+ "025": "e66407997a9d2f5ab579585f014b5c62dfe9964500df6cf96d24df61d1152f56",
+ "026": "4f864bbce9cc0a4cf70c517bb3bb861e51ef653ebe1dad2c243fc374251a542b",
+ "027": "0fac8a0c70c60f93a6ef9123f768e23199b8530393b6809b7e2652185f064e65",
+ "028": "ae0adaae082b49c86ce5c654a99efa27ec303a565132a825c78e29b9b4371436",
+ "029": "0fd966cd0df90f999f7532ddbbaabc183ebd567b14548b54cec38bc4dfe2631a",
+ "030": "579f82a5176870179fd5300fd9f4c4910c7ff7f930f0fe22810c884de2a2bc97",
+ "031": "f40b45e62e16da81da985b6e3bb366b740b1af931ec75d80eaf96bd7eec93a71",
+ "032": "5f2327b1748ea5d712a3050f8ecff0a788b2c051e32b917f1725c60659e7813c",
+ "033": "bb2f77e0d46ba6b2773476ff451fa95b765fbe2caa3b32343bc614e722360ca1",
+ "034": "f0ba3b24aa7b348bd778003d4578ce200d99f453832c7e90ee703dc45c483af5",
+ "035": "42cd4f52b35a2de3af08d64a5fb7a23164621d80010d62a04c864b42e7a9b744",
+ "036": "9b5c497084ef27293d1cdaf71d46cef0bc8d2558ee102655702f1c9a52197760",
+ "037": "9a231483ee4ecd61458edbe7a24b64d8ba27bc5fb3b552c3770c8202b63bf0bb",
+ "038": "b60d99957dfd9f5a84379a2a20f07ec85571673c8b37517a59c00a0658076e82",
+ "039": "16fec881cd81a8a4e1283b17f8d167b246debc6ed027fe3ba3193f93d5ef6ff6",
+ "040": "73e5a82ce642bbfff8c45cadadd3bfae9cb80c3f918b17ae7a0ab2b1b7ac110d",
+ "041": "44a57b80a855ab098d21cfcf049073657a8fb514b7e844d5496472380e7376ea",
+ "042": "15d9c4012badd7ab3671c02ccedad3bc445d8b41edbc39597bd013d0561b3057",
+ "043": "9b81286c0e7085616a68adb9097dbce07cd5976b00c281bf348775e5ae5f1984",
+ "044": "42b3a9f448b2ba00543981fefa7f63b57d61fe7d97d629cb933fcc2773a2e06d",
+ "045": "5c87dbcb000c7b843d33b136ee2d09a92bef4225f8a95a55d7282b1742473c39",
+ "046": "5e559dcb853e3b4e8ff87e149c7f47b98b69751917269c8ad71b81b4567e10da",
+ "047": "35e82080c750393850480b6fa6a4806de936ce2c069d9e2c9832569d4f7d6d8f",
+ "048": "02c2fcbc8f49d620c17daaefe00870413ef59c6b0ad6f46e5d147c01a3a102e5",
+ "049": "1f872643865542f678cd2e05b04e5df74a3b2f1c2f06e353576869d974c2cc78",
+ "050": "e7788f9cab5175de1787e6bf782b99f4b34e22a7562445ad6ca0265e4a0d687a",
+ "051": "9867646e07373de44bd12ed10642c96db703bffe879f10896117342c84b10c18",
+ "052": "a380a736d68c6e5e65796bd704ff68ea213ac654a35c7bb97b34a5545af02271",
+ "053": "ca4c0ce5dacd50ffa1e3479eb9dfeb1b8e69a6cb0a9c7e922176f9d7c4a80860",
+ "054": "95dc6fbd31a230354f1a751d479068f0b5353c922cb8b7e0bee64c72e400763f",
+ "055": "1fa9088cd872e0d90c6af5beec2b3300b92e8ece34935d4f4acd58b7b0dca3fe",
+ "056": "2bc804203a22667dce889d88415ea2b4596948965e814fdab4d42ce015c23016",
+ "057": "1aa0ab56cddcc07d34dc49739705b8f7c86b776f9d77ec0273a8ce43a89c2e8a",
+ "058": "572bc3a726c4d44cb153ba37e3e4c440f5e94a160c6572200fe042f0104e4ae9",
+ "059": "df89fa397cdb08b97601dd1577db0269587c03ebd6b66f2fd0535a729e219277",
+ "060": "da113cf036792a628b8d44678653a3f88dd8acbac1f0185eef117dd425d0defd",
+ "061": "e5e506ae3f9967ea1be8e3beeb5af02aaf196526840b374ba12ec82f5ac2d801",
+ "062": "b85e150b792c28879da6c9592b7bd7252ba8f8b71b47852f0e52f7d23431cee2",
+ "063": "6f01d46b2a87aeba0634629dade606bfcfbebe7e0689f9d284275639c9dd5380",
+ "064": "2b2b6ca116511aa5e2286147995f0770b232037de962457bfe9016c4b9a160d9",
+ "065": "670c7f911eda7a9612846650b273776ad3cb333086b4b619f85eec708e33f5b6",
+ "066": "707e09dbbe97fb176ed42bf070333f3ca628c45850e4879b81ee1100c2ef58bf",
+ "067": "ceacb1b4a93012cc475b02c61862be44e9539b403c1476371a87fd520674ef31",
+ "068": "08064e36f3728182372ccf97c732e1b82472fdcc796a58a7c2c1443a888c9c4a",
+ "069": "eef33da4d69c885b9c42500edbea6b029eeb5656a4dc9f3e6a431ad06b444dec",
+ "070": "6a34f2a8df01fd37da75b40fee002e46043ccdc4720b572957d5c44c1a42b183",
+ "071": "6d05e6deaa76e5febe0ebd1970ffa5f321ad09b6cf3f65479c539d363b0f1172",
+ "072": "eb4249e4e4a3e9311385ec5c86668322271b0bac0ad4952765c15dfd255afb3d",
+ "073": "97bcd0fad628096a65eb12e0c5db3dd365d789a586517889c6aa92cfccb580c3",
+ "074": "7db44a9c0bbaa77702f994b8daea9ad46640ad050af350766ba54c9abd8b9506",
+ "075": "587cb4b4ea563b39f477ee3274e761b2105deb74079f80f1cf98f895a910c3e3",
+ "076": "2732404a36c0501d5a989e2ea7a7e9336be2c282523fbb9e5e4587882e0c3fe7",
+ "077": "d6bf9b2d55db4680145c32d2f0793316d2fbe1c05da93ff557933a51c9e2662e",
+ "078": "8b5544b8d4f4b42f7d115a7efcbd17777f751f63d023daea21f239a9bf457ff4",
+ "079": "3f0e09cff735eccac6fef6ee8d86c74b3e83ff6b6e80cec27a8337938305a0ca",
+ "080": "6666a6845ec9e0eef087b713402c93bbf2ff0f67eddbd218c6b0786281f6b2d8",
+ "081": "574e324e980c65e83e921b97719134b5eeda7d5945cfb5f6376e8f8a0d00de0d",
+ "082": "11a24ea55821569aeb9af5fa67def3a11939eb81a145cd4cf844247d99398558",
+ "083": "a364c773403657279eb511b54d0a0f482007e9cd3f1154c71a02976f2ab4f733",
+ "084": "24c6dcd36f93087c143f58bd3bcea05df65efeec26f183244f16e69ba310bd51",
+ "085": "5adbd1a9b1252d8877619ea1bcc63e73603b3c9efbb3f1f24cd4906055a6cd02",
+ "086": "f99038ae9d1f044ac86558e1f413c3ccd730cae620a76931394d817fc00b7837",
+ "087": "12d7723f9c354430940fd23df4425755c906e68d31d987b4a06bd0355e1e1317",
+ "088": "e3f766af35372e2a08c2db89ec415fb8a89f8db483978b270f49f81334b20749",
+ "089": "7c11ab2c2ccedcff9edc7b9421d81c7d608d04bd90a9d795dcac8dd9d62613a1",
+ "090": "e2bf7cc8d99e7d617a882cdeee0e054637bf76e81e5f5d4627c1adb33945b486",
+ "091": "3889ae84ac09a2526b50db4c4a3c6aa55d0c1902441a667908005c8e5feab105",
+ "092": "ea1d86e63142b56a1856093d97da1202b5f506554237160473e2c71c678c4d33",
+ "093": "8a7f1bdb44b30d3a5a5ae97294ee0ebcbe852ce7a3cc69946d627bae3b44dc4d",
+ "094": "fb3409891b5e5f49ce85bc1d3b8fffb41bf450e236249ec133fef3d3dc114192",
+ "095": "d4dea8257e65089b6ae8b0e858d99fbc3705e18ef7ba698199cfdcd6e6c19ec9",
+ "096": "7c353d2fc1f38ad9f3a9f24829ca7800e50210c7faa385045efd0c59282fa09d",
+ "097": "cb1b54a466170223b341711a368bb8fbbbeeb2ee6a1a1e8422e8eaad60cbb177",
+ "098": "9aaecc2a5133ce5dbac3b041977b2c296f842c9fd4f68fa4caff58f729bae448",
+ "099": "c22b1ff720c33cd555adfd4a45fbf4f994926b610c9abbe5d9085df1cfc9e615"
+ }
+}
diff --git a/test/vision/analysis/2026-10-07-ling30-budget/openrouter-endpoints.json b/test/vision/analysis/2026-10-07-ling30-budget/openrouter-endpoints.json
new file mode 100644
index 000000000..652846d40
--- /dev/null
+++ b/test/vision/analysis/2026-10-07-ling30-budget/openrouter-endpoints.json
@@ -0,0 +1,132 @@
+{
+ "retrievedAt": "2026-10-07T00:18:39.126Z",
+ "url": "https://openrouter.ai/api/v1/models/inclusionai/ling-3.0-flash-vl/endpoints",
+ "status": 200,
+ "data": {
+ "data": {
+ "id": "inclusionai/ling-3.0-flash-vl",
+ "name": "inclusionAI: Ling 3.0 Flash VL",
+ "created": 1789056114,
+ "description": "Ling 3.0 Flash VL builds on Ling 3.0 Flash (124B total / 5.5B active MoE from InclusionAI), further strengthening its language capabilities while adding native visual perception and advanced visual...",
+ "architecture": {
+ "tokenizer": "Other",
+ "instruct_type": null,
+ "modality": "text+image+video->text",
+ "input_modalities": [
+ "text",
+ "image",
+ "video"
+ ],
+ "output_modalities": [
+ "text"
+ ]
+ },
+ "endpoints": [
+ {
+ "name": "Novita | inclusionai/ling-3.0-flash-vl-20260910",
+ "model_id": "inclusionai/ling-3.0-flash-vl",
+ "model_name": "inclusionAI: Ling 3.0 Flash VL",
+ "context_length": 262144,
+ "pricing": {
+ "prompt": "0.000000021",
+ "completion": "0.0000000616",
+ "input_cache_read": "0.0000000042",
+ "discount": 0.72
+ },
+ "provider_name": "Novita",
+ "tag": "novita/bf16",
+ "quantization": "bf16",
+ "max_completion_tokens": 32768,
+ "max_prompt_tokens": null,
+ "supported_parameters": [
+ "reasoning",
+ "include_reasoning",
+ "max_tokens",
+ "temperature",
+ "top_p",
+ "stop",
+ "frequency_penalty",
+ "presence_penalty",
+ "seed",
+ "top_k",
+ "repetition_penalty",
+ "tools",
+ "tool_choice",
+ "logprobs",
+ "top_logprobs"
+ ],
+ "supports_tool_choice": {
+ "none": true,
+ "auto": true,
+ "required": true,
+ "function": true
+ },
+ "status": 0,
+ "uptime_last_30m": 99.92563476043769,
+ "uptime_last_5m": 99.87740089906008,
+ "uptime_last_1d": 99.65869803990948,
+ "supports_implicit_caching": false,
+ "native_tools": {},
+ "supports_voice_cloning": false,
+ "supports_multiple_audio_references": false,
+ "supports_image_reference": false,
+ "latency_last_30m": null,
+ "throughput_last_30m": null
+ },
+ {
+ "name": "DeepInfra | inclusionai/ling-3.0-flash-vl-20260910",
+ "model_id": "inclusionai/ling-3.0-flash-vl",
+ "model_name": "inclusionAI: Ling 3.0 Flash VL",
+ "context_length": 131072,
+ "pricing": {
+ "prompt": "0.00000006",
+ "completion": "0.00000018",
+ "input_cache_read": "0.000000012",
+ "discount": 0
+ },
+ "provider_name": "DeepInfra",
+ "tag": "deepinfra/fp16",
+ "quantization": "fp16",
+ "max_completion_tokens": 32768,
+ "max_prompt_tokens": null,
+ "supported_parameters": [
+ "reasoning",
+ "include_reasoning",
+ "max_tokens",
+ "temperature",
+ "top_p",
+ "stop",
+ "frequency_penalty",
+ "presence_penalty",
+ "repetition_penalty",
+ "top_k",
+ "seed",
+ "min_p",
+ "logit_bias",
+ "tools",
+ "tool_choice",
+ "structured_outputs",
+ "response_format"
+ ],
+ "supports_tool_choice": {
+ "none": true,
+ "auto": true,
+ "required": true,
+ "function": true
+ },
+ "status": 0,
+ "uptime_last_30m": 100,
+ "uptime_last_5m": 100,
+ "uptime_last_1d": 99.84113590735838,
+ "supports_implicit_caching": false,
+ "native_tools": {},
+ "supports_voice_cloning": false,
+ "supports_multiple_audio_references": false,
+ "supports_image_reference": false,
+ "latency_last_30m": null,
+ "throughput_last_30m": null
+ }
+ ]
+ }
+ }
+}
diff --git a/test/vision/analysis/2026-10-07-ling30-budget/regression.json b/test/vision/analysis/2026-10-07-ling30-budget/regression.json
new file mode 100644
index 000000000..de825c18b
--- /dev/null
+++ b/test/vision/analysis/2026-10-07-ling30-budget/regression.json
@@ -0,0 +1,7 @@
+{
+ "verifiedHistoricalModels": 6,
+ "verifiedHistoricalResponses": 600,
+ "sourceRevision": "d25be94f4981c2fe5ff1d220f8b698ee72e88296",
+ "promptImageAndQuestionMatches": true,
+ "savedScoresExactlyReproduced": true
+}
diff --git a/test/vision/analysis/2026-10-07-ling30-budget/rubrics.json b/test/vision/analysis/2026-10-07-ling30-budget/rubrics.json
new file mode 100644
index 000000000..d7bbb3cd6
--- /dev/null
+++ b/test/vision/analysis/2026-10-07-ling30-budget/rubrics.json
@@ -0,0 +1,6752 @@
+{
+ "100": {
+ "schemaVersion": 1,
+ "id": "100",
+ "image": "images/100.png",
+ "threshold": 0.74,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "account",
+ "label",
+ "receipt",
+ "contract",
+ "telemetry",
+ "document"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Telemetry capture",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "visible-fragment",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Node ••7? / region eu-?",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "unknown",
+ "dimension": "unknowns",
+ "section": 6,
+ "kind": "any",
+ "values": [
+ "unknown",
+ "unreadable",
+ "blurred",
+ "smudged",
+ "redacted",
+ "cannot read"
+ ],
+ "weight": 3,
+ "critical": true
+ },
+ {
+ "id": "no-guess",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "Node ••7? / region eu-?"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Transcribe only Node ••7? / region eu-? and explicitly say node and region identifiers are partially unreadable."
+ },
+ "001": {
+ "schemaVersion": 1,
+ "id": "001",
+ "image": "images/001.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "sign in",
+ "login",
+ "identity verification",
+ "session expired"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Welcome back",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Sign in",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "email-input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Email",
+ "empty"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "missing",
+ "email",
+ "password"
+ ],
+ "weight": 2,
+ "critical": false
+ }
+ ],
+ "successRubric": "Identify the Welcome back authentication surface, quote Sign in, and report missing email and password."
+ },
+ "002": {
+ "schemaVersion": 1,
+ "id": "002",
+ "image": "images/002.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "search",
+ "results"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Search",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "tab",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "All",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "count",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "24 results",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "query",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Search",
+ "camera"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Read the active query camera, selected tab All, and exact count 24 results."
+ },
+ "003": {
+ "schemaVersion": 1,
+ "id": "003",
+ "image": "images/003.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "checkout",
+ "cart",
+ "order",
+ "purchase"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Your cart",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "item",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Canvas tote",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "total",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "$24.00",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Checkout",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Quote $24.00 and Checkout; correctly report whether the action is disabled and why."
+ },
+ "004": {
+ "schemaVersion": 1,
+ "id": "004",
+ "image": "images/004.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "form",
+ "profile",
+ "shipping",
+ "invite",
+ "dataset",
+ "tax"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Create profile",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Save profile",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Display name",
+ "empty",
+ "e.g. Maya"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "error",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Display name is required",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "invalid",
+ "required",
+ "incomplete",
+ "missing",
+ "format"
+ ],
+ "weight": 1,
+ "critical": true
+ }
+ ],
+ "successRubric": "Name Display name, preserve its current value and placeholder, and quote Display name is required."
+ },
+ "005": {
+ "schemaVersion": 1,
+ "id": "005",
+ "image": "images/005.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "files",
+ "dialog",
+ "modal"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "title",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Delete file?",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "primary",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Delete",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "secondary",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Cancel",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "modal-signal",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "all",
+ "values": [
+ "modal",
+ "This action cannot be undone."
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "overlay-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "modal",
+ "overlay",
+ "dialog"
+ ],
+ "weight": 1,
+ "critical": true
+ }
+ ],
+ "successRubric": "Recognize a blocking modal, quote Delete file?, Delete, and Cancel."
+ },
+ "006": {
+ "schemaVersion": 1,
+ "id": "006",
+ "image": "images/006.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "notes",
+ "settings",
+ "upload",
+ "export"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Notes",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Done",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "toast",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Saved",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Quote Saved, and classify the visible state."
+ },
+ "007": {
+ "schemaVersion": 1,
+ "id": "007",
+ "image": "images/007.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "loading",
+ "processing",
+ "reconnecting",
+ "import"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": null,
+ "kind": "literal",
+ "value": "Loading messages…",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "spinner",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "any",
+ "values": [
+ "loading",
+ "spinner",
+ "processing",
+ "reconnecting",
+ "progress"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "wait-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "loading",
+ "wait",
+ "processing",
+ "reconnecting",
+ "not ready"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Report the loading state and preserve Loading messages… without inventing completion."
+ },
+ "008": {
+ "schemaVersion": 1,
+ "id": "008",
+ "image": "images/008.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "privacy",
+ "cookie",
+ "consent",
+ "video"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "title",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "We use cookies",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "choice-1",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Accept all",
+ "weight": 1,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "choice-2",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Reject all",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "consent-signal",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "any",
+ "values": [
+ "cookie",
+ "consent",
+ "privacy",
+ "overlay",
+ "banner"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "consent-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "cookie",
+ "consent",
+ "overlay",
+ "blocked"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Recognize the consent UI and quote Accept all / Reject all."
+ },
+ "009": {
+ "schemaVersion": 1,
+ "id": "009",
+ "image": "images/009.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "dashboard",
+ "overview",
+ "system health",
+ "incident"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Overview",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "target-metric",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "all",
+ "values": [
+ "Orders",
+ "86"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Read Orders = 86 and report the alert without guessing unknown values."
+ },
+ "010": {
+ "schemaVersion": 1,
+ "id": "010",
+ "image": "images/010.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "chart",
+ "dashboard",
+ "weekly signups"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Weekly signups",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "highest-label",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "Tue"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "highest-value",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "18"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Use the rendered bars, not source order: Tue is tallest."
+ },
+ "011": {
+ "schemaVersion": 1,
+ "id": "011",
+ "image": "images/011.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "table",
+ "orders",
+ "members",
+ "deployment",
+ "invoice",
+ "audit"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Recent orders",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "target-row",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "#1043",
+ "Pending"
+ ],
+ "weight": 3,
+ "critical": true
+ }
+ ],
+ "successRubric": "Keep row association intact: #1043 — Pending."
+ },
+ "012": {
+ "schemaVersion": 1,
+ "id": "012",
+ "image": "images/012.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "email",
+ "compose",
+ "message"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "New message",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Send",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "to-input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "To",
+ "empty"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "subject-input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Subject",
+ "empty"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Report the compose fields exactly and surface the empty required fields."
+ },
+ "013": {
+ "schemaVersion": 1,
+ "id": "013",
+ "image": "images/013.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "board",
+ "kanban",
+ "project",
+ "sprint",
+ "incident"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Project board",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "card-column",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "Write brief",
+ "To do"
+ ],
+ "weight": 3,
+ "critical": true
+ }
+ ],
+ "successRubric": "Preserve the spatial association: Write brief is in To do."
+ },
+ "014": {
+ "schemaVersion": 1,
+ "id": "014",
+ "image": "images/014.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "calendar",
+ "schedule"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "August 2026",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "selected-event",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "21",
+ "Design review",
+ "10:00"
+ ],
+ "weight": 3,
+ "critical": true
+ }
+ ],
+ "successRubric": "Associate day 21 with Design review at 10:00."
+ },
+ "015": {
+ "schemaVersion": 1,
+ "id": "015",
+ "image": "images/015.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "map",
+ "tracking",
+ "route",
+ "hotel",
+ "café"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Nearby cafés",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "selection",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "Pine Café",
+ "4 min walk"
+ ],
+ "weight": 3,
+ "critical": true
+ },
+ {
+ "id": "pin-count",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "2",
+ "two"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Read the selected Pine Café card, its 4 min walk detail, and 2 pins."
+ },
+ "016": {
+ "schemaVersion": 1,
+ "id": "016",
+ "image": "images/016.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "gallery",
+ "archive",
+ "collection",
+ "comparison"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Street archive",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "caption",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Talad Noi",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "badge",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Color photo",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "visual-content",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "Talad Noi"
+ ],
+ "weight": 2,
+ "critical": false
+ }
+ ],
+ "successRubric": "Combine the rendered image with its UI labels; do not infer a place or date beyond Talad Noi."
+ },
+ "017": {
+ "schemaVersion": 1,
+ "id": "017",
+ "image": "images/017.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "sign",
+ "payment",
+ "order",
+ "security",
+ "login",
+ "checkout",
+ "settings"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Bienvenue",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Continuer",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Adresse e-mail",
+ "empty"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Preserve Unicode exactly: Bienvenue; Adresse e-mail; Continuer."
+ },
+ "018": {
+ "schemaVersion": 1,
+ "id": "018",
+ "image": "images/018.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "account",
+ "pricing",
+ "report",
+ "editor",
+ "trading"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Account",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "target",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Save changes",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "cover",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Help",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "occlusion",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "cover",
+ "overlap",
+ "overlay",
+ "obstruct",
+ "block"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Detect the visual obstruction: Help interferes with Save changes."
+ },
+ "019": {
+ "schemaVersion": 1,
+ "id": "019",
+ "image": "images/019.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "security",
+ "verification",
+ "captcha",
+ "challenge"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Security check",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "label",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "I’m not a robot",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Continue",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "captcha-signal",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "any",
+ "values": [
+ "CAPTCHA",
+ "challenge",
+ "verification"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "security-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "verification",
+ "challenge",
+ "CAPTCHA",
+ "disabled"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Identify the synthetic challenge and disabled state; do not solve or invent unreadable characters."
+ },
+ "020": {
+ "schemaVersion": 1,
+ "id": "020",
+ "image": "images/020.png",
+ "threshold": 0.68,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "account",
+ "label",
+ "receipt",
+ "contract",
+ "telemetry",
+ "document"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Account ID",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "visible-fragment",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "ACCT-1042",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Transcribe only ACCT-1042."
+ },
+ "021": {
+ "schemaVersion": 1,
+ "id": "021",
+ "image": "images/021.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "sign in",
+ "login",
+ "identity verification",
+ "session expired"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Sign in to Atlas",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Continue",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "email-input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Email",
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "missing",
+ "password"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "disabled",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "any",
+ "values": [
+ "disabled",
+ "unavailable"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Identify the Sign in to Atlas authentication surface, quote Continue, and report missing password."
+ },
+ "022": {
+ "schemaVersion": 1,
+ "id": "022",
+ "image": "images/022.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "search",
+ "results"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Product search",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "tab",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Products",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "count",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "18 results",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "query",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Search",
+ "wireless keyboard"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "chip",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "In stock",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Read the active query wireless keyboard, selected tab Products, and exact count 18 results."
+ },
+ "023": {
+ "schemaVersion": 1,
+ "id": "023",
+ "image": "images/023.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "checkout",
+ "cart",
+ "order",
+ "purchase"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Checkout",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "item",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Desk lamp",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "total",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "$67.40",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Place order",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "banner",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Add a delivery address",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "blocked",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "delivery",
+ "address"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Quote $67.40 and Place order; correctly report whether the action is disabled and why."
+ },
+ "024": {
+ "schemaVersion": 1,
+ "id": "024",
+ "image": "images/024.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "form",
+ "profile",
+ "shipping",
+ "invite",
+ "dataset",
+ "tax"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Shipping details",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Continue",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Postal code",
+ "10A2",
+ "5 digits"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "error",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Enter a valid postal code",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "invalid",
+ "required",
+ "incomplete",
+ "missing",
+ "format"
+ ],
+ "weight": 1,
+ "critical": true
+ }
+ ],
+ "successRubric": "Name Postal code, preserve its current value and placeholder, and quote Enter a valid postal code."
+ },
+ "025": {
+ "schemaVersion": 1,
+ "id": "025",
+ "image": "images/025.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "team settings",
+ "dialog",
+ "modal"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "title",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Leave workspace?",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "primary",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Leave",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "secondary",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Stay",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "modal-signal",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "all",
+ "values": [
+ "modal",
+ "You will lose access to 12 projects."
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "overlay-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "modal",
+ "overlay",
+ "dialog"
+ ],
+ "weight": 1,
+ "critical": true
+ }
+ ],
+ "successRubric": "Recognize a blocking modal, quote Leave workspace?, Leave, and Stay."
+ },
+ "026": {
+ "schemaVersion": 1,
+ "id": "026",
+ "image": "images/026.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "preferences",
+ "settings",
+ "upload",
+ "export"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Preferences",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Close",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "toast",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Settings updated",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "detail",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Synced just now",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Quote Settings updated and Synced just now, and classify the visible state."
+ },
+ "027": {
+ "schemaVersion": 1,
+ "id": "027",
+ "image": "images/027.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "loading",
+ "processing",
+ "reconnecting",
+ "import"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Preparing dashboard",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "spinner",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "any",
+ "values": [
+ "loading",
+ "spinner",
+ "processing",
+ "reconnecting",
+ "progress"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "wait-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "loading",
+ "wait",
+ "processing",
+ "reconnecting",
+ "not ready"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "detail",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Fetching latest metrics",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "progress",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "38%",
+ "weight": 2,
+ "critical": false,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Report the loading state and preserve 38% without inventing completion."
+ },
+ "028": {
+ "schemaVersion": 1,
+ "id": "028",
+ "image": "images/028.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "privacy",
+ "cookie",
+ "consent",
+ "video"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "title",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Privacy choices",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "choice-1",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Manage settings",
+ "weight": 1,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "choice-2",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Save choices",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "consent-signal",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "any",
+ "values": [
+ "cookie",
+ "consent",
+ "privacy",
+ "overlay",
+ "banner"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "consent-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "cookie",
+ "consent",
+ "overlay",
+ "blocked"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Recognize the consent UI and quote Manage settings / Save choices."
+ },
+ "029": {
+ "schemaVersion": 1,
+ "id": "029",
+ "image": "images/029.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "dashboard",
+ "overview",
+ "system health",
+ "incident"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Sales dashboard",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "target-metric",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "all",
+ "values": [
+ "Conversion",
+ "3.8%"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Read Conversion = 3.8% and report the alert without guessing unknown values."
+ },
+ "030": {
+ "schemaVersion": 1,
+ "id": "030",
+ "image": "images/030.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "chart",
+ "dashboard",
+ "quarterly revenue"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Quarterly revenue",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "highest-label",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "Q4"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "highest-value",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "44",
+ "$44K"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Use the rendered bars, not source order: Q4 is tallest and is labeled $44K."
+ },
+ "031": {
+ "schemaVersion": 1,
+ "id": "031",
+ "image": "images/031.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "table",
+ "orders",
+ "members",
+ "deployment",
+ "invoice",
+ "audit"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Team members",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "target-row",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "Noah Park",
+ "Suspended"
+ ],
+ "weight": 3,
+ "critical": true
+ }
+ ],
+ "successRubric": "Keep row association intact: Noah Park — Suspended."
+ },
+ "032": {
+ "schemaVersion": 1,
+ "id": "032",
+ "image": "images/032.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "email",
+ "compose",
+ "message"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Compose",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Send",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "to-input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "To",
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "subject-input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Subject",
+ "Project update"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "status",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Draft saved",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Report the compose fields exactly and surface Draft saved."
+ },
+ "033": {
+ "schemaVersion": 1,
+ "id": "033",
+ "image": "images/033.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "board",
+ "kanban",
+ "project",
+ "sprint",
+ "incident"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Sprint 18",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "card-column",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "Fix checkout",
+ "In progress"
+ ],
+ "weight": 3,
+ "critical": true
+ }
+ ],
+ "successRubric": "Preserve the spatial association: Fix checkout is in In progress."
+ },
+ "034": {
+ "schemaVersion": 1,
+ "id": "034",
+ "image": "images/034.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "calendar",
+ "schedule"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Team calendar",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "selected-event",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "24",
+ "Sprint planning",
+ "14:30"
+ ],
+ "weight": 3,
+ "critical": true
+ }
+ ],
+ "successRubric": "Associate day 24 with Sprint planning at 14:30."
+ },
+ "035": {
+ "schemaVersion": 1,
+ "id": "035",
+ "image": "images/035.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "map",
+ "tracking",
+ "route",
+ "hotel",
+ "café"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Hotel map",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "selection",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "Harbor House",
+ "$142/night"
+ ],
+ "weight": 3,
+ "critical": true
+ },
+ {
+ "id": "pin-count",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "3",
+ "three"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Read the selected Harbor House card, its $142/night detail, and 3 pins."
+ },
+ "036": {
+ "schemaVersion": 1,
+ "id": "036",
+ "image": "images/036.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "gallery",
+ "archive",
+ "collection",
+ "comparison"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Winter collection",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "caption",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Snowbound avenue",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "badge",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Historic photo",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "visual-content",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "snow",
+ "winter"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Combine the rendered image with its UI labels; do not infer a place or date beyond Snowbound avenue."
+ },
+ "037": {
+ "schemaVersion": 1,
+ "id": "037",
+ "image": "images/037.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "sign",
+ "payment",
+ "order",
+ "security",
+ "login",
+ "checkout",
+ "settings"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Willkommen zurück",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Anmelden",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Passwort",
+ "••••••"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "status",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Passwort ist falsch",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Preserve Unicode exactly: Willkommen zurück; Passwort; Anmelden; Passwort ist falsch."
+ },
+ "038": {
+ "schemaVersion": 1,
+ "id": "038",
+ "image": "images/038.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "account",
+ "pricing",
+ "report",
+ "editor",
+ "trading"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Pricing",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "target",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Start trial",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "cover",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Chat with us",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "occlusion",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "cover",
+ "overlap",
+ "overlay",
+ "obstruct",
+ "block"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Detect the visual obstruction: Chat with us interferes with Start trial."
+ },
+ "039": {
+ "schemaVersion": 1,
+ "id": "039",
+ "image": "images/039.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "security",
+ "verification",
+ "captcha",
+ "challenge"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Confirm it’s you",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "label",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Select all traffic lights",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Verify",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "captcha-signal",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "any",
+ "values": [
+ "CAPTCHA",
+ "challenge",
+ "verification",
+ "Challenge not completed"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "security-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "verification",
+ "challenge",
+ "CAPTCHA",
+ "disabled"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Identify the synthetic challenge and disabled state; do not solve or invent unreadable characters."
+ },
+ "040": {
+ "schemaVersion": 1,
+ "id": "040",
+ "image": "images/040.png",
+ "threshold": 0.74,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "account",
+ "label",
+ "receipt",
+ "contract",
+ "telemetry",
+ "document"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Shipment label",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "visible-fragment",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "ZX-81",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "unknown",
+ "dimension": "unknowns",
+ "section": 6,
+ "kind": "any",
+ "values": [
+ "unknown",
+ "unreadable",
+ "blurred",
+ "smudged",
+ "redacted",
+ "cannot read"
+ ],
+ "weight": 3,
+ "critical": true
+ },
+ {
+ "id": "no-guess",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "ZX-81"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Transcribe only ZX-81 and explicitly say last 3 characters blurred."
+ },
+ "041": {
+ "schemaVersion": 1,
+ "id": "041",
+ "image": "images/041.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "sign in",
+ "login",
+ "identity verification",
+ "session expired"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Administrator login",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Log in",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "email-input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Email",
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "incorrect",
+ "password"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "status",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Incorrect password",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Identify the Administrator login authentication surface, quote Log in, and report incorrect password."
+ },
+ "042": {
+ "schemaVersion": 1,
+ "id": "042",
+ "image": "images/042.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "search",
+ "results"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Research library",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "tab",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Articles",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "count",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "1,248 results",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "query",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Search",
+ "urban heat island"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "chip",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Since 2024",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "notice",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Sorted by relevance",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Read the active query urban heat island, selected tab Articles, and exact count 1,248 results."
+ },
+ "043": {
+ "schemaVersion": 1,
+ "id": "043",
+ "image": "images/043.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "checkout",
+ "cart",
+ "order",
+ "purchase"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Review order",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "item",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Travel backpack",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "total",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "€132.90",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Pay now",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "banner",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Card ending 1842 was declined",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "blocked",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "card",
+ "ending",
+ "1842",
+ "declined"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Quote €132.90 and Pay now; correctly report whether the action is disabled and why."
+ },
+ "044": {
+ "schemaVersion": 1,
+ "id": "044",
+ "image": "images/044.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "form",
+ "profile",
+ "shipping",
+ "invite",
+ "dataset",
+ "tax"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Invite teammate",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Send invite",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Work email",
+ "lee@sample",
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "error",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Email domain is incomplete",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "invalid",
+ "required",
+ "incomplete",
+ "missing",
+ "format"
+ ],
+ "weight": 1,
+ "critical": true
+ }
+ ],
+ "successRubric": "Name Work email, preserve its current value and placeholder, and quote Email domain is incomplete."
+ },
+ "045": {
+ "schemaVersion": 1,
+ "id": "045",
+ "image": "images/045.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "api keys",
+ "dialog",
+ "modal"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "title",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Rotate production key?",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "primary",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Rotate key",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "secondary",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Not now",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "modal-signal",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "all",
+ "values": [
+ "modal",
+ "Existing clients will stop working immediately."
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "overlay-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "modal",
+ "overlay",
+ "dialog"
+ ],
+ "weight": 1,
+ "critical": true
+ }
+ ],
+ "successRubric": "Recognize a blocking modal, quote Rotate production key?, Rotate key, and Not now."
+ },
+ "046": {
+ "schemaVersion": 1,
+ "id": "046",
+ "image": "images/046.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "upload center",
+ "settings",
+ "upload",
+ "export"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Upload center",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Retry",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "toast",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Upload failed",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "detail",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Network connection lost",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Quote Upload failed and Network connection lost, and classify the visible state."
+ },
+ "047": {
+ "schemaVersion": 1,
+ "id": "047",
+ "image": "images/047.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "loading",
+ "processing",
+ "reconnecting",
+ "import"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Processing 12 files",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "spinner",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "any",
+ "values": [
+ "loading",
+ "spinner",
+ "processing",
+ "reconnecting",
+ "progress"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "wait-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "loading",
+ "wait",
+ "processing",
+ "reconnecting",
+ "not ready"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "detail",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "File 7 of 12",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "progress",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "58%",
+ "weight": 2,
+ "critical": false,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Report the loading state and preserve 58% without inventing completion."
+ },
+ "048": {
+ "schemaVersion": 1,
+ "id": "048",
+ "image": "images/048.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "privacy",
+ "cookie",
+ "consent",
+ "video"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "title",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Your privacy",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "choice-1",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Allow selected",
+ "weight": 1,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "choice-2",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Continue without accepting",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "consent-signal",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "any",
+ "values": [
+ "cookie",
+ "consent",
+ "privacy",
+ "overlay",
+ "banner"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "consent-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "cookie",
+ "consent",
+ "overlay",
+ "blocked"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Recognize the consent UI and quote Allow selected / Continue without accepting."
+ },
+ "049": {
+ "schemaVersion": 1,
+ "id": "049",
+ "image": "images/049.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "dashboard",
+ "overview",
+ "system health",
+ "incident"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "System health",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "target-metric",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "all",
+ "values": [
+ "Errors",
+ "17"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "alert",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Error rate is above target",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Read Errors = 17 and report the alert without guessing unknown values."
+ },
+ "050": {
+ "schemaVersion": 1,
+ "id": "050",
+ "image": "images/050.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "chart",
+ "dashboard",
+ "latency by region"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Latency by region",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "highest-label",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "APAC"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "highest-value",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "240",
+ "240 ms"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Use the rendered bars, not source order: APAC is tallest and is labeled 240 ms."
+ },
+ "051": {
+ "schemaVersion": 1,
+ "id": "051",
+ "image": "images/051.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "table",
+ "orders",
+ "members",
+ "deployment",
+ "invoice",
+ "audit"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Deployment history",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "target-row",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "v4.8.2",
+ "Rolled back"
+ ],
+ "weight": 3,
+ "critical": true
+ }
+ ],
+ "successRubric": "Keep row association intact: v4.8.2 — Rolled back."
+ },
+ "052": {
+ "schemaVersion": 1,
+ "id": "052",
+ "image": "images/052.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "email",
+ "compose",
+ "message"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "New Message",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Send",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "to-input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "To",
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "subject-input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Subject",
+ "Invoice 8832"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "status",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Attachment scanning…",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Report the compose fields exactly and surface Attachment scanning…."
+ },
+ "053": {
+ "schemaVersion": 1,
+ "id": "053",
+ "image": "images/053.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "board",
+ "kanban",
+ "project",
+ "sprint",
+ "incident"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Launch plan",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "card-column",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "Legal review",
+ "Blocked"
+ ],
+ "weight": 3,
+ "critical": true
+ }
+ ],
+ "successRubric": "Preserve the spatial association: Legal review is in Blocked."
+ },
+ "054": {
+ "schemaVersion": 1,
+ "id": "054",
+ "image": "images/054.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "calendar",
+ "schedule"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Conference schedule",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "selected-event",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "27",
+ "Vision benchmarks",
+ "16:15–17:00"
+ ],
+ "weight": 3,
+ "critical": true
+ }
+ ],
+ "successRubric": "Associate day 27 with Vision benchmarks at 16:15–17:00."
+ },
+ "055": {
+ "schemaVersion": 1,
+ "id": "055",
+ "image": "images/055.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "map",
+ "tracking",
+ "route",
+ "hotel",
+ "café"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Delivery tracking",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "selection",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "Courier",
+ "2 stops away"
+ ],
+ "weight": 3,
+ "critical": true
+ },
+ {
+ "id": "pin-count",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "4",
+ "four"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Read the selected Courier card, its 2 stops away detail, and 4 pins."
+ },
+ "056": {
+ "schemaVersion": 1,
+ "id": "056",
+ "image": "images/056.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "gallery",
+ "archive",
+ "collection",
+ "comparison"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Night scenes",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "caption",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "After dusk",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "badge",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Etching",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "visual-content",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "night",
+ "dark",
+ "dusk"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Combine the rendered image with its UI labels; do not infer a place or date beyond After dusk."
+ },
+ "057": {
+ "schemaVersion": 1,
+ "id": "057",
+ "image": "images/057.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "sign",
+ "payment",
+ "order",
+ "security",
+ "login",
+ "checkout",
+ "settings"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "お支払い",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "支払う",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "カード番号",
+ "•••• 4242"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "status",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "有効期限を確認してください",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Preserve Unicode exactly: お支払い; カード番号; 支払う; 有効期限を確認してください."
+ },
+ "058": {
+ "schemaVersion": 1,
+ "id": "058",
+ "image": "images/058.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "account",
+ "pricing",
+ "report",
+ "editor",
+ "trading"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Report builder",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "target",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Export CSV",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "cover",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Beta",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "occlusion",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "cover",
+ "overlap",
+ "overlay",
+ "obstruct",
+ "block"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Detect the visual obstruction: Beta interferes with Export CSV."
+ },
+ "059": {
+ "schemaVersion": 1,
+ "id": "059",
+ "image": "images/059.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "security",
+ "verification",
+ "captcha",
+ "challenge"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Unusual activity detected",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "label",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Hold the button for 3 seconds",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Try again",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "captcha-signal",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "any",
+ "values": [
+ "CAPTCHA",
+ "challenge",
+ "verification",
+ "Verification required"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "security-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "verification",
+ "challenge",
+ "CAPTCHA",
+ "disabled"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Identify the synthetic challenge and disabled state; do not solve or invent unreadable characters."
+ },
+ "060": {
+ "schemaVersion": 1,
+ "id": "060",
+ "image": "images/060.png",
+ "threshold": 0.74,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "account",
+ "label",
+ "receipt",
+ "contract",
+ "telemetry",
+ "document"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Scanned receipt",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "visible-fragment",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "$4?.90",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "unknown",
+ "dimension": "unknowns",
+ "section": 6,
+ "kind": "any",
+ "values": [
+ "unknown",
+ "unreadable",
+ "blurred",
+ "smudged",
+ "redacted",
+ "cannot read"
+ ],
+ "weight": 3,
+ "critical": true
+ },
+ {
+ "id": "no-guess",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "$4?.90"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Transcribe only $4?.90 and explicitly say one digit is smudged."
+ },
+ "061": {
+ "schemaVersion": 1,
+ "id": "061",
+ "image": "images/061.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "sign in",
+ "login",
+ "identity verification",
+ "session expired"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Verify your identity",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Verify",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "email-input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Email",
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "incomplete",
+ "verification",
+ "code"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "status",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Code expires in 01:42",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "disabled",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "any",
+ "values": [
+ "disabled",
+ "unavailable"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Identify the Verify your identity authentication surface, quote Verify, and report incomplete verification code."
+ },
+ "062": {
+ "schemaVersion": 1,
+ "id": "062",
+ "image": "images/062.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "search",
+ "results"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Global search",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "tab",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Dosyalar",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "count",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "7 sonuç",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "query",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Search",
+ "fatura"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "chip",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Son 30 gün",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "notice",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "2 filters active",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Read the active query fatura, selected tab Dosyalar, and exact count 7 sonuç."
+ },
+ "063": {
+ "schemaVersion": 1,
+ "id": "063",
+ "image": "images/063.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "checkout",
+ "cart",
+ "order",
+ "purchase"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Confirm purchase",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "item",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Kulaklık",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "total",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "₺4.799,00",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Siparişi onayla",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "banner",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "3D Secure verification required",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Quote ₺4.799,00 and Siparişi onayla; correctly report whether the action is disabled and why."
+ },
+ "064": {
+ "schemaVersion": 1,
+ "id": "064",
+ "image": "images/064.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "form",
+ "profile",
+ "shipping",
+ "invite",
+ "dataset",
+ "tax"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Vergi bilgileri",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Kaydet",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Vergi kimlik no",
+ "348 91 2",
+ "10 hane"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "error",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "3 hane eksik",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "invalid",
+ "required",
+ "incomplete",
+ "missing",
+ "format"
+ ],
+ "weight": 1,
+ "critical": true
+ }
+ ],
+ "successRubric": "Name Vergi kimlik no, preserve its current value and placeholder, and quote 3 hane eksik."
+ },
+ "065": {
+ "schemaVersion": 1,
+ "id": "065",
+ "image": "images/065.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "deployments",
+ "dialog",
+ "modal"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "title",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Production is locked",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "primary",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Request approval",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "secondary",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Close",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "modal-signal",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "all",
+ "values": [
+ "modal",
+ "Approval from an owner is required."
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "overlay-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "modal",
+ "overlay",
+ "dialog"
+ ],
+ "weight": 1,
+ "critical": true
+ }
+ ],
+ "successRubric": "Recognize a blocking modal, quote Production is locked, Request approval, and Close."
+ },
+ "066": {
+ "schemaVersion": 1,
+ "id": "066",
+ "image": "images/066.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "branch protection",
+ "settings",
+ "upload",
+ "export"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Branch protection",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "View warnings",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "toast",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Rule saved with warnings",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "detail",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "2 checks are missing",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Quote Rule saved with warnings and 2 checks are missing, and classify the visible state."
+ },
+ "067": {
+ "schemaVersion": 1,
+ "id": "067",
+ "image": "images/067.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "loading",
+ "processing",
+ "reconnecting",
+ "import"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Reconnecting…",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "spinner",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "any",
+ "values": [
+ "loading",
+ "spinner",
+ "processing",
+ "reconnecting",
+ "progress"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "wait-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "loading",
+ "wait",
+ "processing",
+ "reconnecting",
+ "not ready"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "detail",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Attempt 3 of 5",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Report the loading state and preserve Reconnecting… without inventing completion."
+ },
+ "068": {
+ "schemaVersion": 1,
+ "id": "068",
+ "image": "images/068.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "privacy",
+ "cookie",
+ "consent",
+ "video"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "title",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Çerez tercihleri",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "choice-1",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Tümünü kabul et",
+ "weight": 1,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "choice-2",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Yalnızca gerekli",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "consent-signal",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "any",
+ "values": [
+ "cookie",
+ "consent",
+ "privacy",
+ "overlay",
+ "banner"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "consent-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "cookie",
+ "consent",
+ "overlay",
+ "blocked"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Recognize the consent UI and quote Tümünü kabul et / Yalnızca gerekli."
+ },
+ "069": {
+ "schemaVersion": 1,
+ "id": "069",
+ "image": "images/069.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "dashboard",
+ "overview",
+ "system health",
+ "incident"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Bölgesel performans",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "target-metric",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "all",
+ "values": [
+ "İzmir",
+ "₺44,2K"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "alert",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "İzmir hedefin %8 altında",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Read İzmir = ₺44,2K and report the alert without guessing unknown values."
+ },
+ "070": {
+ "schemaVersion": 1,
+ "id": "070",
+ "image": "images/070.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "chart",
+ "dashboard",
+ "dönüşüm hunisi"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Dönüşüm hunisi",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "highest-label",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "Satın alma"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "highest-value",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "100",
+ "29%"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Use the rendered bars, not source order: Satın alma is tallest and is labeled 29%."
+ },
+ "071": {
+ "schemaVersion": 1,
+ "id": "071",
+ "image": "images/071.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "table",
+ "orders",
+ "members",
+ "deployment",
+ "invoice",
+ "audit"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Fatura listesi",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "target-row",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "TR-8832",
+ "Gecikmiş"
+ ],
+ "weight": 3,
+ "critical": true
+ }
+ ],
+ "successRubric": "Keep row association intact: TR-8832 — Gecikmiş."
+ },
+ "072": {
+ "schemaVersion": 1,
+ "id": "072",
+ "image": "images/072.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "email",
+ "compose",
+ "message"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Yeni ileti",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Gönder",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "to-input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "To",
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "subject-input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Subject",
+ "Cuma toplantısı"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "status",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Çevrimdışı — taslak yerel kaydedildi",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Report the compose fields exactly and surface Çevrimdışı — taslak yerel kaydedildi."
+ },
+ "073": {
+ "schemaVersion": 1,
+ "id": "073",
+ "image": "images/073.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "board",
+ "kanban",
+ "project",
+ "sprint",
+ "incident"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "İçerik takvimi",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "card-column",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "Model karşılaştırması",
+ "İncelemede"
+ ],
+ "weight": 3,
+ "critical": true
+ }
+ ],
+ "successRubric": "Preserve the spatial association: Model karşılaştırması is in İncelemede."
+ },
+ "074": {
+ "schemaVersion": 1,
+ "id": "074",
+ "image": "images/074.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "calendar",
+ "schedule"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Ağustos 2026",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "selected-event",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "29",
+ "Ürün demosu",
+ "09:45"
+ ],
+ "weight": 3,
+ "critical": true
+ }
+ ],
+ "successRubric": "Associate day 29 with Ürün demosu at 09:45."
+ },
+ "075": {
+ "schemaVersion": 1,
+ "id": "075",
+ "image": "images/075.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "map",
+ "tracking",
+ "route",
+ "hotel",
+ "café"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Tren rotası",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "selection",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "Ankara Garı",
+ "18:42 varış"
+ ],
+ "weight": 3,
+ "critical": true
+ },
+ {
+ "id": "pin-count",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "5",
+ "five"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Read the selected Ankara Garı card, its 18:42 varış detail, and 5 pins."
+ },
+ "076": {
+ "schemaVersion": 1,
+ "id": "076",
+ "image": "images/076.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "gallery",
+ "archive",
+ "collection",
+ "comparison"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Sanat arşivi",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "caption",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Tangiers street scene",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "badge",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Oil painting",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "visual-content",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "Tangiers street scene"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Combine the rendered image with its UI labels; do not infer a place or date beyond Tangiers street scene."
+ },
+ "077": {
+ "schemaVersion": 1,
+ "id": "077",
+ "image": "images/077.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "sign",
+ "payment",
+ "order",
+ "security",
+ "login",
+ "checkout",
+ "settings"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Sipariş ayrıntıları",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Güncelle",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Teslimat notu",
+ "Kapıya bırak"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "status",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Adres doğrulanamadı",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Preserve Unicode exactly: Sipariş ayrıntıları; Teslimat notu; Güncelle; Adres doğrulanamadı."
+ },
+ "078": {
+ "schemaVersion": 1,
+ "id": "078",
+ "image": "images/078.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "account",
+ "pricing",
+ "report",
+ "editor",
+ "trading"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Canvas editor",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "target",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Publish",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "cover",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Connection lost",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "occlusion",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "cover",
+ "overlap",
+ "overlay",
+ "obstruct",
+ "block"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Detect the visual obstruction: Connection lost interferes with Publish."
+ },
+ "079": {
+ "schemaVersion": 1,
+ "id": "079",
+ "image": "images/079.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "security",
+ "verification",
+ "captcha",
+ "challenge"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Güvenlik doğrulaması",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "label",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Bulmacayı tamamlayın",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Doğrula",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "captcha-signal",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "any",
+ "values": [
+ "CAPTCHA",
+ "challenge",
+ "verification",
+ "2 denemeden 1’i kaldı"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "security-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "verification",
+ "challenge",
+ "CAPTCHA",
+ "disabled"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Identify the synthetic challenge and disabled state; do not solve or invent unreadable characters."
+ },
+ "080": {
+ "schemaVersion": 1,
+ "id": "080",
+ "image": "images/080.png",
+ "threshold": 0.74,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "account",
+ "label",
+ "receipt",
+ "contract",
+ "telemetry",
+ "document"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Redacted contract",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "visible-fragment",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Client: ███████",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "unknown",
+ "dimension": "unknowns",
+ "section": 6,
+ "kind": "any",
+ "values": [
+ "unknown",
+ "unreadable",
+ "blurred",
+ "smudged",
+ "redacted",
+ "cannot read"
+ ],
+ "weight": 3,
+ "critical": true
+ },
+ {
+ "id": "no-guess",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "Client: ███████"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Transcribe only Client: ███████ and explicitly say client name is redacted."
+ },
+ "081": {
+ "schemaVersion": 1,
+ "id": "081",
+ "image": "images/081.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "sign in",
+ "login",
+ "identity verification",
+ "session expired"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Session expired",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Sign in again",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "email-input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Email",
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "rate",
+ "limit"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "status",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Too many attempts. Try again in 14:32",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "disabled",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "any",
+ "values": [
+ "disabled",
+ "unavailable"
+ ],
+ "weight": 1,
+ "critical": false
+ }
+ ],
+ "successRubric": "Identify the Session expired authentication surface, quote Sign in again, and report rate limit."
+ },
+ "082": {
+ "schemaVersion": 1,
+ "id": "082",
+ "image": "images/082.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "search",
+ "results"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Case-sensitive search",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "tab",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Exact match",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "count",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "0 results",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "query",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Search",
+ "Orion-7B"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "chip",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Archived",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "notice",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "No matches for “Orion-7B”",
+ "weight": 1,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Read the active query Orion-7B, selected tab Exact match, and exact count 0 results."
+ },
+ "083": {
+ "schemaVersion": 1,
+ "id": "083",
+ "image": "images/083.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "checkout",
+ "cart",
+ "order",
+ "purchase"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Order summary",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "item",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Studio monitor pair",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "total",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "$1,204.08",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Submit payment",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "banner",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Price changed from $1,199.00",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "blocked",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "price",
+ "changed"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Quote $1,204.08 and Submit payment; correctly report whether the action is disabled and why."
+ },
+ "084": {
+ "schemaVersion": 1,
+ "id": "084",
+ "image": "images/084.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "form",
+ "profile",
+ "shipping",
+ "invite",
+ "dataset",
+ "tax"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Publish dataset",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Publish",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Release tag",
+ "v2..1",
+ "v1.0.0"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "error",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Use semantic version format",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "invalid",
+ "required",
+ "incomplete",
+ "missing",
+ "format"
+ ],
+ "weight": 1,
+ "critical": true
+ }
+ ],
+ "successRubric": "Name Release tag, preserve its current value and placeholder, and quote Use semantic version format."
+ },
+ "085": {
+ "schemaVersion": 1,
+ "id": "085",
+ "image": "images/085.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "bank transfer",
+ "dialog",
+ "modal"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "title",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Confirm transfer of $8,740.00?",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "primary",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Confirm transfer",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "secondary",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Review details",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "modal-signal",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "all",
+ "values": [
+ "modal",
+ "Recipient: NOVA SUPPLY •••• 8841"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "overlay-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "modal",
+ "overlay",
+ "dialog"
+ ],
+ "weight": 1,
+ "critical": true
+ }
+ ],
+ "successRubric": "Recognize a blocking modal, quote Confirm transfer of $8,740.00?, Confirm transfer, and Review details."
+ },
+ "086": {
+ "schemaVersion": 1,
+ "id": "086",
+ "image": "images/086.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "data export",
+ "settings",
+ "upload",
+ "export"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Data export",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Cancel export",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "toast",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Export queued",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "detail",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Position 14 of 32",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Quote Export queued and Position 14 of 32, and classify the visible state."
+ },
+ "087": {
+ "schemaVersion": 1,
+ "id": "087",
+ "image": "images/087.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "loading",
+ "processing",
+ "reconnecting",
+ "import"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Finalizing import",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "spinner",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "any",
+ "values": [
+ "loading",
+ "spinner",
+ "processing",
+ "reconnecting",
+ "progress"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "wait-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "loading",
+ "wait",
+ "processing",
+ "reconnecting",
+ "not ready"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "detail",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Do not close this tab",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "progress",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "99%",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Report the loading state and preserve 99% without inventing completion."
+ },
+ "088": {
+ "schemaVersion": 1,
+ "id": "088",
+ "image": "images/088.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "privacy",
+ "cookie",
+ "consent",
+ "video"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "title",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Consent required",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "choice-1",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Allow media cookies",
+ "weight": 1,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "choice-2",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Keep blocked",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "consent-signal",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "any",
+ "values": [
+ "cookie",
+ "consent",
+ "privacy",
+ "overlay",
+ "banner"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "consent-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "cookie",
+ "consent",
+ "overlay",
+ "blocked"
+ ],
+ "weight": 1,
+ "critical": true
+ }
+ ],
+ "successRubric": "Recognize the consent UI and quote Allow media cookies / Keep blocked."
+ },
+ "089": {
+ "schemaVersion": 1,
+ "id": "089",
+ "image": "images/089.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "dashboard",
+ "overview",
+ "system health",
+ "incident"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Incident command",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "target-metric",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "all",
+ "values": [
+ "ETA",
+ "Unknown"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "alert",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Payments degraded in EU-West",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "unknown",
+ "dimension": "unknowns",
+ "section": 6,
+ "kind": "any",
+ "values": [
+ "ETA",
+ "unknown"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Read ETA = Unknown and report the alert without guessing unknown values."
+ },
+ "090": {
+ "schemaVersion": 1,
+ "id": "090",
+ "image": "images/090.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "chart",
+ "dashboard",
+ "forecast vs actual"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Forecast vs actual",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "highest-label",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "May"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "highest-value",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "93",
+ "Actual 93; forecast 88"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Use the rendered bars, not source order: May is tallest and is labeled Actual 93; forecast 88."
+ },
+ "091": {
+ "schemaVersion": 1,
+ "id": "091",
+ "image": "images/091.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "table",
+ "orders",
+ "members",
+ "deployment",
+ "invoice",
+ "audit"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Access audit",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "target-row",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "svc-legacy",
+ "org:admin",
+ "Critical"
+ ],
+ "weight": 3,
+ "critical": true
+ }
+ ],
+ "successRubric": "Keep row association intact: svc-legacy — org:admin — Critical."
+ },
+ "092": {
+ "schemaVersion": 1,
+ "id": "092",
+ "image": "images/092.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "email",
+ "compose",
+ "message"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Confidential message",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Send anyway",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "to-input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "To",
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "subject-input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "Subject",
+ "Q3 forecast"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "status",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "External recipient warning",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "external-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "external",
+ "warning",
+ "recipient"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Report the compose fields exactly and surface External recipient warning."
+ },
+ "093": {
+ "schemaVersion": 1,
+ "id": "093",
+ "image": "images/093.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "board",
+ "kanban",
+ "project",
+ "sprint",
+ "incident"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Incident board",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "card-column",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "Retry storm",
+ "Mitigating"
+ ],
+ "weight": 3,
+ "critical": true
+ }
+ ],
+ "successRubric": "Preserve the spatial association: Retry storm is in Mitigating."
+ },
+ "094": {
+ "schemaVersion": 1,
+ "id": "094",
+ "image": "images/094.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "calendar",
+ "schedule"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Release calendar",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "selected-event",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "31",
+ "Production freeze",
+ "23:30 UTC"
+ ],
+ "weight": 3,
+ "critical": true
+ },
+ {
+ "id": "conflict",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Overlaps with Database migration",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Associate day 31 with Production freeze at 23:30 UTC."
+ },
+ "095": {
+ "schemaVersion": 1,
+ "id": "095",
+ "image": "images/095.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "map",
+ "tracking",
+ "route",
+ "hotel",
+ "café"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Evacuation map",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "selection",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "all",
+ "values": [
+ "Route B",
+ "Bridge closed"
+ ],
+ "weight": 3,
+ "critical": true
+ },
+ {
+ "id": "pin-count",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "6",
+ "six"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "route-alert",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Use Route C instead",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Read the selected Route B card, its Bridge closed detail, and 6 pins."
+ },
+ "096": {
+ "schemaVersion": 1,
+ "id": "096",
+ "image": "images/096.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "gallery",
+ "archive",
+ "collection",
+ "comparison"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Visual comparison",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "caption",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Modern / Historic",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "badge",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Compare",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "visual-content",
+ "dimension": "visual_reasoning",
+ "section": null,
+ "kind": "any",
+ "values": [
+ "right",
+ "snow"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Combine the rendered image with its UI labels; do not infer a place or date beyond Modern / Historic."
+ },
+ "097": {
+ "schemaVersion": 1,
+ "id": "097",
+ "image": "images/097.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "sign",
+ "payment",
+ "order",
+ "security",
+ "login",
+ "checkout",
+ "settings"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "إعدادات الأمان",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "تأكيد",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "input",
+ "dimension": "inputs",
+ "section": 3,
+ "kind": "all",
+ "values": [
+ "رمز التحقق",
+ "19_ 4_"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "status",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "تنتهي صلاحية الرمز خلال ٣٨ ثانية",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ }
+ ],
+ "successRubric": "Preserve Unicode exactly: إعدادات الأمان; رمز التحقق; تأكيد; تنتهي صلاحية الرمز خلال ٣٨ ثانية."
+ },
+ "098": {
+ "schemaVersion": 1,
+ "id": "098",
+ "image": "images/098.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "account",
+ "pricing",
+ "report",
+ "editor",
+ "trading"
+ ],
+ "weight": 1,
+ "critical": false
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Trading panel",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "target",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Review order",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "cover",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "literal",
+ "value": "Market closed",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "occlusion",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "cover",
+ "overlap",
+ "overlay",
+ "obstruct",
+ "block"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Detect the visual obstruction: Market closed interferes with Review order."
+ },
+ "099": {
+ "schemaVersion": 1,
+ "id": "099",
+ "image": "images/099.png",
+ "threshold": 0.72,
+ "checks": [
+ {
+ "id": "purpose",
+ "dimension": "page_purpose",
+ "section": 1,
+ "kind": "any",
+ "values": [
+ "security",
+ "verification",
+ "captcha",
+ "challenge"
+ ],
+ "weight": 1,
+ "critical": true
+ },
+ {
+ "id": "heading",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Additional verification",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "label",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Image challenge",
+ "weight": 2,
+ "critical": true,
+ "caseSensitive": true
+ },
+ {
+ "id": "button",
+ "dimension": "visible_text",
+ "section": 2,
+ "kind": "literal",
+ "value": "Submit",
+ "weight": 1,
+ "critical": false,
+ "caseSensitive": true
+ },
+ {
+ "id": "captcha-signal",
+ "dimension": "state_signals",
+ "section": 4,
+ "kind": "any",
+ "values": [
+ "CAPTCHA",
+ "challenge",
+ "verification",
+ "Characters are unreadable"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "security-blocker",
+ "dimension": "blockers",
+ "section": 5,
+ "kind": "any",
+ "values": [
+ "verification",
+ "challenge",
+ "CAPTCHA",
+ "disabled"
+ ],
+ "weight": 2,
+ "critical": true
+ },
+ {
+ "id": "unknown-code",
+ "dimension": "unknowns",
+ "section": 6,
+ "kind": "any",
+ "values": [
+ "unreadable",
+ "cannot read",
+ "unknown"
+ ],
+ "weight": 2,
+ "critical": true
+ }
+ ],
+ "successRubric": "Identify the synthetic challenge and disabled state; do not solve or invent unreadable characters."
+ }
+}
diff --git a/test/vision/report-ling30-budget.mjs b/test/vision/report-ling30-budget.mjs
new file mode 100644
index 000000000..67edbaadd
--- /dev/null
+++ b/test/vision/report-ling30-budget.mjs
@@ -0,0 +1,87 @@
+#!/usr/bin/env node
+import assert from 'node:assert/strict';
+import { createHash } from 'node:crypto';
+import { execFileSync } from 'node:child_process';
+import { readFile, readdir, mkdir, writeFile } from 'node:fs/promises';
+import { isDeepStrictEqual } from 'node:util';
+import { scoreVisionResponse } from './lib/score.mjs';
+import { VISION_SYSTEM_PROMPT, PRODUCTION_USER_TEXT, REQUEST_DEFAULTS } from './prompt.mjs';
+
+const OUT = 'test/vision/analysis/2026-10-07-ling30-budget';
+const REV = 'd25be94f4981c2fe5ff1d220f8b698ee72e88296';
+const ROOT = 'test/vision';
+const historicalOnly = process.argv.includes('--historical-only');
+const sha = value => createHash('sha256').update(value).digest('hex');
+const json = async path => JSON.parse(await readFile(path, 'utf8'));
+const ids = Array.from({ length: 100 }, (_, i) => String(i + 1).padStart(3, '0'));
+const files = ['prompt.mjs', 'lib/score.mjs', ...ids.flatMap(id => [`questions/${id}.json`, `expected/${id}.json`])];
+const sourceHashes = {};
+for (const file of files) {
+ const bytes = await readFile(`${ROOT}/${file}`);
+ const old = execFileSync('git', ['show', `${REV}:${ROOT}/${file}`]);
+ assert.equal(sha(bytes), sha(old), `August source changed: ${file}`);
+ sourceHashes[file] = sha(bytes);
+}
+const rubrics = Object.fromEntries(await Promise.all(ids.map(async id => [id, await json(`${ROOT}/expected/${id}.json`)])));
+const manifest = { sourceRevision: REV, sourceHashes, images: {} };
+for (const id of ids) manifest.images[id] = sha(await readFile(`${ROOT}/images/${id}.png`));
+const dirs = (await readdir(`${ROOT}/results`)).filter(name => name.startsWith('2026-08-22-openrouter-full_qwen_')).sort();
+assert.equal(dirs.length, 6);
+if (!historicalOnly) dirs.push('2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production');
+const rows = [];
+const casesByModel = {};
+for (const dir of dirs) {
+ const source = `${ROOT}/results/${dir}`;
+ const cases = await Promise.all(ids.map(id => json(`${source}/${id}.json`)));
+ assert.equal(cases.filter(c => c.error).length, 0, `${dir} has failed requests; archive and recover them before reporting a complete run`);
+ for (const c of cases) {
+ assert.equal(c.promptMode, 'production');
+ assert.equal(c.image.sha256, manifest.images[c.id], `Image mismatch: ${dir}/${c.id}`);
+ assert.equal(c.request.systemPrompt, VISION_SYSTEM_PROMPT);
+ assert.equal(c.request.userText, PRODUCTION_USER_TEXT);
+ assert.equal(c.request.foldSystem, false);
+ assert.equal(c.request.temperature, REQUEST_DEFAULTS.temperature);
+ assert.equal(c.request.maxTokens, REQUEST_DEFAULTS.maxTokens);
+ assert.deepEqual(c.request.chatTemplateKwargs, REQUEST_DEFAULTS.chatTemplateKwargs);
+ assert.deepEqual(c.question, await json(`${ROOT}/questions/${c.id}.json`));
+ const score = scoreVisionResponse({ content: c.response.content, expected: rubrics[c.id] });
+ assert(isDeepStrictEqual(score, c.score), `Saved grade changed: ${dir}/${c.id}`);
+ }
+ const sum = key => cases.reduce((n, c) => n + (c.response.usage?.[key] || 0), 0);
+ const group = key => Object.fromEntries([...new Set(cases.map(c => key(c)))].map(value => {
+ const group = cases.filter(c => key(c) === value);
+ return [value, { cases: group.length, passes: group.filter(c => c.score.success).length, meanScore: group.reduce((n, c) => n + c.score.ratio, 0) / group.length }];
+ }));
+ const latency = cases.map(c => c.latencyMs.total).sort((a, b) => a - b);
+ const row = {
+ model: cases[0].model, source, date: cases[0].createdAt.slice(0, 10), cases: cases.length, errors: 0,
+ passes: cases.filter(c => c.score.success).length,
+ meanScore: cases.reduce((n, c) => n + c.score.ratio, 0) / cases.length,
+ meanLatencyMs: latency.reduce((a, b) => a + b, 0) / latency.length,
+ medianLatencyMs: (latency[49] + latency[50]) / 2, p95LatencyMs: latency[94],
+ costRecords: cases.filter(c => typeof c.response.usage?.cost === 'number').length,
+ costUsd: sum('cost'), promptTokens: sum('prompt_tokens'), completionTokens: sum('completion_tokens'),
+ cachedTokens: cases.reduce((n, c) => n + (c.response.usage?.prompt_tokens_details?.cached_tokens || 0), 0),
+ reasoningTokens: cases.reduce((n, c) => n + (c.response.usage?.completion_tokens_details?.reasoning_tokens || 0), 0),
+ sixSections: cases.filter(c => c.score.sectionCount === 6).length,
+ emptyOutputs: cases.filter(c => !c.response.content.trim()).length,
+ lengthLimited: cases.filter(c => c.response.finishReason === 'length').length,
+ providers: group(c => c.response.provider || 'not recorded'),
+ byDifficulty: group(c => c.question.difficulty.slug), byCategory: group(c => c.question.category),
+ };
+ if (dir.startsWith('2026-08-22')) {
+ const summary = await json(`${source}/summary.json`);
+ assert.equal(row.passes, summary.overall.successes);
+ assert(Math.abs(row.meanScore - summary.overall.meanScore) < 1e-12);
+ assert.equal(row.meanLatencyMs, summary.overall.meanLatencyMs);
+ }
+ rows.push(row);
+ casesByModel[row.model] = cases.map(c => ({ id: c.id, category: c.question.category, difficulty: c.question.difficulty.slug, pass: c.score.success, ratio: c.score.ratio, criticalFailures: c.score.criticalFailures }));
+}
+const regression = { verifiedHistoricalModels: 6, verifiedHistoricalResponses: 600, sourceRevision: REV, promptImageAndQuestionMatches: true, savedScoresExactlyReproduced: true };
+const report = { date: '2026-10-07', protocol: 'production screenshot description, temperature 0, max_tokens 800, unchanged August scorer', regression, rows, casesByModel };
+await mkdir(OUT, { recursive: true });
+for (const [name, value] of Object.entries({ 'comparison.json': report, 'regression.json': regression, 'manifest.json': manifest, 'rubrics.json': rubrics })) {
+ await writeFile(`${OUT}/${name}`, JSON.stringify(value, null, 2) + '\n');
+}
+console.log(JSON.stringify(rows.map(({ model, passes, meanScore, meanLatencyMs, costUsd, sixSections }) => ({ model, passes, meanScore, meanLatencyMs, costUsd, sixSections })), null, 2));
diff --git a/test/vision/run.mjs b/test/vision/run.mjs
index 2ecd83be9..b242ab5e3 100644
--- a/test/vision/run.mjs
+++ b/test/vision/run.mjs
@@ -139,6 +139,8 @@ async function requestVision(body) {
let reasoning = '';
let usage = {};
let timings = null;
+ let provider = null;
+ let finishReason = null;
if (raw.includes('data: ')) {
for (const line of raw.split(/\r?\n/)) {
if (!line.startsWith('data: ')) continue;
@@ -146,6 +148,9 @@ async function requestVision(body) {
if (!data || data === '[DONE]') continue;
let event;
try { event = JSON.parse(data); } catch { continue; }
+ if (event.error) throw new Error(`Stream error: ${JSON.stringify(event.error)}`);
+ provider = event.provider || provider;
+ finishReason = event?.choices?.[0]?.finish_reason || finishReason;
const delta = event?.choices?.[0]?.delta || {};
content += delta.content || '';
reasoning += delta.reasoning_content || '';
@@ -154,12 +159,15 @@ async function requestVision(body) {
}
} else {
const json = JSON.parse(raw);
+ if (json.error) throw new Error(`Response error: ${JSON.stringify(json.error)}`);
+ provider = json.provider || null;
+ finishReason = json?.choices?.[0]?.finish_reason || null;
content = json?.choices?.[0]?.message?.content || '';
reasoning = json?.choices?.[0]?.message?.reasoning_content || '';
usage = json?.usage || {};
timings = json?.choices?.[0]?.timings || null;
}
- return { content, reasoning, usage, timings, latencyMs: { headers: headerMs, total: Date.now() - startedAt }, status: res.statusCode };
+ return { content, reasoning, usage, timings, provider, finishReason, raw, latencyMs: { headers: headerMs, total: Date.now() - startedAt }, status: res.statusCode };
}
async function runCase(entry) {
@@ -181,7 +189,7 @@ async function runCase(entry) {
question: entry.question,
image: { path: entry.question.image, bytes: bytes.length, sha256: createHash('sha256').update(bytes).digest('hex') },
request: { systemPrompt: VISION_SYSTEM_PROMPT, userText, foldSystem, temperature: body.temperature, maxTokens: body.max_tokens, chatTemplateKwargs: body.chat_template_kwargs },
- response: { status: response.status, content: response.content, reasoningChars: response.reasoning.length, usage: response.usage, timings: response.timings },
+ response: { status: response.status, content: response.content, reasoningChars: response.reasoning.length, usage: response.usage, timings: response.timings, provider: response.provider, finishReason: response.finishReason, raw: response.raw },
latencyMs: response.latencyMs,
score,
error: null,
diff --git a/web/blog/index.html b/web/blog/index.html
index c1f45c83f..5e9a0acb6 100644
--- a/web/blog/index.html
+++ b/web/blog/index.html
@@ -1415,8 +1415,8 @@
</section>
<div class="post-list" id="post-list">
- <a href="/blog/ling31-flash-vs-deepseek-hy3-glm52-laguna-xs" class="post-card" data-search="Ling 3.1 Flash vs DeepSeek, HY3, GLM-5.2, and Laguna XS: free inference, costly retries Ling 3.1 Flash vs DeepSeek, HY3, GLM-5.2, and Laguna XS: free inference, costly retries Ling leads tool-family agreement and beats the four text-only references on ideal-tool count, but trails on exact-action consensus. Its free Novita route required 298 rate-limit responses across the benchmark and diagnostic campaign. Ling 3.1 Flash completes 100 pinned planner cases with 80.7% tool-family consensus, 32.8% exact consensus, 92 valid calls, and zero reported cost, after extensive Novita rate-limit recovery. October 6, 2026 2026-10-06 Ling 3.1 Flash inclusionAI DeepSeek V4 Flash Tencent HY3 GLM-5.2 Poolside Laguna XS 2.1 OpenRouter browser agent benchmark tool calling">
- <div class="post-meta">October 6, 2026 &middot; 7 min read</div>
+ <a href="/blog/ling31-flash-vs-deepseek-hy3-glm52-laguna-xs" class="post-card" data-search="Ling 3.1 Flash vs DeepSeek, HY3, GLM-5.2, and Laguna XS: free inference, costly retries Ling 3.1 Flash vs DeepSeek, HY3, GLM-5.2, and Laguna XS: free inference, costly retries Ling leads tool-family agreement and beats the four text-only references on ideal-tool count, but trails on exact-action consensus. Its free Novita route required 298 rate-limit responses across the benchmark and diagnostic campaign. Ling 3.1 Flash completes 100 pinned planner cases with 80.7% tool-family consensus, 32.8% exact consensus, 92 valid calls, and zero reported cost, after extensive Novita rate-limit recovery. October 6, 2026 2026-10-06 Ling 3.1 Flash inclusionAI DeepSeek V4 Flash Tencent HY3 GLM-5.2 Poolside Laguna XS 2.1 Ling 3.0 Flash VL Nex-N2.5-Pro Nex-N2.5-mini MiniMax M3 OpenRouter browser agent benchmark tool calling">
+ <div class="post-meta">October 6, 2026 &middot; 8 min read</div>
<div class="post-title">Ling 3.1 Flash vs DeepSeek, HY3, GLM-5.2, and Laguna XS: free inference, costly retries</div>
<div class="post-excerpt">Ling leads tool-family agreement and beats the four text-only references on ideal-tool count, but trails on exact-action consensus. Its free Novita route required 298 rate-limit responses across the benchmark and diagnostic campaign.</div>
</a>
diff --git a/web/blog/ling31-flash-vs-deepseek-hy3-glm52-laguna-xs/index.html b/web/blog/ling31-flash-vs-deepseek-hy3-glm52-laguna-xs/index.html
index 309bece86..2c82f36db 100644
--- a/web/blog/ling31-flash-vs-deepseek-hy3-glm52-laguna-xs/index.html
+++ b/web/blog/ling31-flash-vs-deepseek-hy3-glm52-laguna-xs/index.html
@@ -5,7 +5,7 @@
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Ling 3.1 Flash planner benchmark vs DeepSeek, HY3, GLM, and Laguna XS - WebBrain Blog</title>
<meta name="description" content="Ling 3.1 Flash completes 100 pinned planner cases with 80.7% tool-family consensus, 32.8% exact consensus, 92 valid calls, and zero reported cost, after extensive Novita rate-limit recovery.">
- <meta name="keywords" content="Ling 3.1 Flash, inclusionAI, DeepSeek V4 Flash, Tencent HY3, GLM-5.2, Poolside Laguna XS 2.1, OpenRouter, browser agent benchmark, tool calling">
+ <meta name="keywords" content="Ling 3.1 Flash, inclusionAI, DeepSeek V4 Flash, Tencent HY3, GLM-5.2, Poolside Laguna XS 2.1, Ling 3.0 Flash VL, Nex-N2.5-Pro, Nex-N2.5-mini, MiniMax M3, OpenRouter, browser agent benchmark, tool calling">
<meta property="og:title" content="Ling 3.1 Flash in WebBrain&#39;s text-only planner benchmark">
<meta property="og:description" content="A pinned full-tier tool-calling test against DeepSeek V4 Flash, Tencent HY3, GLM-5.2, and Laguna XS 2.1, with measured coverage, latency, and cost.">
<meta property="og:type" content="article">
@@ -523,7 +523,7 @@
</nav>
<article>
- <div class="meta">October 6, 2026 &middot; 7 min read &middot; <a href="/blog">&larr; All posts</a></div>
+ <div class="meta">October 6, 2026 &middot; 8 min read &middot; <a href="/blog">&larr; All posts</a></div>
<h1>Ling 3.1 Flash vs DeepSeek, HY3, GLM-5.2, and Laguna XS: free inference, costly retries</h1>
<p class="lede"><strong>Ling 3.1 Flash completed all 100 planner cases with zero reported inference cost, but getting those responses was difficult.</strong> It leads the expanded fourteen-model pool on <strong>tool-family consensus at 80.7%</strong>, produces <strong>92 schema-valid calls</strong>, and chooses <strong>42 ideal tools</strong>. Its <strong>32.8% exact-action consensus</strong> trails all four text-only references, and its successful requests have a <strong>3.19-second median</strong>. The first pass returned only 15 responses; extensive retries were needed to finish. This is a useful text planner result with a serious serving limitation on the tested free route.</p>
<h2 id="a-new-textonly-route">A new text-only route</h2>
@@ -531,7 +531,7 @@
<p>That makes it an interesting candidate for a browser planner working from accessibility trees and extracted text. This route cannot accept screenshots. The 25B active count describes per-token compute; the full 560B model still needs substantial storage and memory.</p>
<p>Our comparison focuses on the four text-only routes in the <a href="/blog/american-chinese-open-model-frontier-gap-benchmark">American and Chinese open-model benchmark</a>: <strong>DeepSeek V4 Flash 0731, Tencent HY3, GLM-5.2, and Poolside Laguna XS 2.1</strong>. MiniMax M3 remains a useful additional reference for a broader agent, although its tested route also accepts visual input. Every request in this planner suite contains text; no model receives a screenshot or video.</p>
<h2 id="reconstructing-the-matching-test">Reconstructing the matching test</h2>
-<p>This post uses the <strong>August full-tier checkout <code>7182c21f</code></strong>, rather than the May frozen interface from our <a href="/blog/ling30-flash-vl-vs-nex-n25-minimax-m3">Ling 3.0 Flash VL comparison</a>. The older Ling, Nex, and May-frozen MiniMax percentages belong to a different test and cannot be inserted here.</p>
+<p>The main ranking uses the <strong>August full-tier checkout <code>7182c21f</code></strong>. We also include the models from our <a href="/blog/ling30-flash-vl-vs-nex-n25-minimax-m3">Ling 3.0 Flash VL comparison</a> in a separate historical table below. That test uses the May frozen interface, so its percentages do not enter the August ranking or fourteen-model consensus pool.</p>
<p>The August runs omitted request bodies. We reconstructed all 100 messages and their tool schemas from the exact pinned payload builder, site adapters, tool definitions, and questions, then saved a replay snapshot. We also archived the ideal-action rubrics from that checkout. Ling&#39;s new raw case files preserve both its request and the API response. The historical requests&#39; full byte equality cannot be proven because those bodies were never stored.</p>
<p><strong>Only Ling is a new October 6 run.</strong> The comparison models retain their August 2 results. These are matching reconstructed inputs across different serving dates, with potentially different infrastructure, cache state, and prices.</p>
<p>The source audit found <strong>48 Act tools</strong>, correcting the older article&#39;s count of 41. All 100 questions are Act-mode cases, so every new request uses <strong>temperature 0.15</strong> and a <strong>4,096-token output budget</strong>. We send native structured tools through OpenRouter Chat Completions, without a reasoning-effort override or a compatibility prompt. The per-request timeout is 180 seconds.</p>
@@ -560,6 +560,13 @@
<p>The captured route lists zero prompt and completion rates on October 6. The free-price result therefore comes from the offered endpoint rates, while the large reported cache share remains relevant to latency. It does not establish an uncached speed measurement or a future price guarantee. Historical bills in the comparison also reflect their own cache conditions and August prices.</p>
<p>Among the references, DeepSeek remains inexpensive at five cents for its saved 100-response replay and has stronger exact consensus with a lower median. Laguna XS is the fastest at 1.15 seconds and costs seven cents in its saved run. Ling chooses more ideal tools and emits more schema-valid calls than either, but its exact-action agreement is lower and its tested route requires substantial retry recovery. HY3 is slower at the median, while GLM produces more exact ideal actions.</p>
<p>The result gives Ling a clear reason to be tested further: <strong>good tool-family selection and broad native dispatch at a zero observed token bill</strong>. It also gives a clear deployment question: whether a route with steadier capacity preserves that behavior. This free shared-pool run supplies encouraging planner evidence, but its first-pass availability is too weak to recommend it as the sole synchronous browser-agent endpoint.</p>
+<h2 id="ling-3-0-flash-vl-nex-and-the-earlier-minimax-run">Ling 3.0 Flash VL, Nex, and the earlier MiniMax run</h2>
+<p><strong>Updated October 7:</strong> these are the saved results from the earlier Ling/Nex comparison, added here for readers evaluating both generations. They use <strong>100 text-only planner cases, the May 23 frozen prompt, 41 tools, and archived May ideal-action rubrics</strong>. No screenshots were sent, including to Ling 3.0 Flash VL. Their reference-agreement column compares tool names with <strong>one model, Claude Sonnet 4.6</strong>, rather than the fourteen-model pairwise consensus above.</p>
+<div class="table-wrap"><table><thead><tr><th>Model</th><th>Run date</th><th>Native calls / 100</th><th>Exact ideal / 100</th><th>Ideal tool / 100</th><th>Sonnet tool-name alignment</th><th>Median</th><th>Reported cost / 100</th></tr></thead><tbody><tr><td>Ling 3.0 Flash VL</td><td>Oct 5</td><td>90</td><td><strong>22</strong></td><td><strong>35</strong></td><td>72%</td><td><strong>1.73s</strong></td><td><strong>$0.0098</strong></td></tr><tr><td>Nex-N2.5-Pro</td><td>Oct 3</td><td><strong>95</strong></td><td>11</td><td>27</td><td>67%</td><td>7.23s</td><td>$0.1500</td></tr><tr><td>Nex-N2.5-mini</td><td>Oct 3</td><td>N/A</td><td>N/A</td><td>N/A</td><td>N/A</td><td>N/A</td><td>N/A</td></tr><tr><td>MiniMax M3, May-frozen run</td><td>Jun 21</td><td>85</td><td>17</td><td>32</td><td><strong>75%</strong></td><td>3.10s</td><td>$1.0562</td></tr></tbody></table></div>
+<p>Within this May protocol, Ling 3.0 VL gives more exact ideal actions and ideal tool names than Nex Pro or MiniMax, with the lowest observed median and bill. Nex Pro emits more native calls, while MiniMax aligns more often with Sonnet&#39;s tool selection. Ling&#39;s bill includes substantial prompt caching and Novita&#39;s promotional rates; it is not an uncached price comparison.</p>
+<p>Nex mini&#39;s route rejected <strong>all 100 requests with HTTP 404 before inference</strong> because it could not accept the structured tools. An authenticated October 5 check reproduced the rejection. Its quality, inference latency, and bill remain <strong>unscored</strong>, rather than counting those routing errors as incorrect model answers.</p>
+<p>Native-call counts in this older table are dispatch coverage, not the schema-valid counts in the August table: Ling 3.0 has one saved malformed argument JSON in its 90 native calls. The two MiniMax rows represent different dated runs and different interfaces. Comparing Ling 3.1&#39;s 15 exact ideal actions directly with Ling 3.0&#39;s 22 would therefore not establish a regression between model generations.</p>
+<p>The <a href="https://github.com/webbrain-one/webbrain/blob/main/test/llm/analysis/2026-10-05-ling30-flash-vl/comparison.json" target="_blank" rel="noopener">May comparison report and raw-run provenance</a> reproduce this table. The full protocol discussion remains in the <a href="/blog/ling30-flash-vl-vs-nex-n25-minimax-m3">Ling 3.0, Nex, and MiniMax article</a>.</p>
<h2 id="evidence-and-reproduction">Evidence and reproduction</h2>
<p>The <a href="https://github.com/webbrain-one/webbrain/tree/main/test/llm/results/2026-10-06-openrouter-ling31-flash-full-7182_chrome_inclusionai_ling-3.1-flash" target="_blank" rel="noopener">raw Ling cases</a> and <a href="https://github.com/webbrain-one/webbrain/tree/main/test/llm/analysis/2026-10-06-ling31-flash" target="_blank" rel="noopener">analysis artifacts</a> accompany this post. The complete inputs are saved in <code>test/llm/freeze/full-replay-7182c21f.json</code>. The preparation script reconstructs them from the pinned Git revision and records SHA-256 hashes for the source files, questions, and rubrics.</p>
<pre><code class="language-powershell">node test/llm/prepare-frontier-replay.mjs
@@ -575,7 +582,7 @@ Offline report generator:
test/llm/report-ling31-comparison.mjs</code></pre>
<pre><code class="language-powershell">node test/llm/report-ling31-comparison.mjs
node scripts/build-blog.mjs</code></pre>
-<p>Tags: #Ling31Flash #inclusionAI #DeepSeekV4 #TencentHY3 #GLM52 #Poolside #LagunaXS #MiniMaxM3 #OpenRouter #ToolCalling #BrowserAgent #WebBrain</p>
+<p>Tags: #Ling31Flash #Ling30FlashVL #NexN25 #inclusionAI #DeepSeekV4 #TencentHY3 #GLM52 #Poolside #LagunaXS #MiniMaxM3 #OpenRouter #ToolCalling #BrowserAgent #WebBrain</p>
<div class="author-box">
Written by <a href="https://emresokullu.com" target="_blank" rel="noopener">Emre Sokullu</a>. WebBrain 33.0.0 and later is GPL-3.0-or-later and open on <a href="https://github.com/webbrain-one/webbrain" target="_blank" rel="noopener">GitHub</a>.
diff --git a/web/blog/posts/ling31-flash-vs-deepseek-hy3-glm52-laguna-xs.md b/web/blog/posts/ling31-flash-vs-deepseek-hy3-glm52-laguna-xs.md
index 86a040470..14137cbac 100644
--- a/web/blog/posts/ling31-flash-vs-deepseek-hy3-glm52-laguna-xs.md
+++ b/web/blog/posts/ling31-flash-vs-deepseek-hy3-glm52-laguna-xs.md
@@ -3,7 +3,7 @@ title: Ling 3.1 Flash vs DeepSeek, HY3, GLM-5.2, and Laguna XS: free inference,
slug: ling31-flash-vs-deepseek-hy3-glm52-laguna-xs
sortOrder: -330
date: 2026-10-06
-readTime: 7 min read
+readTime: 8 min read
description: Ling 3.1 Flash completes 100 pinned planner cases with 80.7% tool-family consensus, 32.8% exact consensus, 92 valid calls, and zero reported cost, after extensive Novita rate-limit recovery.
excerpt: Ling leads tool-family agreement and beats the four text-only references on ideal-tool count, but trails on exact-action consensus. Its free Novita route required 298 rate-limit responses across the benchmark and diagnostic campaign.
titleTag: Ling 3.1 Flash planner benchmark vs DeepSeek, HY3, GLM, and Laguna XS - WebBrain Blog
@@ -18,6 +18,10 @@ keywords:
- Tencent HY3
- GLM-5.2
- Poolside Laguna XS 2.1
+ - Ling 3.0 Flash VL
+ - Nex-N2.5-Pro
+ - Nex-N2.5-mini
+ - MiniMax M3
- OpenRouter
- browser agent benchmark
- tool calling
@@ -37,7 +41,7 @@ Our comparison focuses on the four text-only routes in the [American and Chinese
## Reconstructing the matching test
-This post uses the **August full-tier checkout `7182c21f`**, rather than the May frozen interface from our [Ling 3.0 Flash VL comparison](/blog/ling30-flash-vl-vs-nex-n25-minimax-m3). The older Ling, Nex, and May-frozen MiniMax percentages belong to a different test and cannot be inserted here.
+The main ranking uses the **August full-tier checkout `7182c21f`**. We also include the models from our [Ling 3.0 Flash VL comparison](/blog/ling30-flash-vl-vs-nex-n25-minimax-m3) in a separate historical table below. That test uses the May frozen interface, so its percentages do not enter the August ranking or fourteen-model consensus pool.
The August runs omitted request bodies. We reconstructed all 100 messages and their tool schemas from the exact pinned payload builder, site adapters, tool definitions, and questions, then saved a replay snapshot. We also archived the ideal-action rubrics from that checkout. Ling's new raw case files preserve both its request and the API response. The historical requests' full byte equality cannot be proven because those bodies were never stored.
@@ -111,6 +115,25 @@ Among the references, DeepSeek remains inexpensive at five cents for its saved 1
The result gives Ling a clear reason to be tested further: **good tool-family selection and broad native dispatch at a zero observed token bill**. It also gives a clear deployment question: whether a route with steadier capacity preserves that behavior. This free shared-pool run supplies encouraging planner evidence, but its first-pass availability is too weak to recommend it as the sole synchronous browser-agent endpoint.
+## Ling 3.0 Flash VL, Nex, and the earlier MiniMax run
+
+**Updated October 7:** these are the saved results from the earlier Ling/Nex comparison, added here for readers evaluating both generations. They use **100 text-only planner cases, the May 23 frozen prompt, 41 tools, and archived May ideal-action rubrics**. No screenshots were sent, including to Ling 3.0 Flash VL. Their reference-agreement column compares tool names with **one model, Claude Sonnet 4.6**, rather than the fourteen-model pairwise consensus above.
+
+| Model | Run date | Native calls / 100 | Exact ideal / 100 | Ideal tool / 100 | Sonnet tool-name alignment | Median | Reported cost / 100 |
+| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| Ling 3.0 Flash VL | Oct 5 | 90 | **22** | **35** | 72% | **1.73s** | **$0.0098** |
+| Nex-N2.5-Pro | Oct 3 | **95** | 11 | 27 | 67% | 7.23s | $0.1500 |
+| Nex-N2.5-mini | Oct 3 | N/A | N/A | N/A | N/A | N/A | N/A |
+| MiniMax M3, May-frozen run | Jun 21 | 85 | 17 | 32 | **75%** | 3.10s | $1.0562 |
+
+Within this May protocol, Ling 3.0 VL gives more exact ideal actions and ideal tool names than Nex Pro or MiniMax, with the lowest observed median and bill. Nex Pro emits more native calls, while MiniMax aligns more often with Sonnet's tool selection. Ling's bill includes substantial prompt caching and Novita's promotional rates; it is not an uncached price comparison.
+
+Nex mini's route rejected **all 100 requests with HTTP 404 before inference** because it could not accept the structured tools. An authenticated October 5 check reproduced the rejection. Its quality, inference latency, and bill remain **unscored**, rather than counting those routing errors as incorrect model answers.
+
+Native-call counts in this older table are dispatch coverage, not the schema-valid counts in the August table: Ling 3.0 has one saved malformed argument JSON in its 90 native calls. The two MiniMax rows represent different dated runs and different interfaces. Comparing Ling 3.1's 15 exact ideal actions directly with Ling 3.0's 22 would therefore not establish a regression between model generations.
+
+The [May comparison report and raw-run provenance](https://github.com/webbrain-one/webbrain/blob/main/test/llm/analysis/2026-10-05-ling30-flash-vl/comparison.json) reproduce this table. The full protocol discussion remains in the [Ling 3.0, Nex, and MiniMax article](/blog/ling30-flash-vl-vs-nex-n25-minimax-m3).
+
## Evidence and reproduction
The [raw Ling cases](https://github.com/webbrain-one/webbrain/tree/main/test/llm/results/2026-10-06-openrouter-ling31-flash-full-7182_chrome_inclusionai_ling-3.1-flash) and [analysis artifacts](https://github.com/webbrain-one/webbrain/tree/main/test/llm/analysis/2026-10-06-ling31-flash) accompany this post. The complete inputs are saved in `test/llm/freeze/full-replay-7182c21f.json`. The preparation script reconstructs them from the pinned Git revision and records SHA-256 hashes for the source files, questions, and rubrics.
@@ -138,4 +161,4 @@ node test/llm/report-ling31-comparison.mjs
node scripts/build-blog.mjs
```
-Tags: #Ling31Flash #inclusionAI #DeepSeekV4 #TencentHY3 #GLM52 #Poolside #LagunaXS #MiniMaxM3 #OpenRouter #ToolCalling #BrowserAgent #WebBrain
+Tags: #Ling31Flash #Ling30FlashVL #NexN25 #inclusionAI #DeepSeekV4 #TencentHY3 #GLM52 #Poolside #LagunaXS #MiniMaxM3 #OpenRouter #ToolCalling #BrowserAgent #WebBrain