Skip to content

Open-source AI browser agent for Chrome and Firefox (monorepo) 🧠

Add matching Ling 3.0 VL vision results to HY4 and MiMo comparison

Barack Sokullufaead54f14ca
diff --git a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/README.md b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/README.md
index b486b6e81..0a1bd6bbf 100644
--- a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/README.md
+++ b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/README.md
@@ -4,7 +4,9 @@ Completed October 14, 2026 (Istanbul; raw UTC timestamps begin October 13). **60
New planner results: Pro ranks 3/17 (43.6% exact consensus, 44 ideal tools, 95/95 schema-valid first calls, 6123 ms median, 46683 ms p95, $0.124612667); Flash ranks 7/17 (41.9%, 37, 96/97, 1195 ms, 4259 ms, $0.010978782); HY4 ranks 14/17 (27.5%, 36, 98/98, 5805.5 ms, 11638 ms, $0.2961319824); Liquid ranks 15/17 (24.9%, 23, 94/95, 1832 ms, 6677 ms, $0.0828711). No content-tool fallback was needed. Flash 058 misses `extract_data.type`; Liquid 092 sends an unsupported key enum. Schema validation and consensus inspect the first call only; multi-call responses remain in raw evidence.
-Vision: Flash passes 81/100 (94.3% mean score, 1908 ms median, 4586 ms p95, $0.021394496); Pro passes 77/100 (94.0%, 5728 ms, 11589 ms, $0.0644821588). Both have 100 parsed six-section, nonempty outputs. Pro reports 1438 reasoning tokens despite the disabling kwargs; Flash reports zero. All 400 vision grades, including the two historical references, are reproduced from the unchanged scorer.
+Vision: Flash passes 81/100 (94.3% mean score, 1908 ms median, 4586 ms p95, $0.021394496); Pro passes 77/100 (94.0%, 5728 ms, 11589 ms, $0.0644821588). Both have 100 parsed six-section, nonempty outputs. Pro reports 1438 reasoning tokens despite the disabling kwargs; Flash reports zero. All 500 vision grades, including the three historical references, are reproduced from the unchanged scorer.
+
+The comparison now also reuses Ling 3.0 Flash VL's October 7 vision run: 65/100 passes, 88.1% mean score, 3458.5 ms median, 4878 ms p95, and $0.0053822608. All 100 image hashes, prompt/settings/questions and grades match this vision protocol. Its raw streams are checked against final content, usage, provider and finish reason; its 22 length-limited responses and 33,885 reasoning tokens are retained. No new inference or cost is added. Its earlier May 41-tool text-planner run is incompatible with the 48-tool August planner protocol and is excluded from the 17-model pool.
Requested models:
diff --git a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/audit.json b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/audit.json
index e0421bf2e..4ad7b3191 100644
--- a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/audit.json
+++ b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/audit.json
@@ -11801,6 +11801,706 @@
"path": "test/vision/results/2026-08-27-openrouter-full_z-ai_glm-5.3-flash_production/100.json",
"sha256": "1ea6dd2c9941b5b0992d0cbbe33de246179b85fc28ac3bc7411ddf60f6f518b0"
},
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "001",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/001.json",
+ "sha256": "9396038bc546e4883e5de7a0189e117ee4c39c22c91d7a2eae24e48e2066c117"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "002",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/002.json",
+ "sha256": "2b94ad1d69b6ea8964c583c2b7e1913fb9828fa34e79e816466e88fda3f8cf3a"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "003",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/003.json",
+ "sha256": "f18be35847767ea533cabe53b7e48e0d1a2bf57160fc7cd54f0f8640275ef8df"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "004",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/004.json",
+ "sha256": "5dd8168980634c6bb361939a60d5cc7cc37b03d2215d41da71767a4a6fadecad"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "005",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/005.json",
+ "sha256": "5d69d445308f10d1fca0bb52f48aae4f2d6c555c9f00dc271d166b70b6a9bd1f"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "006",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/006.json",
+ "sha256": "8e5e3a5a7a3c73d6506af4027d516a5b9e1e58c27e8363d9647c7fd3b7e281d4"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "007",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/007.json",
+ "sha256": "f7e6d5819df60afea6697ff8885eeadca19bf0be3b567f7c58e97d1c4d8a1144"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "008",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/008.json",
+ "sha256": "3116e7b2a4010f7cc4f1ccfc23f7746c6f427dfafdb4cef290323bca78ddf995"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "009",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/009.json",
+ "sha256": "fab4aca1fb40abf74c1f085d213bcf935e2735df897b36c0a6d19b55e9d6d0e2"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "010",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/010.json",
+ "sha256": "b7625f942d3c50124aeddd5e2b5c1374e67b2ebf459ca85771705f8017981a8f"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "011",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/011.json",
+ "sha256": "b00795408d3238a0fede80169e6a818889b04b91e84670b16154c79279578bb7"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "012",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/012.json",
+ "sha256": "35f02faabb6a0458b7d0874143553365c51c48fdf3944bcea1c26108cce71313"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "013",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/013.json",
+ "sha256": "c95bb038b33f8cb5c1cd1b93c1fb02a66e2ad5ce70f6065b9843345b66af959d"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "014",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/014.json",
+ "sha256": "ae92af4bf1e72634a248e957ad059a26bf1080dfd1b6fc68703f590b9d6c669b"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "015",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/015.json",
+ "sha256": "c30bf705d9bfc9edb03b872361c9ad3b17cf57b8bb325c331c74f702ff75d9f3"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "016",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/016.json",
+ "sha256": "7de3433f40816b954a5985329f58fe6329a93a5d043d5cfde7fca3b4147ce0d2"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "017",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/017.json",
+ "sha256": "13c8f50db00d9fb04b2a98a72ce89d591375796baa5a577d488ab543167e8e7a"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "018",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/018.json",
+ "sha256": "2072107a3efae4ed32b49d0b05ed4d9ac1ff5c7762c31b95f7091e83e590782f"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "019",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/019.json",
+ "sha256": "60a8d6d092e68285a5d69b1346badbc04fea82aabe46e64adb464c60852d0b05"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "020",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/020.json",
+ "sha256": "2e8d53acce6c7c4a32f2117edc883d7f0a80043476d2c978eb33ca6d1fd44e78"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "021",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/021.json",
+ "sha256": "1edb26d308cdb4758ed26fd8b73c2309d621bde247cba2a15bf4bb105fc3a4aa"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "022",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/022.json",
+ "sha256": "420536120a19404062d156e8f01e8ab88e9f376781c7844feb18883481eb8833"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "023",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/023.json",
+ "sha256": "3150cfaf21887b9d28ebb63061b06bbaf5c57ded96dc3662e5bf7608f6015a16"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "024",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/024.json",
+ "sha256": "d3939686cd1e8477eae0259a923010f35e0f8fd5b4191ac3d3fcf08703ad4098"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "025",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/025.json",
+ "sha256": "a889353164bc1e1976568554f58fb7f41d613a0ffda2f4c0e314d5800838c4e4"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "026",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/026.json",
+ "sha256": "15d8ce4258c1692ee5869403dc19e5e0b84abadc2e50a332d5301dc8bd467245"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "027",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/027.json",
+ "sha256": "3c3e7bf059e32f422be5ac66a208ffc917c47094ce167f095a1918f956fa1110"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "028",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/028.json",
+ "sha256": "925ae559e9fb2895c342f0114f59ee87b5858696ba4ea6db0231166d0f5d2d50"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "029",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/029.json",
+ "sha256": "a5f6d494dd2e317234b802b4793fd9b30c0bc3403843e7ff40d9b431dff52912"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "030",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/030.json",
+ "sha256": "7832d3bfa1f6b5f38904290bb715d957d6e0b4ab9b32ad651a2348a4507c9bd3"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "031",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/031.json",
+ "sha256": "8ed22ee9f9162f9b12ad32bd6dadeb3e52bfdf74ac66a45aeb9e2369d70ffd6d"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "032",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/032.json",
+ "sha256": "01aec8b2e8bc7e171a841663369edcb47ffd40431b8de4a62256740035b1f78b"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "033",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/033.json",
+ "sha256": "3173321ffc4b061003fa570c62f5d27cdc31bf9e7e5376bba9f37898280c50f4"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "034",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/034.json",
+ "sha256": "09f5e80c1f2864f4e1e961d90e1ce3080f379e2cfcfd5ce9e0d28b415accec93"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "035",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/035.json",
+ "sha256": "b9161f4b034fc7494af56642dda31486d0fa965b057cf56db48b2e524ceaecbc"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "036",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/036.json",
+ "sha256": "a22efd38ec2fe9deefb33544d58ad0cbccf14c600583dcaa1c5c1137d656a411"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "037",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/037.json",
+ "sha256": "8e4340f4787616b683f86e7dd0ff3aa5145cbc49254091d7344c9643420682ad"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "038",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/038.json",
+ "sha256": "d999281cba9a4752f3d8fc1392ca1a06f6e07bc4ad372f4d0be6ff7954ea1405"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "039",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/039.json",
+ "sha256": "0d40d841517c786f5f716c668dcebd1a43c9c026956c56849f031e14a73f97c4"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "040",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/040.json",
+ "sha256": "149d24e477f2ef92478e6256a4f2094c2d3de3b809588c1a822f304dce67bbc0"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "041",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/041.json",
+ "sha256": "e72c02eb82e7a9cb5777d9f96e79ae4310175cce04770f501b5324a8f628fd17"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "042",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/042.json",
+ "sha256": "1ab87838c3485d84e2dc109bdf37c8a0ea413473ec1de67368e97f5043a0a9af"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "043",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/043.json",
+ "sha256": "75e325bcf8bfeecd79cd8ab00d4e76f228038fc9c697eb41c18bf9b9f5f3a5e0"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "044",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/044.json",
+ "sha256": "c2e91012dcc16009225f04913cecea18f039c896a53e381fc0d20857ecabde1a"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "045",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/045.json",
+ "sha256": "405c3dc47a4933ad8c4d4e2c15351e1ed0ce279d48ff5f3b8a2bce5fb572de91"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "046",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/046.json",
+ "sha256": "f0d8c1c771ad07ee43c97821281096644da9d1984993fbd06516fa8812d6fe9d"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "047",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/047.json",
+ "sha256": "55d2936c70c70bbbd3767297196f7d3f076c884c9859709a8fa420875edb27ba"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "048",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/048.json",
+ "sha256": "140e74e26c25fb96551d122d24c1fb62947c5eea51d2483ed28d9c9d7653e449"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "049",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/049.json",
+ "sha256": "f2b707c407bb53dcd59f9fc7ee788e2de14a6ec5d87442cf58bf8e79eedc59f6"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "050",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/050.json",
+ "sha256": "bb7fa4b98695a1bb72aa6e388016e566dbc1c196bdb9ea4aee552674a49939e7"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "051",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/051.json",
+ "sha256": "553b975b562d0b6016a2ea1ab4823aae0075918f6c0487cd57d867d107e37b38"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "052",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/052.json",
+ "sha256": "d390d039d1d05604c027f266d13f0e804be483393988883c917a820ae1a4dd9a"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "053",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/053.json",
+ "sha256": "d95f8f7d7e5f968cff1611d60038685dc4863d1faa14abcf32dca2fac6ddcb9c"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "054",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/054.json",
+ "sha256": "089c5afc138b1fbc8e9fda041cfe29836f97753a1959ed7ec1924fb9f016aa6b"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "055",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/055.json",
+ "sha256": "9a3d98dc2e3bf62a3675ed3065d263c4da231247928979658f5a4994dab5d980"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "056",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/056.json",
+ "sha256": "01ed0afcb58cd3756c07f579f402c63a3efd3d2bc910faaaabd60c55d80a120e"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "057",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/057.json",
+ "sha256": "397724aa2f6c16e9f00eae29239a0331de625f51552888afcb43ccfae1b73355"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "058",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/058.json",
+ "sha256": "22391ab0acd87ddb360f17e6ec59dfee3399cecc7b6ba227573dfd38315f86dc"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "059",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/059.json",
+ "sha256": "a0444fec114cc2d639ac36f39cc74c60300576bad52c8bf61c710803df7a32a1"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "060",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/060.json",
+ "sha256": "2276f60682d0b75d4a604ecb5e1d9e7770383bfe81a7d5068612e4263e6e33b3"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "061",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/061.json",
+ "sha256": "255f4aed1e9c63ea78743ab01f299d5a5367363d2dd3cfaa7b84b279aa4a6ead"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "062",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/062.json",
+ "sha256": "7962d8f771908a586944a9934aad68561035a3fe5f4988c7f912da2ab2c80669"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "063",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/063.json",
+ "sha256": "2ab147e0f63c9d789dba1b76645bcfb3f36a1689aec0bd50dbc006a172244a5a"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "064",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/064.json",
+ "sha256": "e99fcfcbeb69c34259d920f0b35daae1d510ba2461bec3f36c405abfb1c46bfc"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "065",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/065.json",
+ "sha256": "ea474fa41d9b8e9768744dc104f8004473e3508eb6e1853fa68cb5e37b14b144"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "066",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/066.json",
+ "sha256": "e5817825062c53d1da95be49255764f6a13f82d9b57fd2094fe0d60153b0d620"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "067",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/067.json",
+ "sha256": "14cb82800a0d272b0de474656a679c99351ad488fad1e633cb9a940dad54f02e"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "068",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/068.json",
+ "sha256": "c7cffdb7c60466e43f3e9a45647a251432fca70759b20a261c440563b1da535e"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "069",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/069.json",
+ "sha256": "26b4cd45d0d8bbe2c22b8eca38ee5bfd0e52cd0345b98106d6ba4807d97a0865"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "070",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/070.json",
+ "sha256": "c10b1748e4490e90a7102c509f6cd7cb62b66a707fd5cf01b25957bd4fd15f7b"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "071",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/071.json",
+ "sha256": "efa59ea041f0303a5af241aef4605feb8f56ab6e52122c9d2133e427f22533de"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "072",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/072.json",
+ "sha256": "ed15543ef867449fffe4fbbae38f057a95675d913143c14726e1db659a16365a"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "073",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/073.json",
+ "sha256": "1451aaf0f2ce4c21bec1a989e16db003d4b7e2c412be90674189fbb72a7d7590"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "074",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/074.json",
+ "sha256": "f45c39124325cc7615ed43ba1f4cedc1764bada33450b45a8b2db2456a07f2d0"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "075",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/075.json",
+ "sha256": "5ac0211fdaaa82b5466f442326ff79c04f84508d8ff4946966946e004a1a5fa1"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "076",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/076.json",
+ "sha256": "053531a4993dd5a28909176790ea710f2c136cd4b701b4abaccc352beebcfee8"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "077",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/077.json",
+ "sha256": "3bc7129f941efa84ad9bc109d20b6536e860f5a07ff464d79a26c064463f28c2"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "078",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/078.json",
+ "sha256": "9043bef151cb6cb63cf3faaa08b5ca23720f0a81f4913c45653b40b6b8d8acb2"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "079",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/079.json",
+ "sha256": "11af02814f4dc597cdc6b8a41c633f203f5ad4944af8b572f6f862e464c0540e"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "080",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/080.json",
+ "sha256": "00ec8d1383212e580ec2906bb93b1b91b860cba20ef13906d4c89d8c0283aaeb"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "081",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/081.json",
+ "sha256": "744618eda17a88119c250c20e2f41333ffebee40bc3406620686b744ef8a3296"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "082",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/082.json",
+ "sha256": "77663dec26c9690afd1c348d0a08fa925babc79446f56e28bb2bf78567e2e382"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "083",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/083.json",
+ "sha256": "6832e0546d0814bd47dbdd86982937986f8bcd47fb3e092d32b49b3416d470bc"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "084",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/084.json",
+ "sha256": "42a62b0d3dd3ee952ffb246907ba336e6457cfc6b49ef2d43d67810f61175cc3"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "085",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/085.json",
+ "sha256": "f99580f8def303a4ab377cda9aeea106afb96c62d3789900527f05a273055bc2"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "086",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/086.json",
+ "sha256": "6fd7621a5dcdddeb3843a3cc6778b2fca52bae179a2d1acb5502b101c1d68f4f"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "087",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/087.json",
+ "sha256": "b852567c230772b7f9e6fbcd0e83b38cb1a2d8d987741c8c140d98e55603a9d6"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "088",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/088.json",
+ "sha256": "b43f2d0ec95969c3e77c823efb6fe4af4873807dc64cd8ffc2a6a3ad3bb04488"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "089",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/089.json",
+ "sha256": "eba5716988a134816f62dce28b5f93d08d4bb75b48c391d38cbeae7f1882ced9"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "090",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/090.json",
+ "sha256": "250c094fb4c2d945c48cf98ddf390dbe9bf8f0f87a6777e9efbf235ed2da65ca"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "091",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/091.json",
+ "sha256": "e4d9a6d9efe11c5da5f9b9a09ad2b704218e14dd4f3b14bdb565ccb9e170619e"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "092",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/092.json",
+ "sha256": "10b3113d6b9deb745cf34da5c18de8055ccc07572b4b7456dc520e9edaa1cad3"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "093",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/093.json",
+ "sha256": "cfa5dd2850d0e1636020c299d2ecebfd85f91b45ee7b25560006fdbe072802e5"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "094",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/094.json",
+ "sha256": "d2aab7d64bbe01e8c63dea52be3d3866fc7fc19af21bb727aadaf61dbd8af281"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "095",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/095.json",
+ "sha256": "1d1be2e8b05958c50e1da01b01d5b5a9c443244118d9eb2d4eb15a3d9a0cd162"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "096",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/096.json",
+ "sha256": "586919d4dd4bc0dd97aa3cf49015bc87a5160ba6d296b352ad85e64886935bb1"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "097",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/097.json",
+ "sha256": "51aa6538b2352e75da4d5a1f594e95a361f4892f5dff768e5a3eb51e6cd0485f"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "098",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/098.json",
+ "sha256": "2714dcc03d23c0e9286c21f213c0ab5e98bdb15d406c5bfbcbba5737fb863587"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "099",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/099.json",
+ "sha256": "df0f0056359b4d96b569951d567147654a9873a535e941529bb7d0359923c870"
+ },
+ {
+ "track": "vision",
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "id": "100",
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production/100.json",
+ "sha256": "9a6385e41bd4f5194fbc29d4189703017d4a6617fe63c15e277101ac559a3324"
+ },
{
"model": "tencent/hy4-preview",
"id": "001",
diff --git a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/blog-verification.json b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/blog-verification.json
index a0ff44809..ca6c1b517 100644
--- a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/blog-verification.json
+++ b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/blog-verification.json
@@ -34,7 +34,7 @@
"p95",
"Cost / 100"
],
- "rows": 4
+ "rows": 5
},
{
"headers": [
@@ -42,7 +42,8 @@
"MiMo-V2.6-Flash",
"Qwen3.8 Flash",
"MiMo-V2.6-Pro",
- "GLM-5.3 Flash"
+ "GLM-5.3 Flash",
+ "Ling 3.0 VL"
],
"rows": 5
},
@@ -63,7 +64,7 @@
"usageMatchesReport": true,
"canonical": "https://webbrain.one/blog/hy4-mimo-v26-lfm25-browser-agent-benchmark",
"jsonLdParsed": 1,
- "internalLinksChecked": 3,
+ "internalLinksChecked": 4,
"mobile": {
"viewport": 390,
"page": 390,
diff --git a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/comparison.json b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/comparison.json
index e79de64aa..5a9fd4538 100644
--- a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/comparison.json
+++ b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/comparison.json
@@ -10,6 +10,7 @@
"historicalPlannerModels": 13,
"newPlannerModels": 4,
"newVisionModels": 2,
+ "historicalVisionModels": 3,
"method": "Leave-one-out consensus across all 17 models, comparing the first proposed tool call per response. Recursive object-key sorting only; preserve argument values, defaults, and extra arguments. Additional tool calls are counted separately and not executed or scored. No-tool/no-tool agrees; rejected API requests are excluded from quality. Consensus is agreement, not ground truth.",
"caveat": "Only the four October 14 planner models and two MiMo vision runs are new. Historical requests were omitted and are reconstructed from pinned source; full historical body equality is unprovable. Serving dates, provider routes, caches and bills differ. No browser tools are executed.",
"results": [
@@ -5265,6 +5266,821 @@
"criticalFailures": []
}
]
+ },
+ {
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "label": "Ling 3.0 Flash VL",
+ "date": "2026-10-07",
+ "rawStreams": true,
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production",
+ "cases": 100,
+ "errors": 0,
+ "passes": 65,
+ "meanScore": 0.8809624264624261,
+ "meanLatencyMs": 3572.28,
+ "medianLatencyMs": 3458.5,
+ "p95LatencyMs": 4878,
+ "costRecords": 100,
+ "usageCost": 0.005382260800000002,
+ "promptTokens": 120800,
+ "completionTokens": 51778,
+ "cachedTokens": 20480,
+ "reasoningTokens": 33885,
+ "sixSections": 98,
+ "emptyOutputs": 0,
+ "lengthLimited": 22,
+ "byDifficulty": {
+ "easy": {
+ "cases": 20,
+ "passes": 14
+ },
+ "basic": {
+ "cases": 20,
+ "passes": 14
+ },
+ "intermediate": {
+ "cases": 20,
+ "passes": 13
+ },
+ "advanced": {
+ "cases": 20,
+ "passes": 14
+ },
+ "challenging": {
+ "cases": 20,
+ "passes": 10
+ }
+ },
+ "byCategory": {
+ "authentication": {
+ "cases": 5,
+ "passes": 2
+ },
+ "search-results": {
+ "cases": 5,
+ "passes": 4
+ },
+ "checkout": {
+ "cases": 5,
+ "passes": 3
+ },
+ "form-validation": {
+ "cases": 5,
+ "passes": 1
+ },
+ "modal-overlay": {
+ "cases": 5,
+ "passes": 0
+ },
+ "toast-notification": {
+ "cases": 5,
+ "passes": 4
+ },
+ "loading-state": {
+ "cases": 5,
+ "passes": 3
+ },
+ "consent-banner": {
+ "cases": 5,
+ "passes": 2
+ },
+ "dashboard": {
+ "cases": 5,
+ "passes": 4
+ },
+ "chart-reading": {
+ "cases": 5,
+ "passes": 5
+ },
+ "data-table": {
+ "cases": 5,
+ "passes": 5
+ },
+ "email-compose": {
+ "cases": 5,
+ "passes": 3
+ },
+ "kanban-board": {
+ "cases": 5,
+ "passes": 5
+ },
+ "calendar": {
+ "cases": 5,
+ "passes": 5
+ },
+ "map-and-travel": {
+ "cases": 5,
+ "passes": 4
+ },
+ "photo-understanding": {
+ "cases": 5,
+ "passes": 4
+ },
+ "multilingual-ocr": {
+ "cases": 5,
+ "passes": 3
+ },
+ "occlusion-and-contrast": {
+ "cases": 5,
+ "passes": 2
+ },
+ "security-challenge": {
+ "cases": 5,
+ "passes": 2
+ },
+ "uncertainty-calibration": {
+ "cases": 5,
+ "passes": 4
+ }
+ },
+ "providers": {
+ "Novita": {
+ "cases": 100,
+ "passes": 65
+ }
+ },
+ "caseScores": [
+ {
+ "id": "001",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": []
+ },
+ {
+ "id": "002",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "003",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "004",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "005",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "006",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "007",
+ "pass": false,
+ "ratio": 0.75,
+ "criticalFailures": [
+ "heading"
+ ]
+ },
+ {
+ "id": "008",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "consent-signal"
+ ]
+ },
+ {
+ "id": "009",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "010",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "011",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "012",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "013",
+ "pass": true,
+ "ratio": 0.9047619047619048,
+ "criticalFailures": []
+ },
+ {
+ "id": "014",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "015",
+ "pass": true,
+ "ratio": 0.75,
+ "criticalFailures": []
+ },
+ {
+ "id": "016",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "017",
+ "pass": true,
+ "ratio": 0.8888888888888888,
+ "criticalFailures": []
+ },
+ {
+ "id": "018",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "cover"
+ ]
+ },
+ {
+ "id": "019",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "label"
+ ]
+ },
+ {
+ "id": "020",
+ "pass": true,
+ "ratio": 0.8333333333333334,
+ "criticalFailures": []
+ },
+ {
+ "id": "021",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "purpose"
+ ]
+ },
+ {
+ "id": "022",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "023",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "banner"
+ ]
+ },
+ {
+ "id": "024",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "025",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "026",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "027",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "028",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "consent-signal"
+ ]
+ },
+ {
+ "id": "029",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "030",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "031",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "032",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "033",
+ "pass": true,
+ "ratio": 0.8571428571428571,
+ "criticalFailures": []
+ },
+ {
+ "id": "034",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "035",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "036",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "037",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "038",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "target"
+ ]
+ },
+ {
+ "id": "039",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "040",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "041",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "042",
+ "pass": true,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": []
+ },
+ {
+ "id": "043",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "044",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "045",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "046",
+ "pass": false,
+ "ratio": 0.3333333333333333,
+ "criticalFailures": [
+ "toast",
+ "detail"
+ ]
+ },
+ {
+ "id": "047",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "048",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "049",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "050",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "051",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "052",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "purpose",
+ "status"
+ ]
+ },
+ {
+ "id": "053",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "054",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "055",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "056",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "057",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "058",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "cover"
+ ]
+ },
+ {
+ "id": "059",
+ "pass": false,
+ "ratio": 0.6363636363636364,
+ "criticalFailures": [
+ "captcha-signal",
+ "security-blocker"
+ ]
+ },
+ {
+ "id": "060",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "061",
+ "pass": true,
+ "ratio": 0.9230769230769231,
+ "criticalFailures": []
+ },
+ {
+ "id": "062",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "063",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "064",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "065",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "066",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "067",
+ "pass": false,
+ "ratio": 0.6666666666666666,
+ "criticalFailures": [
+ "heading"
+ ]
+ },
+ {
+ "id": "068",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "069",
+ "pass": false,
+ "ratio": 0.75,
+ "criticalFailures": [
+ "alert"
+ ]
+ },
+ {
+ "id": "070",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "071",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "072",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "073",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "074",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "075",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "076",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "077",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "078",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "079",
+ "pass": false,
+ "ratio": 0.2727272727272727,
+ "criticalFailures": [
+ "label",
+ "captcha-signal",
+ "security-blocker"
+ ]
+ },
+ {
+ "id": "080",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "visible-fragment"
+ ]
+ },
+ {
+ "id": "081",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "purpose"
+ ]
+ },
+ {
+ "id": "082",
+ "pass": false,
+ "ratio": 0.8333333333333334,
+ "criticalFailures": [
+ "purpose",
+ "notice"
+ ]
+ },
+ {
+ "id": "083",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "blocked"
+ ]
+ },
+ {
+ "id": "084",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "input",
+ "blocker"
+ ]
+ },
+ {
+ "id": "085",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "086",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "087",
+ "pass": true,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": []
+ },
+ {
+ "id": "088",
+ "pass": false,
+ "ratio": 0.5,
+ "criticalFailures": [
+ "title",
+ "choice-1",
+ "consent-blocker"
+ ]
+ },
+ {
+ "id": "089",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "090",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "091",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "092",
+ "pass": false,
+ "ratio": 0.5833333333333334,
+ "criticalFailures": [
+ "purpose",
+ "status",
+ "external-blocker"
+ ]
+ },
+ {
+ "id": "093",
+ "pass": true,
+ "ratio": 0.8571428571428571,
+ "criticalFailures": []
+ },
+ {
+ "id": "094",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "095",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "route-alert"
+ ]
+ },
+ {
+ "id": "096",
+ "pass": false,
+ "ratio": 0.5555555555555556,
+ "criticalFailures": [
+ "caption"
+ ]
+ },
+ {
+ "id": "097",
+ "pass": false,
+ "ratio": 0.5454545454545454,
+ "criticalFailures": [
+ "input",
+ "status"
+ ]
+ },
+ {
+ "id": "098",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "099",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "100",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ }
+ ]
}
],
"pairwise": [
@@ -5745,6 +6561,15 @@
"neitherPass": 16,
"cases": 100
},
+ {
+ "model": "xiaomi/mimo-v2.6-flash",
+ "peer": "inclusionai/ling-3.0-flash-vl",
+ "bothPass": 63,
+ "modelOnlyPass": 18,
+ "peerOnlyPass": 2,
+ "neitherPass": 17,
+ "cases": 100
+ },
{
"model": "qwen/qwen3.8-flash",
"peer": "xiaomi/mimo-v2.6-flash",
@@ -5772,6 +6597,15 @@
"neitherPass": 17,
"cases": 100
},
+ {
+ "model": "qwen/qwen3.8-flash",
+ "peer": "inclusionai/ling-3.0-flash-vl",
+ "bothPass": 62,
+ "modelOnlyPass": 17,
+ "peerOnlyPass": 3,
+ "neitherPass": 18,
+ "cases": 100
+ },
{
"model": "xiaomi/mimo-v2.6-pro",
"peer": "xiaomi/mimo-v2.6-flash",
@@ -5799,6 +6633,15 @@
"neitherPass": 17,
"cases": 100
},
+ {
+ "model": "xiaomi/mimo-v2.6-pro",
+ "peer": "inclusionai/ling-3.0-flash-vl",
+ "bothPass": 62,
+ "modelOnlyPass": 15,
+ "peerOnlyPass": 3,
+ "neitherPass": 20,
+ "cases": 100
+ },
{
"model": "z-ai/glm-5.3-flash",
"peer": "xiaomi/mimo-v2.6-flash",
@@ -5825,6 +6668,51 @@
"peerOnlyPass": 7,
"neitherPass": 17,
"cases": 100
+ },
+ {
+ "model": "z-ai/glm-5.3-flash",
+ "peer": "inclusionai/ling-3.0-flash-vl",
+ "bothPass": 62,
+ "modelOnlyPass": 14,
+ "peerOnlyPass": 3,
+ "neitherPass": 21,
+ "cases": 100
+ },
+ {
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "peer": "xiaomi/mimo-v2.6-flash",
+ "bothPass": 63,
+ "modelOnlyPass": 2,
+ "peerOnlyPass": 18,
+ "neitherPass": 17,
+ "cases": 100
+ },
+ {
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "peer": "qwen/qwen3.8-flash",
+ "bothPass": 62,
+ "modelOnlyPass": 3,
+ "peerOnlyPass": 17,
+ "neitherPass": 18,
+ "cases": 100
+ },
+ {
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "peer": "xiaomi/mimo-v2.6-pro",
+ "bothPass": 62,
+ "modelOnlyPass": 3,
+ "peerOnlyPass": 15,
+ "neitherPass": 20,
+ "cases": 100
+ },
+ {
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "peer": "z-ai/glm-5.3-flash",
+ "bothPass": 62,
+ "modelOnlyPass": 3,
+ "peerOnlyPass": 14,
+ "neitherPass": 21,
+ "cases": 100
}
],
"campaign": {
diff --git a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/config.json b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/config.json
index 0daf9dc1c..f0e89e315 100644
--- a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/config.json
+++ b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/config.json
@@ -54,7 +54,8 @@
"historicalPlannerModels": 13,
"historicalVisionDirs": [
"test/vision/results/2026-08-27-openrouter-full_qwen_qwen3.8-flash_production",
- "test/vision/results/2026-08-27-openrouter-full_z-ai_glm-5.3-flash_production"
+ "test/vision/results/2026-08-27-openrouter-full_z-ai_glm-5.3-flash_production",
+ "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production"
],
"note": "Only October 14 models are new inference. August historical responses are retained, dated separately, and verified against matching pinned inputs. Full historical request byte equality is unprovable because old planner runs omitted bodies."
}
diff --git a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/historical-vision-regression.json b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/historical-vision-regression.json
index 6f1f94c42..3b02d58da 100644
--- a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/historical-vision-regression.json
+++ b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/historical-vision-regression.json
@@ -1,6 +1,6 @@
{
- "models": 2,
- "cases": 200,
+ "models": 3,
+ "cases": 300,
"inputAndScoreMatches": true,
"rows": [
{
@@ -1563,6 +1563,821 @@
"criticalFailures": []
}
]
+ },
+ {
+ "model": "inclusionai/ling-3.0-flash-vl",
+ "label": "Ling 3.0 Flash VL",
+ "date": "2026-10-07",
+ "rawStreams": true,
+ "path": "test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production",
+ "cases": 100,
+ "errors": 0,
+ "passes": 65,
+ "meanScore": 0.8809624264624261,
+ "meanLatencyMs": 3572.28,
+ "medianLatencyMs": 3458.5,
+ "p95LatencyMs": 4878,
+ "costRecords": 100,
+ "usageCost": 0.005382260800000002,
+ "promptTokens": 120800,
+ "completionTokens": 51778,
+ "cachedTokens": 20480,
+ "reasoningTokens": 33885,
+ "sixSections": 98,
+ "emptyOutputs": 0,
+ "lengthLimited": 22,
+ "byDifficulty": {
+ "easy": {
+ "cases": 20,
+ "passes": 14
+ },
+ "basic": {
+ "cases": 20,
+ "passes": 14
+ },
+ "intermediate": {
+ "cases": 20,
+ "passes": 13
+ },
+ "advanced": {
+ "cases": 20,
+ "passes": 14
+ },
+ "challenging": {
+ "cases": 20,
+ "passes": 10
+ }
+ },
+ "byCategory": {
+ "authentication": {
+ "cases": 5,
+ "passes": 2
+ },
+ "search-results": {
+ "cases": 5,
+ "passes": 4
+ },
+ "checkout": {
+ "cases": 5,
+ "passes": 3
+ },
+ "form-validation": {
+ "cases": 5,
+ "passes": 1
+ },
+ "modal-overlay": {
+ "cases": 5,
+ "passes": 0
+ },
+ "toast-notification": {
+ "cases": 5,
+ "passes": 4
+ },
+ "loading-state": {
+ "cases": 5,
+ "passes": 3
+ },
+ "consent-banner": {
+ "cases": 5,
+ "passes": 2
+ },
+ "dashboard": {
+ "cases": 5,
+ "passes": 4
+ },
+ "chart-reading": {
+ "cases": 5,
+ "passes": 5
+ },
+ "data-table": {
+ "cases": 5,
+ "passes": 5
+ },
+ "email-compose": {
+ "cases": 5,
+ "passes": 3
+ },
+ "kanban-board": {
+ "cases": 5,
+ "passes": 5
+ },
+ "calendar": {
+ "cases": 5,
+ "passes": 5
+ },
+ "map-and-travel": {
+ "cases": 5,
+ "passes": 4
+ },
+ "photo-understanding": {
+ "cases": 5,
+ "passes": 4
+ },
+ "multilingual-ocr": {
+ "cases": 5,
+ "passes": 3
+ },
+ "occlusion-and-contrast": {
+ "cases": 5,
+ "passes": 2
+ },
+ "security-challenge": {
+ "cases": 5,
+ "passes": 2
+ },
+ "uncertainty-calibration": {
+ "cases": 5,
+ "passes": 4
+ }
+ },
+ "providers": {
+ "Novita": {
+ "cases": 100,
+ "passes": 65
+ }
+ },
+ "caseScores": [
+ {
+ "id": "001",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": []
+ },
+ {
+ "id": "002",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "003",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "004",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "005",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "006",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "007",
+ "pass": false,
+ "ratio": 0.75,
+ "criticalFailures": [
+ "heading"
+ ]
+ },
+ {
+ "id": "008",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "consent-signal"
+ ]
+ },
+ {
+ "id": "009",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "010",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "011",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "012",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "013",
+ "pass": true,
+ "ratio": 0.9047619047619048,
+ "criticalFailures": []
+ },
+ {
+ "id": "014",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "015",
+ "pass": true,
+ "ratio": 0.75,
+ "criticalFailures": []
+ },
+ {
+ "id": "016",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "017",
+ "pass": true,
+ "ratio": 0.8888888888888888,
+ "criticalFailures": []
+ },
+ {
+ "id": "018",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "cover"
+ ]
+ },
+ {
+ "id": "019",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "label"
+ ]
+ },
+ {
+ "id": "020",
+ "pass": true,
+ "ratio": 0.8333333333333334,
+ "criticalFailures": []
+ },
+ {
+ "id": "021",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "purpose"
+ ]
+ },
+ {
+ "id": "022",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "023",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "banner"
+ ]
+ },
+ {
+ "id": "024",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "025",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "026",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "027",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "028",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "consent-signal"
+ ]
+ },
+ {
+ "id": "029",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "030",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "031",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "032",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "033",
+ "pass": true,
+ "ratio": 0.8571428571428571,
+ "criticalFailures": []
+ },
+ {
+ "id": "034",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "035",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "036",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "037",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "038",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "target"
+ ]
+ },
+ {
+ "id": "039",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "040",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "041",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "042",
+ "pass": true,
+ "ratio": 0.9166666666666666,
+ "criticalFailures": []
+ },
+ {
+ "id": "043",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "044",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "045",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "046",
+ "pass": false,
+ "ratio": 0.3333333333333333,
+ "criticalFailures": [
+ "toast",
+ "detail"
+ ]
+ },
+ {
+ "id": "047",
+ "pass": true,
+ "ratio": 0.9090909090909091,
+ "criticalFailures": []
+ },
+ {
+ "id": "048",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "049",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "050",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "051",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "052",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "purpose",
+ "status"
+ ]
+ },
+ {
+ "id": "053",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "054",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "055",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "056",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "057",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "058",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "cover"
+ ]
+ },
+ {
+ "id": "059",
+ "pass": false,
+ "ratio": 0.6363636363636364,
+ "criticalFailures": [
+ "captcha-signal",
+ "security-blocker"
+ ]
+ },
+ {
+ "id": "060",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "061",
+ "pass": true,
+ "ratio": 0.9230769230769231,
+ "criticalFailures": []
+ },
+ {
+ "id": "062",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "063",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "064",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "input"
+ ]
+ },
+ {
+ "id": "065",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "066",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "067",
+ "pass": false,
+ "ratio": 0.6666666666666666,
+ "criticalFailures": [
+ "heading"
+ ]
+ },
+ {
+ "id": "068",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ },
+ {
+ "id": "069",
+ "pass": false,
+ "ratio": 0.75,
+ "criticalFailures": [
+ "alert"
+ ]
+ },
+ {
+ "id": "070",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "071",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "072",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "073",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "074",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "075",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "076",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "077",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "078",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "079",
+ "pass": false,
+ "ratio": 0.2727272727272727,
+ "criticalFailures": [
+ "label",
+ "captcha-signal",
+ "security-blocker"
+ ]
+ },
+ {
+ "id": "080",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "visible-fragment"
+ ]
+ },
+ {
+ "id": "081",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "purpose"
+ ]
+ },
+ {
+ "id": "082",
+ "pass": false,
+ "ratio": 0.8333333333333334,
+ "criticalFailures": [
+ "purpose",
+ "notice"
+ ]
+ },
+ {
+ "id": "083",
+ "pass": false,
+ "ratio": 0.8461538461538461,
+ "criticalFailures": [
+ "blocked"
+ ]
+ },
+ {
+ "id": "084",
+ "pass": false,
+ "ratio": 0.7,
+ "criticalFailures": [
+ "input",
+ "blocker"
+ ]
+ },
+ {
+ "id": "085",
+ "pass": false,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": [
+ "modal-signal"
+ ]
+ },
+ {
+ "id": "086",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "087",
+ "pass": true,
+ "ratio": 0.8181818181818182,
+ "criticalFailures": []
+ },
+ {
+ "id": "088",
+ "pass": false,
+ "ratio": 0.5,
+ "criticalFailures": [
+ "title",
+ "choice-1",
+ "consent-blocker"
+ ]
+ },
+ {
+ "id": "089",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "090",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "091",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "092",
+ "pass": false,
+ "ratio": 0.5833333333333334,
+ "criticalFailures": [
+ "purpose",
+ "status",
+ "external-blocker"
+ ]
+ },
+ {
+ "id": "093",
+ "pass": true,
+ "ratio": 0.8571428571428571,
+ "criticalFailures": []
+ },
+ {
+ "id": "094",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "095",
+ "pass": false,
+ "ratio": 0.8,
+ "criticalFailures": [
+ "route-alert"
+ ]
+ },
+ {
+ "id": "096",
+ "pass": false,
+ "ratio": 0.5555555555555556,
+ "criticalFailures": [
+ "caption"
+ ]
+ },
+ {
+ "id": "097",
+ "pass": false,
+ "ratio": 0.5454545454545454,
+ "criticalFailures": [
+ "input",
+ "status"
+ ]
+ },
+ {
+ "id": "098",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "099",
+ "pass": true,
+ "ratio": 1,
+ "criticalFailures": []
+ },
+ {
+ "id": "100",
+ "pass": true,
+ "ratio": 0.9,
+ "criticalFailures": []
+ }
+ ]
}
]
}
diff --git a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/report.md b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/report.md
index caf17048c..aa26bc5ee 100644
--- a/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/report.md
+++ b/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25/report.md
@@ -24,3 +24,4 @@ Only the four October 14 planner models and two MiMo vision runs are new. Histor
| Qwen3.8 Flash | 79 | 94.8% | 8.49s | 13.43s | $0.0411 |
| MiMo-V2.6-Pro | 77 | 94.0% | 5.73s | 11.59s | $0.0645 |
| GLM-5.3 Flash | 76 | 93.8% | 5.49s | 8.92s | $0.0211 |
+| Ling 3.0 Flash VL | 65 | 88.1% | 3.46s | 4.88s | $0.0054 |
diff --git a/test/llm/prepare-hy4-mimo26-lfm25.mjs b/test/llm/prepare-hy4-mimo26-lfm25.mjs
index f59af2c10..c0a83024a 100644
--- a/test/llm/prepare-hy4-mimo26-lfm25.mjs
+++ b/test/llm/prepare-hy4-mimo26-lfm25.mjs
@@ -43,6 +43,7 @@ async function prepare() {
historicalVisionDirs: [
'test/vision/results/2026-08-27-openrouter-full_qwen_qwen3.8-flash_production',
'test/vision/results/2026-08-27-openrouter-full_z-ai_glm-5.3-flash_production',
+ 'test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production',
],
note: 'Only October 14 models are new inference. August historical responses are retained, dated separately, and verified against matching pinned inputs. Full historical request byte equality is unprovable because old planner runs omitted bodies.' };
await writeFile(`${OUT}/config.json`, JSON.stringify(config, null, 2) + '\n');
diff --git a/test/llm/report-hy4-mimo26-lfm25.mjs b/test/llm/report-hy4-mimo26-lfm25.mjs
index 9db422027..58001397b 100644
--- a/test/llm/report-hy4-mimo26-lfm25.mjs
+++ b/test/llm/report-hy4-mimo26-lfm25.mjs
@@ -212,6 +212,8 @@ for (const [path, sha256] of Object.entries(visionManifest.sourceHashes)) {
const visionSources = [
{ model: 'qwen/qwen3.8-flash', label: 'Qwen3.8 Flash', date: '2026-08-27', path: 'test/vision/results/2026-08-27-openrouter-full_qwen_qwen3.8-flash_production' },
{ model: 'z-ai/glm-5.3-flash', label: 'GLM-5.3 Flash', date: '2026-08-27', path: 'test/vision/results/2026-08-27-openrouter-full_z-ai_glm-5.3-flash_production' },
+ { model: 'inclusionai/ling-3.0-flash-vl', label: 'Ling 3.0 Flash VL', date: '2026-10-07', rawStreams: true,
+ path: 'test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production' },
];
function vision(source) {
const cases = ids.map(id => read(join(root, source.path, `${id}.json`)));
@@ -227,7 +229,7 @@ function vision(source) {
assert.deepEqual(c.request, { systemPrompt: VISION_SYSTEM_PROMPT, userText: PRODUCTION_USER_TEXT,
foldSystem: false, temperature: 0, maxTokens: 800, chatTemplateKwargs: REQUEST_DEFAULTS.chatTemplateKwargs });
assert.deepEqual(c.score, scoreVisionResponse({ content: c.response.content, expected: visionRubrics[c.id] }));
- if (source.fresh) {
+ if (source.fresh || source.rawStreams) {
const events = c.response.raw.split(/\r?\n/).filter(line => line.startsWith('data: ') && line !== 'data: [DONE]').map(line => JSON.parse(line.slice(6)));
assert(events.every(event => !event.error));
assert(c.response.raw.includes('data: [DONE]'));
@@ -252,7 +254,7 @@ function vision(source) {
cachedTokens: cases.reduce((n,c)=>n+(c.response.usage?.prompt_tokens_details?.cached_tokens||0),0),
reasoningTokens: cases.reduce((n,c)=>n+(c.response.usage?.completion_tokens_details?.reasoning_tokens||0),0),
sixSections: cases.filter(c=>c.score.sectionCount===6).length, emptyOutputs: cases.filter(c=>!c.response.content.trim()).length,
- lengthLimited: source.fresh ? cases.filter(c=>c.response.finishReason==='length').length : null,
+ lengthLimited: cases.some(c=>c.response.finishReason!=null) ? cases.filter(c=>c.response.finishReason==='length').length : null,
byDifficulty: group(c=>c.question.difficulty.slug), byCategory: group(c=>c.question.category), providers: group(c=>c.response.provider||'not recorded'),
caseScores: cases.map(c=>({id:c.id,pass:c.score.success,ratio:c.score.ratio,criticalFailures:c.score.criticalFailures})) };
assert.equal(row.costRecords,100);
@@ -264,10 +266,10 @@ function vision(source) {
return row;
}
const oldVision = visionSources.map(vision);
-writeFileSync(join(out,'historical-vision-regression.json'),JSON.stringify({models:2,cases:200,inputAndScoreMatches:true,rows:oldVision},null,2)+'\n');
+writeFileSync(join(out,'historical-vision-regression.json'),JSON.stringify({models:visionSources.length,cases:oldVision.length*100,inputAndScoreMatches:true,rows:oldVision},null,2)+'\n');
if (process.argv.includes('--historical-only')) {
- console.log('Historical regression passed: 1,300 planner responses and 200 vision responses.');
+ console.log(`Historical regression passed: 1,300 planner responses and ${oldVision.length*100} vision responses.`);
} else {
const newCohort = fresh.map(source => load(source));
for (const entry of newCohort) assert.equal(entry.row.errors,0, `Recover and archive failed requests before ranking: ${entry.row.model}`);
@@ -293,7 +295,7 @@ writeFileSync(join(out,'historical-vision-regression.json'),JSON.stringify({mode
startedAt:fullCampaign.startedAt,finishedAt:fullCampaign.finishedAt,wallTimeMs:Date.parse(fullCampaign.finishedAt)-Date.parse(fullCampaign.startedAt),
concurrency:fullCampaign.concurrency,note:'Six runners execute concurrently. All individual request latencies include network/provider/generation time; historical infrastructure, caches, prices and concurrency conditions differ.' };
const report = {date:DATE,status:'complete',sourceCommit:replay.meta.sourceCommit,replaySha256:hash(readFileSync(replayPath)),
- casesPerPlannerModel:100,actTools:48,consensusModels:17,comparisonsPerModel:1600,historicalPlannerModels:13,newPlannerModels:4,newVisionModels:2,
+ casesPerPlannerModel:100,actTools:48,consensusModels:17,comparisonsPerModel:1600,historicalPlannerModels:13,newPlannerModels:4,newVisionModels:2,historicalVisionModels:visionSources.length,
method:'Leave-one-out consensus across all 17 models, comparing the first proposed tool call per response. Recursive object-key sorting only; preserve argument values, defaults, and extra arguments. Additional tool calls are counted separately and not executed or scored. No-tool/no-tool agrees; rejected API requests are excluded from quality. Consensus is agreement, not ground truth.',
caveat:'Only the four October 14 planner models and two MiMo vision runs are new. Historical requests were omitted and are reconstructed from pinned source; full historical body equality is unprovable. Serving dates, provider routes, caches and bills differ. No browser tools are executed.',
results,primaryComparison:results.filter(r=>selected.has(r.model)),visionResults,pairwise,visionPairwise,campaign};
diff --git a/web/blog/hy4-mimo-v26-lfm25-browser-agent-benchmark/index.html b/web/blog/hy4-mimo-v26-lfm25-browser-agent-benchmark/index.html
index ed9181ce4..7ae587822 100644
--- a/web/blog/hy4-mimo-v26-lfm25-browser-agent-benchmark/index.html
+++ b/web/blog/hy4-mimo-v26-lfm25-browser-agent-benchmark/index.html
@@ -537,7 +537,7 @@
<p>All four planners replay the August full-tier source <strong><code>7182c21f</code></strong>: <strong>100 Chrome Act cases, 48 native tools, temperature 0.15, and max_tokens 4,096</strong>. We apply no reasoning-effort override, prompt folding, or compatibility syntax. Only the requested model ID changes. New files save both the request and raw API response.</p>
<p>The offline audit first reproduces all thirteen original models&#39; consensus, schema-valid-call, ideal-tool, and exact-ideal counts from <strong>1,300 historical responses</strong>. We then add these four complete new runs and rescore the entire <strong>seventeen-model pool</strong>, with sixteen peers and <strong>1,600 comparisons per model</strong>. Old thirteen-model percentages are not copied unchanged into the new table.</p>
<p>Exact consensus compares the first proposed tool and its arguments after recursively sorting object keys. Values, defaults, and extra arguments remain significant. Tool consensus ignores the arguments. Schema validity and the archived canonical ideal action are separate checks. No-tool responses can agree with one another; consensus measures agreement, not ground truth. No browser tool is executed.</p>
-<p><strong>Only the four new planners and two MiMo screenshot runs are October 14 inference.</strong> The planner peers retain August 2 results, and the Qwen/GLM screenshot references retain August 27 results. Their infrastructure, prices, caches, and serving dates differ. Old planner bodies were omitted; source reconstruction verifies the matching intended payload without proving full historical byte equality. Raw UTC timestamps begin October 13; the local test date is October 14 in Istanbul.</p>
+<p><strong>Only the four new planners and two MiMo screenshot runs are October 14 inference.</strong> The planner peers retain August 2 results, the Qwen/GLM screenshot references retain August 27 results, and Ling&#39;s screenshot row reuses its October 7 run. Their infrastructure, prices, caches, and serving dates differ. Old planner bodies were omitted; source reconstruction verifies the matching intended payload without proving full historical byte equality. Raw UTC timestamps begin October 13; the local test date is October 14 in Istanbul.</p>
<h2 id="planner-quality-speed-and-the-actual-bill">Planner quality, speed, and the actual bill</h2>
<p>Ranks below come from all seventeen models; the table shows the four new routes and five useful historical references. The complete pool remains in the saved report.</p>
<div class="table-wrap"><table><thead><tr><th>Rank</th><th>Model</th><th>Exact consensus</th><th>Tool consensus</th><th>Valid / emitted</th><th>Ideal tool</th><th>Exact ideal</th><th>Median</th><th>p95</th><th>Cost / 100</th></tr></thead><tbody><tr><td>1</td><td>DeepSeek V4 Flash 0731</td><td>50.9%</td><td>79.6%</td><td>90 / 90</td><td>39</td><td>18</td><td>1.58s</td><td>5.24s</td><td>$0.0504</td></tr><tr><td>2</td><td>Tencent HY3</td><td>47.4%</td><td>77.8%</td><td>90 / 90</td><td>41</td><td>18</td><td>5.12s</td><td>8.98s</td><td>$0.2459</td></tr><tr><td>3</td><td>MiMo-V2.6-Pro</td><td>43.6%</td><td>78.6%</td><td>95 / 95</td><td>44</td><td>16</td><td>6.12s</td><td>46.68s</td><td>$0.1246</td></tr><tr><td>6</td><td>GLM-5.2</td><td>42.8%</td><td>75.9%</td><td>86 / 89</td><td>39</td><td>20</td><td>1.73s</td><td>4.44s</td><td>$0.5452</td></tr><tr><td>7</td><td>MiMo-V2.6-Flash</td><td>41.9%</td><td>73.0%</td><td>96 / 97</td><td>37</td><td>17</td><td>1.20s</td><td>4.26s</td><td>$0.0110</td></tr><tr><td>9</td><td>MiniMax M3</td><td>40.6%</td><td>74.6%</td><td>86 / 89</td><td>33</td><td>17</td><td>2.86s</td><td>7.34s</td><td>$0.5038</td></tr><tr><td>10</td><td>Poolside Laguna XS 2.1</td><td>39.6%</td><td>73.2%</td><td>88 / 89</td><td>31</td><td>5</td><td>1.15s</td><td>2.15s</td><td>$0.0729</td></tr><tr><td>14</td><td>Tencent HY4 Preview</td><td>27.5%</td><td>78.6%</td><td>98 / 98</td><td>36</td><td>16</td><td>5.81s</td><td>11.64s</td><td>$0.2961</td></tr><tr><td>15</td><td>Liquid LFM2.5-24B-A2B</td><td>24.9%</td><td>68.2%</td><td>94 / 95</td><td>23</td><td>6</td><td>1.83s</td><td>6.68s</td><td>$0.0829</td></tr></tbody></table></div>
@@ -551,15 +551,16 @@
<p>Multi-call replies require the same care: <strong>6, 3, 11, and 1 responses</strong> contain multiple native calls. The benchmark compares and validates their <strong>first</strong> call only. The other calls remain in raw evidence and are not executed or scored. First-call validity is not a claim that every proposed action in a reply is valid.</p>
<h2 id="vision-flash-leads-this-small-cohort">Vision: Flash leads this small cohort</h2>
<p>Both MiMo routes receive <strong>100 actual 1280×720 screenshots</strong>, the original production six-section prompt, fixed user text, temperature <strong>0</strong>, and an <strong>800-token</strong> maximum. The original thinking-disabling kwargs are retained, without an extra reasoning override or provider pin. HY4 and Liquid receive no images and remain unscored for vision.</p>
-<p>The report checks image/prompt/question/rubric/scorer parity and reproduces all 200 saved Qwen/GLM grades. A strict pass combines enough weighted substring evidence with the case threshold and critical-check gate. Six numbered sections are a separate coverage measure, not an independent hard pass requirement. Literal matching can reject a recognized synonym, and the score is not a human measure of general visual intelligence.</p>
-<div class="table-wrap"><table><thead><tr><th>Model</th><th>Strict passes / 100</th><th>Mean rubric</th><th>Median</th><th>p95</th><th>Cost / 100</th></tr></thead><tbody><tr><td>MiMo-V2.6-Flash</td><td>81</td><td>94.3%</td><td>1.91s</td><td>4.59s</td><td>$0.0214</td></tr><tr><td>Qwen3.8 Flash</td><td>79</td><td>94.8%</td><td>8.49s</td><td>13.43s</td><td>$0.0411</td></tr><tr><td>MiMo-V2.6-Pro</td><td>77</td><td>94.0%</td><td>5.73s</td><td>11.59s</td><td>$0.0645</td></tr><tr><td>GLM-5.3 Flash</td><td>76</td><td>93.8%</td><td>5.49s</td><td>8.92s</td><td>$0.0211</td></tr></tbody></table></div>
-<p>Flash&#39;s <strong>81 passes</strong> lead the four rows, versus Pro&#39;s <strong>77</strong>. Its screenshot median is <strong>1.91s</strong>, compared with <strong>5.73s</strong> for Pro, and its $0.0214 bill is roughly one third of Pro&#39;s $0.0645. Qwen and GLM remain dated references; GLM&#39;s old bill reflects its August launch promotion, not a claim that that price remains available today. Small pass gaps from one run each do not establish a stable statistical ranking.</p>
+<p>The report checks image/prompt/question/rubric/scorer parity and reproduces all <strong>300 saved Qwen/GLM/Ling grades</strong>. A strict pass combines enough weighted substring evidence with the case threshold and critical-check gate. Six numbered sections are a separate coverage measure, not an independent hard pass requirement. Literal matching can reject a recognized synonym, and the score is not a human measure of general visual intelligence.</p>
+<div class="table-wrap"><table><thead><tr><th>Model</th><th>Strict passes / 100</th><th>Mean rubric</th><th>Median</th><th>p95</th><th>Cost / 100</th></tr></thead><tbody><tr><td>MiMo-V2.6-Flash</td><td>81</td><td>94.3%</td><td>1.91s</td><td>4.59s</td><td>$0.0214</td></tr><tr><td>Qwen3.8 Flash</td><td>79</td><td>94.8%</td><td>8.49s</td><td>13.43s</td><td>$0.0411</td></tr><tr><td>MiMo-V2.6-Pro</td><td>77</td><td>94.0%</td><td>5.73s</td><td>11.59s</td><td>$0.0645</td></tr><tr><td>GLM-5.3 Flash</td><td>76</td><td>93.8%</td><td>5.49s</td><td>8.92s</td><td>$0.0211</td></tr><tr><td>Ling 3.0 Flash VL</td><td>65</td><td>88.1%</td><td>3.46s</td><td>4.88s</td><td>$0.0054</td></tr></tbody></table></div>
+<p>Flash&#39;s <strong>81 passes</strong> lead the five rows, versus Pro&#39;s <strong>77</strong>. Its screenshot median is <strong>1.91s</strong>, compared with <strong>5.73s</strong> for Pro, and its $0.0214 bill is roughly one third of Pro&#39;s $0.0645. Qwen and GLM remain dated references; GLM&#39;s old bill reflects its August launch promotion, not a claim that that price remains available today. Small pass gaps from one run each do not establish a stable statistical ranking.</p>
+<p><strong>Ling 3.0 Flash VL is a compatible historical vision reference.</strong> Its <a href="/blog/ling30-flash-vl-compared-with-budget-vision-models">October 7 screenshot test</a> used the same 100 image hashes, system/user prompt, temperature 0, 800-token maximum, thinking-disabling kwargs, and unchanged scorer. Its 65 passes cost $0.0054, roughly a quarter of Flash&#39;s bill, with a 3.46-second median. That bill reflects Novita&#39;s dated promotion and cache state. Ling&#39;s earlier text-planner test used the May 41-tool protocol, so it does not enter the seventeen-model planner ranking above. No new Ling inference was run for this addition.</p>
<p>Qwen still has the highest <strong>mean weighted rubric score, 94.8%</strong>, versus Flash&#39;s 94.3%. Flash and Qwen both pass 75 screenshots; Flash alone passes six and Qwen alone passes four. Flash and Pro also share 75 passes, with six Flash-only and two Pro-only passes. Strict threshold passes and average recovered evidence describe different aspects of the same responses.</p>
<p>Strict-pass rates by difficulty, twenty images per band:</p>
-<div class="table-wrap"><table><thead><tr><th>Band</th><th>MiMo-V2.6-Flash</th><th>Qwen3.8 Flash</th><th>MiMo-V2.6-Pro</th><th>GLM-5.3 Flash</th></tr></thead><tbody><tr><td>Easy</td><td>85%</td><td>70%</td><td>85%</td><td>80%</td></tr><tr><td>Basic</td><td>75%</td><td>85%</td><td>80%</td><td>70%</td></tr><tr><td>Intermediate</td><td>90%</td><td>85%</td><td>80%</td><td>85%</td></tr><tr><td>Advanced</td><td>75%</td><td>80%</td><td>65%</td><td>80%</td></tr><tr><td>Challenging</td><td>80%</td><td>75%</td><td>75%</td><td>65%</td></tr></tbody></table></div>
+<div class="table-wrap"><table><thead><tr><th>Band</th><th>MiMo-V2.6-Flash</th><th>Qwen3.8 Flash</th><th>MiMo-V2.6-Pro</th><th>GLM-5.3 Flash</th><th>Ling 3.0 VL</th></tr></thead><tbody><tr><td>Easy</td><td>85%</td><td>70%</td><td>85%</td><td>80%</td><td>70%</td></tr><tr><td>Basic</td><td>75%</td><td>85%</td><td>80%</td><td>70%</td><td>70%</td></tr><tr><td>Intermediate</td><td>90%</td><td>85%</td><td>80%</td><td>85%</td><td>65%</td></tr><tr><td>Advanced</td><td>75%</td><td>80%</td><td>65%</td><td>80%</td><td>70%</td></tr><tr><td>Challenging</td><td>80%</td><td>75%</td><td>75%</td><td>65%</td><td>50%</td></tr></tbody></table></div>
<p>Flash scores <strong>4/5 on multilingual OCR</strong>, compared with <strong>2/5</strong> for Pro. It also handles all five checkout, email, chart, photo, and map cases. Pro wins the calendar comparison 4/5 to 3/5 and occlusion/contrast 5/5 to 4/5. Each category has only five cases, so one screenshot changes its percentage by twenty points.</p>
<p>The shared failures remain important: <strong>both MiMo models score 0/5 on modal overlays and 1/5 on form validation</strong>, just as both Qwen3.8 and GLM-5.3 did in their saved runs. Flash&#39;s higher aggregate pass count does not remove these specific contract weaknesses.</p>
-<p>All new screenshot outputs are nonempty and have six parsed sections. <strong>Neither MiMo run hits the token cap.</strong> Pro reports <strong>1,438 reasoning tokens</strong> within completion usage despite the disabling kwargs; Flash reports zero. Historical finish reasons were not captured, so the report leaves their clipping counts unknown. New raw streams preserve the reasoning evidence; historical <code>reasoningChars: 0</code> fields are an old parser convention.</p>
+<p>All new screenshot outputs are nonempty and have six parsed sections. <strong>Neither MiMo run hits the token cap.</strong> Pro reports <strong>1,438 reasoning tokens</strong> within completion usage despite the disabling kwargs; Flash reports zero. Ling&#39;s saved run reports <strong>33,885 reasoning tokens and 22 length-limited responses</strong>, with 98 parsed six-section outputs, under that same budget. Its route did not eliminate reported reasoning either; the score therefore measures the delivered configuration, not the model&#39;s ceiling with different reasoning controls. Qwen/GLM historical finish reasons were not captured, so their clipping counts remain unknown. The MiMo and Ling raw streams preserve the reasoning evidence; historical <code>reasoningChars: 0</code> fields are an old parser convention.</p>
<h2 id="cache-provider-and-budget-context">Cache, provider, and budget context</h2>
<p>Planner token usage explains why list prices alone miss much of the bill:</p>
<div class="table-wrap"><table><thead><tr><th>Planner</th><th>Prompt tokens</th><th>Completion tokens</th><th>Reasoning tokens within completion</th><th>Cached prompt share</th></tr></thead><tbody><tr><td>Tencent HY4 Preview</td><td>2,317,050</td><td>12,780</td><td>9,158</td><td>89.1%</td></tr><tr><td>MiMo-V2.6-Pro</td><td>2,334,035</td><td>13,593</td><td>9,517</td><td>89.5%</td></tr><tr><td>MiMo-V2.6-Flash</td><td>2,333,851</td><td>3,159</td><td>0</td><td>98.2%</td></tr><tr><td>Liquid LFM2.5-24B-A2B</td><td>2,412,778</td><td>30,106</td><td>28,066</td><td>97.6%</td></tr></tbody></table></div>
@@ -567,11 +568,11 @@
<p>HY4&#39;s 100 planner responses identify Tencent. Flash uses Makora for both suites; Liquid uses its own provider. Pro&#39;s planner responses use DeepInfra and Novita, while all Pro screenshots use DeepInfra. Routes were chosen automatically. HY4&#39;s captured Tencent endpoint also has time-of-day pricing; we use the API&#39;s reported charge rather than applying a single headline rate to every token.</p>
<p>Six independent runners execute concurrently, with three requests per planner model and two per vision model. Median latency uses the average of the two middle successful requests; p95 is the nearest-rank 95th percentile. Both include network, provider, and generation time. They are not isolated model-compute or uncached-throughput measurements.</p>
<p>All reference latencies are recomputed with that same procedure. A median can differ slightly from an earlier article that selected the lower middle observation, such as Qwen&#39;s screenshot median of 8.49 seconds here versus 8.42 seconds in the original post.</p>
-<p>The campaign records <strong>606 Chat Completions requests: 600 scored cases plus six separate preflights</strong>, with zero API errors and zero retries. The scored runs cost <strong>$0.6005</strong> in total; including preflights, the observed bill is <strong>$0.6335</strong>. No planner or screenshot response ends with a length limit. API keys and authorization headers are excluded from the evidence.</p>
+<p>The October 14 campaign records <strong>606 Chat Completions requests: 600 scored cases plus six separate preflights</strong>, with zero API errors and zero retries. The scored runs cost <strong>$0.6005</strong> in total; including preflights, the observed bill is <strong>$0.6335</strong>. None of those 600 new responses ends with a length limit. The reused historical reference runs are excluded from this campaign bill. API keys and authorization headers are excluded from the evidence.</p>
<p>For this workload, <strong>Flash is the strongest new budget default to investigate</strong>: cheap, fast first dispatch, the highest screenshot pass count here, and broad multimodal input. HY4 and Pro still deserve assessment through their ideal-tool choices and argument validity, rather than being reduced to one consensus rank. Liquid is the local-agent-oriented member of the group, with a useful hosted dispatch result; assessing its on-device advantage requires an actual local run. These conclusions concern the measured opening action and screenshot contract, not long-horizon task completion.</p>
<h2 id="raw-results-and-reproduction">Raw results and reproduction</h2>
<p>The <a href="https://github.com/webbrain-one/webbrain/tree/main/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25" target="_blank" rel="noopener">analysis directory</a> contains the seventeen-model report, case hashes, archived ideal actions, historical regressions, pairwise pass differences, provider metadata, campaign timings, and all six runners&#39; logs. The raw <a href="https://github.com/webbrain-one/webbrain/tree/main/test/llm/results" target="_blank" rel="noopener">planner results</a> use the <code>2026-10-14-openrouter-hy4-mimo26-lfm25-full-7182</code> tag; raw <a href="https://github.com/webbrain-one/webbrain/tree/main/test/vision/results" target="_blank" rel="noopener">MiMo vision results</a> use <code>2026-10-14-openrouter-mimo26-vision</code>. Separate preflight directories are committed alongside them.</p>
-<p>Every new planner case saves the complete request and API response. Every new vision case saves raw SSE, final text, usage, provider, finish reason, latency, prompt metadata, and image hashes; the shared PNGs remain in the corpus. All 600 new case results and six preflights accompany this post.</p>
+<p>Every new planner case saves the complete request and API response. Every new vision case saves raw SSE, final text, usage, provider, finish reason, latency, prompt metadata, and image hashes; the shared PNGs remain in the corpus. All 600 new case results and six preflights accompany this post. The <a href="https://github.com/webbrain-one/webbrain/tree/main/test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production" target="_blank" rel="noopener">100 raw Ling vision cases</a> are already committed; this update rechecks and reuses them.</p>
<pre><code class="language-powershell">node test/llm/report-hy4-mimo26-lfm25.mjs
npm run test:vision:validate
node scripts/build-blog.mjs</code></pre>
diff --git a/web/blog/posts/hy4-mimo-v26-lfm25-browser-agent-benchmark.md b/web/blog/posts/hy4-mimo-v26-lfm25-browser-agent-benchmark.md
index 520819dfb..adc2d5ba1 100644
--- a/web/blog/posts/hy4-mimo-v26-lfm25-browser-agent-benchmark.md
+++ b/web/blog/posts/hy4-mimo-v26-lfm25-browser-agent-benchmark.md
@@ -54,7 +54,7 @@ The offline audit first reproduces all thirteen original models' consensus, sche
Exact consensus compares the first proposed tool and its arguments after recursively sorting object keys. Values, defaults, and extra arguments remain significant. Tool consensus ignores the arguments. Schema validity and the archived canonical ideal action are separate checks. No-tool responses can agree with one another; consensus measures agreement, not ground truth. No browser tool is executed.
-**Only the four new planners and two MiMo screenshot runs are October 14 inference.** The planner peers retain August 2 results, and the Qwen/GLM screenshot references retain August 27 results. Their infrastructure, prices, caches, and serving dates differ. Old planner bodies were omitted; source reconstruction verifies the matching intended payload without proving full historical byte equality. Raw UTC timestamps begin October 13; the local test date is October 14 in Istanbul.
+**Only the four new planners and two MiMo screenshot runs are October 14 inference.** The planner peers retain August 2 results, the Qwen/GLM screenshot references retain August 27 results, and Ling's screenshot row reuses its October 7 run. Their infrastructure, prices, caches, and serving dates differ. Old planner bodies were omitted; source reconstruction verifies the matching intended payload without proving full historical byte equality. Raw UTC timestamps begin October 13; the local test date is October 14 in Istanbul.
## Planner quality, speed, and the actual bill
@@ -92,7 +92,7 @@ Multi-call replies require the same care: **6, 3, 11, and 1 responses** contain
Both MiMo routes receive **100 actual 1280×720 screenshots**, the original production six-section prompt, fixed user text, temperature **0**, and an **800-token** maximum. The original thinking-disabling kwargs are retained, without an extra reasoning override or provider pin. HY4 and Liquid receive no images and remain unscored for vision.
-The report checks image/prompt/question/rubric/scorer parity and reproduces all 200 saved Qwen/GLM grades. A strict pass combines enough weighted substring evidence with the case threshold and critical-check gate. Six numbered sections are a separate coverage measure, not an independent hard pass requirement. Literal matching can reject a recognized synonym, and the score is not a human measure of general visual intelligence.
+The report checks image/prompt/question/rubric/scorer parity and reproduces all **300 saved Qwen/GLM/Ling grades**. A strict pass combines enough weighted substring evidence with the case threshold and critical-check gate. Six numbered sections are a separate coverage measure, not an independent hard pass requirement. Literal matching can reject a recognized synonym, and the score is not a human measure of general visual intelligence.
| Model | Strict passes / 100 | Mean rubric | Median | p95 | Cost / 100 |
| --- | ---: | ---: | ---: | ---: | ---: |
@@ -100,26 +100,29 @@ The report checks image/prompt/question/rubric/scorer parity and reproduces all
| Qwen3.8 Flash | 79 | 94.8% | 8.49s | 13.43s | $0.0411 |
| MiMo-V2.6-Pro | 77 | 94.0% | 5.73s | 11.59s | $0.0645 |
| GLM-5.3 Flash | 76 | 93.8% | 5.49s | 8.92s | $0.0211 |
+| Ling 3.0 Flash VL | 65 | 88.1% | 3.46s | 4.88s | $0.0054 |
-Flash's **81 passes** lead the four rows, versus Pro's **77**. Its screenshot median is **1.91s**, compared with **5.73s** for Pro, and its $0.0214 bill is roughly one third of Pro's $0.0645. Qwen and GLM remain dated references; GLM's old bill reflects its August launch promotion, not a claim that that price remains available today. Small pass gaps from one run each do not establish a stable statistical ranking.
+Flash's **81 passes** lead the five rows, versus Pro's **77**. Its screenshot median is **1.91s**, compared with **5.73s** for Pro, and its $0.0214 bill is roughly one third of Pro's $0.0645. Qwen and GLM remain dated references; GLM's old bill reflects its August launch promotion, not a claim that that price remains available today. Small pass gaps from one run each do not establish a stable statistical ranking.
+
+**Ling 3.0 Flash VL is a compatible historical vision reference.** Its [October 7 screenshot test](/blog/ling30-flash-vl-compared-with-budget-vision-models) used the same 100 image hashes, system/user prompt, temperature 0, 800-token maximum, thinking-disabling kwargs, and unchanged scorer. Its 65 passes cost $0.0054, roughly a quarter of Flash's bill, with a 3.46-second median. That bill reflects Novita's dated promotion and cache state. Ling's earlier text-planner test used the May 41-tool protocol, so it does not enter the seventeen-model planner ranking above. No new Ling inference was run for this addition.
Qwen still has the highest **mean weighted rubric score, 94.8%**, versus Flash's 94.3%. Flash and Qwen both pass 75 screenshots; Flash alone passes six and Qwen alone passes four. Flash and Pro also share 75 passes, with six Flash-only and two Pro-only passes. Strict threshold passes and average recovered evidence describe different aspects of the same responses.
Strict-pass rates by difficulty, twenty images per band:
-| Band | MiMo-V2.6-Flash | Qwen3.8 Flash | MiMo-V2.6-Pro | GLM-5.3 Flash |
-| --- | ---: | ---: | ---: | ---: |
-| Easy | 85% | 70% | 85% | 80% |
-| Basic | 75% | 85% | 80% | 70% |
-| Intermediate | 90% | 85% | 80% | 85% |
-| Advanced | 75% | 80% | 65% | 80% |
-| Challenging | 80% | 75% | 75% | 65% |
+| Band | MiMo-V2.6-Flash | Qwen3.8 Flash | MiMo-V2.6-Pro | GLM-5.3 Flash | Ling 3.0 VL |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Easy | 85% | 70% | 85% | 80% | 70% |
+| Basic | 75% | 85% | 80% | 70% | 70% |
+| Intermediate | 90% | 85% | 80% | 85% | 65% |
+| Advanced | 75% | 80% | 65% | 80% | 70% |
+| Challenging | 80% | 75% | 75% | 65% | 50% |
Flash scores **4/5 on multilingual OCR**, compared with **2/5** for Pro. It also handles all five checkout, email, chart, photo, and map cases. Pro wins the calendar comparison 4/5 to 3/5 and occlusion/contrast 5/5 to 4/5. Each category has only five cases, so one screenshot changes its percentage by twenty points.
The shared failures remain important: **both MiMo models score 0/5 on modal overlays and 1/5 on form validation**, just as both Qwen3.8 and GLM-5.3 did in their saved runs. Flash's higher aggregate pass count does not remove these specific contract weaknesses.
-All new screenshot outputs are nonempty and have six parsed sections. **Neither MiMo run hits the token cap.** Pro reports **1,438 reasoning tokens** within completion usage despite the disabling kwargs; Flash reports zero. Historical finish reasons were not captured, so the report leaves their clipping counts unknown. New raw streams preserve the reasoning evidence; historical `reasoningChars: 0` fields are an old parser convention.
+All new screenshot outputs are nonempty and have six parsed sections. **Neither MiMo run hits the token cap.** Pro reports **1,438 reasoning tokens** within completion usage despite the disabling kwargs; Flash reports zero. Ling's saved run reports **33,885 reasoning tokens and 22 length-limited responses**, with 98 parsed six-section outputs, under that same budget. Its route did not eliminate reported reasoning either; the score therefore measures the delivered configuration, not the model's ceiling with different reasoning controls. Qwen/GLM historical finish reasons were not captured, so their clipping counts remain unknown. The MiMo and Ling raw streams preserve the reasoning evidence; historical `reasoningChars: 0` fields are an old parser convention.
## Cache, provider, and budget context
@@ -140,7 +143,7 @@ Six independent runners execute concurrently, with three requests per planner mo
All reference latencies are recomputed with that same procedure. A median can differ slightly from an earlier article that selected the lower middle observation, such as Qwen's screenshot median of 8.49 seconds here versus 8.42 seconds in the original post.
-The campaign records **606 Chat Completions requests: 600 scored cases plus six separate preflights**, with zero API errors and zero retries. The scored runs cost **$0.6005** in total; including preflights, the observed bill is **$0.6335**. No planner or screenshot response ends with a length limit. API keys and authorization headers are excluded from the evidence.
+The October 14 campaign records **606 Chat Completions requests: 600 scored cases plus six separate preflights**, with zero API errors and zero retries. The scored runs cost **$0.6005** in total; including preflights, the observed bill is **$0.6335**. None of those 600 new responses ends with a length limit. The reused historical reference runs are excluded from this campaign bill. API keys and authorization headers are excluded from the evidence.
For this workload, **Flash is the strongest new budget default to investigate**: cheap, fast first dispatch, the highest screenshot pass count here, and broad multimodal input. HY4 and Pro still deserve assessment through their ideal-tool choices and argument validity, rather than being reduced to one consensus rank. Liquid is the local-agent-oriented member of the group, with a useful hosted dispatch result; assessing its on-device advantage requires an actual local run. These conclusions concern the measured opening action and screenshot contract, not long-horizon task completion.
@@ -148,7 +151,7 @@ For this workload, **Flash is the strongest new budget default to investigate**:
The [analysis directory](https://github.com/webbrain-one/webbrain/tree/main/test/llm/analysis/2026-10-14-hy4-mimo26-lfm25) contains the seventeen-model report, case hashes, archived ideal actions, historical regressions, pairwise pass differences, provider metadata, campaign timings, and all six runners' logs. The raw [planner results](https://github.com/webbrain-one/webbrain/tree/main/test/llm/results) use the `2026-10-14-openrouter-hy4-mimo26-lfm25-full-7182` tag; raw [MiMo vision results](https://github.com/webbrain-one/webbrain/tree/main/test/vision/results) use `2026-10-14-openrouter-mimo26-vision`. Separate preflight directories are committed alongside them.
-Every new planner case saves the complete request and API response. Every new vision case saves raw SSE, final text, usage, provider, finish reason, latency, prompt metadata, and image hashes; the shared PNGs remain in the corpus. All 600 new case results and six preflights accompany this post.
+Every new planner case saves the complete request and API response. Every new vision case saves raw SSE, final text, usage, provider, finish reason, latency, prompt metadata, and image hashes; the shared PNGs remain in the corpus. All 600 new case results and six preflights accompany this post. The [100 raw Ling vision cases](https://github.com/webbrain-one/webbrain/tree/main/test/vision/results/2026-10-07-openrouter-ling30-budget_inclusionai_ling-3.0-flash-vl_production) are already committed; this update rechecks and reuses them.
```powershell
node test/llm/report-hy4-mimo26-lfm25.mjs