HarnessRouter Community Edition: the self-hosted, Apache-2.0 edition of the unified interface for agent harnesses. Run Codex, Claude Code, Hermes, PI, DSH, and more through one API, with sessions, streaming, files, cancellation, and failure handling. Implements the Unified Harness Protocol (UHP), an open standard. Your keys, your infrastructure.
Harness support matrix
The run's notes, per column, are in support-matrix-notes.md.
Scenarios: first turn, follow-up in the same session, switch model mid-session, artifact (a file the task must produce), recycle (the sandbox is let go on purpose, then a follow-up must recall the first message). pass = ran and answered as asked, FAIL = failed (reason in the notes), n/a = not run.
Provider: agentzero-default
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| agentzero | claude-fable-5 | FAIL | n/a | n/a | n/a | n/a | My TokenRouter | first: The turn failed: the provider declined the request (finish_reason content_filter) ; retested once; first try: first The turn failed: the provider declined the request (finish_reason content_filter ; cache 0% of input |
| agentzero | claude-fable-5-1 | FAIL | n/a | n/a | n/a | n/a | My TokenRouter | first: The turn failed: the provider declined the request (finish_reason content_filter) ; retested once; first try: first The turn failed: the provider declined the request (finish_reason content_filter ; cache 0% of input |
| agentzero | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Anthropic, OpenRouter | no prompt cache reads (finding below) ; served as claude-haiku-4-5-20251001 (the provider's alias of the same model) |
| agentzero | claude-opus-4.7 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) ; served as claude-opus-4-7 (the provider's alias of the same model) |
| agentzero | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) ; served as claude-opus-4-8 (the provider's alias of the same model) |
| agentzero | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | FAIL | FAIL | My TokenRouter, OpenRouter | artifact: The turn failed: the provider declined the request (finish_reason content_filter) ; recycle: The turn failed: the provider declined the request (finish_reason content_filter) ; retested once; first try: artifact The turn failed: the provider declined the request (finish_reason content_filter; recycle The turn failed: the provider declined the request (finish_reason content_filter ; cache 0% of input |
| agentzero | claude-opus-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | no prompt cache reads (finding below) ; served as anthropic/claude-opus-5.5 (the provider's alias of the same model) |
| agentzero | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Anthropic, OpenRouter | no prompt cache reads (finding below) ; served as claude-sonnet-4-6 (the provider's alias of the same model) |
| agentzero | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) |
| agentzero | claude-sonnet-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) ; served as claude-sonnet-5-5 (the provider's alias of the same model) |
| agentzero | deepseek-v4-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as deepseek-v4.1-flash (finding below) ; cache 63% of input |
| agentzero | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 59% of input |
| agentzero | deepseek-v4.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 63% of input |
| agentzero | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | FAIL | My TokenRouter, OpenRouter | recycle: answered without M1-gemini-3-flash-preview: What exact word did I ask you to reply with in my very first message of this task? Reply with ju ; retested once; first try: recycle answered without M1-gemini-3-flash-preview: What exact word did I ask you to rep ; cache 29% of input |
| agentzero | gemini-3.1-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as google/gemini-3.1-flash-lite (the provider's alias of the same model) ; cache 44% of input |
| agentzero | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 29% of input |
| agentzero | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 44% of input |
| agentzero | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 44% of input |
| agentzero | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 19% of input |
| agentzero | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 29% of input |
| agentzero | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 29% of input |
| agentzero | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 71% of input |
| agentzero | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 46% of input |
| agentzero | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Custom OpenAI Chat, OpenRouter | served as gpt-5.2-2025-12-11 (the provider's alias of the same model) ; cache 60% of input |
| agentzero | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as gpt-5.4-2026-03-05 (the provider's alias of the same model) ; cache 58% of input |
| agentzero | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | Custom OpenAI Chat, OpenRouter | served as gpt-5.4-mini-2026-03-17 (the provider's alias of the same model) ; cache 58% of input |
| agentzero | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2, OpenRouter | served as gpt-5.5-2026-04-24 (the provider's alias of the same model) ; cache 56% of input |
| agentzero | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 61% of input |
| agentzero | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 62% of input |
| agentzero | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 66% of input |
| agentzero | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 61% of input |
| agentzero | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 66% of input |
| agentzero | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter, My TokenRouter | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 62% of input |
| agentzero | grok-4.20 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as x-ai/grok-4.20 (the provider's alias of the same model) ; cache 47% of input |
| agentzero | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as x-ai/grok-4.3 (the provider's alias of the same model) ; cache 17% of input |
| agentzero | grok-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 34% of input |
| agentzero | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 6% of input |
| agentzero | grok-build-0.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as x-ai/grok-build-0.1 (the provider's alias of the same model) ; cache 32% of input |
| agentzero | hunyuan-4-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as tencent/hy4-preview (the provider's alias of the same model) ; cache 55% of input |
| agentzero | kimi-k2.7-code | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 62% of input |
| agentzero | kimi-k3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 0% of input |
| agentzero | llama-3.3-70b | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | served as meta-llama/llama-3.3-70b-instruct (the provider's alias of the same model) ; cache 32% of input |
| agentzero | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as mistralai/mistral-medium-3-5 (the provider's alias of the same model) ; cache 63% of input |
| agentzero | muse-glimmer-30b | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-glimmer-30b (the provider's alias of the same model) ; retested once; first try: artifact The turn failed: HandledException: Agent stopped after 5 consecutive unusable mo ; cache 60% of input |
| agentzero | muse-spark-1.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-spark-1.1 (the provider's alias of the same model) ; cache 66% of input |
| agentzero | muse-spark-1.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-spark-1.2 (the provider's alias of the same model) ; cache 67% of input |
| agentzero | muse-spark-1.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-spark-1.3 (the provider's alias of the same model) ; cache 0% of input |
| agentzero | nemotron-3-super | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) ; cache 0% of input |
| agentzero | nemotron-3.5-lightning | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) ; cache 15% of input |
| agentzero | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 57% of input |
| agentzero | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 47% of input |
| agentzero | qwen3.8-27b | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as alibaba/qwen3.8-27b (the provider's alias of the same model) ; cache 51% of input |
| agentzero | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 64% of input |
| agentzero | qwen3.8-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 62% of input |
| agentzero | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as stepfun/step-3.7-flash (the provider's alias of the same model) ; cache 60% of input |
55 pairs, 257 of 262 scenario runs passed; 1 pairs served by another connection or as another model are findings, not counted.
Findings, pairs served by a connection other than the one under test or as a model other than the id asked for:
- agentzero x claude-haiku-4.5: no prompt cache reads over 42994 input tokens (every call paid full price)
- agentzero x claude-opus-4.7: no prompt cache reads over 55588 input tokens (every call paid full price)
- agentzero x claude-opus-4.8: no prompt cache reads over 55260 input tokens (every call paid full price)
- agentzero x claude-opus-5.5: no prompt cache reads over 55502 input tokens (every call paid full price)
- agentzero x claude-sonnet-4.6: no prompt cache reads over 42617 input tokens (every call paid full price)
- agentzero x claude-sonnet-5: no prompt cache reads over 55459 input tokens (every call paid full price)
- agentzero x claude-sonnet-5.5: no prompt cache reads over 55454 input tokens (every call paid full price)
- agentzero x deepseek-v4-flash: served as deepseek-v4.1-flash (the CLI reports the model it ran)
Provider: anthropic
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| agentzero | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Anthropic | no prompt cache reads (finding below) ; served as claude-haiku-5-5 (the provider's alias of the same model) |
| aider | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | FAIL | Anthropic | no prompt cache reads (finding below) ; served as claude-haiku-5-5 (the provider's alias of the same model) ; recycle: answered without M1-claude-haiku-5.5: What exact word did I ask you to reply with in my very first message of this task? Reply with just tha |
| claude-code | claude-fable-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| claude-code | claude-haiku-4.5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| claude-code | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Anthropic | served as claude-haiku-5-5 (the provider's alias of the same model) ; cache 100% of input |
| claude-code | claude-opus-4.7 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| claude-code | claude-opus-4.8 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| claude-code | claude-opus-5 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | |
| claude-code | claude-sonnet-4.6 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| claude-code | claude-sonnet-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| cline | claude-fable-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run on the bare base, partner inside the column ; retested once; first try: first [{"connection": "integration:Anthropic", "status": "failed", "error": "Not Found |
| cline | claude-haiku-4.5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run on the bare base, partner inside the column ; retested once; first try: first [{"connection": "integration:Anthropic", "status": "failed", "error": "Not Found |
| cline | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Anthropic | no prompt cache reads (finding below) ; 4 turn(s) unlabelled (finding below) |
| cline | claude-opus-4.7 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run on the bare base, partner inside the column ; retested once; first try: first [{"connection": "integration:Anthropic", "status": "failed", "error": "Not Found |
| cline | claude-opus-4.8 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run on the bare base, partner inside the column ; retested once; first try: first [{"connection": "integration:Anthropic", "status": "failed", "error": "Not Found |
| cline | claude-opus-5 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | re-run on the bare base, partner inside the column ; retested once; first try: first [{"connection": "integration:Anthropic", "status": "failed", "error": "Not Found |
| cline | claude-sonnet-4.6 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run on the bare base, partner inside the column ; retested once; first try: first [{"connection": "integration:Anthropic", "status": "failed", "error": "Not Found |
| cline | claude-sonnet-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run on the bare base, partner inside the column ; retested once; first try: first [{"connection": "integration:Anthropic", "status": "failed", "error": "Not Found |
| docker-agent | claude-fable-5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| docker-agent | claude-fable-5-1 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| docker-agent | claude-haiku-4.5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| docker-agent | claude-haiku-5.5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| docker-agent | claude-opus-4.8 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| docker-agent | claude-opus-5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| docker-agent | claude-opus-5.5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| docker-agent | claude-sonnet-4.6 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| docker-agent | claude-sonnet-5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| docker-agent | claude-sonnet-5.5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| dsh | claude-fable-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| dsh | claude-haiku-4.5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| dsh | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Anthropic | 4 turn(s) unlabelled (finding below) ; cache 100% of input |
| dsh | claude-opus-4.7 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| dsh | claude-opus-4.8 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| dsh | claude-opus-5 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | |
| dsh | claude-sonnet-4.6 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| dsh | claude-sonnet-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| fast-agent | claude-fable-5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| fast-agent | claude-fable-5-1 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| fast-agent | claude-haiku-4.5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| fast-agent | claude-haiku-5.5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| fast-agent | claude-opus-4.7 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| fast-agent | claude-opus-4.8 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| fast-agent | claude-opus-5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| fast-agent | claude-opus-5.5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| fast-agent | claude-sonnet-4.6 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| fast-agent | claude-sonnet-5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| fast-agent | claude-sonnet-5.5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| goose | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Anthropic | served as claude-haiku-5-5 (the provider's alias of the same model) ; cache 64% of input |
| grok | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Anthropic | no prompt cache reads (finding below) ; served as claude-haiku-5-5 (the provider's alias of the same model) |
| hermes | claude-fable-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| hermes | claude-haiku-4.5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| hermes | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Anthropic | served as claude-haiku-5-5 (the provider's alias of the same model) ; cache 100% of input |
| hermes | claude-opus-4.7 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| hermes | claude-opus-4.8 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| hermes | claude-opus-5 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | |
| hermes | claude-sonnet-4.6 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| hermes | claude-sonnet-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| kilo | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Anthropic | served as claude-haiku-5-5 (the provider's alias of the same model) ; cache 99% of input |
| kimi | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Anthropic | no prompt cache reads (finding below) ; served as claude-haiku-5-5 (the provider's alias of the same model) |
| minimax | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Anthropic | no prompt cache reads (finding below) ; served as claude-haiku-5-5 (the provider's alias of the same model) |
| omp | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Anthropic | served as claude-haiku-5-5 (the provider's alias of the same model) ; cache 100% of input |
| opencode | claude-fable-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run on the bare base, partner inside the column ; retested once; first try: first [{"connection": "integration:Anthropic", "status": "failed", "error": "Not Found |
| opencode | claude-haiku-4.5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | the model answered the first turn with a capabilities blurb instead of the word; one more try ; retested once; first try: first [{"connection": "integration:Anthropic", "status": "failed", "error": "Not Found |
| opencode | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Anthropic | served as claude-haiku-5-5 (the provider's alias of the same model) ; cache 100% of input |
| opencode | claude-opus-4.7 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run on the bare base, partner inside the column ; retested once; first try: first [{"connection": "integration:Anthropic", "status": "failed", "error": "Not Found |
| opencode | claude-opus-4.8 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run on the bare base, partner inside the column ; retested once; first try: first [{"connection": "integration:Anthropic", "status": "failed", "error": "Not Found |
| opencode | claude-opus-5 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | re-run on the bare base, partner inside the column ; retested once; first try: first [{"connection": "integration:Anthropic", "status": "failed", "error": "Not Found |
| opencode | claude-sonnet-4.6 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run on the bare base, partner inside the column ; retested once; first try: first [{"connection": "integration:Anthropic", "status": "failed", "error": "Not Found |
| opencode | claude-sonnet-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run on the bare base, partner inside the column ; retested once; first try: first [{"connection": "integration:Anthropic", "status": "failed", "error": "Not Found |
| openhands | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Anthropic | served as claude-haiku-5-5 (the provider's alias of the same model) ; cache 99% of input |
| pi | claude-fable-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| pi | claude-haiku-4.5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| pi | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Anthropic | served as claude-haiku-5-5 (the provider's alias of the same model) ; cache 83% of input |
| pi | claude-opus-4.7 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| pi | claude-opus-4.8 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| pi | claude-opus-5 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | |
| pi | claude-sonnet-4.6 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| pi | claude-sonnet-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| qwen | claude-fable-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run with the Anthropic base carrying /v1 (0.13.9) ; retested once; first try: artifact no file card (files: none); Create a file named hello-qwen.txt containing exactl; recycle answered without M1-claude-fable-5: What exact word did I ask you to reply with |
| qwen | claude-haiku-4.5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run with the Anthropic base carrying /v1 (0.13.9) ; retested once; first try: artifact no file card (files: none); Create a file named hello-qwen.txt containing exactl; recycle answered without M1-claude-haiku-4.5: What exact word did I ask you to reply wit |
| qwen | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Anthropic | no prompt cache reads (finding below) ; served as claude-haiku-5-5 (the provider's alias of the same model) |
| qwen | claude-opus-4.7 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run with the Anthropic base carrying /v1 (0.13.9) ; retested once; first try: artifact no file card (files: none); Create a file named hello-qwen.txt containing exactl; recycle answered without M1-claude-opus-4.7: What exact word did I ask you to reply with |
| qwen | claude-opus-4.8 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run with the Anthropic base carrying /v1 (0.13.9) ; retested once; first try: artifact no file card (files: none); Create a file named hello-qwen.txt containing exactl; recycle answered without M1-claude-opus-4.8: What exact word did I ask you to reply with |
| qwen | claude-opus-5 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | re-run with the Anthropic base carrying /v1 (0.13.9) ; retested once; first try: artifact no file card (files: none); Create a file named hello-qwen.txt containing exactl; recycle answered without M1-claude-opus-5: What exact word did I ask you to reply with i |
| qwen | claude-sonnet-4.6 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run with the Anthropic base carrying /v1 (0.13.9) ; retested once; first try: artifact no file card (files: none); Create a file named hello-qwen.txt containing exactl; recycle answered without M1-claude-sonnet-4.6: What exact word did I ask you to reply wi |
| qwen | claude-sonnet-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | re-run with the Anthropic base carrying /v1 (0.13.9) ; retested once; first try: artifact no file card (files: none); Create a file named hello-qwen.txt containing exactl; recycle answered without M1-claude-sonnet-5: What exact word did I ask you to reply with |
| vibe | claude-fable-5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| vibe | claude-fable-5-1 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| vibe | claude-haiku-4.5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| vibe | claude-haiku-5.5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| vibe | claude-opus-4.8 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| vibe | claude-opus-5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| vibe | claude-opus-5.5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| vibe | claude-sonnet-4.6 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| vibe | claude-sonnet-5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
| vibe | claude-sonnet-5.5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | no prompt cache reads (finding below) |
96 pairs, 469 of 470 scenario runs passed; 2 pairs served by another connection or as another model are findings, not counted.
Findings, pairs served by a connection other than the one under test or as a model other than the id asked for:
- agentzero x claude-haiku-5.5: no prompt cache reads over 56589 input tokens (every call paid full price)
- aider x claude-haiku-5.5: no prompt cache reads over 24331 input tokens (every call paid full price)
- cline x claude-haiku-5.5: 4 turn(s) report no served model, so rule 2 could not judge them
- cline x claude-haiku-5.5: no prompt cache reads over 51690 input tokens (every call paid full price)
- docker-agent x claude-fable-5: no prompt cache reads over 30158 input tokens (every call paid full price)
- docker-agent x claude-fable-5-1: no prompt cache reads over 30058 input tokens (every call paid full price)
- docker-agent x claude-haiku-4.5: no prompt cache reads over 25246 input tokens (every call paid full price)
- docker-agent x claude-haiku-5.5: no prompt cache reads over 30210 input tokens (every call paid full price)
- docker-agent x claude-opus-4.8: no prompt cache reads over 30198 input tokens (every call paid full price)
- docker-agent x claude-opus-5: no prompt cache reads over 30092 input tokens (every call paid full price)
- docker-agent x claude-opus-5.5: no prompt cache reads over 30148 input tokens (every call paid full price)
- docker-agent x claude-sonnet-4.6: no prompt cache reads over 25247 input tokens (every call paid full price)
- docker-agent x claude-sonnet-5: no prompt cache reads over 30446 input tokens (every call paid full price)
- docker-agent x claude-sonnet-5.5: no prompt cache reads over 30206 input tokens (every call paid full price)
- dsh x claude-haiku-5.5: 4 turn(s) report no served model, so rule 2 could not judge them
- fast-agent x claude-fable-5: no prompt cache reads over 19472 input tokens (every call paid full price)
- fast-agent x claude-fable-5-1: no prompt cache reads over 19510 input tokens (every call paid full price)
- fast-agent x claude-haiku-4.5: no prompt cache reads over 17113 input tokens (every call paid full price)
- fast-agent x claude-haiku-5.5: no prompt cache reads over 19580 input tokens (every call paid full price)
- fast-agent x claude-opus-4.7: no prompt cache reads over 21630 input tokens (every call paid full price)
- fast-agent x claude-opus-4.8: no prompt cache reads over 19684 input tokens (every call paid full price)
- fast-agent x claude-opus-5: no prompt cache reads over 19508 input tokens (every call paid full price)
- fast-agent x claude-opus-5.5: no prompt cache reads over 19766 input tokens (every call paid full price)
- fast-agent x claude-sonnet-4.6: no prompt cache reads over 17242 input tokens (every call paid full price)
- fast-agent x claude-sonnet-5: no prompt cache reads over 19986 input tokens (every call paid full price)
- fast-agent x claude-sonnet-5.5: no prompt cache reads over 19582 input tokens (every call paid full price)
- grok x claude-haiku-5.5: no prompt cache reads over 95190 input tokens (every call paid full price)
- kimi x claude-haiku-5.5: no prompt cache reads over 184580 input tokens (every call paid full price)
- minimax x claude-haiku-5.5: no prompt cache reads over 83845 input tokens (every call paid full price)
- qwen x claude-haiku-5.5: no prompt cache reads over 570021 input tokens (every call paid full price)
- vibe x claude-fable-5: no prompt cache reads over 82556 input tokens (every call paid full price)
- vibe x claude-fable-5-1: no prompt cache reads over 82886 input tokens (every call paid full price)
- vibe x claude-haiku-4.5: no prompt cache reads over 66575 input tokens (every call paid full price)
- vibe x claude-haiku-5.5: no prompt cache reads over 82368 input tokens (every call paid full price)
- vibe x claude-opus-4.8: no prompt cache reads over 82836 input tokens (every call paid full price)
- vibe x claude-opus-5: no prompt cache reads over 82766 input tokens (every call paid full price)
- vibe x claude-opus-5.5: no prompt cache reads over 82980 input tokens (every call paid full price)
- vibe x claude-sonnet-4.6: no prompt cache reads over 66703 input tokens (every call paid full price)
- vibe x claude-sonnet-5: no prompt cache reads over 82898 input tokens (every call paid full price)
- vibe x claude-sonnet-5.5: no prompt cache reads over 82610 input tokens (every call paid full price)
Not run in this column, 113 pairs the provider serves that the harness did not run, with the reason:
- agentzero x claude-fable-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x claude-fable-5-1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x claude-haiku-4.5: not run, not run in this column
- agentzero x claude-opus-4.7: not run, not run in this column
- agentzero x claude-opus-4.8: not run, not run in this column
- agentzero x claude-opus-5: not run, not run in this column
- agentzero x claude-opus-5.5: not run, not run in this column
- agentzero x claude-sonnet-4.6: not run, not run in this column
- agentzero x claude-sonnet-5: not run, not run in this column
- agentzero x claude-sonnet-5.5: not run, not run in this column
- aider x claude-fable-5: not run, not run in this column
- aider x claude-fable-5-1: not run, not run in this column
- aider x claude-haiku-4.5: not run, not run in this column
- aider x claude-opus-4.7: not run, not run in this column
- aider x claude-opus-4.8: not run, not run in this column
- aider x claude-opus-5: not run, not run in this column
- aider x claude-opus-5.5: not run, not run in this column
- aider x claude-sonnet-4.6: not run, not run in this column
- aider x claude-sonnet-5: not run, not run in this column
- aider x claude-sonnet-5.5: not run, not run in this column
- claude-code x claude-fable-5-1: not run, not run in this column
- claude-code x claude-opus-5.5: not run, not run in this column
- claude-code x claude-sonnet-5.5: not run, not run in this column
- cline x claude-fable-5-1: not run, not run in this column
- cline x claude-opus-5.5: not run, not run in this column
- cline x claude-sonnet-5.5: not run, not run in this column
- docker-agent x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x claude-fable-5-1: not run, not run in this column
- dsh x claude-opus-5.5: not run, not run in this column
- dsh x claude-sonnet-5.5: not run, not run in this column
- goose x claude-fable-5: not run, not run in this column
- goose x claude-fable-5-1: not run, not run in this column
- goose x claude-haiku-4.5: not run, not run in this column
- goose x claude-opus-4.7: not run, not run in this column
- goose x claude-opus-4.8: not run, not run in this column
- goose x claude-opus-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-opus-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-sonnet-4.6: not run, not run in this column
- goose x claude-sonnet-5: not run, not run in this column
- goose x claude-sonnet-5.5: not run, not run in this column
- grok x claude-fable-5: not run, not run in this column
- grok x claude-fable-5-1: not run, not run in this column
- grok x claude-haiku-4.5: not run, not run in this column
- grok x claude-opus-4.7: not run, not run in this column
- grok x claude-opus-4.8: not run, not run in this column
- grok x claude-opus-5: not run, not run in this column
- grok x claude-opus-5.5: not run, not run in this column
- grok x claude-sonnet-4.6: not run, not run in this column
- grok x claude-sonnet-5: not run, not run in this column
- grok x claude-sonnet-5.5: not run, not run in this column
- hermes x claude-fable-5-1: not run, not run in this column
- hermes x claude-opus-5.5: not run, not run in this column
- hermes x claude-sonnet-5.5: not run, not run in this column
- kilo x claude-fable-5: not run, not run in this column
- kilo x claude-fable-5-1: not run, not run in this column
- kilo x claude-haiku-4.5: not run, not run in this column
- kilo x claude-opus-4.7: not run, not run in this column
- kilo x claude-opus-4.8: not run, not run in this column
- kilo x claude-opus-5: not run, not run in this column
- kilo x claude-opus-5.5: not run, not run in this column
- kilo x claude-sonnet-4.6: not run, not run in this column
- kilo x claude-sonnet-5: not run, not run in this column
- kilo x claude-sonnet-5.5: not run, not run in this column
- kimi x claude-fable-5: not run, not run in this column
- kimi x claude-fable-5-1: not run, not run in this column
- kimi x claude-haiku-4.5: not run, not run in this column
- kimi x claude-opus-4.7: not run, not run in this column
- kimi x claude-opus-4.8: not run, not run in this column
- kimi x claude-opus-5: not run, not run in this column
- kimi x claude-opus-5.5: not run, not run in this column
- kimi x claude-sonnet-4.6: not run, not run in this column
- kimi x claude-sonnet-5: not run, not run in this column
- kimi x claude-sonnet-5.5: not run, not run in this column
- minimax x claude-fable-5: not run, not run in this column
- minimax x claude-fable-5-1: not run, not run in this column
- minimax x claude-haiku-4.5: not run, not run in this column
- minimax x claude-opus-4.7: not run, not run in this column
- minimax x claude-opus-4.8: not run, not run in this column
- minimax x claude-opus-5: not run, not run in this column
- minimax x claude-opus-5.5: not run, not run in this column
- minimax x claude-sonnet-4.6: not run, not run in this column
- minimax x claude-sonnet-5: not run, not run in this column
- minimax x claude-sonnet-5.5: not run, not run in this column
- omp x claude-fable-5: not run, not run in this column
- omp x claude-fable-5-1: not run, not run in this column
- omp x claude-haiku-4.5: not run, not run in this column
- omp x claude-opus-4.7: not run, not run in this column
- omp x claude-opus-4.8: not run, not run in this column
- omp x claude-opus-5: not run, not run in this column
- omp x claude-opus-5.5: not run, not run in this column
- omp x claude-sonnet-4.6: not run, not run in this column
- omp x claude-sonnet-5: not run, not run in this column
- omp x claude-sonnet-5.5: not run, not run in this column
- opencode x claude-fable-5-1: not run, not run in this column
- opencode x claude-opus-5.5: not run, not run in this column
- opencode x claude-sonnet-5.5: not run, not run in this column
- openhands x claude-fable-5: not run, not run in this column
- openhands x claude-fable-5-1: not run, not run in this column
- openhands x claude-haiku-4.5: not run, not run in this column
- openhands x claude-opus-4.7: not run, not run in this column
- openhands x claude-opus-4.8: not run, not run in this column
- openhands x claude-opus-5: not run, not run in this column
- openhands x claude-opus-5.5: not run, not run in this column
- openhands x claude-sonnet-4.6: not run, not run in this column
- openhands x claude-sonnet-5: not run, not run in this column
- openhands x claude-sonnet-5.5: not run, not run in this column
- pi x claude-fable-5-1: not run, not run in this column
- pi x claude-opus-5.5: not run, not run in this column
- pi x claude-sonnet-5.5: not run, not run in this column
- qwen x claude-fable-5-1: not run, not run in this column
- qwen x claude-opus-5.5: not run, not run in this column
- qwen x claude-sonnet-5.5: not run, not run in this column
- vibe x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
Provider: azure-e2
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| aider | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | FAIL | Azure OpenAI E2 | recycle: answered without M1-gpt-6-luna: What exact word did I ask you to reply with in my very first message of this task? Reply with just that word ; served as gpt-6-luna-2026-09-22 (the provider's alias of the same model) ; recycle: answered without M1-gpt-6-luna: What |
| aider | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | Azure OpenAI E2 | served as gpt-6-sol-2026-09-22 (the provider's alias of the same model) |
| aider | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | served as gpt-6.1-sol-2026-09-29 (the provider's alias of the same model) ; cache 50% of input |
| cline | gpt-5.2 | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI E2 | retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Res |
| cline | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Res |
| cline | gpt-5.4-mini | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Res |
| cline | gpt-5.5 | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Res |
| cline | gpt-5.6-luna | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Res |
| cline | gpt-5.6-sol | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Res |
| cline | gpt-5.6-terra | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Res |
| cline | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| cline | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | Azure OpenAI E2 | |
| cline | gpt-6.1-sol | FAIL | n/a | n/a | n/a | n/a | Azure OpenAI E2 | 1 turn(s) unlabelled (finding below) ; first: The turn failed: Unsupported value: 'reasoning_effort' does not support 'none' with this model. Supported values are: 'low', 'medium', 'high |
| codex | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Rec |
| codex | gpt-5.3-codex | pass | pass | pass (gpt-5.5) | FAIL | FAIL | Azure OpenAI E2 | artifact: Codex cannot run gpt-5.3-codex in a task that has already used gpt-5.5: its tools are not available there. Start a new task for gpt-5.3-code ; recycle: Codex cannot run gpt-5.3-codex in a task that has already used gpt-5.5: its tools are not available there. Start a new task for gpt-5.3-code ; re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Rec |
| codex | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Rec |
| codex | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Rec |
| codex | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Rec |
| codex | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | FAIL | Azure OpenAI E2 | recycle: answered without M1-gpt-5.6-luna: What exact word did I ask you to reply with in my very first message of this task? Reply with just that wo ; re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Rec |
| codex | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Azure OpenAI E2 | retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Rec |
| codex | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Rec |
| codex | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| codex | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| codex | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| codex | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | 4 turn(s) unlabelled (finding below) ; cache 50% of input |
| dcode | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2 | cache 49% of input |
| dcode | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2 | cache 66% of input |
| dcode | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2 | cache 64% of input |
| dcode | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2 | cache 48% of input |
| dcode | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2 | cache 66% of input |
| dcode | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2 | cache 66% of input |
| dcode | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2 | cache 71% of input |
| dcode | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2 | cache 66% of input |
| dcode | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2 | cache 71% of input |
| dcode | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 66% of input |
| docker-agent | gpt-5.2 | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 76% of input |
| docker-agent | gpt-5.4 | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 99% of input |
| docker-agent | gpt-5.4-mini | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 98% of input |
| docker-agent | gpt-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 78% of input |
| docker-agent | gpt-5.6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 100% of input |
| docker-agent | gpt-5.6-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 100% of input |
| docker-agent | gpt-5.6-terra | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 100% of input |
| docker-agent | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 100% of input |
| docker-agent | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | Azure OpenAI E2 | cache 100% of input |
| dsh | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| dsh | gpt-5.3-codex | pass | pass | pass (gpt-5.5) | pass | pass | Azure OpenAI E2 | retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| dsh | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| dsh | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| dsh | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| dsh | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| dsh | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| dsh | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| dsh | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| dsh | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| dsh | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| dsh | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | 4 turn(s) unlabelled (finding below) ; cache 100% of input |
| fast-agent | gpt-5.2 | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| fast-agent | gpt-5.4 | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| fast-agent | gpt-5.4-mini | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| fast-agent | gpt-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| fast-agent | gpt-5.6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| fast-agent | gpt-5.6-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| fast-agent | gpt-5.6-terra | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| fast-agent | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| fast-agent | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | Azure OpenAI E2 | |
| goose | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 52% of input |
| hermes | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "API |
| hermes | gpt-5.3-codex | pass | pass | pass (gpt-5.5) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "API |
| hermes | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "API |
| hermes | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "API |
| hermes | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "API |
| hermes | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "API |
| hermes | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "API |
| hermes | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "API |
| hermes | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| hermes | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| hermes | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| hermes | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | 4 turn(s) unlabelled (finding below) ; cache 0% of input |
| kimi | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | served as gpt-6-luna-2026-09-22 (the provider's alias of the same model) |
| kimi | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | Azure OpenAI E2 | served as gpt-6-sol-2026-09-22 (the provider's alias of the same model) |
| kimi | gpt-6.1-sol | FAIL | n/a | n/a | n/a | n/a | Azure OpenAI E2 | 1 turn(s) unlabelled (finding below) ; first: The turn failed: error: failed to run prompt: provider.api_error: 400 Unsupported value: 'reasoning_effort' does not support 'none' with thi |
| omp | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| omp | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| omp | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| omp | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | 1 turn(s) unlabelled (finding below) ; cache 100% of input |
| opencode | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Res |
| opencode | gpt-5.3-codex | pass | pass | pass (gpt-5.5) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Res |
| opencode | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Res |
| opencode | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Res |
| opencode | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Res |
| opencode | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Res |
| opencode | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Res |
| opencode | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Res |
| opencode | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| opencode | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| opencode | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| opencode | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 100% of input |
| openhands | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| openhands | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | Azure OpenAI E2 | |
| openhands | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 0% of input |
| pi | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| pi | gpt-5.3-codex | pass | pass | pass (gpt-5.5) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| pi | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| pi | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| pi | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| pi | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| pi | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| pi | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: first [{"connection": "integration:Azure OpenAI E2", "status": "failed", "error": "Ope |
| pi | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | |
| pi | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| pi | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| pi | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 100% of input |
| qwen | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: artifact no file card (files: none); Create a file named hello-qwen.txt containing exactl; recycle answered without M1-gpt-5.2: What exact word did I ask you to reply with in my v |
| qwen | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: artifact no file card (files: none); Create a file named hello-qwen.txt containing exactl; recycle answered without M1-gpt-5.4: What exact word did I ask you to reply with in my v |
| qwen | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: artifact no file card (files: none); Create a file named hello-qwen.txt containing exactl; recycle answered without M1-gpt-5.4-mini: What exact word did I ask you to reply with in |
| qwen | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: artifact no file card (files: none); Create a file named hello-qwen.txt containing exactl; recycle answered without M1-gpt-5.5: What exact word did I ask you to reply with in my v |
| qwen | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: artifact no file card (files: none); Create a file named hello-qwen.txt containing exactl; recycle answered without M1-gpt-5.6-luna: What exact word did I ask you to reply with in |
| qwen | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: artifact no file card (files: none); Create a file named hello-qwen.txt containing exactl; recycle answered without M1-gpt-5.6-sol: What exact word did I ask you to reply with in |
| qwen | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | re-run with the E2 base carrying /openai/v1 ; retested once; first try: artifact no file card (files: none); Create a file named hello-qwen.txt containing exactl; recycle answered without M1-gpt-5.6-terra: What exact word did I ask you to reply with i |
| qwen | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | served as gpt-6-luna-2026-09-22 (the provider's alias of the same model) |
| qwen | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | Azure OpenAI E2 | served as gpt-6-sol-2026-09-22 (the provider's alias of the same model) |
| qwen | gpt-6.1-sol | FAIL | n/a | n/a | n/a | n/a | Azure OpenAI E2 | 1 turn(s) unlabelled (finding below) ; first: The turn failed: API Error: 400 Unsupported value: 'reasoning_effort' does not support 'none' with this model. Supported values are: 'low', |
| vibe | gpt-5.2 | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 17% of input |
| vibe | gpt-5.4 | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 34% of input |
| vibe | gpt-5.4-mini | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 32% of input |
| vibe | gpt-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 16% of input |
| vibe | gpt-5.6-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 17% of input |
| vibe | gpt-5.6-terra | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 33% of input |
| vibe | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | Azure OpenAI E2 | cache 17% of input |
| vibe | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | Azure OpenAI E2 | cache 29% of input |
130 pairs, 611 of 615 scenario runs passed; 7 pairs served by another connection or as another model are findings, not counted.
Findings, pairs served by a connection other than the one under test or as a model other than the id asked for:
- cline x gpt-6.1-sol: 1 turn(s) report no served model, so rule 2 could not judge them
- codex x gpt-6.1-sol: 4 turn(s) report no served model, so rule 2 could not judge them
- dsh x gpt-6.1-sol: 4 turn(s) report no served model, so rule 2 could not judge them
- hermes x gpt-6.1-sol: 4 turn(s) report no served model, so rule 2 could not judge them
- kimi x gpt-6.1-sol: 1 turn(s) report no served model, so rule 2 could not judge them
- omp x gpt-6.1-sol: 1 turn(s) report no served model, so rule 2 could not judge them
- qwen x gpt-6.1-sol: 1 turn(s) report no served model, so rule 2 could not judge them
Not run in this column, 62 pairs the provider serves that the harness did not run, with the reason:
- aider x gpt-5.2: not run, not run in this column
- aider x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- aider x gpt-5.4: not run, not run in this column
- aider x gpt-5.4-mini: not run, not run in this column
- aider x gpt-5.5: not run, not run in this column
- aider x gpt-5.6-luna: not run, not run in this column
- aider x gpt-5.6-sol: not run, not run in this column
- aider x gpt-5.6-terra: not run, not run in this column
- aider x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- dcode x gpt-5.3-codex: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x gpt-6-astra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- docker-agent x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- docker-agent x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- fast-agent x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- fast-agent x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- fast-agent x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- goose x gpt-5.2: not run, not run in this column
- goose x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-5.4: not run, not run in this column
- goose x gpt-5.4-mini: not run, not run in this column
- goose x gpt-5.5: not run, not run in this column
- goose x gpt-5.6-luna: not run, not run in this column
- goose x gpt-5.6-sol: not run, not run in this column
- goose x gpt-5.6-terra: not run, not run in this column
- goose x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-luna: not run, not run in this column
- goose x gpt-6-sol: not run, not run in this column
- kimi x gpt-5.2: not run, not run in this column
- kimi x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- kimi x gpt-5.4: not run, not run in this column
- kimi x gpt-5.4-mini: not run, not run in this column
- kimi x gpt-5.5: not run, not run in this column
- kimi x gpt-5.6-luna: not run, not run in this column
- kimi x gpt-5.6-sol: not run, not run in this column
- kimi x gpt-5.6-terra: not run, not run in this column
- kimi x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- omp x gpt-5.2: not run, not run in this column
- omp x gpt-5.3-codex: not run, not run in this column
- omp x gpt-5.4: not run, not run in this column
- omp x gpt-5.4-mini: not run, not run in this column
- omp x gpt-5.5: not run, not run in this column
- omp x gpt-5.6-luna: not run, not run in this column
- omp x gpt-5.6-sol: not run, not run in this column
- omp x gpt-5.6-terra: not run, not run in this column
- openhands x gpt-5.2: not run, not run in this column
- openhands x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- openhands x gpt-5.4: not run, not run in this column
- openhands x gpt-5.4-mini: not run, not run in this column
- openhands x gpt-5.5: not run, not run in this column
- openhands x gpt-5.6-luna: not run, not run in this column
- openhands x gpt-5.6-sol: not run, not run in this column
- openhands x gpt-5.6-terra: not run, not run in this column
- openhands x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- vibe x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- vibe x gpt-5.6-luna: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- vibe x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
Provider: azure-openai
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| cline | gpt-5.2 | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI | |
| cline | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| cline | gpt-5.4-mini | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI | |
| cline | gpt-5.5 | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI | |
| cline | gpt-5.6-luna | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI | |
| cline | gpt-5.6-sol | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI | |
| cline | gpt-5.6-terra | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI | |
| codex | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | re-run on 0.13.7 (a Codex history is kept whole under the same account) ; retested once; first try: |
| codex | gpt-5.3-codex | pass | pass | pass (gpt-5.6-sol) | FAIL | FAIL | Azure OpenAI | artifact: Codex cannot run gpt-5.3-codex in a task that has already used gpt-5.6-sol: its tools are not available there. Start a new task for gpt-5.3- ; recycle: Codex cannot run gpt-5.3-codex in a task that has already used gpt-5.6-sol: its tools are not available there. Start a new task for gpt-5.3- ; deployment gpt-5.3-codex added to the resource 2026-09-06, then re-run ; retested once; first try: first [{"connection": "integration:Azure OpenAI", "status": "failed", "error": "Reconn |
| codex | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | re-run on 0.13.7 (a Codex history is kept whole under the same account) ; retested once; first try: switch [{"connection": "integration:Azure OpenAI", "status": "failed", "error": "{\n ; artifact [{"connection": "integration:Azure OpenAI", "status": "failed", "error": "Error ; recycle [{"connection": "integration:Azure OpenAI", "status": "failed", "error": "Error |
| codex | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | re-run on 0.13.7 (a Codex history is kept whole under the same account) ; retested once; first try: switch [{"connection": "integration:Azure OpenAI", "status": "failed", "error": "{\n ; artifact [{"connection": "integration:Azure OpenAI", "status": "failed", "error": "Error ; recycle [{"connection": "integration:Azure OpenAI", "status": "failed", "error": "Error |
| codex | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | re-run on 0.13.7 (a Codex history is kept whole under the same account) ; retested once; first try: |
| codex | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | re-run on 0.13.7 (a Codex history is kept whole under the same account) ; retested once; first try: recycle answered without M1-gpt-5.6-luna: What exact word did I ask you to reply with in |
| codex | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Azure OpenAI | re-run on 0.13.7 (a Codex history is kept whole under the same account) ; retested once; first try: |
| codex | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | re-run on 0.13.7 (a Codex history is kept whole under the same account) ; retested once; first try: |
| dsh | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| dsh | gpt-5.3-codex | pass | pass | n/a | pass | pass | Azure OpenAI | deployment gpt-5.3-codex added to the resource 2026-09-06, then re-run ; retested once; first try: first [{"connection": "integration:Azure OpenAI", "status": "failed", "error": "OpenAI |
| dsh | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| dsh | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| dsh | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| dsh | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| dsh | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Azure OpenAI | |
| dsh | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| hermes | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| hermes | gpt-5.3-codex | pass | pass | n/a | pass | pass | Azure OpenAI | deployment gpt-5.3-codex added to the resource 2026-09-06, then re-run ; retested once; first try: first [{"connection": "integration:Azure OpenAI", "status": "failed", "error": "HTTP 4 |
| hermes | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| hermes | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| hermes | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| hermes | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| hermes | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Azure OpenAI | |
| hermes | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| opencode | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| opencode | gpt-5.3-codex | pass | pass | n/a | pass | pass | Azure OpenAI | deployment gpt-5.3-codex added to the resource 2026-09-06, then re-run ; retested once; first try: first [{"connection": "integration:Azure OpenAI", "status": "failed", "error": "The AP |
| opencode | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| opencode | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| opencode | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| opencode | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| opencode | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Azure OpenAI | |
| opencode | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| pi | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| pi | gpt-5.3-codex | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | deployment gpt-5.3-codex added to the resource 2026-09-06, then re-run ; retested once; first try: first [{"connection": "integration:Azure OpenAI", "status": "failed", "error": "OpenAI |
| pi | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| pi | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| pi | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| pi | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| pi | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Azure OpenAI | |
| pi | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| qwen | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| qwen | gpt-5.3-codex | pass | pass | n/a | FAIL | FAIL | Azure OpenAI | artifact: no file card (files: none); Create a file named hello-qwen.txt containing exactly the word HELLO, then reply DONE. QWEN CODE [API Error: 400 ; recycle: answered without M1-gpt-5.3-codex: What exact word did I ask you to reply with in my very first message of this task? Reply with just that w ; deployment gpt-5.3-codex added to the resource 2026-09-06, then re-run ; retested once; first try: artifact no file card (files: none); PI Error: 404 The API deployment for this resource d; recycle answered without M1-gpt-5.3-codex: Reply with just that word. QWEN CODE [API Er |
| qwen | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| qwen | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| qwen | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| qwen | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI | |
| qwen | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Azure OpenAI | |
| qwen | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI |
55 pairs, 267 of 271 scenario runs passed.
Not run in this column, 29 pairs the provider serves that the harness did not run, with the reason:
- cline x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x gpt-6-luna: not run, not run in this column
- cline x gpt-6-sol: not run, not run in this column
- cline x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- codex x gpt-6-astra: not run, not run in this column
- codex x gpt-6-luna: not run, not run in this column
- codex x gpt-6-sol: not run, not run in this column
- codex x gpt-6.1-sol: not run, not run in this column
- dsh x gpt-6-astra: not run, not run in this column
- dsh x gpt-6-luna: not run, not run in this column
- dsh x gpt-6-sol: not run, not run in this column
- dsh x gpt-6.1-sol: not run, not run in this column
- hermes x gpt-6-astra: not run, not run in this column
- hermes x gpt-6-luna: not run, not run in this column
- hermes x gpt-6-sol: not run, not run in this column
- hermes x gpt-6.1-sol: not run, not run in this column
- opencode x gpt-6-astra: not run, not run in this column
- opencode x gpt-6-luna: not run, not run in this column
- opencode x gpt-6-sol: not run, not run in this column
- opencode x gpt-6.1-sol: not run, not run in this column
- pi x gpt-6-astra: not run, not run in this column
- pi x gpt-6-luna: not run, not run in this column
- pi x gpt-6-sol: not run, not run in this column
- pi x gpt-6.1-sol: not run, not run in this column
- qwen x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x gpt-6-luna: not run, not run in this column
- qwen x gpt-6-sol: not run, not run in this column
- qwen x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
Provider: cheetahclaws-tokenrouter
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| cheetahclaws | claude-sonnet-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | |
| cheetahclaws | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as gpt-5.4-2026-03-05 (the provider's alias of the same model) |
| cheetahclaws | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Custom OpenAI Chat, My TokenRouter | served as gpt-5.4-mini-2026-03-17 (the provider's alias of the same model) |
3 pairs, 15 of 15 scenario runs passed.
Provider: codex-azure-e2
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| codex | gpt-5.2 | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| codex | gpt-5.3-codex | pass | pass | n/a | pass | pass | Azure OpenAI E2 | |
| codex | gpt-5.4 | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| codex | gpt-5.4-mini | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| codex | gpt-5.5 | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| codex | gpt-5.6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | retested once; first try: recycle answered without M1-gpt-5.6-luna: What exact word did I ask you to reply with in |
| codex | gpt-5.6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| codex | gpt-5.6-terra | pass | pass | pass (gpt-6-astra) | pass | pass | Azure OpenAI E2 | |
| codex | gpt-6-astra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 |
9 pairs, 44 of 44 scenario runs passed.
Not run in this column, 3 pairs the provider serves that the harness did not run, with the reason:
- codex x gpt-6-luna: not run, not run in this column
- codex x gpt-6-sol: not run, not run in this column
- codex x gpt-6.1-sol: not run, not run in this column
Provider: codex-openai
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| codex | gpt-5.2 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| codex | gpt-5.3-codex | pass | pass | n/a | pass | pass | OpenAI | |
| codex | gpt-5.4 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| codex | gpt-5.4-mini | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| codex | gpt-5.5 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| codex | gpt-5.6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| codex | gpt-5.6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| codex | gpt-5.6-terra | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| codex | gpt-6-astra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI |
9 pairs, 44 of 44 scenario runs passed.
Not run in this column, 3 pairs the provider serves that the harness did not run, with the reason:
- codex x gpt-6-luna: not run, not run in this column
- codex x gpt-6-sol: not run, not run in this column
- codex x gpt-6.1-sol: not run, not run in this column
Provider: codex-openrouter
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| codex | gpt-5.2 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| codex | gpt-5.3-codex | pass | pass | n/a | pass | pass | OpenRouter | |
| codex | gpt-5.4 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| codex | gpt-5.4-mini | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| codex | gpt-5.5 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| codex | gpt-5.6-luna | pass | pass | pass (gpt-6-astra) | pass | FAIL | OpenRouter | recycle: answered without M1-gpt-5.6-luna: What exact word did I ask you to reply with in my very first message of this task? Reply with just that wo ; retested once; first try: recycle answered without M1-gpt-5.6-luna: What exact word did I ask you to reply with in |
| codex | gpt-5.6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| codex | gpt-5.6-terra | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| codex | gpt-6-astra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter |
9 pairs, 43 of 44 scenario runs passed.
Not run in this column, 60 pairs the provider serves that the harness did not run, with the reason:
- codex x claude-fable-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-fable-5-1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-haiku-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-haiku-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-4.8: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-4.6: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4-pro: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3-flash-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.1-flash-lite: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.1-pro-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.5-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.5-flash-lite: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.6-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-4-argon: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x glm-5.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x glm-5.3-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gpt-6-luna: not run, not run in this column
- codex x gpt-6-sol: not run, not run in this column
- codex x gpt-6.1-sol: not run, not run in this column
- codex x grok-4.20: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.6: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-build-0.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x kimi-k3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x llama-3.3-70b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x llama-4-maverick: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x mistral-medium-3.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-glimmer-30b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-spark-1.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-spark-1.2: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-spark-1.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-wonder-1.0: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x nemotron-3-super: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x nemotron-3.5-lightning: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-plus: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-27b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x step-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
Provider: codex-tokenrouter
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| codex | gpt-5.2 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| codex | gpt-5.3-codex | pass | pass | n/a | pass | pass | My TokenRouter | |
| codex | gpt-5.4 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| codex | gpt-5.4-mini | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| codex | gpt-5.5 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| codex | gpt-5.6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| codex | gpt-5.6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| codex | gpt-5.6-terra | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| codex | gpt-6-astra | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter |
9 pairs, 44 of 44 scenario runs passed.
Not run in this column, 40 pairs the provider serves that the harness did not run, with the reason:
- codex x claude-fable-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-fable-5-1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-haiku-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-4.8: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-4.6: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4-pro: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3-flash-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.1-pro-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.5-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.5-flash-lite: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.6-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x glm-5.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x glm-5.3-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gpt-6-luna: not run, not run in this column
- codex x gpt-6-sol: not run, not run in this column
- codex x gpt-6.1-sol: not run, not run in this column
- codex x grok-4.20: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.6: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-build-0.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x mistral-medium-3.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x nemotron-3-super: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x nemotron-3.5-lightning: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-plus: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x step-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
Provider: codex-vercel
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| codex | gpt-5.2 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| codex | gpt-5.3-codex | pass | pass | n/a | pass | pass | Vercel AI Gateway | |
| codex | gpt-5.4 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| codex | gpt-5.4-mini | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| codex | gpt-5.5 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| codex | gpt-5.6-luna | pass | pass | pass (gpt-6-astra) | pass | FAIL | Vercel AI Gateway | recycle: answered without M1-gpt-5.6-luna: What exact word did I ask you to reply with in my very first message of this task? Reply with just that wo ; retested once; first try: recycle answered without M1-gpt-5.6-luna: What exact word did I ask you to reply with in |
| codex | gpt-5.6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| codex | gpt-5.6-terra | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| codex | gpt-6-astra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway |
9 pairs, 43 of 44 scenario runs passed.
Not run in this column, 55 pairs the provider serves that the harness did not run, with the reason:
- codex x claude-fable-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-fable-5-1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-haiku-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-haiku-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-4.8: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-4.6: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4-pro: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3-flash-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.1-flash-lite: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.1-pro-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.5-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.5-flash-lite: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.6-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-4-argon: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x glm-5.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x glm-5.3-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gpt-6-luna: not run, not run in this column
- codex x gpt-6-sol: not run, not run in this column
- codex x gpt-6.1-sol: not run, not run in this column
- codex x grok-4.20: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.6: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-build-0.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x kimi-k3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x mistral-medium-3.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-glimmer-30b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-spark-1.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-spark-1.2: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-spark-1.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-wonder-1.0: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x nemotron-3-super: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-plus: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-27b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x step-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
Provider: custom-openai-chat
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| docker-agent | gpt-5.2 | pass | pass | pass (gpt-5.4-mini) | pass | pass | Custom OpenAI Chat | cache 78% of input |
| docker-agent | gpt-5.4-mini | pass | pass | pass (gpt-5.2) | pass | pass | Custom OpenAI Chat | cache 85% of input |
| fast-agent | gpt-5.2 | pass | pass | pass (gpt-5.4-mini) | pass | pass | Custom OpenAI Chat | |
| fast-agent | gpt-5.4-mini | pass | pass | pass (gpt-5.2) | pass | pass | Custom OpenAI Chat | |
| vibe | gpt-5.2 | pass | pass | pass (gpt-5.4-mini) | pass | pass | Custom OpenAI Chat | cache 41% of input |
| vibe | gpt-5.4-mini | pass | pass | pass (gpt-5.2) | pass | pass | Custom OpenAI Chat | cache 42% of input |
6 pairs, 30 of 30 scenario runs passed.
Provider: custom-openai-responses
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| dcode | gpt-5.3-codex | pass | pass | pass (gpt-5.5) | pass | pass | Custom OpenAI Responses | cache 73% of input |
| dcode | gpt-5.4-mini | pass | pass | pass (gpt-5.5) | pass | pass | Custom OpenAI Responses | cache 81% of input |
| dcode | gpt-5.5 | pass | pass | pass (gpt-5.4-mini) | pass | pass | Custom OpenAI Responses | cache 61% of input |
3 pairs, 15 of 15 scenario runs passed.
Provider: dsh-anthropic
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| dsh | claude-fable-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Anthropic, Vercel AI Gateway | cache 100% of input |
| dsh | claude-fable-5-1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Anthropic, Vercel AI Gateway | cache 100% of input |
| dsh | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Anthropic, Vercel AI Gateway | cache 100% of input |
| dsh | claude-opus-4.7 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Anthropic, Vercel AI Gateway | cache 100% of input |
| dsh | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Anthropic, Vercel AI Gateway | cache 100% of input |
| dsh | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Anthropic, Vercel AI Gateway | cache 100% of input |
| dsh | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Anthropic, Vercel AI Gateway | cache 100% of input |
| dsh | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Anthropic, Vercel AI Gateway | cache 100% of input |
8 pairs, 40 of 40 scenario runs passed.
Not run in this column, 3 pairs the provider serves that the harness did not run, with the reason:
- dsh x claude-haiku-5.5: not run, not run in this column
- dsh x claude-opus-5.5: not run, not run in this column
- dsh x claude-sonnet-5.5: not run, not run in this column
Provider: dsh-azure-e2
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| dsh | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2, Vercel AI Gateway | cache 78% of input |
| dsh | gpt-5.3-codex | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2, Vercel AI Gateway | cache 78% of input |
| dsh | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2, Vercel AI Gateway | cache 59% of input |
| dsh | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2, Vercel AI Gateway | cache 76% of input |
| dsh | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2, Vercel AI Gateway | cache 76% of input |
| dsh | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2, Vercel AI Gateway | cache 100% of input |
| dsh | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2, Vercel AI Gateway | cache 100% of input |
| dsh | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2, Vercel AI Gateway | cache 100% of input |
| dsh | gpt-6-astra | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2, Vercel AI Gateway | cache 100% of input |
9 pairs, 45 of 45 scenario runs passed.
Not run in this column, 3 pairs the provider serves that the harness did not run, with the reason:
- dsh x gpt-6-luna: not run, not run in this column
- dsh x gpt-6-sol: not run, not run in this column
- dsh x gpt-6.1-sol: not run, not run in this column
Provider: dsh-google
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| dsh | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | Google AI Studio, Vercel AI Gateway | cache 39% of input |
| dsh | gemini-3.1-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | Google AI Studio, Vercel AI Gateway | cache 58% of input |
| dsh | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | Google AI Studio, Vercel AI Gateway | cache 45% of input |
| dsh | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Google AI Studio, Vercel AI Gateway | cache 60% of input |
| dsh | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | Google AI Studio, Vercel AI Gateway | cache 44% of input |
| dsh | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Google AI Studio, Vercel AI Gateway | cache 60% of input |
| dsh | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| dsh | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Google AI Studio, Vercel AI Gateway | cache 60% of input |
8 pairs, 40 of 40 scenario runs passed.
Not run in this column, 1 pairs the provider serves that the harness did not run, with the reason:
- dsh x gemini-4-argon: not run, not run in this column
Provider: dsh-openai
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| dsh | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI, Vercel AI Gateway | cache 77% of input |
| dsh | gpt-5.3-codex | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI, Vercel AI Gateway | cache 78% of input |
| dsh | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI, Vercel AI Gateway | cache 73% of input |
| dsh | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI, Vercel AI Gateway | cache 78% of input |
| dsh | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI, Vercel AI Gateway | cache 73% of input |
| dsh | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI, Vercel AI Gateway | cache 100% of input |
| dsh | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI, Vercel AI Gateway | cache 100% of input |
| dsh | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI, Vercel AI Gateway | cache 100% of input |
| dsh | gpt-6-astra | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI, Vercel AI Gateway | cache 100% of input |
9 pairs, 45 of 45 scenario runs passed.
Not run in this column, 3 pairs the provider serves that the harness did not run, with the reason:
- dsh x gpt-6-luna: not run, not run in this column
- dsh x gpt-6-sol: not run, not run in this column
- dsh x gpt-6.1-sol: not run, not run in this column
Provider: dsh-openrouter
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| dsh | claude-fable-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | claude-fable-5-1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | claude-haiku-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | claude-opus-4.7 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | claude-opus-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | claude-sonnet-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | deepseek-v4-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 78% of input |
| dsh | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 79% of input |
| dsh | deepseek-v4.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 0% of input |
| dsh | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 20% of input |
| dsh | gemini-3.1-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 0% of input |
| dsh | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 0% of input |
| dsh | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 54% of input |
| dsh | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 30% of input |
| dsh | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 44% of input |
| dsh | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 20% of input |
| dsh | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 20% of input |
| dsh | gemini-4-argon | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 45% of input |
| dsh | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 79% of input |
| dsh | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 78% of input |
| dsh | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 78% of input |
| dsh | gpt-5.3-codex | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 69% of input |
| dsh | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 70% of input |
| dsh | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 78% of input |
| dsh | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 70% of input |
| dsh | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | gpt-6-astra | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 100% of input |
| dsh | gpt-6.1-sol | pass | pass | pass (gpt-6-astra) | pass | pass | openrouter | cache 100% of input |
| dsh | grok-4.20 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 80% of input |
| dsh | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 60% of input |
| dsh | grok-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 79% of input |
| dsh | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 59% of input |
| dsh | grok-build-0.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 79% of input |
| dsh | hunyuan-4-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 78% of input |
| dsh | kimi-k2.7-code | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 78% of input |
| dsh | kimi-k3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 80% of input |
| dsh | llama-3.3-70b | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 85% of input |
| dsh | llama-4-maverick | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 60% of input |
| dsh | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 79% of input |
| dsh | muse-glimmer-30b | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 60% of input |
| dsh | muse-spark-1.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 59% of input |
| dsh | muse-spark-1.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 20% of input |
| dsh | muse-spark-1.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 1% of input |
| dsh | muse-wonder-1.0 | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 21% of input |
| dsh | nemotron-3-super | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 37% of input |
| dsh | nemotron-3.5-lightning | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 55% of input |
| dsh | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 77% of input |
| dsh | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 76% of input |
| dsh | qwen3.8-27b | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 82% of input |
| dsh | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 82% of input |
| dsh | qwen3.8-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 77% of input |
| dsh | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | openrouter | cache 76% of input |
61 pairs, 305 of 305 scenario runs passed.
Not run in this column, 8 pairs the provider serves that the harness did not run, with the reason:
- dsh x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
Provider: dsh-tokenrouter
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| dsh | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, Vercel AI Gateway | cache 44% of input |
| dsh | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 72% of input |
| dsh | deepseek-v4.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 79% of input |
| dsh | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 41% of input |
| dsh | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 37% of input |
| dsh | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 46% of input |
| dsh | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 30% of input |
| dsh | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 54% of input |
| dsh | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 41% of input |
| dsh | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 41% of input |
| dsh | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 80% of input |
| dsh | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 78% of input |
| dsh | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 68% of input |
| dsh | gpt-5.3-codex | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 69% of input |
| dsh | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 78% of input |
| dsh | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 78% of input |
| dsh | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 69% of input |
| dsh | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 100% of input |
| dsh | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 100% of input |
| dsh | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 100% of input |
| dsh | gpt-6-astra | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 100% of input |
| dsh | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 100% of input |
| dsh | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 100% of input |
| dsh | gpt-6.1-sol | pass | pass | pass (gpt-6-astra) | pass | pass | tokenrouter | cache 100% of input |
| dsh | grok-4.20 | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 66% of input |
| dsh | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 60% of input |
| dsh | grok-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 80% of input |
| dsh | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 73% of input |
| dsh | grok-build-0.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 97% of input |
| dsh | hunyuan-4-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 79% of input |
| dsh | kimi-k2.7-code | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 78% of input |
| dsh | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 79% of input |
| dsh | nemotron-3-super | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 0% of input |
| dsh | nemotron-3.5-lightning | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 36% of input |
| dsh | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 77% of input |
| dsh | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 77% of input |
| dsh | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 80% of input |
| dsh | qwen3.8-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 78% of input |
| dsh | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | tokenrouter | cache 76% of input |
39 pairs, 195 of 195 scenario runs passed.
Not run in this column, 10 pairs the provider serves that the harness did not run, with the reason:
- dsh x claude-fable-5: not run, not run in this column
- dsh x claude-fable-5-1: not run, not run in this column
- dsh x claude-opus-4.7: not run, not run in this column
- dsh x claude-opus-4.8: not run, not run in this column
- dsh x claude-opus-5: not run, not run in this column
- dsh x claude-opus-5.5: not run, not run in this column
- dsh x claude-sonnet-4.6: not run, not run in this column
- dsh x claude-sonnet-5: not run, not run in this column
- dsh x claude-sonnet-5.5: not run, not run in this column
- dsh x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
Provider: dsh-vercel
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| dsh | claude-fable-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | claude-fable-5-1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | claude-haiku-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | claude-opus-4.7 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | claude-opus-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | claude-sonnet-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | deepseek-v4-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 79% of input |
| dsh | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 79% of input |
| dsh | deepseek-v4.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 59% of input |
| dsh | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 42% of input |
| dsh | gemini-3.1-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 15% of input |
| dsh | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | retested once; first try: recycle answered without M1-gemini-3.1-pro-preview: DONE ; cache 40% of input |
| dsh | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 50% of input |
| dsh | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 46% of input |
| dsh | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 40% of input |
| dsh | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 41% of input |
| dsh | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 30% of input |
| dsh | gemini-4-argon | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 44% of input |
| dsh | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 80% of input |
| dsh | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 0% of input |
| dsh | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 68% of input |
| dsh | gpt-5.3-codex | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 69% of input |
| dsh | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 78% of input |
| dsh | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 78% of input |
| dsh | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 69% of input |
| dsh | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | gpt-6-astra | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 100% of input |
| dsh | gpt-6.1-sol | pass | pass | pass (gpt-6-astra) | pass | pass | vercel | cache 100% of input |
| dsh | grok-4.20 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 71% of input |
| dsh | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 79% of input |
| dsh | grok-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 40% of input |
| dsh | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 79% of input |
| dsh | grok-build-0.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 86% of input |
| dsh | hunyuan-4-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 79% of input |
| dsh | kimi-k2.7-code | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 78% of input |
| dsh | kimi-k3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 79% of input |
| dsh | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 0% of input |
| dsh | muse-glimmer-30b | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 60% of input |
| dsh | muse-spark-1.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 60% of input |
| dsh | muse-spark-1.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 79% of input |
| dsh | muse-spark-1.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 39% of input |
| dsh | muse-wonder-1.0 | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 39% of input |
| dsh | nemotron-3-super | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 0% of input |
| dsh | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 76% of input |
| dsh | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 76% of input |
| dsh | qwen3.8-27b | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 72% of input |
| dsh | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 77% of input |
| dsh | qwen3.8-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 71% of input |
| dsh | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | vercel | cache 57% of input |
58 pairs, 290 of 290 scenario runs passed.
Not run in this column, 6 pairs the provider serves that the harness did not run, with the reason:
- dsh x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
Provider: gemini-cli
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| gemini | gemini-3-flash-preview | pass | pass | pass (gemini-3.5-flash) | pass | pass | Google AI Studio | |
| gemini | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.5-flash) | pass | pass | Google AI Studio | |
| gemini | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.5-flash) | pass | pass | Google AI Studio | |
| gemini | gemini-3.5-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | Google AI Studio | |
| gemini | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.5-flash) | pass | pass | Google AI Studio | |
| gemini | gemini-3.6-flash | pass | pass | pass (gemini-3.5-flash) | pass | pass | Google AI Studio | |
| gemini | gemini-3.7-flash | pass | pass | pass (gemini-3.5-flash) | pass | pass | Google AI Studio | |
| gemini | gemini-3.8-flash | pass | pass | pass (gemini-3.5-flash) | pass | pass | Google AI Studio |
8 pairs, 40 of 40 scenario runs passed.
Provider: gemini-google
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| cline | gemini-3-flash-preview | pass | pass | pass (gemini-3.8-flash) | pass | pass | Google AI Studio | |
| cline | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.8-flash) | pass | pass | Google AI Studio | |
| cline | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.8-flash) | pass | pass | Google AI Studio | |
| cline | gemini-3.5-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | Google AI Studio | |
| cline | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.8-flash) | pass | pass | Google AI Studio | |
| cline | gemini-3.7-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | Google AI Studio | |
| cline | gemini-3.8-flash | pass | pass | pass (gemini-3.7-flash) | pass | pass | Google AI Studio | |
| dsh | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| dsh | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| dsh | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| dsh | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| dsh | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| dsh | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | Google AI Studio | |
| dsh | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| dsh | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| hermes | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| hermes | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| hermes | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| hermes | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| hermes | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | retested once; first try: followup [{"connection": "integration:Google AI Studio", "status": "failed", "error": "\u |
| hermes | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | Google AI Studio | |
| hermes | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| hermes | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| opencode | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | opencode rows re-run on 0.13.20 (opencode rides the loopback relay, #97) ; retested once; first try: artifact [{"connection": "integration:Google AI Studio", "status": "failed", "error": "Ba |
| opencode | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | opencode rows re-run on 0.13.20 (opencode rides the loopback relay, #97) ; retested once; first try: artifact [{"connection": "integration:Google AI Studio", "status": "failed", "error": "Ba |
| opencode | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | opencode rows re-run on 0.13.20 (opencode rides the loopback relay, #97) ; retested once; first try: artifact [{"connection": "integration:Google AI Studio", "status": "failed", "error": "Ba |
| opencode | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | opencode rows re-run on 0.13.20 (opencode rides the loopback relay, #97) ; retested once; first try: artifact [{"connection": "integration:Google AI Studio", "status": "failed", "error": "Ba |
| opencode | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | opencode rows re-run on 0.13.20 (opencode rides the loopback relay, #97) ; retested once; first try: artifact [{"connection": "integration:Google AI Studio", "status": "failed", "error": "Ba |
| opencode | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | Google AI Studio | opencode rows re-run on 0.13.20 (opencode rides the loopback relay, #97) ; retested once; first try: artifact [{"connection": "integration:Google AI Studio", "status": "failed", "error": "Ba |
| opencode | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | opencode rows re-run on 0.13.20 (opencode rides the loopback relay, #97) ; retested once; first try: artifact [{"connection": "integration:Google AI Studio", "status": "failed", "error": "Ba |
| opencode | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | opencode rows re-run on 0.13.20 (opencode rides the loopback relay, #97) ; retested once; first try: artifact [{"connection": "integration:Google AI Studio", "status": "failed", "error": "Ba |
| pi | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| pi | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| pi | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| pi | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| pi | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| pi | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | Google AI Studio | |
| pi | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| pi | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| qwen | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| qwen | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| qwen | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| qwen | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| qwen | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| qwen | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | Google AI Studio | |
| qwen | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| qwen | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio |
47 pairs, 235 of 235 scenario runs passed.
Not run in this column, 7 pairs the provider serves that the harness did not run, with the reason:
- cline x gemini-3.6-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x gemini-4-argon: not run, not run in this column
- dsh x gemini-4-argon: not run, not run in this column
- hermes x gemini-4-argon: not run, not run in this column
- opencode x gemini-4-argon: not run, not run in this column
- pi x gemini-4-argon: not run, not run in this column
- qwen x gemini-4-argon: not run, not run in this column
Provider: gemini-openrouter
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| cline | gemini-3-flash-preview | pass | pass | pass (gemini-3.8-flash) | pass | pass | OpenRouter | |
| cline | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.8-flash) | pass | pass | OpenRouter | |
| cline | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.8-flash) | pass | pass | OpenRouter | |
| cline | gemini-3.5-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | OpenRouter | |
| cline | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.8-flash) | pass | pass | OpenRouter | |
| cline | gemini-3.7-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | OpenRouter | |
| cline | gemini-3.8-flash | pass | pass | pass (gemini-3.7-flash) | pass | pass | OpenRouter | |
| dsh | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| dsh | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| dsh | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| dsh | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| dsh | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| dsh | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | OpenRouter | |
| dsh | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| dsh | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| hermes | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| hermes | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| hermes | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| hermes | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| hermes | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| hermes | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | OpenRouter | |
| hermes | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| hermes | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| opencode | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| opencode | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| opencode | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| opencode | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| opencode | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| opencode | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | OpenRouter | |
| opencode | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| opencode | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| pi | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| pi | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| pi | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| pi | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| pi | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| pi | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | OpenRouter | |
| pi | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| pi | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| qwen | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| qwen | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| qwen | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| qwen | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| qwen | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| qwen | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | OpenRouter | |
| qwen | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter | |
| qwen | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | OpenRouter |
47 pairs, 235 of 235 scenario runs passed.
Not run in this column, 367 pairs the provider serves that the harness did not run, with the reason:
- cline x claude-fable-5: not run, not run in this column
- cline x claude-fable-5-1: not run, not run in this column
- cline x claude-haiku-4.5: not run, not run in this column
- cline x claude-haiku-5.5: not run, not run in this column
- cline x claude-opus-4.7: not run, not run in this column
- cline x claude-opus-4.8: not run, not run in this column
- cline x claude-opus-5: not run, not run in this column
- cline x claude-opus-5.5: not run, not run in this column
- cline x claude-sonnet-4.6: not run, not run in this column
- cline x claude-sonnet-5: not run, not run in this column
- cline x claude-sonnet-5.5: not run, not run in this column
- cline x deepseek-v4-flash: not run, not run in this column
- cline x deepseek-v4-pro: not run, not run in this column
- cline x deepseek-v4.1-flash: not run, not run in this column
- cline x gemini-3.6-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x gemini-4-argon: not run, not run in this column
- cline x glm-5.3: not run, not run in this column
- cline x glm-5.3-flash: not run, not run in this column
- cline x gpt-5.2: not run, not run in this column
- cline x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x gpt-5.4: not run, not run in this column
- cline x gpt-5.4-mini: not run, not run in this column
- cline x gpt-5.5: not run, not run in this column
- cline x gpt-5.6-luna: not run, not run in this column
- cline x gpt-5.6-sol: not run, not run in this column
- cline x gpt-5.6-terra: not run, not run in this column
- cline x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x gpt-6-luna: not run, not run in this column
- cline x gpt-6-sol: not run, not run in this column
- cline x gpt-6.1-sol: not run, not run in this column
- cline x grok-4.20: not run, not run in this column
- cline x grok-4.3: not run, not run in this column
- cline x grok-4.5: not run, not run in this column
- cline x grok-4.6: not run, not run in this column
- cline x grok-build-0.1: not run, not run in this column
- cline x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x hunyuan-4-preview: not run, not run in this column
- cline x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x kimi-k2.7-code: not run, not run in this column
- cline x kimi-k3: not run, not run in this column
- cline x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x llama-3.3-70b: not run, not run in this column
- cline x llama-4-maverick: not run, not run in this column
- cline x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x mistral-medium-3.5: not run, not run in this column
- cline x muse-glimmer-30b: not run, not run in this column
- cline x muse-spark-1.1: not run, not run in this column
- cline x muse-spark-1.2: not run, not run in this column
- cline x muse-spark-1.3: not run, not run in this column
- cline x muse-wonder-1.0: not run, not run in this column
- cline x nemotron-3-super: not run, not run in this column
- cline x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x nemotron-3.5-lightning: not run, not run in this column
- cline x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x qwen3.7-max: not run, not run in this column
- cline x qwen3.7-plus: not run, not run in this column
- cline x qwen3.8-27b: not run, not run in this column
- cline x qwen3.8-flash: not run, not run in this column
- cline x qwen3.8-max: not run, not run in this column
- cline x step-3.7-flash: not run, not run in this column
- dsh x claude-fable-5: not run, not run in this column
- dsh x claude-fable-5-1: not run, not run in this column
- dsh x claude-haiku-4.5: not run, not run in this column
- dsh x claude-haiku-5.5: not run, not run in this column
- dsh x claude-opus-4.7: not run, not run in this column
- dsh x claude-opus-4.8: not run, not run in this column
- dsh x claude-opus-5: not run, not run in this column
- dsh x claude-opus-5.5: not run, not run in this column
- dsh x claude-sonnet-4.6: not run, not run in this column
- dsh x claude-sonnet-5: not run, not run in this column
- dsh x claude-sonnet-5.5: not run, not run in this column
- dsh x deepseek-v4-flash: not run, not run in this column
- dsh x deepseek-v4-pro: not run, not run in this column
- dsh x deepseek-v4.1-flash: not run, not run in this column
- dsh x gemini-4-argon: not run, not run in this column
- dsh x glm-5.3: not run, not run in this column
- dsh x glm-5.3-flash: not run, not run in this column
- dsh x gpt-5.2: not run, not run in this column
- dsh x gpt-5.3-codex: not run, not run in this column
- dsh x gpt-5.4: not run, not run in this column
- dsh x gpt-5.4-mini: not run, not run in this column
- dsh x gpt-5.5: not run, not run in this column
- dsh x gpt-5.6-luna: not run, not run in this column
- dsh x gpt-5.6-sol: not run, not run in this column
- dsh x gpt-5.6-terra: not run, not run in this column
- dsh x gpt-6-astra: not run, not run in this column
- dsh x gpt-6-luna: not run, not run in this column
- dsh x gpt-6-sol: not run, not run in this column
- dsh x gpt-6.1-sol: not run, not run in this column
- dsh x grok-4.20: not run, not run in this column
- dsh x grok-4.3: not run, not run in this column
- dsh x grok-4.5: not run, not run in this column
- dsh x grok-4.6: not run, not run in this column
- dsh x grok-build-0.1: not run, not run in this column
- dsh x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x hunyuan-4-preview: not run, not run in this column
- dsh x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x kimi-k2.7-code: not run, not run in this column
- dsh x kimi-k3: not run, not run in this column
- dsh x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x llama-3.3-70b: not run, not run in this column
- dsh x llama-4-maverick: not run, not run in this column
- dsh x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x mistral-medium-3.5: not run, not run in this column
- dsh x muse-glimmer-30b: not run, not run in this column
- dsh x muse-spark-1.1: not run, not run in this column
- dsh x muse-spark-1.2: not run, not run in this column
- dsh x muse-spark-1.3: not run, not run in this column
- dsh x muse-wonder-1.0: not run, not run in this column
- dsh x nemotron-3-super: not run, not run in this column
- dsh x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x nemotron-3.5-lightning: not run, not run in this column
- dsh x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x qwen3.7-max: not run, not run in this column
- dsh x qwen3.7-plus: not run, not run in this column
- dsh x qwen3.8-27b: not run, not run in this column
- dsh x qwen3.8-flash: not run, not run in this column
- dsh x qwen3.8-max: not run, not run in this column
- dsh x step-3.7-flash: not run, not run in this column
- hermes x claude-fable-5: not run, not run in this column
- hermes x claude-fable-5-1: not run, not run in this column
- hermes x claude-haiku-4.5: not run, not run in this column
- hermes x claude-haiku-5.5: not run, not run in this column
- hermes x claude-opus-4.7: not run, not run in this column
- hermes x claude-opus-4.8: not run, not run in this column
- hermes x claude-opus-5: not run, not run in this column
- hermes x claude-opus-5.5: not run, not run in this column
- hermes x claude-sonnet-4.6: not run, not run in this column
- hermes x claude-sonnet-5: not run, not run in this column
- hermes x claude-sonnet-5.5: not run, not run in this column
- hermes x deepseek-v4-flash: not run, not run in this column
- hermes x deepseek-v4-pro: not run, not run in this column
- hermes x deepseek-v4.1-flash: not run, not run in this column
- hermes x gemini-4-argon: not run, not run in this column
- hermes x glm-5.3: not run, not run in this column
- hermes x glm-5.3-flash: not run, not run in this column
- hermes x gpt-5.2: not run, not run in this column
- hermes x gpt-5.3-codex: not run, not run in this column
- hermes x gpt-5.4: not run, not run in this column
- hermes x gpt-5.4-mini: not run, not run in this column
- hermes x gpt-5.5: not run, not run in this column
- hermes x gpt-5.6-luna: not run, not run in this column
- hermes x gpt-5.6-sol: not run, not run in this column
- hermes x gpt-5.6-terra: not run, not run in this column
- hermes x gpt-6-astra: not run, not run in this column
- hermes x gpt-6-luna: not run, not run in this column
- hermes x gpt-6-sol: not run, not run in this column
- hermes x gpt-6.1-sol: not run, not run in this column
- hermes x grok-4.20: not run, not run in this column
- hermes x grok-4.3: not run, not run in this column
- hermes x grok-4.5: not run, not run in this column
- hermes x grok-4.6: not run, not run in this column
- hermes x grok-build-0.1: not run, not run in this column
- hermes x hunyuan-3: not run, not run in this column
- hermes x hunyuan-4-preview: not run, not run in this column
- hermes x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- hermes x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- hermes x kimi-k2.7-code: not run, not run in this column
- hermes x kimi-k3: not run, not run in this column
- hermes x ling-3.0-flash: not run, not run in this column
- hermes x llama-3.3-70b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- hermes x llama-4-maverick: not run, not run in this column
- hermes x minimax-m3: not run, not run in this column
- hermes x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- hermes x mistral-medium-3.5: not run, not run in this column
- hermes x muse-glimmer-30b: not run, not run in this column
- hermes x muse-spark-1.1: not run, not run in this column
- hermes x muse-spark-1.2: not run, not run in this column
- hermes x muse-spark-1.3: not run, not run in this column
- hermes x muse-wonder-1.0: not run, not run in this column
- hermes x nemotron-3-super: not run, not run in this column
- hermes x nemotron-3-ultra: not run, not run in this column
- hermes x nemotron-3.5-lightning: not run, not run in this column
- hermes x qwen3.7-flash: not run, not run in this column
- hermes x qwen3.7-max: not run, not run in this column
- hermes x qwen3.7-plus: not run, not run in this column
- hermes x qwen3.8-27b: not run, not run in this column
- hermes x qwen3.8-flash: not run, not run in this column
- hermes x qwen3.8-max: not run, not run in this column
- hermes x step-3.7-flash: not run, not run in this column
- opencode x claude-fable-5: not run, not run in this column
- opencode x claude-fable-5-1: not run, not run in this column
- opencode x claude-haiku-4.5: not run, not run in this column
- opencode x claude-haiku-5.5: not run, not run in this column
- opencode x claude-opus-4.7: not run, not run in this column
- opencode x claude-opus-4.8: not run, not run in this column
- opencode x claude-opus-5: not run, not run in this column
- opencode x claude-opus-5.5: not run, not run in this column
- opencode x claude-sonnet-4.6: not run, not run in this column
- opencode x claude-sonnet-5: not run, not run in this column
- opencode x claude-sonnet-5.5: not run, not run in this column
- opencode x deepseek-v4-flash: not run, not run in this column
- opencode x deepseek-v4-pro: not run, not run in this column
- opencode x deepseek-v4.1-flash: not run, not run in this column
- opencode x gemini-4-argon: not run, not run in this column
- opencode x glm-5.3: not run, not run in this column
- opencode x glm-5.3-flash: not run, not run in this column
- opencode x gpt-5.2: not run, not run in this column
- opencode x gpt-5.3-codex: not run, not run in this column
- opencode x gpt-5.4: not run, not run in this column
- opencode x gpt-5.4-mini: not run, not run in this column
- opencode x gpt-5.5: not run, not run in this column
- opencode x gpt-5.6-luna: not run, not run in this column
- opencode x gpt-5.6-sol: not run, not run in this column
- opencode x gpt-5.6-terra: not run, not run in this column
- opencode x gpt-6-astra: not run, not run in this column
- opencode x gpt-6-luna: not run, not run in this column
- opencode x gpt-6-sol: not run, not run in this column
- opencode x gpt-6.1-sol: not run, not run in this column
- opencode x grok-4.20: not run, not run in this column
- opencode x grok-4.3: not run, not run in this column
- opencode x grok-4.5: not run, not run in this column
- opencode x grok-4.6: not run, not run in this column
- opencode x grok-build-0.1: not run, not run in this column
- opencode x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x hunyuan-4-preview: not run, not run in this column
- opencode x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x kimi-k2.7-code: not run, not run in this column
- opencode x kimi-k3: not run, not run in this column
- opencode x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x llama-3.3-70b: not run, not run in this column
- opencode x llama-4-maverick: not run, not run in this column
- opencode x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x mistral-medium-3.5: not run, not run in this column
- opencode x muse-glimmer-30b: not run, not run in this column
- opencode x muse-spark-1.1: not run, not run in this column
- opencode x muse-spark-1.2: not run, not run in this column
- opencode x muse-spark-1.3: not run, not run in this column
- opencode x muse-wonder-1.0: not run, not run in this column
- opencode x nemotron-3-super: not run, not run in this column
- opencode x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x nemotron-3.5-lightning: not run, not run in this column
- opencode x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x qwen3.7-max: not run, not run in this column
- opencode x qwen3.7-plus: not run, not run in this column
- opencode x qwen3.8-27b: not run, not run in this column
- opencode x qwen3.8-flash: not run, not run in this column
- opencode x qwen3.8-max: not run, not run in this column
- opencode x step-3.7-flash: not run, not run in this column
- pi x claude-fable-5: not run, not run in this column
- pi x claude-fable-5-1: not run, not run in this column
- pi x claude-haiku-4.5: not run, not run in this column
- pi x claude-haiku-5.5: not run, not run in this column
- pi x claude-opus-4.7: not run, not run in this column
- pi x claude-opus-4.8: not run, not run in this column
- pi x claude-opus-5: not run, not run in this column
- pi x claude-opus-5.5: not run, not run in this column
- pi x claude-sonnet-4.6: not run, not run in this column
- pi x claude-sonnet-5: not run, not run in this column
- pi x claude-sonnet-5.5: not run, not run in this column
- pi x deepseek-v4-flash: not run, not run in this column
- pi x deepseek-v4-pro: not run, not run in this column
- pi x deepseek-v4.1-flash: not run, not run in this column
- pi x gemini-4-argon: not run, not run in this column
- pi x glm-5.3: not run, not run in this column
- pi x glm-5.3-flash: not run, not run in this column
- pi x gpt-5.2: not run, not run in this column
- pi x gpt-5.3-codex: not run, not run in this column
- pi x gpt-5.4: not run, not run in this column
- pi x gpt-5.4-mini: not run, not run in this column
- pi x gpt-5.5: not run, not run in this column
- pi x gpt-5.6-luna: not run, not run in this column
- pi x gpt-5.6-sol: not run, not run in this column
- pi x gpt-5.6-terra: not run, not run in this column
- pi x gpt-6-astra: not run, not run in this column
- pi x gpt-6-luna: not run, not run in this column
- pi x gpt-6-sol: not run, not run in this column
- pi x gpt-6.1-sol: not run, not run in this column
- pi x grok-4.20: not run, not run in this column
- pi x grok-4.3: not run, not run in this column
- pi x grok-4.5: not run, not run in this column
- pi x grok-4.6: not run, not run in this column
- pi x grok-build-0.1: not run, not run in this column
- pi x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x hunyuan-4-preview: not run, not run in this column
- pi x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x kimi-k2.7-code: not run, not run in this column
- pi x kimi-k3: not run, not run in this column
- pi x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x llama-3.3-70b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x llama-4-maverick: not run, not run in this column
- pi x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x mistral-medium-3.5: not run, not run in this column
- pi x muse-glimmer-30b: not run, not run in this column
- pi x muse-spark-1.1: not run, not run in this column
- pi x muse-spark-1.2: not run, not run in this column
- pi x muse-spark-1.3: not run, not run in this column
- pi x muse-wonder-1.0: not run, not run in this column
- pi x nemotron-3-super: not run, not run in this column
- pi x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x nemotron-3.5-lightning: not run, not run in this column
- pi x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x qwen3.7-max: not run, not run in this column
- pi x qwen3.7-plus: not run, not run in this column
- pi x qwen3.8-27b: not run, not run in this column
- pi x qwen3.8-flash: not run, not run in this column
- pi x qwen3.8-max: not run, not run in this column
- pi x step-3.7-flash: not run, not run in this column
- qwen x claude-fable-5: not run, not run in this column
- qwen x claude-fable-5-1: not run, not run in this column
- qwen x claude-haiku-4.5: not run, not run in this column
- qwen x claude-haiku-5.5: not run, not run in this column
- qwen x claude-opus-4.7: not run, not run in this column
- qwen x claude-opus-4.8: not run, not run in this column
- qwen x claude-opus-5: not run, not run in this column
- qwen x claude-opus-5.5: not run, not run in this column
- qwen x claude-sonnet-4.6: not run, not run in this column
- qwen x claude-sonnet-5: not run, not run in this column
- qwen x claude-sonnet-5.5: not run, not run in this column
- qwen x deepseek-v4-flash: not run, not run in this column
- qwen x deepseek-v4-pro: not run, not run in this column
- qwen x deepseek-v4.1-flash: not run, not run in this column
- qwen x gemini-4-argon: not run, not run in this column
- qwen x glm-5.3: not run, not run in this column
- qwen x glm-5.3-flash: not run, not run in this column
- qwen x gpt-5.2: not run, not run in this column
- qwen x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x gpt-5.4: not run, not run in this column
- qwen x gpt-5.4-mini: not run, not run in this column
- qwen x gpt-5.5: not run, not run in this column
- qwen x gpt-5.6-luna: not run, not run in this column
- qwen x gpt-5.6-sol: not run, not run in this column
- qwen x gpt-5.6-terra: not run, not run in this column
- qwen x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x gpt-6-luna: not run, not run in this column
- qwen x gpt-6-sol: not run, not run in this column
- qwen x gpt-6.1-sol: not run, not run in this column
- qwen x grok-4.20: not run, not run in this column
- qwen x grok-4.3: not run, not run in this column
- qwen x grok-4.5: not run, not run in this column
- qwen x grok-4.6: not run, not run in this column
- qwen x grok-build-0.1: not run, not run in this column
- qwen x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x hunyuan-4-preview: not run, not run in this column
- qwen x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x kimi-k2.7-code: not run, not run in this column
- qwen x kimi-k3: not run, not run in this column
- qwen x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x llama-3.3-70b: not run, not run in this column
- qwen x llama-4-maverick: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x mistral-medium-3.5: not run, not run in this column
- qwen x muse-glimmer-30b: not run, not run in this column
- qwen x muse-spark-1.1: not run, not run in this column
- qwen x muse-spark-1.2: not run, not run in this column
- qwen x muse-spark-1.3: not run, not run in this column
- qwen x muse-wonder-1.0: not run, not run in this column
- qwen x nemotron-3-super: not run, not run in this column
- qwen x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x nemotron-3.5-lightning: not run, not run in this column
- qwen x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x qwen3.7-max: not run, not run in this column
- qwen x qwen3.7-plus: not run, not run in this column
- qwen x qwen3.8-27b: not run, not run in this column
- qwen x qwen3.8-flash: not run, not run in this column
- qwen x qwen3.8-max: not run, not run in this column
- qwen x step-3.7-flash: not run, not run in this column
Provider: gemini-tokenrouter
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| cline | gemini-3-flash-preview | pass | pass | pass (gemini-3.8-flash) | pass | pass | My TokenRouter | cline rows re-run on 0.13.22 (no anyOf leaves the normaliser, #104) ; retested once; first try: |
| cline | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.8-flash) | pass | pass | My TokenRouter | cline rows re-run on 0.13.22 (no anyOf leaves the normaliser, #104) ; retested once; first try: |
| cline | gemini-3.5-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | My TokenRouter | cline rows re-run on 0.13.22 (no anyOf leaves the normaliser, #104) ; retested once; first try: |
| cline | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.8-flash) | pass | pass | My TokenRouter | cline rows re-run on 0.13.22 (no anyOf leaves the normaliser, #104) ; retested once; first try: |
| cline | gemini-3.7-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | My TokenRouter | cline rows re-run on 0.13.22 (no anyOf leaves the normaliser, #104) ; retested once; first try: |
| cline | gemini-3.8-flash | pass | pass | pass (gemini-3.7-flash) | pass | pass | My TokenRouter | cline rows re-run on 0.13.22 (no anyOf leaves the normaliser, #104) ; retested once; first try: |
| dsh | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| dsh | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| dsh | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| dsh | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| dsh | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | My TokenRouter | |
| dsh | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| dsh | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| hermes | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| hermes | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| hermes | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| hermes | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| hermes | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | My TokenRouter | |
| hermes | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| hermes | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| opencode | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| opencode | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| opencode | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| opencode | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| opencode | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | My TokenRouter | |
| opencode | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| opencode | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| pi | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| pi | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| pi | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| pi | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| pi | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | My TokenRouter | |
| pi | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| pi | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | |
| qwen | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | qwen rows re-run on 0.13.22 (no anyOf leaves the normaliser, #104) ; retested once; first try: |
| qwen | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | qwen rows re-run on 0.13.22 (no anyOf leaves the normaliser, #104) ; retested once; first try: artifact no file card (files: none); claration parameters.fork_turns schema specified oth; recycle answered without M1-gemini-3.1-pro-preview: submit request because agent functi |
| qwen | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | qwen rows re-run on 0.13.22 (no anyOf leaves the normaliser, #104) ; retested once; first try: artifact no file card (files: none); claration parameters.fork_turns schema specified oth; recycle answered without M1-gemini-3.5-flash: submit request because agent functionDecl |
| qwen | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | qwen rows re-run on 0.13.22 (no anyOf leaves the normaliser, #104) ; retested once; first try: artifact no file card (files: none); claration parameters.fork_turns schema specified oth; recycle answered without M1-gemini-3.5-flash-lite: submit request because agent functio |
| qwen | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | My TokenRouter | qwen rows re-run on 0.13.22 (no anyOf leaves the normaliser, #104) ; retested once; first try: artifact no file card (files: none); claration parameters.fork_turns schema specified oth; recycle answered without M1-gemini-3.6-flash: submit request because agent functionDecl |
| qwen | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | qwen rows re-run on 0.13.22 (no anyOf leaves the normaliser, #104) ; retested once; first try: |
| qwen | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | My TokenRouter | qwen rows re-run on 0.13.22 (no anyOf leaves the normaliser, #104) ; retested once; first try: artifact no file card (files: none); claration parameters.fork_turns schema specified oth; recycle answered without M1-gemini-3.8-flash: submit request because agent functionDecl |
41 pairs, 205 of 205 scenario runs passed.
Not run in this column, 253 pairs the provider serves that the harness did not run, with the reason:
- cline x claude-fable-5: not run, not run in this column
- cline x claude-fable-5-1: not run, not run in this column
- cline x claude-haiku-4.5: not run, not run in this column
- cline x claude-opus-4.7: not run, not run in this column
- cline x claude-opus-4.8: not run, not run in this column
- cline x claude-opus-5: not run, not run in this column
- cline x claude-opus-5.5: not run, not run in this column
- cline x claude-sonnet-4.6: not run, not run in this column
- cline x claude-sonnet-5: not run, not run in this column
- cline x claude-sonnet-5.5: not run, not run in this column
- cline x deepseek-v4-pro: not run, not run in this column
- cline x deepseek-v4.1-flash: not run, not run in this column
- cline x gemini-3.6-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x glm-5.3: not run, not run in this column
- cline x glm-5.3-flash: not run, not run in this column
- cline x gpt-5.2: not run, not run in this column
- cline x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x gpt-5.4: not run, not run in this column
- cline x gpt-5.4-mini: not run, not run in this column
- cline x gpt-5.5: not run, not run in this column
- cline x gpt-5.6-luna: not run, not run in this column
- cline x gpt-5.6-sol: not run, not run in this column
- cline x gpt-5.6-terra: not run, not run in this column
- cline x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x gpt-6-luna: not run, not run in this column
- cline x gpt-6-sol: not run, not run in this column
- cline x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- cline x grok-4.20: not run, not run in this column
- cline x grok-4.3: not run, not run in this column
- cline x grok-4.5: not run, not run in this column
- cline x grok-4.6: not run, not run in this column
- cline x grok-build-0.1: not run, not run in this column
- cline x hunyuan-4-preview: not run, not run in this column
- cline x kimi-k2.7-code: not run, not run in this column
- cline x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x mistral-medium-3.5: not run, not run in this column
- cline x nemotron-3-super: not run, not run in this column
- cline x nemotron-3.5-lightning: not run, not run in this column
- cline x qwen3.7-max: not run, not run in this column
- cline x qwen3.7-plus: not run, not run in this column
- cline x qwen3.8-flash: not run, not run in this column
- cline x qwen3.8-max: not run, not run in this column
- cline x step-3.7-flash: not run, not run in this column
- dsh x claude-fable-5: not run, not run in this column
- dsh x claude-fable-5-1: not run, not run in this column
- dsh x claude-haiku-4.5: not run, not run in this column
- dsh x claude-opus-4.7: not run, not run in this column
- dsh x claude-opus-4.8: not run, not run in this column
- dsh x claude-opus-5: not run, not run in this column
- dsh x claude-opus-5.5: not run, not run in this column
- dsh x claude-sonnet-4.6: not run, not run in this column
- dsh x claude-sonnet-5: not run, not run in this column
- dsh x claude-sonnet-5.5: not run, not run in this column
- dsh x deepseek-v4-pro: not run, not run in this column
- dsh x deepseek-v4.1-flash: not run, not run in this column
- dsh x glm-5.3: not run, not run in this column
- dsh x glm-5.3-flash: not run, not run in this column
- dsh x gpt-5.2: not run, not run in this column
- dsh x gpt-5.3-codex: not run, not run in this column
- dsh x gpt-5.4: not run, not run in this column
- dsh x gpt-5.4-mini: not run, not run in this column
- dsh x gpt-5.5: not run, not run in this column
- dsh x gpt-5.6-luna: not run, not run in this column
- dsh x gpt-5.6-sol: not run, not run in this column
- dsh x gpt-5.6-terra: not run, not run in this column
- dsh x gpt-6-astra: not run, not run in this column
- dsh x gpt-6-luna: not run, not run in this column
- dsh x gpt-6-sol: not run, not run in this column
- dsh x gpt-6.1-sol: not run, not run in this column
- dsh x grok-4.20: not run, not run in this column
- dsh x grok-4.3: not run, not run in this column
- dsh x grok-4.5: not run, not run in this column
- dsh x grok-4.6: not run, not run in this column
- dsh x grok-build-0.1: not run, not run in this column
- dsh x hunyuan-4-preview: not run, not run in this column
- dsh x kimi-k2.7-code: not run, not run in this column
- dsh x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x mistral-medium-3.5: not run, not run in this column
- dsh x nemotron-3-super: not run, not run in this column
- dsh x nemotron-3.5-lightning: not run, not run in this column
- dsh x qwen3.7-max: not run, not run in this column
- dsh x qwen3.7-plus: not run, not run in this column
- dsh x qwen3.8-flash: not run, not run in this column
- dsh x qwen3.8-max: not run, not run in this column
- dsh x step-3.7-flash: not run, not run in this column
- hermes x claude-fable-5: not run, not run in this column
- hermes x claude-fable-5-1: not run, not run in this column
- hermes x claude-haiku-4.5: not run, not run in this column
- hermes x claude-opus-4.7: not run, not run in this column
- hermes x claude-opus-4.8: not run, not run in this column
- hermes x claude-opus-5: not run, not run in this column
- hermes x claude-opus-5.5: not run, not run in this column
- hermes x claude-sonnet-4.6: not run, not run in this column
- hermes x claude-sonnet-5: not run, not run in this column
- hermes x claude-sonnet-5.5: not run, not run in this column
- hermes x deepseek-v4-pro: not run, not run in this column
- hermes x deepseek-v4.1-flash: not run, not run in this column
- hermes x glm-5.3: not run, not run in this column
- hermes x glm-5.3-flash: not run, not run in this column
- hermes x gpt-5.2: not run, not run in this column
- hermes x gpt-5.3-codex: not run, not run in this column
- hermes x gpt-5.4: not run, not run in this column
- hermes x gpt-5.4-mini: not run, not run in this column
- hermes x gpt-5.5: not run, not run in this column
- hermes x gpt-5.6-luna: not run, not run in this column
- hermes x gpt-5.6-sol: not run, not run in this column
- hermes x gpt-5.6-terra: not run, not run in this column
- hermes x gpt-6-astra: not run, not run in this column
- hermes x gpt-6-luna: not run, not run in this column
- hermes x gpt-6-sol: not run, not run in this column
- hermes x gpt-6.1-sol: not run, not run in this column
- hermes x grok-4.20: not run, not run in this column
- hermes x grok-4.3: not run, not run in this column
- hermes x grok-4.5: not run, not run in this column
- hermes x grok-4.6: not run, not run in this column
- hermes x grok-build-0.1: not run, not run in this column
- hermes x hunyuan-4-preview: not run, not run in this column
- hermes x kimi-k2.7-code: not run, not run in this column
- hermes x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- hermes x mistral-medium-3.5: not run, not run in this column
- hermes x nemotron-3-super: not run, not run in this column
- hermes x nemotron-3.5-lightning: not run, not run in this column
- hermes x qwen3.7-max: not run, not run in this column
- hermes x qwen3.7-plus: not run, not run in this column
- hermes x qwen3.8-flash: not run, not run in this column
- hermes x qwen3.8-max: not run, not run in this column
- hermes x step-3.7-flash: not run, not run in this column
- opencode x claude-fable-5: not run, not run in this column
- opencode x claude-fable-5-1: not run, not run in this column
- opencode x claude-haiku-4.5: not run, not run in this column
- opencode x claude-opus-4.7: not run, not run in this column
- opencode x claude-opus-4.8: not run, not run in this column
- opencode x claude-opus-5: not run, not run in this column
- opencode x claude-opus-5.5: not run, not run in this column
- opencode x claude-sonnet-4.6: not run, not run in this column
- opencode x claude-sonnet-5: not run, not run in this column
- opencode x claude-sonnet-5.5: not run, not run in this column
- opencode x deepseek-v4-pro: not run, not run in this column
- opencode x deepseek-v4.1-flash: not run, not run in this column
- opencode x glm-5.3: not run, not run in this column
- opencode x glm-5.3-flash: not run, not run in this column
- opencode x gpt-5.2: not run, not run in this column
- opencode x gpt-5.3-codex: not run, not run in this column
- opencode x gpt-5.4: not run, not run in this column
- opencode x gpt-5.4-mini: not run, not run in this column
- opencode x gpt-5.5: not run, not run in this column
- opencode x gpt-5.6-luna: not run, not run in this column
- opencode x gpt-5.6-sol: not run, not run in this column
- opencode x gpt-5.6-terra: not run, not run in this column
- opencode x gpt-6-astra: not run, not run in this column
- opencode x gpt-6-luna: not run, not run in this column
- opencode x gpt-6-sol: not run, not run in this column
- opencode x gpt-6.1-sol: not run, not run in this column
- opencode x grok-4.20: not run, not run in this column
- opencode x grok-4.3: not run, not run in this column
- opencode x grok-4.5: not run, not run in this column
- opencode x grok-4.6: not run, not run in this column
- opencode x grok-build-0.1: not run, not run in this column
- opencode x hunyuan-4-preview: not run, not run in this column
- opencode x kimi-k2.7-code: not run, not run in this column
- opencode x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x mistral-medium-3.5: not run, not run in this column
- opencode x nemotron-3-super: not run, not run in this column
- opencode x nemotron-3.5-lightning: not run, not run in this column
- opencode x qwen3.7-max: not run, not run in this column
- opencode x qwen3.7-plus: not run, not run in this column
- opencode x qwen3.8-flash: not run, not run in this column
- opencode x qwen3.8-max: not run, not run in this column
- opencode x step-3.7-flash: not run, not run in this column
- pi x claude-fable-5: not run, not run in this column
- pi x claude-fable-5-1: not run, not run in this column
- pi x claude-haiku-4.5: not run, not run in this column
- pi x claude-opus-4.7: not run, not run in this column
- pi x claude-opus-4.8: not run, not run in this column
- pi x claude-opus-5: not run, not run in this column
- pi x claude-opus-5.5: not run, not run in this column
- pi x claude-sonnet-4.6: not run, not run in this column
- pi x claude-sonnet-5: not run, not run in this column
- pi x claude-sonnet-5.5: not run, not run in this column
- pi x deepseek-v4-pro: not run, not run in this column
- pi x deepseek-v4.1-flash: not run, not run in this column
- pi x glm-5.3: not run, not run in this column
- pi x glm-5.3-flash: not run, not run in this column
- pi x gpt-5.2: not run, not run in this column
- pi x gpt-5.3-codex: not run, not run in this column
- pi x gpt-5.4: not run, not run in this column
- pi x gpt-5.4-mini: not run, not run in this column
- pi x gpt-5.5: not run, not run in this column
- pi x gpt-5.6-luna: not run, not run in this column
- pi x gpt-5.6-sol: not run, not run in this column
- pi x gpt-5.6-terra: not run, not run in this column
- pi x gpt-6-astra: not run, not run in this column
- pi x gpt-6-luna: not run, not run in this column
- pi x gpt-6-sol: not run, not run in this column
- pi x gpt-6.1-sol: not run, not run in this column
- pi x grok-4.20: not run, not run in this column
- pi x grok-4.3: not run, not run in this column
- pi x grok-4.5: not run, not run in this column
- pi x grok-4.6: not run, not run in this column
- pi x grok-build-0.1: not run, not run in this column
- pi x hunyuan-4-preview: not run, not run in this column
- pi x kimi-k2.7-code: not run, not run in this column
- pi x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x mistral-medium-3.5: not run, not run in this column
- pi x nemotron-3-super: not run, not run in this column
- pi x nemotron-3.5-lightning: not run, not run in this column
- pi x qwen3.7-max: not run, not run in this column
- pi x qwen3.7-plus: not run, not run in this column
- pi x qwen3.8-flash: not run, not run in this column
- pi x qwen3.8-max: not run, not run in this column
- pi x step-3.7-flash: not run, not run in this column
- qwen x claude-fable-5: not run, not run in this column
- qwen x claude-fable-5-1: not run, not run in this column
- qwen x claude-haiku-4.5: not run, not run in this column
- qwen x claude-opus-4.7: not run, not run in this column
- qwen x claude-opus-4.8: not run, not run in this column
- qwen x claude-opus-5: not run, not run in this column
- qwen x claude-opus-5.5: not run, not run in this column
- qwen x claude-sonnet-4.6: not run, not run in this column
- qwen x claude-sonnet-5: not run, not run in this column
- qwen x claude-sonnet-5.5: not run, not run in this column
- qwen x deepseek-v4-pro: not run, not run in this column
- qwen x deepseek-v4.1-flash: not run, not run in this column
- qwen x glm-5.3: not run, not run in this column
- qwen x glm-5.3-flash: not run, not run in this column
- qwen x gpt-5.2: not run, not run in this column
- qwen x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x gpt-5.4: not run, not run in this column
- qwen x gpt-5.4-mini: not run, not run in this column
- qwen x gpt-5.5: not run, not run in this column
- qwen x gpt-5.6-luna: not run, not run in this column
- qwen x gpt-5.6-sol: not run, not run in this column
- qwen x gpt-5.6-terra: not run, not run in this column
- qwen x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x gpt-6-luna: not run, not run in this column
- qwen x gpt-6-sol: not run, not run in this column
- qwen x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- qwen x grok-4.20: not run, not run in this column
- qwen x grok-4.3: not run, not run in this column
- qwen x grok-4.5: not run, not run in this column
- qwen x grok-4.6: not run, not run in this column
- qwen x grok-build-0.1: not run, not run in this column
- qwen x hunyuan-4-preview: not run, not run in this column
- qwen x kimi-k2.7-code: not run, not run in this column
- qwen x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x mistral-medium-3.5: not run, not run in this column
- qwen x nemotron-3-super: not run, not run in this column
- qwen x nemotron-3.5-lightning: not run, not run in this column
- qwen x qwen3.7-max: not run, not run in this column
- qwen x qwen3.7-plus: not run, not run in this column
- qwen x qwen3.8-flash: not run, not run in this column
- qwen x qwen3.8-max: not run, not run in this column
- qwen x step-3.7-flash: not run, not run in this column
Provider: gemini-vercel
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| cline | gemini-3-flash-preview | pass | pass | pass (gemini-3.8-flash) | pass | pass | Vercel AI Gateway | |
| cline | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.8-flash) | pass | pass | Vercel AI Gateway | |
| cline | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.8-flash) | pass | pass | Vercel AI Gateway | |
| cline | gemini-3.5-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | Vercel AI Gateway | |
| cline | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.8-flash) | pass | pass | Vercel AI Gateway | |
| cline | gemini-3.7-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | Vercel AI Gateway | |
| cline | gemini-3.8-flash | pass | pass | pass (gemini-3.7-flash) | pass | pass | Vercel AI Gateway | |
| dsh | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| dsh | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| dsh | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| dsh | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| dsh | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| dsh | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | Vercel AI Gateway | |
| dsh | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| dsh | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| hermes | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| hermes | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| hermes | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| hermes | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| hermes | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| hermes | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | FAIL | Vercel AI Gateway | recycle: answered without M1-gemini-3.6-flash: What exact word did I ask you to reply with in my very first message of this task? Reply with just tha ; retested once; first try: recycle answered without M1-gemini-3.6-flash: What exact word did I ask you to reply wit |
| hermes | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| hermes | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| opencode | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| opencode | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| opencode | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| opencode | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| opencode | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| opencode | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | Vercel AI Gateway | |
| opencode | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| opencode | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| pi | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| pi | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| pi | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| pi | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| pi | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| pi | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | Vercel AI Gateway | |
| pi | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| pi | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| qwen | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| qwen | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| qwen | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| qwen | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| qwen | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| qwen | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | Vercel AI Gateway | |
| qwen | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway | |
| qwen | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Vercel AI Gateway |
47 pairs, 234 of 235 scenario runs passed.
Not run in this column, 337 pairs the provider serves that the harness did not run, with the reason:
- cline x claude-fable-5: not run, not run in this column
- cline x claude-fable-5-1: not run, not run in this column
- cline x claude-haiku-4.5: not run, not run in this column
- cline x claude-haiku-5.5: not run, not run in this column
- cline x claude-opus-4.7: not run, not run in this column
- cline x claude-opus-4.8: not run, not run in this column
- cline x claude-opus-5: not run, not run in this column
- cline x claude-opus-5.5: not run, not run in this column
- cline x claude-sonnet-4.6: not run, not run in this column
- cline x claude-sonnet-5: not run, not run in this column
- cline x claude-sonnet-5.5: not run, not run in this column
- cline x deepseek-v4-flash: not run, not run in this column
- cline x deepseek-v4-pro: not run, not run in this column
- cline x deepseek-v4.1-flash: not run, not run in this column
- cline x gemini-3.6-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x gemini-4-argon: not run, not run in this column
- cline x glm-5.3: not run, not run in this column
- cline x glm-5.3-flash: not run, not run in this column
- cline x gpt-5.2: not run, not run in this column
- cline x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x gpt-5.4: not run, not run in this column
- cline x gpt-5.4-mini: not run, not run in this column
- cline x gpt-5.5: not run, not run in this column
- cline x gpt-5.6-luna: not run, not run in this column
- cline x gpt-5.6-sol: not run, not run in this column
- cline x gpt-5.6-terra: not run, not run in this column
- cline x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x gpt-6-luna: not run, not run in this column
- cline x gpt-6-sol: not run, not run in this column
- cline x gpt-6.1-sol: not run, not run in this column
- cline x grok-4.20: not run, not run in this column
- cline x grok-4.3: not run, not run in this column
- cline x grok-4.5: not run, not run in this column
- cline x grok-4.6: not run, not run in this column
- cline x grok-build-0.1: not run, not run in this column
- cline x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x hunyuan-4-preview: not run, not run in this column
- cline x kimi-k2.7-code: not run, not run in this column
- cline x kimi-k3: not run, not run in this column
- cline x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x mistral-medium-3.5: not run, not run in this column
- cline x muse-glimmer-30b: not run, not run in this column
- cline x muse-spark-1.1: not run, not run in this column
- cline x muse-spark-1.2: not run, not run in this column
- cline x muse-spark-1.3: not run, not run in this column
- cline x muse-wonder-1.0: not run, not run in this column
- cline x nemotron-3-super: not run, not run in this column
- cline x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x qwen3.7-max: not run, not run in this column
- cline x qwen3.7-plus: not run, not run in this column
- cline x qwen3.8-27b: not run, not run in this column
- cline x qwen3.8-flash: not run, not run in this column
- cline x qwen3.8-max: not run, not run in this column
- cline x step-3.7-flash: not run, not run in this column
- dsh x claude-fable-5: not run, not run in this column
- dsh x claude-fable-5-1: not run, not run in this column
- dsh x claude-haiku-4.5: not run, not run in this column
- dsh x claude-haiku-5.5: not run, not run in this column
- dsh x claude-opus-4.7: not run, not run in this column
- dsh x claude-opus-4.8: not run, not run in this column
- dsh x claude-opus-5: not run, not run in this column
- dsh x claude-opus-5.5: not run, not run in this column
- dsh x claude-sonnet-4.6: not run, not run in this column
- dsh x claude-sonnet-5: not run, not run in this column
- dsh x claude-sonnet-5.5: not run, not run in this column
- dsh x deepseek-v4-flash: not run, not run in this column
- dsh x deepseek-v4-pro: not run, not run in this column
- dsh x deepseek-v4.1-flash: not run, not run in this column
- dsh x gemini-4-argon: not run, not run in this column
- dsh x glm-5.3: not run, not run in this column
- dsh x glm-5.3-flash: not run, not run in this column
- dsh x gpt-5.2: not run, not run in this column
- dsh x gpt-5.3-codex: not run, not run in this column
- dsh x gpt-5.4: not run, not run in this column
- dsh x gpt-5.4-mini: not run, not run in this column
- dsh x gpt-5.5: not run, not run in this column
- dsh x gpt-5.6-luna: not run, not run in this column
- dsh x gpt-5.6-sol: not run, not run in this column
- dsh x gpt-5.6-terra: not run, not run in this column
- dsh x gpt-6-astra: not run, not run in this column
- dsh x gpt-6-luna: not run, not run in this column
- dsh x gpt-6-sol: not run, not run in this column
- dsh x gpt-6.1-sol: not run, not run in this column
- dsh x grok-4.20: not run, not run in this column
- dsh x grok-4.3: not run, not run in this column
- dsh x grok-4.5: not run, not run in this column
- dsh x grok-4.6: not run, not run in this column
- dsh x grok-build-0.1: not run, not run in this column
- dsh x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x hunyuan-4-preview: not run, not run in this column
- dsh x kimi-k2.7-code: not run, not run in this column
- dsh x kimi-k3: not run, not run in this column
- dsh x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x mistral-medium-3.5: not run, not run in this column
- dsh x muse-glimmer-30b: not run, not run in this column
- dsh x muse-spark-1.1: not run, not run in this column
- dsh x muse-spark-1.2: not run, not run in this column
- dsh x muse-spark-1.3: not run, not run in this column
- dsh x muse-wonder-1.0: not run, not run in this column
- dsh x nemotron-3-super: not run, not run in this column
- dsh x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x qwen3.7-max: not run, not run in this column
- dsh x qwen3.7-plus: not run, not run in this column
- dsh x qwen3.8-27b: not run, not run in this column
- dsh x qwen3.8-flash: not run, not run in this column
- dsh x qwen3.8-max: not run, not run in this column
- dsh x step-3.7-flash: not run, not run in this column
- hermes x claude-fable-5: not run, not run in this column
- hermes x claude-fable-5-1: not run, not run in this column
- hermes x claude-haiku-4.5: not run, not run in this column
- hermes x claude-haiku-5.5: not run, not run in this column
- hermes x claude-opus-4.7: not run, not run in this column
- hermes x claude-opus-4.8: not run, not run in this column
- hermes x claude-opus-5: not run, not run in this column
- hermes x claude-opus-5.5: not run, not run in this column
- hermes x claude-sonnet-4.6: not run, not run in this column
- hermes x claude-sonnet-5: not run, not run in this column
- hermes x claude-sonnet-5.5: not run, not run in this column
- hermes x deepseek-v4-flash: not run, not run in this column
- hermes x deepseek-v4-pro: not run, not run in this column
- hermes x deepseek-v4.1-flash: not run, not run in this column
- hermes x gemini-4-argon: not run, not run in this column
- hermes x glm-5.3: not run, not run in this column
- hermes x glm-5.3-flash: not run, not run in this column
- hermes x gpt-5.2: not run, not run in this column
- hermes x gpt-5.3-codex: not run, not run in this column
- hermes x gpt-5.4: not run, not run in this column
- hermes x gpt-5.4-mini: not run, not run in this column
- hermes x gpt-5.5: not run, not run in this column
- hermes x gpt-5.6-luna: not run, not run in this column
- hermes x gpt-5.6-sol: not run, not run in this column
- hermes x gpt-5.6-terra: not run, not run in this column
- hermes x gpt-6-astra: not run, not run in this column
- hermes x gpt-6-luna: not run, not run in this column
- hermes x gpt-6-sol: not run, not run in this column
- hermes x gpt-6.1-sol: not run, not run in this column
- hermes x grok-4.20: not run, not run in this column
- hermes x grok-4.3: not run, not run in this column
- hermes x grok-4.5: not run, not run in this column
- hermes x grok-4.6: not run, not run in this column
- hermes x grok-build-0.1: not run, not run in this column
- hermes x hunyuan-3: not run, not run in this column
- hermes x hunyuan-4-preview: not run, not run in this column
- hermes x kimi-k2.7-code: not run, not run in this column
- hermes x kimi-k3: not run, not run in this column
- hermes x ling-3.0-flash: not run, not run in this column
- hermes x minimax-m3: not run, not run in this column
- hermes x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- hermes x mistral-medium-3.5: not run, not run in this column
- hermes x muse-glimmer-30b: not run, not run in this column
- hermes x muse-spark-1.1: not run, not run in this column
- hermes x muse-spark-1.2: not run, not run in this column
- hermes x muse-spark-1.3: not run, not run in this column
- hermes x muse-wonder-1.0: not run, not run in this column
- hermes x nemotron-3-super: not run, not run in this column
- hermes x nemotron-3-ultra: not run, not run in this column
- hermes x qwen3.7-flash: not run, not run in this column
- hermes x qwen3.7-max: not run, not run in this column
- hermes x qwen3.7-plus: not run, not run in this column
- hermes x qwen3.8-27b: not run, not run in this column
- hermes x qwen3.8-flash: not run, not run in this column
- hermes x qwen3.8-max: not run, not run in this column
- hermes x step-3.7-flash: not run, not run in this column
- opencode x claude-fable-5: not run, not run in this column
- opencode x claude-fable-5-1: not run, not run in this column
- opencode x claude-haiku-4.5: not run, not run in this column
- opencode x claude-haiku-5.5: not run, not run in this column
- opencode x claude-opus-4.7: not run, not run in this column
- opencode x claude-opus-4.8: not run, not run in this column
- opencode x claude-opus-5: not run, not run in this column
- opencode x claude-opus-5.5: not run, not run in this column
- opencode x claude-sonnet-4.6: not run, not run in this column
- opencode x claude-sonnet-5: not run, not run in this column
- opencode x claude-sonnet-5.5: not run, not run in this column
- opencode x deepseek-v4-flash: not run, not run in this column
- opencode x deepseek-v4-pro: not run, not run in this column
- opencode x deepseek-v4.1-flash: not run, not run in this column
- opencode x gemini-4-argon: not run, not run in this column
- opencode x glm-5.3: not run, not run in this column
- opencode x glm-5.3-flash: not run, not run in this column
- opencode x gpt-5.2: not run, not run in this column
- opencode x gpt-5.3-codex: not run, not run in this column
- opencode x gpt-5.4: not run, not run in this column
- opencode x gpt-5.4-mini: not run, not run in this column
- opencode x gpt-5.5: not run, not run in this column
- opencode x gpt-5.6-luna: not run, not run in this column
- opencode x gpt-5.6-sol: not run, not run in this column
- opencode x gpt-5.6-terra: not run, not run in this column
- opencode x gpt-6-astra: not run, not run in this column
- opencode x gpt-6-luna: not run, not run in this column
- opencode x gpt-6-sol: not run, not run in this column
- opencode x gpt-6.1-sol: not run, not run in this column
- opencode x grok-4.20: not run, not run in this column
- opencode x grok-4.3: not run, not run in this column
- opencode x grok-4.5: not run, not run in this column
- opencode x grok-4.6: not run, not run in this column
- opencode x grok-build-0.1: not run, not run in this column
- opencode x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x hunyuan-4-preview: not run, not run in this column
- opencode x kimi-k2.7-code: not run, not run in this column
- opencode x kimi-k3: not run, not run in this column
- opencode x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x mistral-medium-3.5: not run, not run in this column
- opencode x muse-glimmer-30b: not run, not run in this column
- opencode x muse-spark-1.1: not run, not run in this column
- opencode x muse-spark-1.2: not run, not run in this column
- opencode x muse-spark-1.3: not run, not run in this column
- opencode x muse-wonder-1.0: not run, not run in this column
- opencode x nemotron-3-super: not run, not run in this column
- opencode x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x qwen3.7-max: not run, not run in this column
- opencode x qwen3.7-plus: not run, not run in this column
- opencode x qwen3.8-27b: not run, not run in this column
- opencode x qwen3.8-flash: not run, not run in this column
- opencode x qwen3.8-max: not run, not run in this column
- opencode x step-3.7-flash: not run, not run in this column
- pi x claude-fable-5: not run, not run in this column
- pi x claude-fable-5-1: not run, not run in this column
- pi x claude-haiku-4.5: not run, not run in this column
- pi x claude-haiku-5.5: not run, not run in this column
- pi x claude-opus-4.7: not run, not run in this column
- pi x claude-opus-4.8: not run, not run in this column
- pi x claude-opus-5: not run, not run in this column
- pi x claude-opus-5.5: not run, not run in this column
- pi x claude-sonnet-4.6: not run, not run in this column
- pi x claude-sonnet-5: not run, not run in this column
- pi x claude-sonnet-5.5: not run, not run in this column
- pi x deepseek-v4-flash: not run, not run in this column
- pi x deepseek-v4-pro: not run, not run in this column
- pi x deepseek-v4.1-flash: not run, not run in this column
- pi x gemini-4-argon: not run, not run in this column
- pi x glm-5.3: not run, not run in this column
- pi x glm-5.3-flash: not run, not run in this column
- pi x gpt-5.2: not run, not run in this column
- pi x gpt-5.3-codex: not run, not run in this column
- pi x gpt-5.4: not run, not run in this column
- pi x gpt-5.4-mini: not run, not run in this column
- pi x gpt-5.5: not run, not run in this column
- pi x gpt-5.6-luna: not run, not run in this column
- pi x gpt-5.6-sol: not run, not run in this column
- pi x gpt-5.6-terra: not run, not run in this column
- pi x gpt-6-astra: not run, not run in this column
- pi x gpt-6-luna: not run, not run in this column
- pi x gpt-6-sol: not run, not run in this column
- pi x gpt-6.1-sol: not run, not run in this column
- pi x grok-4.20: not run, not run in this column
- pi x grok-4.3: not run, not run in this column
- pi x grok-4.5: not run, not run in this column
- pi x grok-4.6: not run, not run in this column
- pi x grok-build-0.1: not run, not run in this column
- pi x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x hunyuan-4-preview: not run, not run in this column
- pi x kimi-k2.7-code: not run, not run in this column
- pi x kimi-k3: not run, not run in this column
- pi x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x mistral-medium-3.5: not run, not run in this column
- pi x muse-glimmer-30b: not run, not run in this column
- pi x muse-spark-1.1: not run, not run in this column
- pi x muse-spark-1.2: not run, not run in this column
- pi x muse-spark-1.3: not run, not run in this column
- pi x muse-wonder-1.0: not run, not run in this column
- pi x nemotron-3-super: not run, not run in this column
- pi x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x qwen3.7-max: not run, not run in this column
- pi x qwen3.7-plus: not run, not run in this column
- pi x qwen3.8-27b: not run, not run in this column
- pi x qwen3.8-flash: not run, not run in this column
- pi x qwen3.8-max: not run, not run in this column
- pi x step-3.7-flash: not run, not run in this column
- qwen x claude-fable-5: not run, not run in this column
- qwen x claude-fable-5-1: not run, not run in this column
- qwen x claude-haiku-4.5: not run, not run in this column
- qwen x claude-haiku-5.5: not run, not run in this column
- qwen x claude-opus-4.7: not run, not run in this column
- qwen x claude-opus-4.8: not run, not run in this column
- qwen x claude-opus-5: not run, not run in this column
- qwen x claude-opus-5.5: not run, not run in this column
- qwen x claude-sonnet-4.6: not run, not run in this column
- qwen x claude-sonnet-5: not run, not run in this column
- qwen x claude-sonnet-5.5: not run, not run in this column
- qwen x deepseek-v4-flash: not run, not run in this column
- qwen x deepseek-v4-pro: not run, not run in this column
- qwen x deepseek-v4.1-flash: not run, not run in this column
- qwen x gemini-4-argon: not run, not run in this column
- qwen x glm-5.3: not run, not run in this column
- qwen x glm-5.3-flash: not run, not run in this column
- qwen x gpt-5.2: not run, not run in this column
- qwen x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x gpt-5.4: not run, not run in this column
- qwen x gpt-5.4-mini: not run, not run in this column
- qwen x gpt-5.5: not run, not run in this column
- qwen x gpt-5.6-luna: not run, not run in this column
- qwen x gpt-5.6-sol: not run, not run in this column
- qwen x gpt-5.6-terra: not run, not run in this column
- qwen x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x gpt-6-luna: not run, not run in this column
- qwen x gpt-6-sol: not run, not run in this column
- qwen x gpt-6.1-sol: not run, not run in this column
- qwen x grok-4.20: not run, not run in this column
- qwen x grok-4.3: not run, not run in this column
- qwen x grok-4.5: not run, not run in this column
- qwen x grok-4.6: not run, not run in this column
- qwen x grok-build-0.1: not run, not run in this column
- qwen x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x hunyuan-4-preview: not run, not run in this column
- qwen x kimi-k2.7-code: not run, not run in this column
- qwen x kimi-k3: not run, not run in this column
- qwen x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x mistral-medium-3.5: not run, not run in this column
- qwen x muse-glimmer-30b: not run, not run in this column
- qwen x muse-spark-1.1: not run, not run in this column
- qwen x muse-spark-1.2: not run, not run in this column
- qwen x muse-spark-1.3: not run, not run in this column
- qwen x muse-wonder-1.0: not run, not run in this column
- qwen x nemotron-3-super: not run, not run in this column
- qwen x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x qwen3.7-max: not run, not run in this column
- qwen x qwen3.7-plus: not run, not run in this column
- qwen x qwen3.8-27b: not run, not run in this column
- qwen x qwen3.8-flash: not run, not run in this column
- qwen x qwen3.8-max: not run, not run in this column
- qwen x step-3.7-flash: not run, not run in this column
Provider: google
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| docker-agent | gemini-3-flash-preview | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | cache 0% of input |
| docker-agent | gemini-3.1-flash-lite | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | cache 0% of input |
| docker-agent | gemini-3.1-pro-preview | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | cache 0% of input |
| docker-agent | gemini-3.5-flash | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | cache 0% of input |
| docker-agent | gemini-3.5-flash-lite | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | cache 0% of input |
| docker-agent | gemini-3.6-flash | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | cache 0% of input |
| docker-agent | gemini-3.8-flash | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | cache 0% of input |
| docker-agent | gemini-4-argon | pass | pass | pass (gemini-3.8-flash) | pass | pass | Google AI Studio | cache 31% of input |
| dsh | gemini-3.6-flash | n/a | n/a | n/a | n/a | n/a | ? | re-run on the sponsored Tier 3 key and 0.13.14 (the relays drop the field Google refuses); first try on the Free-tier ke ; retested once; first try: first [{"connection": "integration:Google AI Studio", "status": "failed", "error": "40 |
| fast-agent | gemini-3-flash-preview | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | |
| fast-agent | gemini-3.1-flash-lite | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | |
| fast-agent | gemini-3.1-pro-preview | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | |
| fast-agent | gemini-3.5-flash | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | |
| fast-agent | gemini-3.5-flash-lite | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | |
| fast-agent | gemini-3.6-flash | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | |
| fast-agent | gemini-3.8-flash | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | |
| hermes | gemini-3.6-flash | n/a | n/a | n/a | n/a | n/a | ? | re-run on the sponsored Tier 3 key and 0.13.14 (the relays drop the field Google refuses); first try on the Free-tier ke ; retested once; first try: first Your openai-api key was refused: API call failed after 3 retries: HTTP 429: [{ |
| opencode | gemini-3.6-flash | pass | pass | n/a | FAIL | pass | Google AI Studio | artifact: [{"connection": "integration:Google AI Studio", "status": "failed", "error": "Bad Request: [{\n "error": {\n "code": 400,\n "mes ; re-run on the sponsored Tier 3 key and 0.13.14 (the relays drop the field Google refuses); first try on the Free-tier ke ; retested once; first try: followup [{"connection": "integration:Google AI Studio", "status": "failed", "error": "To; artifact [{"connection": "integration:Google AI Studio", "status": "failed", "error": "To; recycle [{"connection": "integration:Google AI Studio", "status": "failed", "error": "To |
| pi | gemini-3.6-flash | n/a | n/a | n/a | n/a | n/a | ? | re-run on the sponsored Tier 3 key and 0.13.14 (the relays drop the field Google refuses); first try on the Free-tier ke ; retested once; first try: first [{"connection": "integration:Google AI Studio", "status": "failed", "error": "40 |
| qwen | gemini-3.6-flash | pass | pass | n/a | FAIL | pass | Google AI Studio | artifact: no file card (files: none); he word HELLO, then reply DONE. QWEN CODE I will check if hello-qwen.txt exists and then write "HELLO" to it. Us ; re-run on the sponsored Tier 3 key and 0.13.14 (the relays drop the field Google refuses); first try on the Free-tier ke ; retested once; first try: first Reply with exactly: M1-gemini-3.6-flash QWEN CODE M1-gemini-3.6-flash Working… |
| vibe | gemini-3-flash-preview | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | cache 7% of input |
| vibe | gemini-3.1-flash-lite | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | cache 14% of input |
| vibe | gemini-3.1-pro-preview | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | cache 6% of input |
| vibe | gemini-3.5-flash | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | cache 5% of input |
| vibe | gemini-3.5-flash-lite | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | cache 15% of input |
| vibe | gemini-3.6-flash | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | cache 6% of input |
| vibe | gemini-3.8-flash | pass | pass | pass (gemini-4-argon) | pass | pass | Google AI Studio | cache 0% of input |
| vibe | gemini-4-argon | pass | pass | pass (gemini-3.8-flash) | pass | pass | Google AI Studio | cache 6% of input |
28 pairs, 121 of 123 scenario runs passed.
Not run in this column, 36 pairs the provider serves that the harness did not run, with the reason:
- dsh x gemini-3-flash-preview: not run, not run in this column
- dsh x gemini-3.1-flash-lite: not run, not run in this column
- dsh x gemini-3.1-pro-preview: not run, not run in this column
- dsh x gemini-3.5-flash: not run, not run in this column
- dsh x gemini-3.5-flash-lite: not run, not run in this column
- dsh x gemini-3.8-flash: not run, not run in this column
- dsh x gemini-4-argon: not run, not run in this column
- fast-agent x gemini-4-argon: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- hermes x gemini-3-flash-preview: not run, not run in this column
- hermes x gemini-3.1-flash-lite: not run, not run in this column
- hermes x gemini-3.1-pro-preview: not run, not run in this column
- hermes x gemini-3.5-flash: not run, not run in this column
- hermes x gemini-3.5-flash-lite: not run, not run in this column
- hermes x gemini-3.8-flash: not run, not run in this column
- hermes x gemini-4-argon: not run, not run in this column
- opencode x gemini-3-flash-preview: not run, not run in this column
- opencode x gemini-3.1-flash-lite: not run, not run in this column
- opencode x gemini-3.1-pro-preview: not run, not run in this column
- opencode x gemini-3.5-flash: not run, not run in this column
- opencode x gemini-3.5-flash-lite: not run, not run in this column
- opencode x gemini-3.8-flash: not run, not run in this column
- opencode x gemini-4-argon: not run, not run in this column
- pi x gemini-3-flash-preview: not run, not run in this column
- pi x gemini-3.1-flash-lite: not run, not run in this column
- pi x gemini-3.1-pro-preview: not run, not run in this column
- pi x gemini-3.5-flash: not run, not run in this column
- pi x gemini-3.5-flash-lite: not run, not run in this column
- pi x gemini-3.8-flash: not run, not run in this column
- pi x gemini-4-argon: not run, not run in this column
- qwen x gemini-3-flash-preview: not run, not run in this column
- qwen x gemini-3.1-flash-lite: not run, not run in this column
- qwen x gemini-3.1-pro-preview: not run, not run in this column
- qwen x gemini-3.5-flash: not run, not run in this column
- qwen x gemini-3.5-flash-lite: not run, not run in this column
- qwen x gemini-3.8-flash: not run, not run in this column
- qwen x gemini-4-argon: not run, not run in this column
Provider: goose-anthropic
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| goose | claude-fable-5 | pass | pass | pass (claude-fable-5-1) | pass | pass | Anthropic | |
| goose | claude-fable-5-1 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | |
| goose | claude-haiku-4.5 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | served as claude-haiku-4-5-20251001 (the provider's alias of the same model) |
| goose | claude-opus-4.7 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | served as claude-opus-4-7 (the provider's alias of the same model) |
| goose | claude-opus-4.8 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | served as claude-opus-4-8 (the provider's alias of the same model) |
| goose | claude-sonnet-4.6 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | served as claude-sonnet-4-6 (the provider's alias of the same model) |
| goose | claude-sonnet-5 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic |
7 pairs, 35 of 35 scenario runs passed.
Not run in this column, 4 pairs the provider serves that the harness did not run, with the reason:
- goose x claude-haiku-5.5: not run, not run in this column
- goose x claude-opus-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-opus-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-sonnet-5.5: not run, not run in this column
Provider: goose-azure-e2
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| goose | gpt-5.2 | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI E2 | |
| goose | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | |
| goose | gpt-5.4-mini | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI E2 | |
| goose | gpt-5.5 | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI E2 | |
| goose | gpt-5.6-luna | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI E2 | |
| goose | gpt-5.6-sol | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI E2 | |
| goose | gpt-5.6-terra | pass | pass | pass (gpt-5.4) | pass | pass | Azure OpenAI E2 |
7 pairs, 35 of 35 scenario runs passed.
Not run in this column, 5 pairs the provider serves that the harness did not run, with the reason:
- goose x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-luna: not run, not run in this column
- goose x gpt-6-sol: not run, not run in this column
- goose x gpt-6.1-sol: not run, not run in this column
Provider: goose-harnessrouter
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| goose | claude-fable-5 | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | claude-fable-5-1 | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | claude-haiku-4.5 | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | served as claude-haiku-4-5-20251001 (the provider's alias of the same model) |
| goose | claude-opus-4.7 | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | served as claude-opus-4-7 (the provider's alias of the same model) |
| goose | claude-opus-4.8 | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | served as claude-opus-4-8 (the provider's alias of the same model) |
| goose | claude-sonnet-4.6 | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | served as claude-sonnet-4-6 (the provider's alias of the same model) |
| goose | claude-sonnet-5 | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | deepseek-v4-flash | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | deepseek-v4-pro | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | gemini-3-flash-preview | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | served as google/gemini-3-flash-preview (the provider's alias of the same model) |
| goose | gemini-3.1-pro-preview | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | gemini-3.5-flash | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | gemini-3.5-flash-lite | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | gemini-3.6-flash | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | gemini-3.7-flash | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | served as google/gemini-3.7-flash (the provider's alias of the same model) |
| goose | gemini-3.8-flash | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | glm-5.3 | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | glm-5.3-flash | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | gpt-5.2 | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | served as gpt-5.2-2025-12-11 (the provider's alias of the same model) |
| goose | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | HarnessRouter API test 1 | |
| goose | gpt-5.4-mini | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | served as gpt-5.4-mini-2026-03-17 (the provider's alias of the same model) |
| goose | gpt-5.5 | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | served as gpt-5.5-2026-04-23 (the provider's alias of the same model) |
| goose | gpt-5.6-luna | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | gpt-5.6-sol | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | gpt-5.6-terra | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | kimi-k2.7-code | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | kimi-k3 | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | mistral-medium-3.5 | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | served as mistralai/mistral-medium-3-5 (the provider's alias of the same model) |
| goose | qwen3.7-max | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | qwen3.8-max | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | |
| goose | step-3.7-flash | pass | pass | pass (gpt-5.4) | pass | pass | HarnessRouter API test 1 | served as stepfun/step-3.7-flash (the provider's alias of the same model) |
31 pairs, 155 of 155 scenario runs passed.
Not run in this column, 20 pairs the provider serves that the harness did not run, with the reason:
- goose x claude-opus-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-opus-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-sonnet-5.5: not run, not run in this column
- goose x deepseek-v4.1-flash: not run, not run in this column
- goose x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-luna: not run, not run in this column
- goose x gpt-6-sol: not run, not run in this column
- goose x gpt-6.1-sol: not run, not run in this column
- goose x grok-4.20: not run, not run in this column
- goose x grok-4.3: not run, not run in this column
- goose x grok-4.5: not run, not run in this column
- goose x grok-4.6: not run, not run in this column
- goose x grok-build-0.1: not run, not run in this column
- goose x hunyuan-4-preview: not run, not run in this column
- goose x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x nemotron-3-super: not run, not run in this column
- goose x nemotron-3.5-lightning: not run, not run in this column
- goose x qwen3.7-plus: not run, not run in this column
- goose x qwen3.8-flash: not run, not run in this column
Provider: goose-openai
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| goose | gpt-5.2 | pass | pass | pass (gpt-5.4) | pass | pass | OpenAI | served as gpt-5.2-2025-12-11 (the provider's alias of the same model) |
| goose | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | served as gpt-5.4-2026-03-05 (the provider's alias of the same model) |
| goose | gpt-5.4-mini | pass | pass | pass (gpt-5.4) | pass | pass | OpenAI | served as gpt-5.4-mini-2026-03-17 (the provider's alias of the same model) |
| goose | gpt-5.5 | pass | pass | pass (gpt-5.4) | pass | pass | OpenAI | served as gpt-5.5-2026-04-23 (the provider's alias of the same model) |
| goose | gpt-5.6-luna | pass | pass | pass (gpt-5.4) | pass | pass | OpenAI | |
| goose | gpt-5.6-sol | pass | pass | pass (gpt-5.4) | pass | pass | OpenAI | |
| goose | gpt-5.6-terra | pass | pass | pass (gpt-5.4) | pass | pass | OpenAI |
7 pairs, 35 of 35 scenario runs passed.
Not run in this column, 5 pairs the provider serves that the harness did not run, with the reason:
- goose x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-luna: not run, not run in this column
- goose x gpt-6-sol: not run, not run in this column
- goose x gpt-6.1-sol: not run, not run in this column
Provider: goose-openrouter
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| goose | claude-fable-5 | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as anthropic/claude-fable-5 (the provider's alias of the same model) |
| goose | claude-fable-5-1 | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as anthropic/claude-fable-5.1 (the provider's alias of the same model) |
| goose | claude-haiku-4.5 | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as anthropic/claude-haiku-4.5 (the provider's alias of the same model) |
| goose | claude-opus-4.7 | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as anthropic/claude-opus-4.7 (the provider's alias of the same model) |
| goose | claude-opus-4.8 | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as anthropic/claude-opus-4.8 (the provider's alias of the same model) |
| goose | claude-sonnet-4.6 | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as anthropic/claude-sonnet-4.6 (the provider's alias of the same model) |
| goose | claude-sonnet-5 | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as anthropic/claude-sonnet-5 (the provider's alias of the same model) |
| goose | deepseek-v4-flash | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as deepseek/deepseek-v4-flash (the provider's alias of the same model) |
| goose | deepseek-v4-pro | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as deepseek/deepseek-v4-pro (the provider's alias of the same model) |
| goose | deepseek-v4.1-flash | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as deepseek/deepseek-v4.1-flash (the provider's alias of the same model) |
| goose | gemini-3-flash-preview | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as google/gemini-3-flash-preview (the provider's alias of the same model) |
| goose | gemini-3.1-flash-lite | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as google/gemini-3.1-flash-lite (the provider's alias of the same model) |
| goose | gemini-3.1-pro-preview | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as google/gemini-3.1-pro-preview (the provider's alias of the same model) |
| goose | gemini-3.5-flash | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as google/gemini-3.5-flash (the provider's alias of the same model) |
| goose | gemini-3.5-flash-lite | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as google/gemini-3.5-flash-lite (the provider's alias of the same model) |
| goose | gemini-3.6-flash | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as google/gemini-3.6-flash (the provider's alias of the same model) |
| goose | gemini-3.7-flash | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as google/gemini-3.7-flash (the provider's alias of the same model) |
| goose | gemini-3.8-flash | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as google/gemini-3.8-flash (the provider's alias of the same model) |
| goose | glm-5.3 | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as z-ai/glm-5.3 (the provider's alias of the same model) |
| goose | glm-5.3-flash | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as z-ai/glm-5.3-flash (the provider's alias of the same model) |
| goose | gpt-5.2 | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as openai/gpt-5.2 (the provider's alias of the same model) |
| goose | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as openai/gpt-5.4 (the provider's alias of the same model) |
| goose | gpt-5.4-mini | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as openai/gpt-5.4-mini (the provider's alias of the same model) |
| goose | gpt-5.5 | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as openai/gpt-5.5 (the provider's alias of the same model) |
| goose | gpt-5.6-luna | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as openai/gpt-5.6-luna (the provider's alias of the same model) |
| goose | gpt-5.6-sol | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as openai/gpt-5.6-sol (the provider's alias of the same model) |
| goose | gpt-5.6-terra | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as openai/gpt-5.6-terra (the provider's alias of the same model) |
| goose | grok-4.20 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as x-ai/grok-4.20 (the provider's alias of the same model) |
| goose | grok-4.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as x-ai/grok-4.3 (the provider's alias of the same model) |
| goose | grok-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as x-ai/grok-4.5 (the provider's alias of the same model) |
| goose | grok-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as x-ai/grok-4.6 (the provider's alias of the same model) |
| goose | grok-build-0.1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as x-ai/grok-build-0.1 (the provider's alias of the same model) |
| goose | hunyuan-3 | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as tencent/hy3 (the provider's alias of the same model) |
| goose | hunyuan-4-preview | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as tencent/hy4-preview (the provider's alias of the same model) |
| goose | kimi-k2.7-code | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as moonshotai/kimi-k2.7-code (the provider's alias of the same model) |
| goose | kimi-k3 | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as moonshotai/kimi-k3 (the provider's alias of the same model) |
| goose | ling-3.0-flash | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as inclusionai/ling-3.0-flash (the provider's alias of the same model) |
| goose | llama-3.3-70b | pass | pass | pass (gpt-5.6-sol) | pass | FAIL | OpenRouter | served as meta-llama/llama-3.3-70b-instruct (the provider's alias of the same model) ; recycle: answered without M1-llama-3.3-70b: What exact word did I ask you to reply with in my very first message of this task? Reply with just that w |
| goose | llama-4-maverick | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as meta-llama/llama-4-maverick (the provider's alias of the same model) |
| goose | minimax-m3 | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as minimax/minimax-m3 (the provider's alias of the same model) |
| goose | mistral-medium-3.5 | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as mistralai/mistral-medium-3-5 (the provider's alias of the same model) |
| goose | muse-glimmer-30b | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as meta/muse-glimmer-30b (the provider's alias of the same model) |
| goose | muse-spark-1.1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as meta/muse-spark-1.1 (the provider's alias of the same model) |
| goose | muse-spark-1.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as meta/muse-spark-1.2 (the provider's alias of the same model) |
| goose | muse-spark-1.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as meta/muse-spark-1.3 (the provider's alias of the same model) |
| goose | nemotron-3-super | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) |
| goose | nemotron-3-ultra | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as nvidia/nemotron-3-ultra-550b-a55b (the provider's alias of the same model) |
| goose | nemotron-3.5-lightning | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) |
| goose | qwen3.7-flash | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as qwen/qwen3.7-flash (the provider's alias of the same model) |
| goose | qwen3.7-max | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as qwen/qwen3.7-max (the provider's alias of the same model) |
| goose | qwen3.7-plus | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as qwen/qwen3.7-plus (the provider's alias of the same model) |
| goose | qwen3.8-27b | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as qwen/qwen3.8-27b (the provider's alias of the same model) |
| goose | qwen3.8-flash | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as qwen/qwen3.8-flash (the provider's alias of the same model) |
| goose | qwen3.8-max | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as qwen/qwen3.8-max-0902 (the provider's alias of the same model) |
| goose | step-3.7-flash | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | served as stepfun/step-3.7-flash (the provider's alias of the same model) |
55 pairs, 274 of 275 scenario runs passed.
Not run in this column, 14 pairs the provider serves that the harness did not run, with the reason:
- goose x claude-haiku-5.5: not run, not run in this column
- goose x claude-opus-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-opus-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-sonnet-5.5: not run, not run in this column
- goose x gemini-4-argon: not run, not run in this column
- goose x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-luna: not run, not run in this column
- goose x gpt-6-sol: not run, not run in this column
- goose x gpt-6.1-sol: not run, not run in this column
- goose x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x muse-wonder-1.0: not run, not run in this column
Provider: goose-tokenrouter
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| goose | claude-fable-5 | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | claude-fable-5-1 | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | claude-haiku-4.5 | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as claude-haiku-4-5-20251001 (the provider's alias of the same model) |
| goose | claude-opus-4.7 | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as claude-opus-4-7 (the provider's alias of the same model) |
| goose | claude-opus-4.8 | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as claude-opus-4-8 (the provider's alias of the same model) |
| goose | claude-sonnet-4.6 | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as claude-sonnet-4-6 (the provider's alias of the same model) |
| goose | claude-sonnet-5 | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | deepseek-v4-flash | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | deepseek-v4-pro | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | deepseek-v4.1-flash | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as deepseek-flash (the provider's alias of the same model) |
| goose | gemini-3-flash-preview | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as google/gemini-3-flash-preview (the provider's alias of the same model) |
| goose | gemini-3.1-pro-preview | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | gemini-3.5-flash | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | gemini-3.5-flash-lite | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | gemini-3.6-flash | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | gemini-3.7-flash | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as google/gemini-3.7-flash (the provider's alias of the same model) |
| goose | gemini-3.8-flash | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | glm-5.3 | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | glm-5.3-flash | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | gpt-5.2 | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as gpt-5.2-2025-12-11 (the provider's alias of the same model) |
| goose | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | |
| goose | gpt-5.4-mini | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as gpt-5.4-mini-2026-03-17 (the provider's alias of the same model) |
| goose | gpt-5.5 | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as gpt-5.5-2026-04-23 (the provider's alias of the same model) |
| goose | gpt-5.6-luna | pass | pass | pass (gpt-5.4) | pass | FAIL | My TokenRouter | recycle: answered without M1-gpt-5.6-luna: What exact word did I ask you to reply with in my very first message of this task? Reply with just that wo |
| goose | gpt-5.6-sol | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | gpt-5.6-terra | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | grok-4.20 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as x-ai/grok-4.20 (the provider's alias of the same model) |
| goose | grok-4.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as x-ai/grok-4.3 (the provider's alias of the same model) |
| goose | grok-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | |
| goose | grok-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | |
| goose | grok-build-0.1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as x-ai/grok-build-0.1 (the provider's alias of the same model) |
| goose | hunyuan-4-preview | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as tencent/hy4-preview (the provider's alias of the same model) |
| goose | kimi-k2.7-code | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | kimi-k3 | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | mistral-medium-3.5 | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as mistralai/mistral-medium-3-5 (the provider's alias of the same model) |
| goose | nemotron-3-super | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) |
| goose | nemotron-3.5-lightning | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) |
| goose | qwen3.7-max | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | qwen3.7-plus | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as qwen/qwen3.7-plus (the provider's alias of the same model) |
| goose | qwen3.8-flash | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | qwen3.8-max | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| goose | step-3.7-flash | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | served as stepfun/step-3.7-flash (the provider's alias of the same model) |
42 pairs, 209 of 210 scenario runs passed.
Not run in this column, 9 pairs the provider serves that the harness did not run, with the reason:
- goose x claude-opus-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-opus-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-sonnet-5.5: not run, not run in this column
- goose x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-luna: not run, not run in this column
- goose x gpt-6-sol: not run, not run in this column
- goose x gpt-6.1-sol: not run, not run in this column
- goose x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
Provider: goose-vercel
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| goose | claude-fable-5 | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as anthropic/claude-fable-5 (the provider's alias of the same model) |
| goose | claude-fable-5-1 | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as anthropic/claude-fable-5.1 (the provider's alias of the same model) |
| goose | claude-haiku-4.5 | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as anthropic/claude-haiku-4.5 (the provider's alias of the same model) |
| goose | claude-opus-4.7 | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as anthropic/claude-opus-4.7 (the provider's alias of the same model) |
| goose | claude-opus-4.8 | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as anthropic/claude-opus-4.8 (the provider's alias of the same model) |
| goose | claude-sonnet-4.6 | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as anthropic/claude-sonnet-4.6 (the provider's alias of the same model) |
| goose | claude-sonnet-5 | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as anthropic/claude-sonnet-5 (the provider's alias of the same model) |
| goose | deepseek-v4-flash | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as deepseek/deepseek-v4-flash (the provider's alias of the same model) |
| goose | deepseek-v4-pro | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as deepseek/deepseek-v4-pro (the provider's alias of the same model) |
| goose | deepseek-v4.1-flash | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as deepseek/deepseek-v4.1-flash (the provider's alias of the same model) |
| goose | gemini-3-flash-preview | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as google/gemini-3-flash (the provider's alias of the same model) |
| goose | gemini-3.1-flash-lite | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as google/gemini-3.1-flash-lite (the provider's alias of the same model) |
| goose | gemini-3.1-pro-preview | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as google/gemini-3.1-pro-preview (the provider's alias of the same model) |
| goose | gemini-3.5-flash | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as google/gemini-3.5-flash (the provider's alias of the same model) |
| goose | gemini-3.5-flash-lite | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as google/gemini-3.5-flash-lite (the provider's alias of the same model) |
| goose | gemini-3.6-flash | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as google/gemini-3.6-flash (the provider's alias of the same model) |
| goose | gemini-3.7-flash | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as google/gemini-3.7-flash (the provider's alias of the same model) |
| goose | gemini-3.8-flash | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as google/gemini-3.8-flash (the provider's alias of the same model) |
| goose | glm-5.3 | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as zai/glm-5.3 (the provider's alias of the same model) |
| goose | glm-5.3-flash | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as zai/glm-5.3-flash (the provider's alias of the same model) |
| goose | gpt-5.2 | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.2 (the provider's alias of the same model) |
| goose | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.4 (the provider's alias of the same model) |
| goose | gpt-5.4-mini | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.4-mini (the provider's alias of the same model) |
| goose | gpt-5.5 | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.5 (the provider's alias of the same model) |
| goose | gpt-5.6-luna | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.6-luna (the provider's alias of the same model) |
| goose | gpt-5.6-sol | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.6-sol (the provider's alias of the same model) |
| goose | gpt-5.6-terra | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.6-terra (the provider's alias of the same model) |
| goose | grok-4.1-fast | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.1-fast-reasoning (finding below) |
| goose | grok-4.20 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.20-reasoning (the provider's alias of the same model) |
| goose | grok-4.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.3 (the provider's alias of the same model) |
| goose | grok-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.5 (the provider's alias of the same model) |
| goose | grok-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.6 (the provider's alias of the same model) |
| goose | grok-build-0.1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as spacexai/grok-build-0.1 (the provider's alias of the same model) |
| goose | hunyuan-3 | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as tencent/hy3 (the provider's alias of the same model) |
| goose | hunyuan-4-preview | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as tencent/hy4-preview (the provider's alias of the same model) |
| goose | kimi-k2.7-code | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as moonshotai/kimi-k2.7-code (the provider's alias of the same model) |
| goose | kimi-k3 | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as moonshotai/kimi-k3 (the provider's alias of the same model) |
| goose | ling-3.0-flash | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as inclusionai/ling-3.0-flash (the provider's alias of the same model) |
| goose | llama-3.3-70b | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | first: The turn failed: Ran into this error: Request failed: Bad request (400): undefined: This model doesn't support tool use in streaming mode.. |
| goose | llama-4-maverick | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | first: The turn failed: Ran into this error: Request failed: Request failed with status 405 Method Not Allowed at http://127.0.0.1:39571/v1/chat/co |
| goose | llama-4-scout | pass | pass | pass (gpt-5.6-sol) | FAIL | pass | Vercel AI Gateway | served as meta/llama-4-scout (the provider's alias of the same model) ; artifact: no file card (files: none); le named hello-goose.txt containing exactly the word HELLO, then reply DONE. GOOSE The model returned an empty r |
| goose | minimax-m3 | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as minimax/minimax-m3 (the provider's alias of the same model) |
| goose | mistral-medium-3.5 | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as mistral/mistral-medium-3.5 (the provider's alias of the same model) |
| goose | muse-glimmer-30b | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as meta/muse-glimmer-30b (the provider's alias of the same model) |
| goose | muse-spark-1.1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as meta/muse-spark-1.1 (the provider's alias of the same model) |
| goose | muse-spark-1.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as meta/muse-spark-1.2 (the provider's alias of the same model) |
| goose | muse-spark-1.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as meta/muse-spark-1.3 (the provider's alias of the same model) |
| goose | nemotron-3-super | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) |
| goose | nemotron-3-ultra | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as nvidia/nemotron-3-ultra-550b-a55b (the provider's alias of the same model) |
| goose | nemotron-3.5-lightning | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) |
| goose | qwen3.7-flash | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.7-flash (the provider's alias of the same model) |
| goose | qwen3.7-max | pass | pass | pass (gpt-5.4) | FAIL | pass | Vercel AI Gateway | served as alibaba/qwen3.7-max (the provider's alias of the same model) ; artifact: The turn failed: Ran into this error: Server error: Upstream stream ended before terminal chunk. |
Please retry if you think this is a transi | | goose | qwen3.7-plus | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.7-plus (the provider's alias of the same model) | | goose | qwen3.8-27b | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.8-27b (the provider's alias of the same model) | | goose | qwen3.8-flash | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.8-flash (the provider's alias of the same model) | | goose | qwen3.8-max | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.8-max (the provider's alias of the same model) | | goose | step-3.7-flash | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | served as stepfun/step-3.7-flash (the provider's alias of the same model) |
57 pairs, 268 of 272 scenario runs passed; 1 pairs served by another connection or as another model are findings, not counted.
Findings, pairs served by a connection other than the one under test or as a model other than the id asked for:
- goose x grok-4.1-fast: served as spacexai/grok-4.1-fast-reasoning (the CLI reports the model it ran)
Not run in this column, 12 pairs the provider serves that the harness did not run, with the reason:
- goose x claude-haiku-5.5: not run, not run in this column
- goose x claude-opus-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-opus-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-sonnet-5.5: not run, not run in this column
- goose x gemini-4-argon: not run, not run in this column
- goose x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-luna: not run, not run in this column
- goose x gpt-6-sol: not run, not run in this column
- goose x gpt-6.1-sol: not run, not run in this column
- goose x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x muse-wonder-1.0: not run, not run in this column
Provider: grok-default
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| grok | claude-fable-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 8% of input |
| grok | claude-fable-5-1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 8% of input |
| grok | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Anthropic, OpenRouter | served as claude-haiku-4-5-20251001 (the provider's alias of the same model) ; cache 10% of input |
| grok | claude-opus-4.7 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as claude-opus-4-7 (the provider's alias of the same model) ; cache 8% of input |
| grok | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as claude-opus-4-8 (the provider's alias of the same model) ; cache 8% of input |
| grok | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 8% of input |
| grok | claude-opus-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as anthropic/claude-opus-5.5 (the provider's alias of the same model) ; cache 8% of input |
| grok | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Anthropic, OpenRouter | served as claude-sonnet-4-6 (the provider's alias of the same model) ; cache 10% of input |
| grok | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 6% of input |
| grok | claude-sonnet-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as claude-sonnet-5-5 (the provider's alias of the same model) ; cache 8% of input |
| grok | deepseek-v4-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as deepseek-v4.1-flash (finding below) ; cache 78% of input |
| grok | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | FAIL | pass | My TokenRouter, OpenRouter | artifact: The turn failed: Internal error: { "message": "empty response from model (reasoning_only)", "promptUsage": { "inputTokens": 71225, "outputTo ; retested once; first try: artifact The turn failed: Internal error: { "message": "empty response from model (reason ; cache 82% of input |
| grok | deepseek-v4.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 79% of input |
| grok | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 65% of input |
| grok | gemini-3.1-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as google/gemini-3.1-flash-lite (the provider's alias of the same model) ; cache 56% of input |
| grok | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 65% of input |
| grok | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 78% of input |
| grok | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 78% of input |
| grok | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 52% of input |
| grok | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 52% of input |
| grok | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 39% of input |
| grok | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 79% of input |
| grok | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 79% of input |
| grok | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Custom OpenAI Chat, OpenRouter | served as gpt-5.2-2025-12-11 (the provider's alias of the same model) ; cache 76% of input |
| grok | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as gpt-5.4-2026-03-05 (the provider's alias of the same model) ; cache 62% of input |
| grok | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | Custom OpenAI Chat, OpenRouter | served as gpt-5.4-mini-2026-03-17 (the provider's alias of the same model) ; cache 77% of input |
| grok | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2, OpenRouter | served as gpt-5.5-2026-04-24 (the provider's alias of the same model) ; cache 77% of input |
| grok | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 79% of input |
| grok | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 79% of input |
| grok | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 79% of input |
| grok | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 80% of input |
| grok | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 93% of input |
| grok | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter, My TokenRouter | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 66% of input |
| grok | grok-4.20 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as x-ai/grok-4.20 (the provider's alias of the same model) ; cache 63% of input |
| grok | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as x-ai/grok-4.3 (the provider's alias of the same model) ; cache 80% of input |
| grok | grok-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 64% of input |
| grok | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 83% of input |
| grok | grok-build-0.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as x-ai/grok-build-0.1 (the provider's alias of the same model) ; cache 83% of input |
| grok | hunyuan-4-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as tencent/hy4-preview (the provider's alias of the same model) ; cache 82% of input |
| grok | kimi-k2.7-code | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | retested once; first try: artifact no file card (files: none); Create a file named hello-grok.txt containing exactl ; cache 79% of input |
| grok | kimi-k3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | retested once; first try: first The turn failed: Internal error: "serialization error: invalid type: null, expec ; cache 79% of input |
| grok | llama-3.3-70b | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | served as meta-llama/llama-3.3-70b-instruct (the provider's alias of the same model) ; cache 62% of input |
| grok | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as mistralai/mistral-medium-3-5 (the provider's alias of the same model) ; cache 79% of input |
| grok | muse-glimmer-30b | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-glimmer-30b (the provider's alias of the same model) ; cache 62% of input |
| grok | muse-spark-1.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-spark-1.1 (the provider's alias of the same model) ; cache 45% of input |
| grok | muse-spark-1.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-spark-1.2 (the provider's alias of the same model) ; cache 68% of input |
| grok | muse-spark-1.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-spark-1.3 (the provider's alias of the same model) ; cache 28% of input |
| grok | nemotron-3-super | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) ; cache 10% of input |
| grok | nemotron-3.5-lightning | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) ; cache 55% of input |
| grok | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 73% of input |
| grok | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 76% of input |
| grok | qwen3.8-27b | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as alibaba/qwen3.8-27b (the provider's alias of the same model) ; cache 71% of input |
| grok | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 80% of input |
| grok | qwen3.8-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 79% of input |
| grok | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as stepfun/step-3.7-flash (the provider's alias of the same model) ; cache 78% of input |
55 pairs, 269 of 270 scenario runs passed; 1 pairs served by another connection or as another model are findings, not counted.
Findings, pairs served by a connection other than the one under test or as a model other than the id asked for:
- grok x deepseek-v4-flash: served as deepseek-v4.1-flash (the CLI reports the model it ran)
Provider: harnessrouter
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| dcode | claude-fable-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| dcode | claude-fable-5-1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| dcode | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| dcode | claude-haiku-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| dcode | claude-opus-4.7 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| dcode | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| dcode | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| dcode | claude-opus-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| dcode | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| dcode | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| dcode | claude-sonnet-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| dcode | deepseek-v4-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 63% of input |
| dcode | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 71% of input |
| dcode | deepseek-v4.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 17% of input |
| dcode | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 31% of input |
| dcode | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 77% of input |
| dcode | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 14% of input |
| dcode | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 61% of input |
| dcode | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 46% of input |
| dcode | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 66% of input |
| dcode | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 60% of input |
| dcode | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 66% of input |
| dcode | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 66% of input |
| dcode | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 66% of input |
| dcode | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 66% of input |
| dcode | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 71% of input |
| dcode | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 66% of input |
| dcode | grok-4.20 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 21% of input |
| dcode | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 67% of input |
| dcode | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 72% of input |
| dcode | grok-build-0.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 67% of input |
| dcode | kimi-k3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 50% of input |
| dcode | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| dcode | muse-spark-1.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 50% of input |
| dcode | muse-spark-1.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 17% of input |
| dcode | muse-wonder-1.0 | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 33% of input |
| dcode | nemotron-3-super | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| dcode | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 63% of input |
| dcode | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 66% of input |
| dcode | qwen3.8-27b | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 68% of input |
| dcode | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 66% of input |
| dcode | qwen3.8-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 68% of input |
| dcode | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | HarnessRouter API test 1 | cache 66% of input |
| docker-agent | claude-fable-5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| docker-agent | claude-fable-5-1 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| docker-agent | claude-haiku-4.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| docker-agent | claude-haiku-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| docker-agent | claude-opus-4.8 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| docker-agent | claude-opus-5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| docker-agent | claude-opus-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| docker-agent | claude-sonnet-4.6 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| docker-agent | claude-sonnet-5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| docker-agent | claude-sonnet-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| docker-agent | deepseek-v4-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 68% of input |
| docker-agent | deepseek-v4-pro | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 77% of input |
| docker-agent | deepseek-v4.1-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 50% of input |
| docker-agent | gemini-3-flash-preview | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| docker-agent | gemini-3.1-flash-lite | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| docker-agent | gemini-3.1-pro-preview | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 7% of input |
| docker-agent | gemini-3.5-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| docker-agent | gemini-3.5-flash-lite | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| docker-agent | gemini-3.6-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| docker-agent | gemini-3.7-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| docker-agent | gemini-3.8-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| docker-agent | gemini-4-argon | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 32% of input |
| docker-agent | glm-5.3 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 70% of input |
| docker-agent | glm-5.3-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| docker-agent | gpt-5.2 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 78% of input |
| docker-agent | gpt-5.4 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 77% of input |
| docker-agent | gpt-5.4-mini | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 79% of input |
| docker-agent | gpt-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 42% of input |
| docker-agent | gpt-5.6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 67% of input |
| docker-agent | gpt-5.6-sol | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 67% of input |
| docker-agent | gpt-5.6-terra | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 85% of input |
| docker-agent | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 67% of input |
| docker-agent | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | HarnessRouter API test 1 | cache 72% of input |
| docker-agent | grok-4.3 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 75% of input |
| docker-agent | grok-4.6 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 54% of input |
| docker-agent | hunyuan-4-preview | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 70% of input |
| docker-agent | kimi-k2.7-code | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 68% of input |
| docker-agent | kimi-k3 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 70% of input |
| docker-agent | mistral-medium-3.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| docker-agent | muse-glimmer-30b | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 84% of input |
| docker-agent | muse-spark-1.1 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 54% of input |
| docker-agent | muse-spark-1.2 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 37% of input |
| docker-agent | muse-spark-1.3 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 20% of input |
| docker-agent | muse-wonder-1.0 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 52% of input |
| docker-agent | nemotron-3-super | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| docker-agent | qwen3.7-max | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 70% of input |
| docker-agent | qwen3.7-plus | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 70% of input |
| docker-agent | qwen3.8-27b | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 70% of input |
| docker-agent | qwen3.8-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 69% of input |
| docker-agent | qwen3.8-max | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 71% of input |
| docker-agent | step-3.7-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 58% of input |
| fast-agent | claude-fable-5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| fast-agent | claude-fable-5-1 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| fast-agent | claude-haiku-4.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| fast-agent | claude-haiku-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| fast-agent | claude-opus-4.7 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| fast-agent | claude-opus-4.8 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| fast-agent | claude-opus-5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| fast-agent | claude-opus-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| fast-agent | claude-sonnet-4.6 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| fast-agent | claude-sonnet-5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| fast-agent | claude-sonnet-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| fast-agent | deepseek-v4-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 58% of input |
| fast-agent | deepseek-v4-pro | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 69% of input |
| fast-agent | deepseek-v4.1-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 10% of input |
| fast-agent | gemini-3-flash-preview | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| fast-agent | gemini-3.1-flash-lite | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| fast-agent | gemini-3.1-pro-preview | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| fast-agent | gemini-3.5-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| fast-agent | gemini-3.5-flash-lite | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| fast-agent | gemini-3.6-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| fast-agent | gemini-3.7-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| fast-agent | gemini-3.8-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| fast-agent | glm-5.3 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 66% of input |
| fast-agent | glm-5.3-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| fast-agent | gpt-5.2 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 56% of input |
| fast-agent | gpt-5.4 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 30% of input |
| fast-agent | gpt-5.4-mini | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 58% of input |
| fast-agent | gpt-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 30% of input |
| fast-agent | gpt-5.6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 66% of input |
| fast-agent | gpt-5.6-sol | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 65% of input |
| fast-agent | gpt-5.6-terra | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 88% of input |
| fast-agent | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 66% of input |
| fast-agent | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | HarnessRouter API test 1 | cache 66% of input |
| fast-agent | grok-4.20 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 51% of input |
| fast-agent | grok-4.3 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 52% of input |
| fast-agent | grok-4.6 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 68% of input |
| fast-agent | grok-build-0.1 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 77% of input |
| fast-agent | kimi-k3 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 65% of input |
| fast-agent | mistral-medium-3.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| fast-agent | muse-glimmer-30b | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 69% of input |
| fast-agent | muse-spark-1.1 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 58% of input |
| fast-agent | muse-spark-1.2 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 34% of input |
| fast-agent | muse-spark-1.3 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 18% of input |
| fast-agent | muse-wonder-1.0 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 54% of input |
| fast-agent | qwen3.7-max | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 36% of input |
| fast-agent | qwen3.7-plus | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 52% of input |
| fast-agent | qwen3.8-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 61% of input |
| fast-agent | step-3.7-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 52% of input |
| vibe | claude-fable-5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| vibe | claude-fable-5-1 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| vibe | claude-haiku-4.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| vibe | claude-haiku-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| vibe | claude-opus-4.8 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| vibe | claude-opus-5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| vibe | claude-opus-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| vibe | claude-sonnet-4.6 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| vibe | claude-sonnet-5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| vibe | claude-sonnet-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | no prompt cache reads (finding below) |
| vibe | deepseek-v4-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 64% of input |
| vibe | deepseek-v4-pro | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 63% of input |
| vibe | gemini-3-flash-preview | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 15% of input |
| vibe | gemini-3.1-flash-lite | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 16% of input |
| vibe | gemini-3.1-pro-preview | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 26% of input |
| vibe | gemini-3.5-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 26% of input |
| vibe | gemini-3.5-flash-lite | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 16% of input |
| vibe | gemini-3.6-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 24% of input |
| vibe | gemini-3.7-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 22% of input |
| vibe | gemini-3.8-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| vibe | gemini-4-argon | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 25% of input |
| vibe | glm-5.3 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 55% of input |
| vibe | glm-5.3-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 17% of input |
| vibe | gpt-5.2 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 48% of input |
| vibe | gpt-5.4 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 49% of input |
| vibe | gpt-5.4-mini | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 53% of input |
| vibe | gpt-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 49% of input |
| vibe | gpt-5.6-sol | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 50% of input |
| vibe | gpt-5.6-terra | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 50% of input |
| vibe | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 50% of input |
| vibe | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | HarnessRouter API test 1 | cache 50% of input |
| vibe | grok-4.20 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 51% of input |
| vibe | grok-4.3 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 34% of input |
| vibe | grok-4.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 51% of input |
| vibe | grok-4.6 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 50% of input |
| vibe | grok-build-0.1 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 62% of input |
| vibe | hunyuan-4-preview | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 79% of input |
| vibe | kimi-k2.7-code | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 66% of input |
| vibe | kimi-k3 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 67% of input |
| vibe | mistral-medium-3.5 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 29% of input |
| vibe | muse-glimmer-30b | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 66% of input |
| vibe | muse-spark-1.1 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 34% of input |
| vibe | muse-spark-1.2 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 34% of input |
| vibe | muse-spark-1.3 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 34% of input |
| vibe | muse-wonder-1.0 | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 34% of input |
| vibe | nemotron-3-super | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 0% of input |
| vibe | nemotron-3.5-lightning | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 22% of input |
| vibe | qwen3.7-max | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 48% of input |
| vibe | qwen3.7-plus | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 49% of input |
| vibe | qwen3.8-27b | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 51% of input |
| vibe | qwen3.8-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 52% of input |
| vibe | qwen3.8-max | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 51% of input |
| vibe | step-3.7-flash | pass | pass | pass (gpt-6-sol) | pass | pass | HarnessRouter API test 1 | cache 49% of input |
195 pairs, 975 of 975 scenario runs passed.
Findings, pairs served by a connection other than the one under test or as a model other than the id asked for:
- dcode x claude-fable-5: no prompt cache reads over 90248 input tokens (every call paid full price)
- dcode x claude-fable-5-1: no prompt cache reads over 90016 input tokens (every call paid full price)
- dcode x claude-haiku-4.5: no prompt cache reads over 70176 input tokens (every call paid full price)
- dcode x claude-haiku-5.5: no prompt cache reads over 90405 input tokens (every call paid full price)
- dcode x claude-opus-4.7: no prompt cache reads over 91720 input tokens (every call paid full price)
- dcode x claude-opus-4.8: no prompt cache reads over 89870 input tokens (every call paid full price)
- dcode x claude-opus-5: no prompt cache reads over 90538 input tokens (every call paid full price)
- dcode x claude-opus-5.5: no prompt cache reads over 89862 input tokens (every call paid full price)
- dcode x claude-sonnet-4.6: no prompt cache reads over 70613 input tokens (every call paid full price)
- dcode x claude-sonnet-5: no prompt cache reads over 90561 input tokens (every call paid full price)
- dcode x claude-sonnet-5.5: no prompt cache reads over 90610 input tokens (every call paid full price)
- docker-agent x claude-fable-5: no prompt cache reads over 27486 input tokens (every call paid full price)
- docker-agent x claude-fable-5-1: no prompt cache reads over 27562 input tokens (every call paid full price)
- docker-agent x claude-haiku-4.5: no prompt cache reads over 22656 input tokens (every call paid full price)
- docker-agent x claude-haiku-5.5: no prompt cache reads over 27510 input tokens (every call paid full price)
- docker-agent x claude-opus-4.8: no prompt cache reads over 27594 input tokens (every call paid full price)
- docker-agent x claude-opus-5: no prompt cache reads over 27540 input tokens (every call paid full price)
- docker-agent x claude-opus-5.5: no prompt cache reads over 27568 input tokens (every call paid full price)
- docker-agent x claude-sonnet-4.6: no prompt cache reads over 22613 input tokens (every call paid full price)
- docker-agent x claude-sonnet-5: no prompt cache reads over 27898 input tokens (every call paid full price)
- docker-agent x claude-sonnet-5.5: no prompt cache reads over 27574 input tokens (every call paid full price)
- fast-agent x claude-fable-5: no prompt cache reads over 18175 input tokens (every call paid full price)
- fast-agent x claude-fable-5-1: no prompt cache reads over 18284 input tokens (every call paid full price)
- fast-agent x claude-haiku-4.5: no prompt cache reads over 14340 input tokens (every call paid full price)
- fast-agent x claude-haiku-5.5: no prompt cache reads over 16286 input tokens (every call paid full price)
- fast-agent x claude-opus-4.7: no prompt cache reads over 18459 input tokens (every call paid full price)
- fast-agent x claude-opus-4.8: no prompt cache reads over 16443 input tokens (every call paid full price)
- fast-agent x claude-opus-5: no prompt cache reads over 18364 input tokens (every call paid full price)
- fast-agent x claude-opus-5.5: no prompt cache reads over 16165 input tokens (every call paid full price)
- fast-agent x claude-sonnet-4.6: no prompt cache reads over 14381 input tokens (every call paid full price)
- fast-agent x claude-sonnet-5: no prompt cache reads over 18682 input tokens (every call paid full price)
- fast-agent x claude-sonnet-5.5: no prompt cache reads over 16251 input tokens (every call paid full price)
- vibe x claude-fable-5: no prompt cache reads over 77061 input tokens (every call paid full price)
- vibe x claude-fable-5-1: no prompt cache reads over 77211 input tokens (every call paid full price)
- vibe x claude-haiku-4.5: no prompt cache reads over 61217 input tokens (every call paid full price)
- vibe x claude-haiku-5.5: no prompt cache reads over 77637 input tokens (every call paid full price)
- vibe x claude-opus-4.8: no prompt cache reads over 77088 input tokens (every call paid full price)
- vibe x claude-opus-5: no prompt cache reads over 77010 input tokens (every call paid full price)
- vibe x claude-opus-5.5: no prompt cache reads over 77552 input tokens (every call paid full price)
- vibe x claude-sonnet-4.6: no prompt cache reads over 61386 input tokens (every call paid full price)
- vibe x claude-sonnet-5: no prompt cache reads over 77462 input tokens (every call paid full price)
- vibe x claude-sonnet-5.5: no prompt cache reads over 77545 input tokens (every call paid full price)
Not run in this column, 39 pairs the provider serves that the harness did not run, with the reason:
- dcode x gemini-3-flash-preview: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x gemini-3.1-pro-preview: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x gemini-3.5-flash: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x gemini-3.5-flash-lite: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x gemini-3.6-flash: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x gemini-3.8-flash: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x gpt-5.3-codex: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x gpt-6-astra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x nemotron-3.5-lightning: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- docker-agent x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- docker-agent x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- docker-agent x grok-4.20: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x grok-build-0.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x nemotron-3.5-lightning: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- fast-agent x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- fast-agent x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- fast-agent x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x nemotron-3-super: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x nemotron-3.5-lightning: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x qwen3.8-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x deepseek-v4.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- vibe x gpt-5.6-luna: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- vibe x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- vibe x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
Provider: hermes-azure-e2
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| hermes | gpt-6-astra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 |
1 pairs, 5 of 5 scenario runs passed.
Not run in this column, 11 pairs the provider serves that the harness did not run, with the reason:
- hermes x gpt-5.2: not run, not run in this column
- hermes x gpt-5.3-codex: not run, not run in this column
- hermes x gpt-5.4: not run, not run in this column
- hermes x gpt-5.4-mini: not run, not run in this column
- hermes x gpt-5.5: not run, not run in this column
- hermes x gpt-5.6-luna: not run, not run in this column
- hermes x gpt-5.6-sol: not run, not run in this column
- hermes x gpt-5.6-terra: not run, not run in this column
- hermes x gpt-6-luna: not run, not run in this column
- hermes x gpt-6-sol: not run, not run in this column
- hermes x gpt-6.1-sol: not run, not run in this column
Provider: kilo-default
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| kilo | claude-fable-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) |
| kilo | claude-fable-5-1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) |
| kilo | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Custom Anthropic, OpenRouter | served as claude-haiku-4-5-20251001 (the provider's alias of the same model) ; retested once; first try: ; cache 99% of input |
| kilo | claude-opus-4.7 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) ; served as claude-opus-4-7 (the provider's alias of the same model) |
| kilo | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) ; served as claude-opus-4-8 (the provider's alias of the same model) ; retested once; first try: |
| kilo | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | FAIL | FAIL | My TokenRouter, OpenRouter | artifact: The turn failed: the provider declined the request (finish_reason content_filter) ; recycle: The turn failed: the provider declined the request (finish_reason content_filter) ; retested once; first try: artifact The turn failed: the provider declined the request (finish_reason content_filter; recycle The turn failed: the provider declined the request (finish_reason content_filter ; cache 0% of input |
| kilo | claude-opus-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as anthropic/claude-opus-5.5 (the provider's alias of the same model) ; cache 79% of input |
| kilo | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Custom Anthropic, OpenRouter | served as claude-sonnet-4-6 (the provider's alias of the same model) ; retested once; first try: ; cache 99% of input |
| kilo | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) |
| kilo | claude-sonnet-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) ; served as claude-sonnet-5-5 (the provider's alias of the same model) |
| kilo | deepseek-v4-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as deepseek-v4.1-flash (finding below) ; cache 77% of input |
| kilo | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 76% of input |
| kilo | deepseek-v4.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 82% of input |
| kilo | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 64% of input |
| kilo | gemini-3.1-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as google/gemini-3.1-flash-lite (the provider's alias of the same model) ; cache 38% of input |
| kilo | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 48% of input |
| kilo | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 74% of input |
| kilo | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 37% of input |
| kilo | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 66% of input |
| kilo | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 64% of input |
| kilo | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 48% of input |
| kilo | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 80% of input |
| kilo | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 79% of input |
| kilo | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Custom OpenAI Chat, OpenRouter | served as gpt-5.2-2025-12-11 (the provider's alias of the same model) ; cache 78% of input |
| kilo | gpt-5.3-codex | pass | pass | n/a | pass | pass | Azure OpenAI E2 | cache 78% of input |
| kilo | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 79% of input |
| kilo | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | Custom OpenAI Chat, OpenRouter | served as gpt-5.4-mini-2026-03-17 (the provider's alias of the same model) ; cache 77% of input |
| kilo | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2, OpenRouter | cache 78% of input |
| kilo | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 100% of input |
| kilo | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 100% of input |
| kilo | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 100% of input |
| kilo | gpt-6-astra | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 100% of input |
| kilo | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 100% of input |
| kilo | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 100% of input |
| kilo | gpt-6.1-sol | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter, My TokenRouter | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 100% of input |
| kilo | grok-4.20 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as x-ai/grok-4.20 (the provider's alias of the same model) ; cache 80% of input |
| kilo | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as x-ai/grok-4.3 (the provider's alias of the same model) ; cache 80% of input |
| kilo | grok-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 60% of input |
| kilo | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 80% of input |
| kilo | grok-build-0.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as x-ai/grok-build-0.1 (the provider's alias of the same model) ; cache 69% of input |
| kilo | hunyuan-4-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as tencent/hy4-preview (the provider's alias of the same model) ; cache 79% of input |
| kilo | kimi-k2.7-code | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 79% of input |
| kilo | kimi-k3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 59% of input |
| kilo | llama-3.3-70b | pass | pass | pass (gpt-6.1-sol) | FAIL | pass | OpenRouter | served as meta-llama/llama-3.3-70b-instruct (the provider's alias of the same model) ; artifact: no file card (files: none); xactly the word HELLO, then reply DONE. KILO CODE write.write(content="HELLO", filePath="/data/workspaces/hsesse ; retested once; first try: first etch Webfetch Webfetch Webfetch Webfetch Webfetch Webfetch Webfetch Webfetch Web ; cache 60% of input |
| kilo | llama-4-maverick | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | served as meta-llama/llama-4-maverick (the provider's alias of the same model) ; cache 80% of input |
| kilo | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as mistralai/mistral-medium-3-5 (the provider's alias of the same model) ; cache 79% of input |
| kilo | muse-glimmer-30b | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-glimmer-30b (the provider's alias of the same model) ; cache 60% of input |
| kilo | muse-spark-1.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-spark-1.1 (the provider's alias of the same model) ; cache 59% of input |
| kilo | muse-spark-1.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-spark-1.2 (the provider's alias of the same model) ; cache 79% of input |
| kilo | muse-spark-1.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-spark-1.3 (the provider's alias of the same model) ; cache 20% of input |
| kilo | nemotron-3-super | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) ; cache 0% of input |
| kilo | nemotron-3.5-lightning | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) ; cache 54% of input |
| kilo | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 76% of input |
| kilo | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 77% of input |
| kilo | qwen3.8-27b | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as alibaba/qwen3.8-27b (the provider's alias of the same model) ; cache 73% of input |
| kilo | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 76% of input |
| kilo | qwen3.8-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 76% of input |
| kilo | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as stepfun/step-3.7-flash (the provider's alias of the same model) ; cache 81% of input |
58 pairs, 281 of 284 scenario runs passed; 1 pairs served by another connection or as another model are findings, not counted.
Findings, pairs served by a connection other than the one under test or as a model other than the id asked for:
- kilo x claude-fable-5: no prompt cache reads over 100948 input tokens (every call paid full price)
- kilo x claude-fable-5-1: no prompt cache reads over 100755 input tokens (every call paid full price)
- kilo x claude-opus-4.7: no prompt cache reads over 102906 input tokens (every call paid full price)
- kilo x claude-opus-4.8: no prompt cache reads over 101079 input tokens (every call paid full price)
- kilo x claude-sonnet-5: no prompt cache reads over 100846 input tokens (every call paid full price)
- kilo x claude-sonnet-5.5: no prompt cache reads over 100807 input tokens (every call paid full price)
- kilo x deepseek-v4-flash: served as deepseek-v4.1-flash (the CLI reports the model it ran)
Provider: minimax-default
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| minimax | claude-fable-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) |
| minimax | claude-fable-5-1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) |
| minimax | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Anthropic, OpenRouter | no prompt cache reads (finding below) ; served as claude-haiku-4-5-20251001 (the provider's alias of the same model) |
| minimax | claude-opus-4.7 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) ; served as claude-opus-4-7 (the provider's alias of the same model) |
| minimax | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) ; served as claude-opus-4-8 (the provider's alias of the same model) |
| minimax | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) |
| minimax | claude-opus-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | no prompt cache reads (finding below) ; served as anthropic/claude-opus-5.5 (the provider's alias of the same model) |
| minimax | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Anthropic, OpenRouter | no prompt cache reads (finding below) ; served as claude-sonnet-4-6 (the provider's alias of the same model) |
| minimax | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) |
| minimax | claude-sonnet-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | no prompt cache reads (finding below) ; served as claude-sonnet-5-5 (the provider's alias of the same model) |
| minimax | deepseek-v4-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as deepseek-v4.1-flash (finding below) ; cache 61% of input |
| minimax | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 64% of input |
| minimax | deepseek-v4.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 72% of input |
| minimax | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 57% of input |
| minimax | gemini-3.1-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as google/gemini-3.1-flash-lite (the provider's alias of the same model) ; cache 57% of input |
| minimax | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 28% of input |
| minimax | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 57% of input |
| minimax | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | FAIL | pass | My TokenRouter, OpenRouter | artifact: The turn failed: Conversation history could not be safely updated. Please retry. ; retested once; first try: artifact The turn failed: Conversation history could not be safely updated. Please retry. ; cache 43% of input |
| minimax | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 32% of input |
| minimax | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 21% of input |
| minimax | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 29% of input |
| minimax | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 68% of input |
| minimax | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 67% of input |
| minimax | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Custom OpenAI Chat, OpenRouter | served as gpt-5.2-2025-12-11 (the provider's alias of the same model) ; cache 66% of input |
| minimax | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as gpt-5.4-2026-03-05 (the provider's alias of the same model) ; cache 66% of input |
| minimax | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | Custom OpenAI Chat, OpenRouter | served as gpt-5.4-mini-2026-03-17 (the provider's alias of the same model) ; cache 70% of input |
| minimax | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Azure OpenAI E2, OpenRouter | served as gpt-5.5-2026-04-24 (the provider's alias of the same model) ; cache 65% of input |
| minimax | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 71% of input |
| minimax | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 71% of input |
| minimax | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | FAIL | pass | My TokenRouter, OpenRouter | artifact: no file card (files: none); Create a file named hello-minimax.txt containing exactly the word HELLO, then reply DONE. MINIMAX CODE DONE ; retested once; first try: artifact no file card (files: none); Create a file named hello-minimax.txt containing exa ; cache 59% of input |
| minimax | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 71% of input |
| minimax | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 71% of input |
| minimax | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter, My TokenRouter | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 66% of input |
| minimax | grok-4.20 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as x-ai/grok-4.20 (the provider's alias of the same model) ; cache 58% of input |
| minimax | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as x-ai/grok-4.3 (the provider's alias of the same model) ; cache 68% of input |
| minimax | grok-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 68% of input |
| minimax | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 47% of input |
| minimax | grok-build-0.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as x-ai/grok-build-0.1 (the provider's alias of the same model) ; cache 72% of input |
| minimax | hunyuan-4-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as tencent/hy4-preview (the provider's alias of the same model) ; cache 67% of input |
| minimax | kimi-k2.7-code | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 66% of input |
| minimax | kimi-k3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 77% of input |
| minimax | llama-3.3-70b | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | served as meta-llama/llama-3.3-70b-instruct (the provider's alias of the same model) ; retested once; first try: artifact the cards do not match the record: cards [model.txt,model.txt,hello-minimax.txt] ; cache 63% of input |
| minimax | minimax-m3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as minimax/minimax-m3 (the provider's alias of the same model) ; cache 68% of input |
| minimax | minimax-m3.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, Vercel AI Gateway | cache 73% of input |
| minimax | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as mistralai/mistral-medium-3-5 (the provider's alias of the same model) ; cache 68% of input |
| minimax | muse-glimmer-30b | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-glimmer-30b (the provider's alias of the same model) ; cache 50% of input |
| minimax | muse-spark-1.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-spark-1.1 (the provider's alias of the same model) ; cache 50% of input |
| minimax | muse-spark-1.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-spark-1.2 (the provider's alias of the same model) ; cache 51% of input |
| minimax | muse-spark-1.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as meta/muse-spark-1.3 (the provider's alias of the same model) ; cache 0% of input |
| minimax | nemotron-3-super | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) ; cache 0% of input |
| minimax | nemotron-3.5-lightning | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) ; cache 49% of input |
| minimax | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 63% of input |
| minimax | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 68% of input |
| minimax | qwen3.8-27b | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway, OpenRouter | served as alibaba/qwen3.8-27b (the provider's alias of the same model) ; cache 60% of input |
| minimax | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 69% of input |
| minimax | qwen3.8-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | cache 66% of input |
| minimax | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter, OpenRouter | served as stepfun/step-3.7-flash (the provider's alias of the same model) ; cache 65% of input |
57 pairs, 278 of 280 scenario runs passed; 1 pairs served by another connection or as another model are findings, not counted.
Findings, pairs served by a connection other than the one under test or as a model other than the id asked for:
- minimax x claude-fable-5: no prompt cache reads over 81472 input tokens (every call paid full price)
- minimax x claude-fable-5-1: no prompt cache reads over 81293 input tokens (every call paid full price)
- minimax x claude-haiku-4.5: no prompt cache reads over 62761 input tokens (every call paid full price)
- minimax x claude-opus-4.7: no prompt cache reads over 83407 input tokens (every call paid full price)
- minimax x claude-opus-4.8: no prompt cache reads over 81281 input tokens (every call paid full price)
- minimax x claude-opus-5: no prompt cache reads over 81303 input tokens (every call paid full price)
- minimax x claude-opus-5.5: no prompt cache reads over 81338 input tokens (every call paid full price)
- minimax x claude-sonnet-4.6: no prompt cache reads over 62404 input tokens (every call paid full price)
- minimax x claude-sonnet-5: no prompt cache reads over 81655 input tokens (every call paid full price)
- minimax x claude-sonnet-5.5: no prompt cache reads over 81476 input tokens (every call paid full price)
- minimax x deepseek-v4-flash: served as deepseek-v4.1-flash (the CLI reports the model it ran)
Provider: omp-anthropic
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| omp | claude-fable-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| omp | claude-fable-5-1 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
| omp | claude-haiku-4.5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | served as claude-haiku-4-5-20251001 (the provider's alias of the same model) |
| omp | claude-opus-4.7 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | served as claude-opus-4-7 (the provider's alias of the same model) |
| omp | claude-opus-4.8 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | served as claude-opus-4-8 (the provider's alias of the same model) |
| omp | claude-opus-5 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | |
| omp | claude-sonnet-4.6 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | served as claude-sonnet-4-6 (the provider's alias of the same model) |
| omp | claude-sonnet-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic |
8 pairs, 40 of 40 scenario runs passed.
Not run in this column, 3 pairs the provider serves that the harness did not run, with the reason:
- omp x claude-haiku-5.5: not run, not run in this column
- omp x claude-opus-5.5: not run, not run in this column
- omp x claude-sonnet-5.5: not run, not run in this column
Provider: omp-azure-e2
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| omp | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | |
| omp | gpt-5.3-codex | pass | pass | n/a | pass | pass | Azure OpenAI E2 | |
| omp | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | |
| omp | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | |
| omp | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | |
| omp | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | |
| omp | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Azure OpenAI E2 | |
| omp | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 |
8 pairs, 39 of 39 scenario runs passed.
Not run in this column, 4 pairs the provider serves that the harness did not run, with the reason:
- omp x gpt-6-astra: not run, not run in this column
- omp x gpt-6-luna: not run, not run in this column
- omp x gpt-6-sol: not run, not run in this column
- omp x gpt-6.1-sol: not run, not run in this column
Provider: omp-google
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| omp | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| omp | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| omp | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| omp | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| omp | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| omp | gemini-3.6-flash | pass | pass | pass (gemini-3.8-flash) | pass | pass | Google AI Studio | |
| omp | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| omp | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio |
8 pairs, 40 of 40 scenario runs passed.
Not run in this column, 1 pairs the provider serves that the harness did not run, with the reason:
- omp x gemini-4-argon: not run, not run in this column
Provider: omp-openai
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| omp | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| omp | gpt-5.3-codex | pass | pass | n/a | pass | pass | OpenAI | |
| omp | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| omp | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| omp | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| omp | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| omp | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | OpenAI | |
| omp | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI |
8 pairs, 39 of 39 scenario runs passed.
Not run in this column, 4 pairs the provider serves that the harness did not run, with the reason:
- omp x gpt-6-astra: not run, not run in this column
- omp x gpt-6-luna: not run, not run in this column
- omp x gpt-6-sol: not run, not run in this column
- omp x gpt-6.1-sol: not run, not run in this column
Provider: omp-openrouter
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| omp | claude-fable-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-fable-5 (the provider's alias of the same model) |
| omp | claude-fable-5-1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-fable-5.1 (the provider's alias of the same model) |
| omp | claude-haiku-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-haiku-4.5 (the provider's alias of the same model) |
| omp | claude-opus-4.7 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-opus-4.7 (the provider's alias of the same model) |
| omp | claude-opus-4.8 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-opus-4.8 (the provider's alias of the same model) |
| omp | claude-opus-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-opus-5 (the provider's alias of the same model) |
| omp | claude-sonnet-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-sonnet-4.6 (the provider's alias of the same model) |
| omp | claude-sonnet-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-sonnet-5 (the provider's alias of the same model) |
| omp | deepseek-v4-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as deepseek/deepseek-v4-flash (the provider's alias of the same model) |
| omp | deepseek-v4-pro | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as deepseek/deepseek-v4-pro (the provider's alias of the same model) |
| omp | deepseek-v4.1-flash | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as deepseek/deepseek-v4.1-flash (the provider's alias of the same model) |
| omp | gemini-3-flash-preview | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3-flash-preview (the provider's alias of the same model) |
| omp | gemini-3.1-flash-lite | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3.1-flash-lite (the provider's alias of the same model) |
| omp | gemini-3.1-pro-preview | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3.1-pro-preview (the provider's alias of the same model) |
| omp | gemini-3.5-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3.5-flash (the provider's alias of the same model) |
| omp | gemini-3.5-flash-lite | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3.5-flash-lite (the provider's alias of the same model) |
| omp | gemini-3.6-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3.6-flash (the provider's alias of the same model) |
| omp | gemini-3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3.7-flash (the provider's alias of the same model) |
| omp | gemini-3.8-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3.8-flash (the provider's alias of the same model) |
| omp | glm-5.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as z-ai/glm-5.3 (the provider's alias of the same model) |
| omp | glm-5.3-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as z-ai/glm-5.3-flash (the provider's alias of the same model) |
| omp | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as openai/gpt-5.2 (the provider's alias of the same model) |
| omp | gpt-5.3-codex | pass | pass | n/a | pass | pass | OpenRouter | served as openai/gpt-5.3-codex (the provider's alias of the same model) |
| omp | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as openai/gpt-5.4 (the provider's alias of the same model) |
| omp | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as openai/gpt-5.4-mini (the provider's alias of the same model) |
| omp | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as openai/gpt-5.5 (the provider's alias of the same model) |
| omp | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as openai/gpt-5.6-luna (the provider's alias of the same model) |
| omp | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | OpenRouter | served as openai/gpt-5.6-sol (the provider's alias of the same model) |
| omp | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as openai/gpt-5.6-terra (the provider's alias of the same model) |
| omp | grok-4.20 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as x-ai/grok-4.20 (the provider's alias of the same model) |
| omp | grok-4.3 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as x-ai/grok-4.3 (the provider's alias of the same model) |
| omp | grok-4.5 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as x-ai/grok-4.5 (the provider's alias of the same model) |
| omp | grok-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as x-ai/grok-4.6 (the provider's alias of the same model) |
| omp | grok-build-0.1 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as x-ai/grok-build-0.1 (the provider's alias of the same model) |
| omp | hunyuan-4-preview | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as tencent/hy4-preview (the provider's alias of the same model) |
| omp | kimi-k2.7-code | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as moonshotai/kimi-k2.7-code (the provider's alias of the same model) |
| omp | kimi-k3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as moonshotai/kimi-k3 (the provider's alias of the same model) |
| omp | llama-3.3-70b | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta-llama/llama-3.3-70b-instruct (the provider's alias of the same model) |
| omp | llama-4-maverick | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta-llama/llama-4-maverick (the provider's alias of the same model) |
| omp | mistral-medium-3.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as mistralai/mistral-medium-3-5 (the provider's alias of the same model) |
| omp | muse-glimmer-30b | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta/muse-glimmer-30b (the provider's alias of the same model) |
| omp | muse-spark-1.1 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta/muse-spark-1.1 (the provider's alias of the same model) |
| omp | muse-spark-1.2 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta/muse-spark-1.2 (the provider's alias of the same model) |
| omp | muse-spark-1.3 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta/muse-spark-1.3 (the provider's alias of the same model) |
| omp | nemotron-3-super | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) |
| omp | nemotron-3.5-lightning | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) |
| omp | qwen3.7-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as qwen/qwen3.7-max (the provider's alias of the same model) |
| omp | qwen3.7-plus | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as qwen/qwen3.7-plus (the provider's alias of the same model) |
| omp | qwen3.8-27b | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as qwen/qwen3.8-27b (the provider's alias of the same model) |
| omp | qwen3.8-flash | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as qwen/qwen3.8-flash (the provider's alias of the same model) |
| omp | qwen3.8-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as qwen/qwen3.8-max-0902 (the provider's alias of the same model) |
| omp | step-3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as stepfun/step-3.7-flash (the provider's alias of the same model) |
52 pairs, 259 of 259 scenario runs passed.
Not run in this column, 17 pairs the provider serves that the harness did not run, with the reason:
- omp x claude-haiku-5.5: not run, not run in this column
- omp x claude-opus-5.5: not run, not run in this column
- omp x claude-sonnet-5.5: not run, not run in this column
- omp x gemini-4-argon: not run, not run in this column
- omp x gpt-6-astra: not run, not run in this column
- omp x gpt-6-luna: not run, not run in this column
- omp x gpt-6-sol: not run, not run in this column
- omp x gpt-6.1-sol: not run, not run in this column
- omp x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x muse-wonder-1.0: not run, not run in this column
- omp x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
Provider: omp-tokenrouter
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| omp | claude-fable-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as anthropic/claude-fable-5 (the provider's alias of the same model) |
| omp | claude-fable-5-1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as anthropic/claude-fable-5.1 (the provider's alias of the same model) |
| omp | claude-haiku-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as anthropic/claude-haiku-4.5 (the provider's alias of the same model) |
| omp | claude-opus-4.7 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as anthropic/claude-opus-4.7 (the provider's alias of the same model) |
| omp | claude-opus-4.8 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as anthropic/claude-opus-4.8 (the provider's alias of the same model) |
| omp | claude-opus-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as anthropic/claude-opus-5 (the provider's alias of the same model) |
| omp | claude-sonnet-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as anthropic/claude-sonnet-4.6 (the provider's alias of the same model) |
| omp | claude-sonnet-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as anthropic/claude-sonnet-5 (the provider's alias of the same model) |
| omp | deepseek-v4-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as deepseek/deepseek-v4-flash (the provider's alias of the same model) |
| omp | deepseek-v4-pro | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as deepseek/deepseek-v4-pro (the provider's alias of the same model) |
| omp | deepseek-v4.1-flash | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as deepseek/deepseek-v4.1-flash (the provider's alias of the same model) |
| omp | gemini-3-flash-preview | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as google/gemini-3-flash-preview (the provider's alias of the same model) |
| omp | gemini-3.1-pro-preview | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as google/gemini-3.1-pro-preview (the provider's alias of the same model) |
| omp | gemini-3.5-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as google/gemini-3.5-flash (the provider's alias of the same model) |
| omp | gemini-3.5-flash-lite | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as google/gemini-3.5-flash-lite (the provider's alias of the same model) |
| omp | gemini-3.6-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as google/gemini-3.6-flash (the provider's alias of the same model) |
| omp | gemini-3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as google/gemini-3.7-flash (the provider's alias of the same model) |
| omp | gemini-3.8-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as google/gemini-3.8-flash (the provider's alias of the same model) |
| omp | glm-5.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as z-ai/glm-5.3 (the provider's alias of the same model) |
| omp | glm-5.3-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as z-ai/glm-5.3-flash (the provider's alias of the same model) |
| omp | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as openai/gpt-5.2 (the provider's alias of the same model) |
| omp | gpt-5.3-codex | pass | pass | n/a | pass | pass | My TokenRouter | served as openai/gpt-5.3-codex (the provider's alias of the same model) |
| omp | gpt-5.4 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as openai/gpt-5.4 (the provider's alias of the same model) |
| omp | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as openai/gpt-5.4-mini (the provider's alias of the same model) |
| omp | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as openai/gpt-5.5 (the provider's alias of the same model) |
| omp | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as openai/gpt-5.6-luna (the provider's alias of the same model) |
| omp | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | My TokenRouter | served as openai/gpt-5.6-sol (the provider's alias of the same model) |
| omp | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as openai/gpt-5.6-terra (the provider's alias of the same model) |
| omp | grok-4.20 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as x-ai/grok-4.20-beta (the provider's alias of the same model) |
| omp | grok-4.3 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as x-ai/grok-4.3 (the provider's alias of the same model) |
| omp | grok-4.5 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as x-ai/grok-4.5 (the provider's alias of the same model) |
| omp | grok-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as x-ai/grok-4.6 (the provider's alias of the same model) |
| omp | grok-build-0.1 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as x-ai/grok-build-0.1 (the provider's alias of the same model) |
| omp | hunyuan-4-preview | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as tencent/hy4-preview (the provider's alias of the same model) |
| omp | kimi-k2.7-code | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as moonshotai/kimi-k2.7-code (the provider's alias of the same model) |
| omp | kimi-k3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as moonshotai/kimi-k3 (the provider's alias of the same model) |
| omp | mistral-medium-3.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as mistralai/mistral-medium-3-5 (the provider's alias of the same model) |
| omp | nemotron-3-super | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) |
| omp | nemotron-3.5-lightning | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) |
| omp | qwen3.7-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as qwen/qwen3.7-max (the provider's alias of the same model) |
| omp | qwen3.7-plus | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as qwen/qwen3.7-plus (the provider's alias of the same model) |
| omp | qwen3.8-flash | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as qwen/qwen3.8-flash (the provider's alias of the same model) |
| omp | qwen3.8-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as qwen/qwen3.8-max (the provider's alias of the same model) |
| omp | step-3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as stepfun/step-3.7-flash (the provider's alias of the same model) |
44 pairs, 219 of 219 scenario runs passed.
Not run in this column, 7 pairs the provider serves that the harness did not run, with the reason:
- omp x claude-opus-5.5: not run, not run in this column
- omp x claude-sonnet-5.5: not run, not run in this column
- omp x gpt-6-astra: not run, not run in this column
- omp x gpt-6-luna: not run, not run in this column
- omp x gpt-6-sol: not run, not run in this column
- omp x gpt-6.1-sol: not run, not run in this column
- omp x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
Provider: omp-vercel
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| omp | claude-fable-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-fable-5 (the provider's alias of the same model) |
| omp | claude-fable-5-1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-fable-5.1 (the provider's alias of the same model) |
| omp | claude-haiku-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-haiku-4.5 (the provider's alias of the same model) |
| omp | claude-opus-4.7 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-opus-4.7 (the provider's alias of the same model) |
| omp | claude-opus-4.8 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-opus-4.8 (the provider's alias of the same model) |
| omp | claude-opus-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-opus-5 (the provider's alias of the same model) |
| omp | claude-sonnet-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-sonnet-4.6 (the provider's alias of the same model) |
| omp | claude-sonnet-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-sonnet-5 (the provider's alias of the same model) |
| omp | deepseek-v4-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as deepseek/deepseek-v4-flash (the provider's alias of the same model) |
| omp | deepseek-v4-pro | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as deepseek/deepseek-v4-pro (the provider's alias of the same model) |
| omp | deepseek-v4.1-flash | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as deepseek/deepseek-v4.1-flash (the provider's alias of the same model) |
| omp | gemini-3-flash-preview | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3-flash (the provider's alias of the same model) |
| omp | gemini-3.1-flash-lite | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3.1-flash-lite (the provider's alias of the same model) |
| omp | gemini-3.1-pro-preview | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3.1-pro-preview (the provider's alias of the same model) |
| omp | gemini-3.5-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3.5-flash (the provider's alias of the same model) |
| omp | gemini-3.5-flash-lite | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3.5-flash-lite (the provider's alias of the same model) |
| omp | gemini-3.6-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3.6-flash (the provider's alias of the same model) |
| omp | gemini-3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3.7-flash (the provider's alias of the same model) |
| omp | gemini-3.8-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3.8-flash (the provider's alias of the same model) |
| omp | glm-5.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as zai/glm-5.3 (the provider's alias of the same model) |
| omp | glm-5.3-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as zai/glm-5.3-flash (the provider's alias of the same model) |
| omp | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.2 (the provider's alias of the same model) |
| omp | gpt-5.3-codex | pass | pass | n/a | pass | pass | Vercel AI Gateway | served as openai/gpt-5.3-codex (the provider's alias of the same model) |
| omp | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.4 (the provider's alias of the same model) |
| omp | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.4-mini (the provider's alias of the same model) |
| omp | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.5 (the provider's alias of the same model) |
| omp | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.6-luna (the provider's alias of the same model) |
| omp | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.6-sol (the provider's alias of the same model) |
| omp | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.6-terra (the provider's alias of the same model) |
| omp | grok-4.1-fast | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | served as spacexai/grok-4.1-fast-reasoning (finding below) ; first: The turn failed: Stream error occurred |
| omp | grok-4.20 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.20-reasoning (the provider's alias of the same model) |
| omp | grok-4.3 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.3 (the provider's alias of the same model) |
| omp | grok-4.5 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.5 (the provider's alias of the same model) |
| omp | grok-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.6 (the provider's alias of the same model) |
| omp | grok-build-0.1 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-build-0.1 (the provider's alias of the same model) |
| omp | hunyuan-4-preview | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as tencent/hy4-preview (the provider's alias of the same model) |
| omp | kimi-k2.7-code | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as moonshotai/kimi-k2.7-code (the provider's alias of the same model) |
| omp | kimi-k3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as moonshotai/kimi-k3 (the provider's alias of the same model) |
| omp | llama-3.3-70b | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | served as meta/llama-3.3-70b (the provider's alias of the same model) ; first: The turn failed: 400 undefined: The maximum tokens you requested exceeds the model limit of 8192. Try again with a maximum tokens value that |
| omp | llama-4-maverick | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | served as meta/llama-4-maverick (the provider's alias of the same model) ; first: The turn failed: 405 Tool calling is not supported for model: meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 |
| Tool calling is not supporte | ||||||||
| omp | llama-4-scout | pass | pass | pass (gpt-6-astra) | FAIL | FAIL | Vercel AI Gateway | served as meta/llama-4-scout (the provider's alias of the same model) ; artifact: no file card (files: none); Create a file named hello-omp.txt containing exactly the word HELLO, then reply DONE. OH MY PI WRITE { "path": " ; recycle: answered without M1-llama-4-scout: What exact word did I ask you to reply with in my very first message of this task? Reply with just that w |
| omp | mistral-medium-3.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as mistral/mistral-medium-3.5 (the provider's alias of the same model) |
| omp | muse-glimmer-30b | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as meta/muse-glimmer-30b (the provider's alias of the same model) |
| omp | muse-spark-1.1 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as meta/muse-spark-1.1 (the provider's alias of the same model) |
| omp | muse-spark-1.2 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as meta/muse-spark-1.2 (the provider's alias of the same model) |
| omp | muse-spark-1.3 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as meta/muse-spark-1.3 (the provider's alias of the same model) |
| omp | nemotron-3-super | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) |
| omp | nemotron-3.5-lightning | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) |
| omp | qwen3.7-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.7-max (the provider's alias of the same model) |
| omp | qwen3.7-plus | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.7-plus (the provider's alias of the same model) |
| omp | qwen3.8-27b | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.8-27b (the provider's alias of the same model) |
| omp | qwen3.8-flash | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.8-flash (the provider's alias of the same model) |
| omp | qwen3.8-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.8-max (the provider's alias of the same model) |
| omp | step-3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as stepfun/step-3.7-flash (the provider's alias of the same model) |
54 pairs, 252 of 256 scenario runs passed; 1 pairs served by another connection or as another model are findings, not counted.
Findings, pairs served by a connection other than the one under test or as a model other than the id asked for:
- omp x grok-4.1-fast: served as spacexai/grok-4.1-fast-reasoning (the CLI reports the model it ran)
Not run in this column, 15 pairs the provider serves that the harness did not run, with the reason:
- omp x claude-haiku-5.5: not run, not run in this column
- omp x claude-opus-5.5: not run, not run in this column
- omp x claude-sonnet-5.5: not run, not run in this column
- omp x gemini-4-argon: not run, not run in this column
- omp x gpt-6-astra: not run, not run in this column
- omp x gpt-6-luna: not run, not run in this column
- omp x gpt-6-sol: not run, not run in this column
- omp x gpt-6.1-sol: not run, not run in this column
- omp x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x muse-wonder-1.0: not run, not run in this column
- omp x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
Provider: openai
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| aider | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| aider | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | OpenAI | |
| aider | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 45% of input |
| cline | gpt-5.2 | pass | pass | pass (gpt-5.4) | pass | pass | OpenAI | |
| cline | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| cline | gpt-5.4-mini | pass | pass | pass (gpt-5.4) | pass | pass | OpenAI | |
| cline | gpt-5.5 | pass | pass | pass (gpt-5.4) | pass | pass | OpenAI | |
| cline | gpt-5.6-luna | pass | pass | pass (gpt-5.4) | pass | pass | OpenAI | |
| cline | gpt-5.6-sol | pass | pass | pass (gpt-5.4) | pass | pass | OpenAI | |
| cline | gpt-5.6-terra | pass | pass | pass (gpt-5.4) | pass | pass | OpenAI | |
| cline | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| cline | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | OpenAI | |
| cline | gpt-6.1-sol | FAIL | n/a | n/a | n/a | n/a | OpenAI | 1 turn(s) unlabelled (finding below) ; first: The turn failed: Unsupported value: 'reasoning_effort' does not support 'none' with this model. Supported values are: 'low', 'medium', 'high |
| codex | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| codex | gpt-5.3-codex | pass | pass | pass (gpt-5.6-luna) | FAIL | FAIL | OpenAI | artifact: Codex cannot run gpt-5.3-codex in a task that has already used gpt-5.6-luna: its tools are not available there. Start a new task for gpt-5.3 ; recycle: Codex cannot run gpt-5.3-codex in a task that has already used gpt-5.6-luna: its tools are not available there. Start a new task for gpt-5.3 ; retested once; first try: artifact Codex cannot run gpt-5.3-codex in a task that has already used gpt-5.6-sol: its ; recycle Codex cannot run gpt-5.3-codex in a task that has already used gpt-5.6-sol: its |
| codex | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| codex | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| codex | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| codex | gpt-5.6-luna | pass | pass | FAIL (gpt-5.3-codex) | pass | FAIL | OpenAI | switch: Codex cannot run gpt-5.3-codex in a task that has already used gpt-5.6-luna: its tools are not available there. Start a new task for gpt-5.3 ; recycle: answered without M1-gpt-5.6-luna: What exact word did I ask you to reply with in my very first message of this task? Reply with just that wo ; retested once; first try: recycle answered without M1-gpt-5.6-luna: What exact word did I ask you to reply with in |
| codex | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | OpenAI | |
| codex | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| codex | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| codex | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | FAIL | OpenAI | recycle: answered without M1-gpt-6-luna: What exact word did I ask you to reply with in my very first message of this task? Reply with just that word ; recycle: answered without M1-gpt-6-luna: What exact word did I ask you to reply with in my very first message of this ta |
| codex | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| codex | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | 4 turn(s) unlabelled (finding below) ; cache 58% of input |
| dcode | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI | cache 59% of input |
| dcode | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI | cache 68% of input |
| dcode | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI | cache 73% of input |
| dcode | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI | cache 68% of input |
| dcode | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI | cache 71% of input |
| dcode | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI | cache 66% of input |
| dcode | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI | cache 71% of input |
| dcode | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI | cache 66% of input |
| dcode | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenAI | cache 66% of input |
| dcode | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 66% of input |
| docker-agent | gpt-5.2 | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 90% of input |
| docker-agent | gpt-5.4 | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 92% of input |
| docker-agent | gpt-5.4-mini | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 88% of input |
| docker-agent | gpt-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 51% of input |
| docker-agent | gpt-5.6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 100% of input |
| docker-agent | gpt-5.6-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 100% of input |
| docker-agent | gpt-5.6-terra | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 100% of input |
| docker-agent | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 100% of input |
| docker-agent | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | OpenAI | cache 100% of input |
| dsh | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| dsh | gpt-5.3-codex | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| dsh | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| dsh | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| dsh | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| dsh | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| dsh | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | OpenAI | |
| dsh | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| dsh | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| dsh | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| dsh | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| dsh | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | 4 turn(s) unlabelled (finding below) ; cache 100% of input |
| fast-agent | gpt-5.2 | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| fast-agent | gpt-5.4 | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| fast-agent | gpt-5.4-mini | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| fast-agent | gpt-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| fast-agent | gpt-5.6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| fast-agent | gpt-5.6-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| fast-agent | gpt-5.6-terra | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| fast-agent | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| fast-agent | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | OpenAI | |
| goose | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 52% of input |
| hermes | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| hermes | gpt-5.3-codex | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| hermes | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| hermes | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| hermes | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| hermes | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| hermes | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | OpenAI | |
| hermes | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| hermes | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| hermes | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| hermes | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| hermes | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 0% of input |
| kimi | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| kimi | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | OpenAI | |
| kimi | gpt-6.1-sol | FAIL | n/a | n/a | n/a | n/a | OpenAI | 1 turn(s) unlabelled (finding below) ; first: The turn failed: error: failed to run prompt: provider.api_error: 400 Unsupported value: 'reasoning_effort' does not support 'none' with thi |
| omp | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| omp | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| omp | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| omp | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | 1 turn(s) unlabelled (finding below) ; cache 100% of input |
| opencode | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| opencode | gpt-5.3-codex | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| opencode | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| opencode | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| opencode | gpt-5.5 | pass | pass | n/a | pass | pass | OpenAI | retested once; first try: artifact no file card (files: none); Create a file named hello-opencode.txt containing ex |
| opencode | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| opencode | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | OpenAI | |
| opencode | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| opencode | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| opencode | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| opencode | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| opencode | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 100% of input |
| openhands | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| openhands | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | OpenAI | |
| openhands | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 0% of input |
| pi | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| pi | gpt-5.3-codex | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| pi | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| pi | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| pi | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| pi | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| pi | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | OpenAI | |
| pi | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| pi | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| pi | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| pi | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | OpenAI | |
| pi | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 100% of input |
| qwen | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| qwen | gpt-5.3-codex | pass | pass | n/a | FAIL | FAIL | OpenAI | artifact: no file card (files: none); word HELLO, then reply DONE. QWEN CODE [API Error: 404 This model is not supported in the v1/chat/completions e ; recycle: answered without M1-gpt-5.3-codex: ith in my very first message of this task? Reply with just that word. QWEN CODE [API Error: 404 This mode ; retested once; first try: artifact no file card (files: none); word HELLO, then reply DONE. QWEN CODE [API Error: ; recycle answered without M1-gpt-5.3-codex: ith in my very first message of this task? Re |
| qwen | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| qwen | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| qwen | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| qwen | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| qwen | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | OpenAI | |
| qwen | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
| qwen | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | |
| qwen | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | OpenAI | |
| qwen | gpt-6.1-sol | FAIL | n/a | n/a | n/a | n/a | OpenAI | 1 turn(s) unlabelled (finding below) ; first: The turn failed: API Error: 400 Unsupported value: 'reasoning_effort' does not support 'none' with this model. Supported values are: 'low', |
| vibe | gpt-5.2 | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 35% of input |
| vibe | gpt-5.4 | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 31% of input |
| vibe | gpt-5.4-mini | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 32% of input |
| vibe | gpt-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 31% of input |
| vibe | gpt-5.6-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 17% of input |
| vibe | gpt-5.6-terra | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 17% of input |
| vibe | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenAI | cache 17% of input |
| vibe | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | OpenAI | cache 29% of input |
131 pairs, 616 of 623 scenario runs passed; 6 pairs served by another connection or as another model are findings, not counted.
Findings, pairs served by a connection other than the one under test or as a model other than the id asked for:
- cline x gpt-6.1-sol: 1 turn(s) report no served model, so rule 2 could not judge them
- codex x gpt-6.1-sol: 4 turn(s) report no served model, so rule 2 could not judge them
- dsh x gpt-6.1-sol: 4 turn(s) report no served model, so rule 2 could not judge them
- kimi x gpt-6.1-sol: 1 turn(s) report no served model, so rule 2 could not judge them
- omp x gpt-6.1-sol: 1 turn(s) report no served model, so rule 2 could not judge them
- qwen x gpt-6.1-sol: 1 turn(s) report no served model, so rule 2 could not judge them
Not run in this column, 61 pairs the provider serves that the harness did not run, with the reason:
- aider x gpt-5.2: not run, not run in this column
- aider x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- aider x gpt-5.4: not run, not run in this column
- aider x gpt-5.4-mini: not run, not run in this column
- aider x gpt-5.5: not run, not run in this column
- aider x gpt-5.6-luna: not run, not run in this column
- aider x gpt-5.6-sol: not run, not run in this column
- aider x gpt-5.6-terra: not run, not run in this column
- aider x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- dcode x gpt-5.3-codex: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x gpt-6-astra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- docker-agent x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- docker-agent x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- fast-agent x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- fast-agent x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- fast-agent x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- goose x gpt-5.2: not run, not run in this column
- goose x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-5.4: not run, not run in this column
- goose x gpt-5.4-mini: not run, not run in this column
- goose x gpt-5.5: not run, not run in this column
- goose x gpt-5.6-luna: not run, not run in this column
- goose x gpt-5.6-sol: not run, not run in this column
- goose x gpt-5.6-terra: not run, not run in this column
- goose x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-luna: not run, not run in this column
- goose x gpt-6-sol: not run, not run in this column
- kimi x gpt-5.2: not run, not run in this column
- kimi x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- kimi x gpt-5.4: not run, not run in this column
- kimi x gpt-5.4-mini: not run, not run in this column
- kimi x gpt-5.5: not run, not run in this column
- kimi x gpt-5.6-luna: not run, not run in this column
- kimi x gpt-5.6-sol: not run, not run in this column
- kimi x gpt-5.6-terra: not run, not run in this column
- kimi x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- omp x gpt-5.2: not run, not run in this column
- omp x gpt-5.3-codex: not run, not run in this column
- omp x gpt-5.4: not run, not run in this column
- omp x gpt-5.4-mini: not run, not run in this column
- omp x gpt-5.5: not run, not run in this column
- omp x gpt-5.6-luna: not run, not run in this column
- omp x gpt-5.6-sol: not run, not run in this column
- omp x gpt-5.6-terra: not run, not run in this column
- openhands x gpt-5.2: not run, not run in this column
- openhands x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- openhands x gpt-5.4: not run, not run in this column
- openhands x gpt-5.4-mini: not run, not run in this column
- openhands x gpt-5.5: not run, not run in this column
- openhands x gpt-5.6-luna: not run, not run in this column
- openhands x gpt-5.6-sol: not run, not run in this column
- openhands x gpt-5.6-terra: not run, not run in this column
- openhands x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- vibe x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- vibe x gpt-5.6-luna: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- vibe x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
Provider: openrouter
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| agentzero | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) |
| aider | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) |
| aider | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6-luna (the provider's alias of the same model) |
| aider | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | FAIL | OpenRouter | recycle: answered without M1-gpt-6-sol: What exact word did I ask you to reply with in my very first message of this task? Reply with just that word. ; served as openai/gpt-6-sol (the provider's alias of the same model) ; recycle: answered without M1-gpt-6-sol: What exact |
| aider | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 51% of input |
| cheetahclaws | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) |
| cheetahclaws | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6-luna (the provider's alias of the same model) |
| cheetahclaws | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | OpenRouter | served as openai/gpt-6-sol (the provider's alias of the same model) |
| cheetahclaws | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 17% of input |
| cline | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | no prompt cache reads (finding below) ; 4 turn(s) unlabelled (finding below) |
| cline | deepseek-v4.1-flash | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | |
| cline | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | |
| cline | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | OpenRouter | |
| cline | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | 4 turn(s) unlabelled (finding below) ; cache 65% of input |
| cline | grok-4.20 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| cline | grok-4.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| cline | grok-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| cline | grok-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| cline | grok-build-0.1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| cline | hunyuan-4-preview | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | |
| cline | llama-3.3-70b | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| cline | llama-4-maverick | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| cline | muse-glimmer-30b | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| cline | muse-spark-1.1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| cline | muse-spark-1.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| cline | muse-spark-1.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| cline | nemotron-3-super | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | |
| cline | nemotron-3.5-lightning | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | |
| cline | qwen3.7-plus | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | |
| cline | qwen3.8-27b | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | |
| cline | qwen3.8-flash | pass | pass | pass (gpt-5.4) | pass | pass | OpenRouter | |
| codex | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | |
| codex | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| codex | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| codex | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | 4 turn(s) unlabelled (finding below) ; cache 50% of input |
| dcode | claude-fable-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | no prompt cache reads (finding below) |
| dcode | claude-fable-5-1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | no prompt cache reads (finding below) |
| dcode | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | no prompt cache reads (finding below) |
| dcode | claude-haiku-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | no prompt cache reads (finding below) |
| dcode | claude-opus-4.7 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | no prompt cache reads (finding below) |
| dcode | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | no prompt cache reads (finding below) |
| dcode | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | no prompt cache reads (finding below) |
| dcode | claude-opus-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | no prompt cache reads (finding below) |
| dcode | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | no prompt cache reads (finding below) |
| dcode | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | no prompt cache reads (finding below) |
| dcode | claude-sonnet-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | no prompt cache reads (finding below) |
| dcode | deepseek-v4-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 67% of input |
| dcode | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 67% of input |
| dcode | deepseek-v4.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 76% of input |
| dcode | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 50% of input |
| dcode | gemini-3.1-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 62% of input |
| dcode | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 49% of input |
| dcode | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 61% of input |
| dcode | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 31% of input |
| dcode | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 49% of input |
| dcode | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 49% of input |
| dcode | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 24% of input |
| dcode | gemini-4-argon | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 41% of input |
| dcode | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 67% of input |
| dcode | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 67% of input |
| dcode | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 60% of input |
| dcode | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 61% of input |
| dcode | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 66% of input |
| dcode | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 60% of input |
| dcode | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 66% of input |
| dcode | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 66% of input |
| dcode | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 66% of input |
| dcode | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 66% of input |
| dcode | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 66% of input |
| dcode | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | cache 66% of input |
| dcode | grok-4.20 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 73% of input |
| dcode | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 51% of input |
| dcode | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 72% of input |
| dcode | grok-build-0.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 67% of input |
| dcode | kimi-k3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 67% of input |
| dcode | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 71% of input |
| dcode | muse-spark-1.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 1% of input |
| dcode | muse-spark-1.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 17% of input |
| dcode | muse-wonder-1.0 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 17% of input |
| dcode | nemotron-3-super | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 0% of input |
| dcode | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 63% of input |
| dcode | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 66% of input |
| dcode | qwen3.8-27b | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 68% of input |
| dcode | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 67% of input |
| dcode | qwen3.8-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 67% of input |
| dcode | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 66% of input |
| docker-agent | claude-fable-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | claude-fable-5-1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 26% of input |
| docker-agent | claude-haiku-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | claude-opus-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | claude-sonnet-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | deepseek-v4-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 19% of input |
| docker-agent | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | deepseek-v4.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 71% of input |
| docker-agent | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 0% of input |
| docker-agent | gemini-3.1-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 0% of input |
| docker-agent | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 0% of input |
| docker-agent | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 0% of input |
| docker-agent | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 0% of input |
| docker-agent | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 0% of input |
| docker-agent | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 0% of input |
| docker-agent | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 0% of input |
| docker-agent | gemini-4-argon | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 27% of input |
| docker-agent | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 80% of input |
| docker-agent | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 80% of input |
| docker-agent | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 86% of input |
| docker-agent | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 87% of input |
| docker-agent | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 92% of input |
| docker-agent | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 73% of input |
| docker-agent | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | cache 100% of input |
| docker-agent | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 53% of input |
| docker-agent | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 82% of input |
| docker-agent | hunyuan-4-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 80% of input |
| docker-agent | kimi-k2.7-code | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 76% of input |
| docker-agent | kimi-k3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 81% of input |
| docker-agent | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 80% of input |
| docker-agent | muse-glimmer-30b | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 90% of input |
| docker-agent | muse-spark-1.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 68% of input |
| docker-agent | muse-spark-1.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 52% of input |
| docker-agent | muse-spark-1.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 33% of input |
| docker-agent | muse-wonder-1.0 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 24% of input |
| docker-agent | nemotron-3-super | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 0% of input |
| docker-agent | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 80% of input |
| docker-agent | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 80% of input |
| docker-agent | qwen3.8-27b | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 80% of input |
| docker-agent | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 80% of input |
| docker-agent | qwen3.8-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 78% of input |
| docker-agent | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 79% of input |
| dsh | claude-fable-5 | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | claude-haiku-4.5 | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | 4 turn(s) unlabelled (finding below) ; cache 100% of input |
| dsh | claude-opus-4.7 | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | claude-opus-4.8 | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | claude-opus-5 | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | claude-sonnet-4.6 | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | claude-sonnet-5 | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | deepseek-v4-flash | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | deepseek-v4-pro | pass | pass | pass (deepseek-v4-flash) | pass | pass | OpenRouter | |
| dsh | gemini-3.6-flash | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | glm-5.3 | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | glm-5.3-flash | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | gpt-5.2 | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | gpt-5.3-codex | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | gpt-5.4 | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | gpt-5.4-mini | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | gpt-5.5 | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | gpt-5.6-luna | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | gpt-5.6-sol | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | gpt-5.6-terra | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | |
| dsh | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| dsh | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| dsh | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | 4 turn(s) unlabelled (finding below) ; cache 100% of input |
| dsh | kimi-k2.7-code | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | kimi-k3 | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | mistral-medium-3.5 | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | qwen3.7-max | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | qwen3.8-max | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| dsh | step-3.7-flash | pass | pass | pass (deepseek-v4-pro) | pass | pass | OpenRouter | |
| fast-agent | claude-fable-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 54% of input |
| fast-agent | claude-fable-5-1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 69% of input |
| fast-agent | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | no prompt cache reads (finding below) |
| fast-agent | claude-haiku-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 70% of input |
| fast-agent | claude-opus-4.7 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 70% of input |
| fast-agent | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 70% of input |
| fast-agent | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 54% of input |
| fast-agent | claude-opus-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 70% of input |
| fast-agent | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 68% of input |
| fast-agent | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 54% of input |
| fast-agent | claude-sonnet-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 70% of input |
| fast-agent | deepseek-v4-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | deepseek-v4.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gemini-3.1-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | |
| fast-agent | grok-4.20 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | grok-build-0.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | kimi-k3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | muse-glimmer-30b | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | muse-spark-1.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | muse-spark-1.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | muse-spark-1.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | muse-wonder-1.0 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| fast-agent | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | |
| goose | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) |
| goose | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 52% of input |
| grok | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) |
| hermes | claude-fable-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | claude-haiku-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) |
| hermes | claude-opus-4.7 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | claude-opus-4.8 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | claude-opus-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | claude-sonnet-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | claude-sonnet-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | deepseek-v4-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | deepseek-v4-pro | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | deepseek-v4.1-flash | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| hermes | gemini-3.6-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | glm-5.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | glm-5.3-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | gpt-5.3-codex | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | OpenRouter | |
| hermes | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | |
| hermes | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| hermes | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| hermes | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | 4 turn(s) unlabelled (finding below) ; cache 0% of input |
| hermes | grok-4.20 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| hermes | grok-4.3 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| hermes | grok-4.5 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| hermes | grok-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| hermes | grok-build-0.1 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| hermes | hunyuan-3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | hunyuan-4-preview | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | |
| hermes | kimi-k2.7-code | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | kimi-k3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | ling-3.0-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | |
| hermes | llama-3.3-70b | pass | pass | FAIL (gpt-6-astra) | pass | pass | OpenRouter | switch: The turn failed: ↻ Resumed session 20260913_060549_70cbeb (2 user messages, 4 total messages) |
session_id: 20260913_060549_70cbeb | | hermes | llama-4-maverick | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | | | hermes | minimax-m3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | hermes | mistral-medium-3.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | hermes | muse-glimmer-30b | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | | | hermes | muse-spark-1.1 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | | | hermes | muse-spark-1.2 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | | | hermes | muse-spark-1.3 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | | | hermes | nemotron-3-super | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | | | hermes | nemotron-3-ultra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | hermes | nemotron-3.5-lightning | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | | | hermes | qwen3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | hermes | qwen3.7-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | hermes | qwen3.7-plus | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | | | hermes | qwen3.8-27b | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | | | hermes | qwen3.8-flash | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | | | hermes | qwen3.8-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | hermes | step-3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | kilo | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) ; cache 69% of input | | kimi | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) | | kimi | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6-luna (the provider's alias of the same model) | | kimi | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | OpenRouter | served as openai/gpt-6-sol (the provider's alias of the same model) | | kimi | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 50% of input | | minimax | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) | | minimax | minimax-m3.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter, Vercel AI Gateway | cache 60% of input | | omp | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) | | omp | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6-astra (the provider's alias of the same model) | | omp | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as openai/gpt-6-luna (the provider's alias of the same model) | | omp | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as openai/gpt-6-sol (the provider's alias of the same model) | | omp | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | 1 turn(s) unlabelled (finding below) ; served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 100% of input | | opencode | claude-fable-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | claude-haiku-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) ; cache 68% of input | | opencode | claude-opus-4.7 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | claude-opus-4.8 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | claude-opus-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | claude-sonnet-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | claude-sonnet-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | deepseek-v4-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | deepseek-v4-pro | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | deepseek-v4.1-flash | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as deepseek/deepseek-v4.1-flash (the provider's alias of the same model) | | opencode | gemini-3.6-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | glm-5.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | glm-5.3-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | gpt-5.3-codex | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | OpenRouter | | | opencode | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6-astra (the provider's alias of the same model) | | opencode | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as openai/gpt-6-luna (the provider's alias of the same model) | | opencode | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as openai/gpt-6-sol (the provider's alias of the same model) | | opencode | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 100% of input | | opencode | grok-4.20 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as x-ai/grok-4.20 (the provider's alias of the same model) | | opencode | grok-4.3 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as x-ai/grok-4.3 (the provider's alias of the same model) | | opencode | grok-4.5 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as x-ai/grok-4.5 (the provider's alias of the same model) | | opencode | grok-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as x-ai/grok-4.6 (the provider's alias of the same model) | | opencode | grok-build-0.1 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as x-ai/grok-build-0.1 (the provider's alias of the same model) | | opencode | hunyuan-4-preview | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as tencent/hy4-preview (the provider's alias of the same model) | | opencode | kimi-k2.7-code | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | kimi-k3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | llama-3.3-70b | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta-llama/llama-3.3-70b-instruct (the provider's alias of the same model) | | opencode | llama-4-maverick | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta-llama/llama-4-maverick (the provider's alias of the same model) | | opencode | mistral-medium-3.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | muse-glimmer-30b | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta/muse-glimmer-30b (the provider's alias of the same model) | | opencode | muse-spark-1.1 | pass | pass | pass (gpt-6-astra) | pass | FAIL | OpenRouter | served as meta/muse-spark-1.1 (the provider's alias of the same model) ; recycle: answered without M1-muse-spark-1.1: What exact word did I ask you to reply with in my very first message of this task? Reply with just that | | opencode | muse-spark-1.2 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta/muse-spark-1.2 (the provider's alias of the same model) | | opencode | muse-spark-1.3 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta/muse-spark-1.3 (the provider's alias of the same model) | | opencode | nemotron-3-super | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) | | opencode | nemotron-3.5-lightning | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) | | opencode | qwen3.7-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | qwen3.7-plus | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as qwen/qwen3.7-plus (the provider's alias of the same model) | | opencode | qwen3.8-27b | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as qwen/qwen3.8-27b (the provider's alias of the same model) | | opencode | qwen3.8-flash | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as qwen/qwen3.8-flash (the provider's alias of the same model) | | opencode | qwen3.8-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | opencode | step-3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | openhands | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) ; cache 69% of input | | openhands | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6-luna (the provider's alias of the same model) | | openhands | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | OpenRouter | served as openai/gpt-6-sol (the provider's alias of the same model) | | openhands | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 66% of input | | pi | claude-fable-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | claude-haiku-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) | | pi | claude-opus-4.7 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | claude-opus-4.8 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | claude-opus-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | pi | claude-sonnet-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | claude-sonnet-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | deepseek-v4-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | deepseek-v4-pro | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | deepseek-v4.1-flash | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as deepseek/deepseek-v4.1-flash (the provider's alias of the same model) | | pi | gemini-3.6-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | glm-5.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | glm-5.3-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | pi | gpt-5.3-codex | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | pi | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | pi | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | pi | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | pi | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | pi | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | OpenRouter | | | pi | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | | | pi | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6-astra (the provider's alias of the same model) | | pi | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as openai/gpt-6-luna (the provider's alias of the same model) | | pi | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as openai/gpt-6-sol (the provider's alias of the same model) | | pi | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 100% of input | | pi | grok-4.20 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as x-ai/grok-4.20 (the provider's alias of the same model) | | pi | grok-4.3 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as x-ai/grok-4.3 (the provider's alias of the same model) | | pi | grok-4.5 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as x-ai/grok-4.5 (the provider's alias of the same model) | | pi | grok-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as x-ai/grok-4.6 (the provider's alias of the same model) | | pi | grok-build-0.1 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as x-ai/grok-build-0.1 (the provider's alias of the same model) | | pi | hunyuan-4-preview | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as tencent/hy4-preview (the provider's alias of the same model) | | pi | kimi-k2.7-code | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | kimi-k3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | llama-3.3-70b | pass | pass | pass (gpt-6-astra) | FAIL | pass | OpenRouter | served as meta-llama/llama-3.3-70b-instruct (the provider's alias of the same model) ; artifact: no file card (files: output.txt,output.txt); Create a file named hello-pi.txt containing exactly the word HELLO, then reply DONE. PI write(" | | pi | llama-4-maverick | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta-llama/llama-4-maverick (the provider's alias of the same model) | | pi | mistral-medium-3.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | muse-glimmer-30b | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta/muse-glimmer-30b (the provider's alias of the same model) | | pi | muse-spark-1.1 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta/muse-spark-1.1 (the provider's alias of the same model) | | pi | muse-spark-1.2 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta/muse-spark-1.2 (the provider's alias of the same model) | | pi | muse-spark-1.3 | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as meta/muse-spark-1.3 (the provider's alias of the same model) | | pi | nemotron-3-super | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) | | pi | nemotron-3.5-lightning | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) | | pi | qwen3.7-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | qwen3.7-plus | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as qwen/qwen3.7-plus (the provider's alias of the same model) | | pi | qwen3.8-27b | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as qwen/qwen3.8-27b (the provider's alias of the same model) | | pi | qwen3.8-flash | pass | pass | pass (gpt-6-astra) | pass | pass | OpenRouter | served as qwen/qwen3.8-flash (the provider's alias of the same model) | | pi | qwen3.8-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | pi | step-3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | retested once; first try: | | qwen | claude-fable-5 | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | claude-haiku-4.5 | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | OpenRouter | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) | | qwen | claude-opus-4.7 | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | claude-opus-4.8 | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | claude-opus-5 | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | claude-sonnet-4.6 | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | claude-sonnet-5 | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | deepseek-v4-flash | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | deepseek-v4-pro | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | deepseek-v4.1-flash | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | served as deepseek/deepseek-v4.1-flash (the provider's alias of the same model) | | qwen | gemini-3.6-flash | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | glm-5.3 | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | glm-5.3-flash | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | gpt-5.2 | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | gpt-5.4 | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | gpt-5.4-mini | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | gpt-5.5 | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | gpt-5.6-luna | pass | pass | pass (qwen3.7-max) | pass | FAIL | OpenRouter | recycle: answered without M1-gpt-5.6-luna: What exact word did I ask you to reply with in my very first message of this task? Reply with just that wo ; retested once; first try: recycle answered without M1-gpt-5.6-luna: What exact word did I ask you to reply with in | | qwen | gpt-5.6-sol | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | gpt-5.6-terra | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6-luna (the provider's alias of the same model) | | qwen | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | OpenRouter | served as openai/gpt-6-sol (the provider's alias of the same model) | | qwen | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 25% of input | | qwen | grok-4.20 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as x-ai/grok-4.20 (the provider's alias of the same model) | | qwen | grok-4.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as x-ai/grok-4.3 (the provider's alias of the same model) | | qwen | grok-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as x-ai/grok-4.5 (the provider's alias of the same model) | | qwen | grok-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as x-ai/grok-4.6 (the provider's alias of the same model) | | qwen | grok-build-0.1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as x-ai/grok-build-0.1 (the provider's alias of the same model) | | qwen | hunyuan-4-preview | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | served as tencent/hy4-preview (the provider's alias of the same model) | | qwen | kimi-k2.7-code | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | kimi-k3 | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | llama-3.3-70b | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as meta-llama/llama-3.3-70b-instruct (the provider's alias of the same model) | | qwen | llama-4-maverick | pass | pass | pass (gpt-5.6-sol) | FAIL | pass | OpenRouter | served as meta-llama/llama-4-maverick (the provider's alias of the same model) ; artifact: no file card (files: none); create 'hello-qwen.txt'. [tool_call: write_file for file_path '/data/workspaces/hsess77f149599a1d465fb237581431a | | qwen | mistral-medium-3.5 | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | muse-glimmer-30b | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as meta/muse-glimmer-30b (the provider's alias of the same model) | | qwen | muse-spark-1.1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as meta/muse-spark-1.1 (the provider's alias of the same model) | | qwen | muse-spark-1.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as meta/muse-spark-1.2 (the provider's alias of the same model) | | qwen | muse-spark-1.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as meta/muse-spark-1.3 (the provider's alias of the same model) | | qwen | nemotron-3-super | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) | | qwen | nemotron-3.5-lightning | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) | | qwen | qwen3.7-max | pass | pass | pass (qwen3.8-max) | pass | pass | OpenRouter | | | qwen | qwen3.7-plus | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | served as qwen/qwen3.7-plus (the provider's alias of the same model) | | qwen | qwen3.8-27b | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | served as qwen/qwen3.8-27b (the provider's alias of the same model) | | qwen | qwen3.8-flash | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | served as qwen/qwen3.8-flash (the provider's alias of the same model) | | qwen | qwen3.8-max | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | qwen | step-3.7-flash | pass | pass | pass (qwen3.7-max) | pass | pass | OpenRouter | | | vibe | claude-fable-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 71% of input | | vibe | claude-fable-5-1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 71% of input | | vibe | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 69% of input | | vibe | claude-haiku-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 71% of input | | vibe | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 71% of input | | vibe | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 71% of input | | vibe | claude-opus-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 71% of input | | vibe | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 69% of input | | vibe | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 71% of input | | vibe | claude-sonnet-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 71% of input | | vibe | deepseek-v4-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 29% of input | | vibe | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 33% of input | | vibe | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 58% of input | | vibe | gemini-3.1-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 52% of input | | vibe | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 0% of input | | vibe | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 43% of input | | vibe | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 43% of input | | vibe | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 32% of input | | vibe | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 14% of input | | vibe | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 22% of input | | vibe | gemini-4-argon | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 53% of input | | vibe | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 64% of input | | vibe | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 50% of input | | vibe | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 48% of input | | vibe | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 55% of input | | vibe | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 49% of input | | vibe | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 49% of input | | vibe | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 50% of input | | vibe | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 50% of input | | vibe | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 64% of input | | vibe | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 57% of input | | vibe | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | OpenRouter | cache 50% of input | | vibe | grok-4.20 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 57% of input | | vibe | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 33% of input | | vibe | grok-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 51% of input | | vibe | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 57% of input | | vibe | grok-build-0.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 62% of input | | vibe | hunyuan-4-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 55% of input | | vibe | kimi-k2.7-code | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 66% of input | | vibe | kimi-k3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 16% of input | | vibe | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 61% of input | | vibe | muse-glimmer-30b | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 49% of input | | vibe | muse-spark-1.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 34% of input | | vibe | muse-spark-1.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 34% of input | | vibe | muse-spark-1.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 34% of input | | vibe | muse-wonder-1.0 | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 34% of input | | vibe | nemotron-3-super | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 0% of input | | vibe | nemotron-3.5-lightning | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 22% of input | | vibe | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 48% of input | | vibe | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 32% of input | | vibe | qwen3.8-27b | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 35% of input | | vibe | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 61% of input | | vibe | qwen3.8-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 53% of input | | vibe | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | OpenRouter | cache 65% of input |
490 pairs, 2409 of 2415 scenario runs passed; 7 pairs served by another connection or as another model are findings, not counted.
Findings, pairs served by a connection other than the one under test or as a model other than the id asked for:
- agentzero x claude-haiku-5.5: no prompt cache reads over 56612 input tokens (every call paid full price)
- aider x claude-haiku-5.5: no prompt cache reads over 24403 input tokens (every call paid full price)
- cheetahclaws x claude-haiku-5.5: no prompt cache reads over 75368 input tokens (every call paid full price)
- cline x claude-haiku-5.5: 4 turn(s) report no served model, so rule 2 could not judge them
- cline x claude-haiku-5.5: no prompt cache reads over 51682 input tokens (every call paid full price)
- cline x gpt-6.1-sol: 4 turn(s) report no served model, so rule 2 could not judge them
- codex x gpt-6.1-sol: 4 turn(s) report no served model, so rule 2 could not judge them
- dcode x claude-fable-5: no prompt cache reads over 89782 input tokens (every call paid full price)
- dcode x claude-fable-5-1: no prompt cache reads over 90714 input tokens (every call paid full price)
- dcode x claude-haiku-4.5: no prompt cache reads over 70158 input tokens (every call paid full price)
- dcode x claude-haiku-5.5: no prompt cache reads over 89537 input tokens (every call paid full price)
- dcode x claude-opus-4.7: no prompt cache reads over 92662 input tokens (every call paid full price)
- dcode x claude-opus-4.8: no prompt cache reads over 90046 input tokens (every call paid full price)
- dcode x claude-opus-5: no prompt cache reads over 90654 input tokens (every call paid full price)
- dcode x claude-opus-5.5: no prompt cache reads over 90132 input tokens (every call paid full price)
- dcode x claude-sonnet-4.6: no prompt cache reads over 70043 input tokens (every call paid full price)
- dcode x claude-sonnet-5: no prompt cache reads over 89806 input tokens (every call paid full price)
- dcode x claude-sonnet-5.5: no prompt cache reads over 90786 input tokens (every call paid full price)
- dsh x claude-haiku-5.5: 4 turn(s) report no served model, so rule 2 could not judge them
- dsh x gpt-6.1-sol: 4 turn(s) report no served model, so rule 2 could not judge them
- fast-agent x claude-haiku-4.5: no prompt cache reads over 14349 input tokens (every call paid full price)
- goose x claude-haiku-5.5: no prompt cache reads over 179206 input tokens (every call paid full price)
- grok x claude-haiku-5.5: no prompt cache reads over 95095 input tokens (every call paid full price)
- hermes x claude-haiku-5.5: no prompt cache reads over 124046 input tokens (every call paid full price)
- hermes x gpt-6.1-sol: 4 turn(s) report no served model, so rule 2 could not judge them
- kimi x claude-haiku-5.5: no prompt cache reads over 184668 input tokens (every call paid full price)
- minimax x claude-haiku-5.5: no prompt cache reads over 83953 input tokens (every call paid full price)
- omp x claude-haiku-5.5: no prompt cache reads over 169082 input tokens (every call paid full price)
- omp x gpt-6.1-sol: 1 turn(s) report no served model, so rule 2 could not judge them
- pi x claude-haiku-5.5: no prompt cache reads over 25529 input tokens (every call paid full price)
- qwen x claude-haiku-5.5: no prompt cache reads over 570679 input tokens (every call paid full price)
Not run in this column, 959 pairs the provider serves that the harness did not run, with the reason:
- agentzero x claude-fable-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x claude-fable-5-1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x claude-haiku-4.5: not run, not run in this column
- agentzero x claude-opus-4.7: not run, not run in this column
- agentzero x claude-opus-4.8: not run, not run in this column
- agentzero x claude-opus-5: not run, not run in this column
- agentzero x claude-opus-5.5: not run, not run in this column
- agentzero x claude-sonnet-4.6: not run, not run in this column
- agentzero x claude-sonnet-5: not run, not run in this column
- agentzero x claude-sonnet-5.5: not run, not run in this column
- agentzero x deepseek-v4-flash: not run, not run in this column
- agentzero x deepseek-v4-pro: not run, not run in this column
- agentzero x deepseek-v4.1-flash: not run, not run in this column
- agentzero x gemini-3-flash-preview: not run, not run in this column
- agentzero x gemini-3.1-flash-lite: not run, not run in this column
- agentzero x gemini-3.1-pro-preview: not run, not run in this column
- agentzero x gemini-3.5-flash: not run, not run in this column
- agentzero x gemini-3.5-flash-lite: not run, not run in this column
- agentzero x gemini-3.6-flash: not run, not run in this column
- agentzero x gemini-3.7-flash: not run, not run in this column
- agentzero x gemini-3.8-flash: not run, not run in this column
- agentzero x gemini-4-argon: not run, not run in this column
- agentzero x glm-5.3: not run, not run in this column
- agentzero x glm-5.3-flash: not run, not run in this column
- agentzero x gpt-5.2: not run, not run in this column
- agentzero x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- agentzero x gpt-5.4: not run, not run in this column
- agentzero x gpt-5.4-mini: not run, not run in this column
- agentzero x gpt-5.5: not run, not run in this column
- agentzero x gpt-5.6-luna: not run, not run in this column
- agentzero x gpt-5.6-sol: not run, not run in this column
- agentzero x gpt-5.6-terra: not run, not run in this column
- agentzero x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- agentzero x gpt-6-luna: not run, not run in this column
- agentzero x gpt-6-sol: not run, not run in this column
- agentzero x gpt-6.1-sol: not run, not run in this column
- agentzero x grok-4.20: not run, not run in this column
- agentzero x grok-4.3: not run, not run in this column
- agentzero x grok-4.5: not run, not run in this column
- agentzero x grok-4.6: not run, not run in this column
- agentzero x grok-build-0.1: not run, not run in this column
- agentzero x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x hunyuan-4-preview: not run, not run in this column
- agentzero x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x kimi-k2.7-code: not run, not run in this column
- agentzero x kimi-k3: not run, not run in this column
- agentzero x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x llama-3.3-70b: not run, not run in this column
- agentzero x llama-4-maverick: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x mistral-medium-3.5: not run, not run in this column
- agentzero x muse-glimmer-30b: not run, not run in this column
- agentzero x muse-spark-1.1: not run, not run in this column
- agentzero x muse-spark-1.2: not run, not run in this column
- agentzero x muse-spark-1.3: not run, not run in this column
- agentzero x muse-wonder-1.0: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x nemotron-3-super: not run, not run in this column
- agentzero x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x nemotron-3.5-lightning: not run, not run in this column
- agentzero x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x qwen3.7-max: not run, not run in this column
- agentzero x qwen3.7-plus: not run, not run in this column
- agentzero x qwen3.8-27b: not run, not run in this column
- agentzero x qwen3.8-flash: not run, not run in this column
- agentzero x qwen3.8-max: not run, not run in this column
- agentzero x step-3.7-flash: not run, not run in this column
- aider x claude-fable-5: not run, not run in this column
- aider x claude-fable-5-1: not run, not run in this column
- aider x claude-haiku-4.5: not run, not run in this column
- aider x claude-opus-4.7: not run, not run in this column
- aider x claude-opus-4.8: not run, not run in this column
- aider x claude-opus-5: not run, not run in this column
- aider x claude-opus-5.5: not run, not run in this column
- aider x claude-sonnet-4.6: not run, not run in this column
- aider x claude-sonnet-5: not run, not run in this column
- aider x claude-sonnet-5.5: not run, not run in this column
- aider x deepseek-v4-flash: not run, not run in this column
- aider x deepseek-v4-pro: not run, not run in this column
- aider x deepseek-v4.1-flash: not run, not run in this column
- aider x gemini-3-flash-preview: not run, not run in this column
- aider x gemini-3.1-flash-lite: not run, not run in this column
- aider x gemini-3.1-pro-preview: not run, not run in this column
- aider x gemini-3.5-flash: not run, not run in this column
- aider x gemini-3.5-flash-lite: not run, not run in this column
- aider x gemini-3.6-flash: not run, not run in this column
- aider x gemini-3.7-flash: not run, not run in this column
- aider x gemini-3.8-flash: not run, not run in this column
- aider x gemini-4-argon: not run, not run in this column
- aider x glm-5.3: not run, not run in this column
- aider x glm-5.3-flash: not run, not run in this column
- aider x gpt-5.2: not run, not run in this column
- aider x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- aider x gpt-5.4: not run, not run in this column
- aider x gpt-5.4-mini: not run, not run in this column
- aider x gpt-5.5: not run, not run in this column
- aider x gpt-5.6-luna: not run, not run in this column
- aider x gpt-5.6-sol: not run, not run in this column
- aider x gpt-5.6-terra: not run, not run in this column
- aider x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- aider x grok-4.20: not run, not run in this column
- aider x grok-4.3: not run, not run in this column
- aider x grok-4.5: not run, not run in this column
- aider x grok-4.6: not run, not run in this column
- aider x grok-build-0.1: not run, not run in this column
- aider x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x hunyuan-4-preview: not run, not run in this column
- aider x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x kimi-k2.7-code: not run, not run in this column
- aider x kimi-k3: not run, not run in this column
- aider x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x llama-3.3-70b: not run, not run in this column
- aider x llama-4-maverick: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x mistral-medium-3.5: not run, not run in this column
- aider x muse-glimmer-30b: not run, not run in this column
- aider x muse-spark-1.1: not run, not run in this column
- aider x muse-spark-1.2: not run, not run in this column
- aider x muse-spark-1.3: not run, not run in this column
- aider x muse-wonder-1.0: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x nemotron-3-super: not run, not run in this column
- aider x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x nemotron-3.5-lightning: not run, not run in this column
- aider x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x qwen3.7-max: not run, not run in this column
- aider x qwen3.7-plus: not run, not run in this column
- aider x qwen3.8-27b: not run, not run in this column
- aider x qwen3.8-flash: not run, not run in this column
- aider x qwen3.8-max: not run, not run in this column
- aider x step-3.7-flash: not run, not run in this column
- cheetahclaws x claude-fable-5: not run, not run in this column
- cheetahclaws x claude-fable-5-1: not run, not run in this column
- cheetahclaws x claude-haiku-4.5: not run, not run in this column
- cheetahclaws x claude-opus-4.7: not run, not run in this column
- cheetahclaws x claude-opus-4.8: not run, not run in this column
- cheetahclaws x claude-opus-5: not run, not run in this column
- cheetahclaws x claude-opus-5.5: not run, not run in this column
- cheetahclaws x claude-sonnet-4.6: not run, not run in this column
- cheetahclaws x claude-sonnet-5: not run, not run in this column
- cheetahclaws x claude-sonnet-5.5: not run, not run in this column
- cheetahclaws x deepseek-v4-flash: not run, not run in this column
- cheetahclaws x deepseek-v4-pro: not run, not run in this column
- cheetahclaws x deepseek-v4.1-flash: not run, not run in this column
- cheetahclaws x gemini-3-flash-preview: not run, not run in this column
- cheetahclaws x gemini-3.1-flash-lite: not run, not run in this column
- cheetahclaws x gemini-3.1-pro-preview: not run, not run in this column
- cheetahclaws x gemini-3.5-flash: not run, not run in this column
- cheetahclaws x gemini-3.5-flash-lite: not run, not run in this column
- cheetahclaws x gemini-3.6-flash: not run, not run in this column
- cheetahclaws x gemini-3.7-flash: not run, not run in this column
- cheetahclaws x gemini-3.8-flash: not run, not run in this column
- cheetahclaws x gemini-4-argon: not run, not run in this column
- cheetahclaws x glm-5.3: not run, not run in this column
- cheetahclaws x glm-5.3-flash: not run, not run in this column
- cheetahclaws x gpt-5.2: not run, not run in this column
- cheetahclaws x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cheetahclaws x gpt-5.4: not run, not run in this column
- cheetahclaws x gpt-5.4-mini: not run, not run in this column
- cheetahclaws x gpt-5.5: not run, not run in this column
- cheetahclaws x gpt-5.6-luna: not run, not run in this column
- cheetahclaws x gpt-5.6-sol: not run, not run in this column
- cheetahclaws x gpt-5.6-terra: not run, not run in this column
- cheetahclaws x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cheetahclaws x grok-4.20: not run, not run in this column
- cheetahclaws x grok-4.3: not run, not run in this column
- cheetahclaws x grok-4.5: not run, not run in this column
- cheetahclaws x grok-4.6: not run, not run in this column
- cheetahclaws x grok-build-0.1: not run, not run in this column
- cheetahclaws x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x hunyuan-4-preview: not run, not run in this column
- cheetahclaws x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x kimi-k2.7-code: not run, not run in this column
- cheetahclaws x kimi-k3: not run, not run in this column
- cheetahclaws x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x llama-3.3-70b: not run, not run in this column
- cheetahclaws x llama-4-maverick: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x mistral-medium-3.5: not run, not run in this column
- cheetahclaws x muse-glimmer-30b: not run, not run in this column
- cheetahclaws x muse-spark-1.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x muse-spark-1.2: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x muse-spark-1.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x muse-wonder-1.0: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x nemotron-3-super: not run, not run in this column
- cheetahclaws x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x nemotron-3.5-lightning: not run, not run in this column
- cheetahclaws x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x qwen3.7-max: not run, not run in this column
- cheetahclaws x qwen3.7-plus: not run, not run in this column
- cheetahclaws x qwen3.8-27b: not run, not run in this column
- cheetahclaws x qwen3.8-flash: not run, not run in this column
- cheetahclaws x qwen3.8-max: not run, not run in this column
- cheetahclaws x step-3.7-flash: not run, not run in this column
- cline x claude-fable-5: not run, not run in this column
- cline x claude-fable-5-1: not run, not run in this column
- cline x claude-haiku-4.5: not run, not run in this column
- cline x claude-opus-4.7: not run, not run in this column
- cline x claude-opus-4.8: not run, not run in this column
- cline x claude-opus-5: not run, not run in this column
- cline x claude-opus-5.5: not run, not run in this column
- cline x claude-sonnet-4.6: not run, not run in this column
- cline x claude-sonnet-5: not run, not run in this column
- cline x claude-sonnet-5.5: not run, not run in this column
- cline x deepseek-v4-flash: not run, not run in this column
- cline x deepseek-v4-pro: not run, not run in this column
- cline x gemini-3-flash-preview: not run, not run in this column
- cline x gemini-3.1-flash-lite: not run, not run in this column
- cline x gemini-3.1-pro-preview: not run, not run in this column
- cline x gemini-3.5-flash: not run, not run in this column
- cline x gemini-3.5-flash-lite: not run, not run in this column
- cline x gemini-3.6-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x gemini-3.7-flash: not run, not run in this column
- cline x gemini-3.8-flash: not run, not run in this column
- cline x gemini-4-argon: not run, not run in this column
- cline x glm-5.3: not run, not run in this column
- cline x glm-5.3-flash: not run, not run in this column
- cline x gpt-5.2: not run, not run in this column
- cline x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x gpt-5.4: not run, not run in this column
- cline x gpt-5.4-mini: not run, not run in this column
- cline x gpt-5.5: not run, not run in this column
- cline x gpt-5.6-luna: not run, not run in this column
- cline x gpt-5.6-sol: not run, not run in this column
- cline x gpt-5.6-terra: not run, not run in this column
- cline x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x kimi-k2.7-code: not run, not run in this column
- cline x kimi-k3: not run, not run in this column
- cline x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x mistral-medium-3.5: not run, not run in this column
- cline x muse-wonder-1.0: not run, not run in this column
- cline x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x qwen3.7-max: not run, not run in this column
- cline x qwen3.8-max: not run, not run in this column
- cline x step-3.7-flash: not run, not run in this column
- codex x claude-fable-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-fable-5-1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-haiku-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-haiku-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-4.8: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-4.6: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4-pro: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3-flash-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.1-flash-lite: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.1-pro-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.5-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.5-flash-lite: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.6-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-4-argon: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x glm-5.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x glm-5.3-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gpt-5.2: not run, not run in this column
- codex x gpt-5.3-codex: not run, not run in this column
- codex x gpt-5.4: not run, not run in this column
- codex x gpt-5.4-mini: not run, not run in this column
- codex x gpt-5.5: not run, not run in this column
- codex x gpt-5.6-luna: not run, not run in this column
- codex x gpt-5.6-sol: not run, not run in this column
- codex x gpt-5.6-terra: not run, not run in this column
- codex x grok-4.20: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.6: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-build-0.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x kimi-k3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x llama-3.3-70b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x llama-4-maverick: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x mistral-medium-3.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-glimmer-30b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-spark-1.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-spark-1.2: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-spark-1.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-wonder-1.0: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x nemotron-3-super: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x nemotron-3.5-lightning: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-plus: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-27b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x step-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x gpt-5.3-codex: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x gpt-6-astra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x llama-3.3-70b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x llama-4-maverick: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x muse-glimmer-30b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x muse-spark-1.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x nemotron-3.5-lightning: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- docker-agent x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- docker-agent x grok-4.20: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x grok-build-0.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x llama-3.3-70b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x llama-4-maverick: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x nemotron-3.5-lightning: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x claude-fable-5-1: not run, not run in this column
- dsh x claude-opus-5.5: not run, not run in this column
- dsh x claude-sonnet-5.5: not run, not run in this column
- dsh x deepseek-v4.1-flash: not run, not run in this column
- dsh x gemini-3-flash-preview: not run, not run in this column
- dsh x gemini-3.1-flash-lite: not run, not run in this column
- dsh x gemini-3.1-pro-preview: not run, not run in this column
- dsh x gemini-3.5-flash: not run, not run in this column
- dsh x gemini-3.5-flash-lite: not run, not run in this column
- dsh x gemini-3.7-flash: not run, not run in this column
- dsh x gemini-3.8-flash: not run, not run in this column
- dsh x gemini-4-argon: not run, not run in this column
- dsh x grok-4.20: not run, not run in this column
- dsh x grok-4.3: not run, not run in this column
- dsh x grok-4.5: not run, not run in this column
- dsh x grok-4.6: not run, not run in this column
- dsh x grok-build-0.1: not run, not run in this column
- dsh x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x hunyuan-4-preview: not run, not run in this column
- dsh x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x llama-3.3-70b: not run, not run in this column
- dsh x llama-4-maverick: not run, not run in this column
- dsh x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x muse-glimmer-30b: not run, not run in this column
- dsh x muse-spark-1.1: not run, not run in this column
- dsh x muse-spark-1.2: not run, not run in this column
- dsh x muse-spark-1.3: not run, not run in this column
- dsh x muse-wonder-1.0: not run, not run in this column
- dsh x nemotron-3-super: not run, not run in this column
- dsh x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x nemotron-3.5-lightning: not run, not run in this column
- dsh x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x qwen3.7-plus: not run, not run in this column
- dsh x qwen3.8-27b: not run, not run in this column
- dsh x qwen3.8-flash: not run, not run in this column
- fast-agent x gemini-4-argon: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- fast-agent x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- fast-agent x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x llama-3.3-70b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x llama-4-maverick: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x nemotron-3-super: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x nemotron-3.5-lightning: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x qwen3.8-27b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x qwen3.8-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-fable-5: not run, not run in this column
- goose x claude-fable-5-1: not run, not run in this column
- goose x claude-haiku-4.5: not run, not run in this column
- goose x claude-opus-4.7: not run, not run in this column
- goose x claude-opus-4.8: not run, not run in this column
- goose x claude-opus-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-opus-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-sonnet-4.6: not run, not run in this column
- goose x claude-sonnet-5: not run, not run in this column
- goose x claude-sonnet-5.5: not run, not run in this column
- goose x deepseek-v4-flash: not run, not run in this column
- goose x deepseek-v4-pro: not run, not run in this column
- goose x deepseek-v4.1-flash: not run, not run in this column
- goose x gemini-3-flash-preview: not run, not run in this column
- goose x gemini-3.1-flash-lite: not run, not run in this column
- goose x gemini-3.1-pro-preview: not run, not run in this column
- goose x gemini-3.5-flash: not run, not run in this column
- goose x gemini-3.5-flash-lite: not run, not run in this column
- goose x gemini-3.6-flash: not run, not run in this column
- goose x gemini-3.7-flash: not run, not run in this column
- goose x gemini-3.8-flash: not run, not run in this column
- goose x gemini-4-argon: not run, not run in this column
- goose x glm-5.3: not run, not run in this column
- goose x glm-5.3-flash: not run, not run in this column
- goose x gpt-5.2: not run, not run in this column
- goose x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-5.4: not run, not run in this column
- goose x gpt-5.4-mini: not run, not run in this column
- goose x gpt-5.5: not run, not run in this column
- goose x gpt-5.6-luna: not run, not run in this column
- goose x gpt-5.6-sol: not run, not run in this column
- goose x gpt-5.6-terra: not run, not run in this column
- goose x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-luna: not run, not run in this column
- goose x gpt-6-sol: not run, not run in this column
- goose x grok-4.20: not run, not run in this column
- goose x grok-4.3: not run, not run in this column
- goose x grok-4.5: not run, not run in this column
- goose x grok-4.6: not run, not run in this column
- goose x grok-build-0.1: not run, not run in this column
- goose x hunyuan-3: not run, not run in this column
- goose x hunyuan-4-preview: not run, not run in this column
- goose x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x kimi-k2.7-code: not run, not run in this column
- goose x kimi-k3: not run, not run in this column
- goose x ling-3.0-flash: not run, not run in this column
- goose x llama-3.3-70b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x llama-4-maverick: not run, not run in this column
- goose x minimax-m3: not run, not run in this column
- goose x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x mistral-medium-3.5: not run, not run in this column
- goose x muse-glimmer-30b: not run, not run in this column
- goose x muse-spark-1.1: not run, not run in this column
- goose x muse-spark-1.2: not run, not run in this column
- goose x muse-spark-1.3: not run, not run in this column
- goose x muse-wonder-1.0: not run, not run in this column
- goose x nemotron-3-super: not run, not run in this column
- goose x nemotron-3-ultra: not run, not run in this column
- goose x nemotron-3.5-lightning: not run, not run in this column
- goose x qwen3.7-flash: not run, not run in this column
- goose x qwen3.7-max: not run, not run in this column
- goose x qwen3.7-plus: not run, not run in this column
- goose x qwen3.8-27b: not run, not run in this column
- goose x qwen3.8-flash: not run, not run in this column
- goose x qwen3.8-max: not run, not run in this column
- goose x step-3.7-flash: not run, not run in this column
- grok x claude-fable-5: not run, not run in this column
- grok x claude-fable-5-1: not run, not run in this column
- grok x claude-haiku-4.5: not run, not run in this column
- grok x claude-opus-4.7: not run, not run in this column
- grok x claude-opus-4.8: not run, not run in this column
- grok x claude-opus-5: not run, not run in this column
- grok x claude-opus-5.5: not run, not run in this column
- grok x claude-sonnet-4.6: not run, not run in this column
- grok x claude-sonnet-5: not run, not run in this column
- grok x claude-sonnet-5.5: not run, not run in this column
- grok x deepseek-v4-flash: not run, not run in this column
- grok x deepseek-v4-pro: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x deepseek-v4.1-flash: not run, not run in this column
- grok x gemini-3-flash-preview: not run, not run in this column
- grok x gemini-3.1-flash-lite: not run, not run in this column
- grok x gemini-3.1-pro-preview: not run, not run in this column
- grok x gemini-3.5-flash: not run, not run in this column
- grok x gemini-3.5-flash-lite: not run, not run in this column
- grok x gemini-3.6-flash: not run, not run in this column
- grok x gemini-3.7-flash: not run, not run in this column
- grok x gemini-3.8-flash: not run, not run in this column
- grok x gemini-4-argon: not run, not run in this column
- grok x glm-5.3: not run, not run in this column
- grok x glm-5.3-flash: not run, not run in this column
- grok x gpt-5.2: not run, not run in this column
- grok x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- grok x gpt-5.4: not run, not run in this column
- grok x gpt-5.4-mini: not run, not run in this column
- grok x gpt-5.5: not run, not run in this column
- grok x gpt-5.6-luna: not run, not run in this column
- grok x gpt-5.6-sol: not run, not run in this column
- grok x gpt-5.6-terra: not run, not run in this column
- grok x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- grok x gpt-6-luna: not run, not run in this column
- grok x gpt-6-sol: not run, not run in this column
- grok x gpt-6.1-sol: not run, not run in this column
- grok x grok-4.20: not run, not run in this column
- grok x grok-4.3: not run, not run in this column
- grok x grok-4.5: not run, not run in this column
- grok x grok-4.6: not run, not run in this column
- grok x grok-build-0.1: not run, not run in this column
- grok x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x hunyuan-4-preview: not run, not run in this column
- grok x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x kimi-k2.7-code: not run, not run in this column
- grok x kimi-k3: not run, not run in this column
- grok x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x llama-3.3-70b: not run, not run in this column
- grok x llama-4-maverick: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x mistral-medium-3.5: not run, not run in this column
- grok x muse-glimmer-30b: not run, not run in this column
- grok x muse-spark-1.1: not run, not run in this column
- grok x muse-spark-1.2: not run, not run in this column
- grok x muse-spark-1.3: not run, not run in this column
- grok x muse-wonder-1.0: not run, not run in this column
- grok x nemotron-3-super: not run, not run in this column
- grok x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x nemotron-3.5-lightning: not run, not run in this column
- grok x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x qwen3.7-max: not run, not run in this column
- grok x qwen3.7-plus: not run, not run in this column
- grok x qwen3.8-27b: not run, not run in this column
- grok x qwen3.8-flash: not run, not run in this column
- grok x qwen3.8-max: not run, not run in this column
- grok x step-3.7-flash: not run, not run in this column
- hermes x claude-fable-5-1: not run, not run in this column
- hermes x claude-opus-5.5: not run, not run in this column
- hermes x claude-sonnet-5.5: not run, not run in this column
- hermes x gemini-3-flash-preview: not run, not run in this column
- hermes x gemini-3.1-flash-lite: not run, not run in this column
- hermes x gemini-3.1-pro-preview: not run, not run in this column
- hermes x gemini-3.5-flash: not run, not run in this column
- hermes x gemini-3.5-flash-lite: not run, not run in this column
- hermes x gemini-3.7-flash: not run, not run in this column
- hermes x gemini-3.8-flash: not run, not run in this column
- hermes x gemini-4-argon: not run, not run in this column
- hermes x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- hermes x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- hermes x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- hermes x muse-wonder-1.0: not run, not run in this column
- kilo x claude-fable-5: not run, not run in this column
- kilo x claude-fable-5-1: not run, not run in this column
- kilo x claude-haiku-4.5: not run, not run in this column
- kilo x claude-opus-4.7: not run, not run in this column
- kilo x claude-opus-4.8: not run, not run in this column
- kilo x claude-opus-5: not run, not run in this column
- kilo x claude-opus-5.5: not run, not run in this column
- kilo x claude-sonnet-4.6: not run, not run in this column
- kilo x claude-sonnet-5: not run, not run in this column
- kilo x claude-sonnet-5.5: not run, not run in this column
- kilo x deepseek-v4-flash: not run, not run in this column
- kilo x deepseek-v4-pro: not run, not run in this column
- kilo x deepseek-v4.1-flash: not run, not run in this column
- kilo x gemini-3-flash-preview: not run, not run in this column
- kilo x gemini-3.1-flash-lite: not run, not run in this column
- kilo x gemini-3.1-pro-preview: not run, not run in this column
- kilo x gemini-3.5-flash: not run, not run in this column
- kilo x gemini-3.5-flash-lite: not run, not run in this column
- kilo x gemini-3.6-flash: not run, not run in this column
- kilo x gemini-3.7-flash: not run, not run in this column
- kilo x gemini-3.8-flash: not run, not run in this column
- kilo x gemini-4-argon: not run, not run in this column
- kilo x glm-5.3: not run, not run in this column
- kilo x glm-5.3-flash: not run, not run in this column
- kilo x gpt-5.2: not run, not run in this column
- kilo x gpt-5.3-codex: not run, not run in this column
- kilo x gpt-5.4: not run, not run in this column
- kilo x gpt-5.4-mini: not run, not run in this column
- kilo x gpt-5.5: not run, not run in this column
- kilo x gpt-5.6-luna: not run, not run in this column
- kilo x gpt-5.6-sol: not run, not run in this column
- kilo x gpt-5.6-terra: not run, not run in this column
- kilo x gpt-6-astra: not run, not run in this column
- kilo x gpt-6-luna: not run, not run in this column
- kilo x gpt-6-sol: not run, not run in this column
- kilo x gpt-6.1-sol: not run, not run in this column
- kilo x grok-4.20: not run, not run in this column
- kilo x grok-4.3: not run, not run in this column
- kilo x grok-4.5: not run, not run in this column
- kilo x grok-4.6: not run, not run in this column
- kilo x grok-build-0.1: not run, not run in this column
- kilo x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kilo x hunyuan-4-preview: not run, not run in this column
- kilo x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kilo x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kilo x kimi-k2.7-code: not run, not run in this column
- kilo x kimi-k3: not run, not run in this column
- kilo x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kilo x llama-3.3-70b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kilo x llama-4-maverick: not run, not run in this column
- kilo x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kilo x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kilo x mistral-medium-3.5: not run, not run in this column
- kilo x muse-glimmer-30b: not run, not run in this column
- kilo x muse-spark-1.1: not run, not run in this column
- kilo x muse-spark-1.2: not run, not run in this column
- kilo x muse-spark-1.3: not run, not run in this column
- kilo x muse-wonder-1.0: not run, not run in this column
- kilo x nemotron-3-super: not run, not run in this column
- kilo x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kilo x nemotron-3.5-lightning: not run, not run in this column
- kilo x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kilo x qwen3.7-max: not run, not run in this column
- kilo x qwen3.7-plus: not run, not run in this column
- kilo x qwen3.8-27b: not run, not run in this column
- kilo x qwen3.8-flash: not run, not run in this column
- kilo x qwen3.8-max: not run, not run in this column
- kilo x step-3.7-flash: not run, not run in this column
- kimi x claude-fable-5: not run, not run in this column
- kimi x claude-fable-5-1: not run, not run in this column
- kimi x claude-haiku-4.5: not run, not run in this column
- kimi x claude-opus-4.7: not run, not run in this column
- kimi x claude-opus-4.8: not run, not run in this column
- kimi x claude-opus-5: not run, not run in this column
- kimi x claude-opus-5.5: not run, not run in this column
- kimi x claude-sonnet-4.6: not run, not run in this column
- kimi x claude-sonnet-5: not run, not run in this column
- kimi x claude-sonnet-5.5: not run, not run in this column
- kimi x deepseek-v4-flash: not run, not run in this column
- kimi x deepseek-v4-pro: not run, not run in this column
- kimi x deepseek-v4.1-flash: not run, not run in this column
- kimi x gemini-3-flash-preview: not run, not run in this column
- kimi x gemini-3.1-flash-lite: not run, not run in this column
- kimi x gemini-3.1-pro-preview: not run, not run in this column
- kimi x gemini-3.5-flash: not run, not run in this column
- kimi x gemini-3.5-flash-lite: not run, not run in this column
- kimi x gemini-3.6-flash: not run, not run in this column
- kimi x gemini-3.7-flash: not run, not run in this column
- kimi x gemini-3.8-flash: not run, not run in this column
- kimi x gemini-4-argon: not run, not run in this column
- kimi x glm-5.3: not run, not run in this column
- kimi x glm-5.3-flash: not run, not run in this column
- kimi x gpt-5.2: not run, not run in this column
- kimi x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- kimi x gpt-5.4: not run, not run in this column
- kimi x gpt-5.4-mini: not run, not run in this column
- kimi x gpt-5.5: not run, not run in this column
- kimi x gpt-5.6-luna: not run, not run in this column
- kimi x gpt-5.6-sol: not run, not run in this column
- kimi x gpt-5.6-terra: not run, not run in this column
- kimi x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- kimi x grok-4.20: not run, not run in this column
- kimi x grok-4.3: not run, not run in this column
- kimi x grok-4.5: not run, not run in this column
- kimi x grok-4.6: not run, not run in this column
- kimi x grok-build-0.1: not run, not run in this column
- kimi x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x hunyuan-4-preview: not run, not run in this column
- kimi x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x kimi-k2.7-code: not run, not run in this column
- kimi x kimi-k3: not run, not run in this column
- kimi x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x llama-3.3-70b: not run, not run in this column
- kimi x llama-4-maverick: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x mistral-medium-3.5: not run, not run in this column
- kimi x muse-glimmer-30b: not run, not run in this column
- kimi x muse-spark-1.1: not run, not run in this column
- kimi x muse-spark-1.2: not run, not run in this column
- kimi x muse-spark-1.3: not run, not run in this column
- kimi x muse-wonder-1.0: not run, not run in this column
- kimi x nemotron-3-super: not run, not run in this column
- kimi x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x nemotron-3.5-lightning: not run, not run in this column
- kimi x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x qwen3.7-max: not run, not run in this column
- kimi x qwen3.7-plus: not run, not run in this column
- kimi x qwen3.8-27b: not run, not run in this column
- kimi x qwen3.8-flash: not run, not run in this column
- kimi x qwen3.8-max: not run, not run in this column
- kimi x step-3.7-flash: not run, not run in this column
- minimax x claude-fable-5: not run, not run in this column
- minimax x claude-fable-5-1: not run, not run in this column
- minimax x claude-haiku-4.5: not run, not run in this column
- minimax x claude-opus-4.7: not run, not run in this column
- minimax x claude-opus-4.8: not run, not run in this column
- minimax x claude-opus-5: not run, not run in this column
- minimax x claude-opus-5.5: not run, not run in this column
- minimax x claude-sonnet-4.6: not run, not run in this column
- minimax x claude-sonnet-5: not run, not run in this column
- minimax x claude-sonnet-5.5: not run, not run in this column
- minimax x deepseek-v4-flash: not run, not run in this column
- minimax x deepseek-v4-pro: not run, not run in this column
- minimax x deepseek-v4.1-flash: not run, not run in this column
- minimax x gemini-3-flash-preview: not run, not run in this column
- minimax x gemini-3.1-flash-lite: not run, not run in this column
- minimax x gemini-3.1-pro-preview: not run, not run in this column
- minimax x gemini-3.5-flash: not run, not run in this column
- minimax x gemini-3.5-flash-lite: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- minimax x gemini-3.6-flash: not run, not run in this column
- minimax x gemini-3.7-flash: not run, not run in this column
- minimax x gemini-3.8-flash: not run, not run in this column
- minimax x gemini-4-argon: not run, not run in this column
- minimax x glm-5.3: not run, not run in this column
- minimax x glm-5.3-flash: not run, not run in this column
- minimax x gpt-5.2: not run, not run in this column
- minimax x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- minimax x gpt-5.4: not run, not run in this column
- minimax x gpt-5.4-mini: not run, not run in this column
- minimax x gpt-5.5: not run, not run in this column
- minimax x gpt-5.6-luna: not run, not run in this column
- minimax x gpt-5.6-sol: not run, not run in this column
- minimax x gpt-5.6-terra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- minimax x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- minimax x gpt-6-luna: not run, not run in this column
- minimax x gpt-6-sol: not run, not run in this column
- minimax x gpt-6.1-sol: not run, not run in this column
- minimax x grok-4.20: not run, not run in this column
- minimax x grok-4.3: not run, not run in this column
- minimax x grok-4.5: not run, not run in this column
- minimax x grok-4.6: not run, not run in this column
- minimax x grok-build-0.1: not run, not run in this column
- minimax x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- minimax x hunyuan-4-preview: not run, not run in this column
- minimax x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- minimax x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- minimax x kimi-k2.7-code: not run, not run in this column
- minimax x kimi-k3: not run, not run in this column
- minimax x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- minimax x llama-3.3-70b: not run, not run in this column
- minimax x llama-4-maverick: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- minimax x minimax-m3: not run, not run in this column
- minimax x mistral-medium-3.5: not run, not run in this column
- minimax x muse-glimmer-30b: not run, not run in this column
- minimax x muse-spark-1.1: not run, not run in this column
- minimax x muse-spark-1.2: not run, not run in this column
- minimax x muse-spark-1.3: not run, not run in this column
- minimax x muse-wonder-1.0: not run, not run in this column
- minimax x nemotron-3-super: not run, not run in this column
- minimax x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- minimax x nemotron-3.5-lightning: not run, not run in this column
- minimax x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- minimax x qwen3.7-max: not run, not run in this column
- minimax x qwen3.7-plus: not run, not run in this column
- minimax x qwen3.8-27b: not run, not run in this column
- minimax x qwen3.8-flash: not run, not run in this column
- minimax x qwen3.8-max: not run, not run in this column
- minimax x step-3.7-flash: not run, not run in this column
- omp x claude-fable-5: not run, not run in this column
- omp x claude-fable-5-1: not run, not run in this column
- omp x claude-haiku-4.5: not run, not run in this column
- omp x claude-opus-4.7: not run, not run in this column
- omp x claude-opus-4.8: not run, not run in this column
- omp x claude-opus-5: not run, not run in this column
- omp x claude-opus-5.5: not run, not run in this column
- omp x claude-sonnet-4.6: not run, not run in this column
- omp x claude-sonnet-5: not run, not run in this column
- omp x claude-sonnet-5.5: not run, not run in this column
- omp x deepseek-v4-flash: not run, not run in this column
- omp x deepseek-v4-pro: not run, not run in this column
- omp x deepseek-v4.1-flash: not run, not run in this column
- omp x gemini-3-flash-preview: not run, not run in this column
- omp x gemini-3.1-flash-lite: not run, not run in this column
- omp x gemini-3.1-pro-preview: not run, not run in this column
- omp x gemini-3.5-flash: not run, not run in this column
- omp x gemini-3.5-flash-lite: not run, not run in this column
- omp x gemini-3.6-flash: not run, not run in this column
- omp x gemini-3.7-flash: not run, not run in this column
- omp x gemini-3.8-flash: not run, not run in this column
- omp x gemini-4-argon: not run, not run in this column
- omp x glm-5.3: not run, not run in this column
- omp x glm-5.3-flash: not run, not run in this column
- omp x gpt-5.2: not run, not run in this column
- omp x gpt-5.3-codex: not run, not run in this column
- omp x gpt-5.4: not run, not run in this column
- omp x gpt-5.4-mini: not run, not run in this column
- omp x gpt-5.5: not run, not run in this column
- omp x gpt-5.6-luna: not run, not run in this column
- omp x gpt-5.6-sol: not run, not run in this column
- omp x gpt-5.6-terra: not run, not run in this column
- omp x grok-4.20: not run, not run in this column
- omp x grok-4.3: not run, not run in this column
- omp x grok-4.5: not run, not run in this column
- omp x grok-4.6: not run, not run in this column
- omp x grok-build-0.1: not run, not run in this column
- omp x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x hunyuan-4-preview: not run, not run in this column
- omp x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x kimi-k2.7-code: not run, not run in this column
- omp x kimi-k3: not run, not run in this column
- omp x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x llama-3.3-70b: not run, not run in this column
- omp x llama-4-maverick: not run, not run in this column
- omp x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x mistral-medium-3.5: not run, not run in this column
- omp x muse-glimmer-30b: not run, not run in this column
- omp x muse-spark-1.1: not run, not run in this column
- omp x muse-spark-1.2: not run, not run in this column
- omp x muse-spark-1.3: not run, not run in this column
- omp x muse-wonder-1.0: not run, not run in this column
- omp x nemotron-3-super: not run, not run in this column
- omp x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x nemotron-3.5-lightning: not run, not run in this column
- omp x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x qwen3.7-max: not run, not run in this column
- omp x qwen3.7-plus: not run, not run in this column
- omp x qwen3.8-27b: not run, not run in this column
- omp x qwen3.8-flash: not run, not run in this column
- omp x qwen3.8-max: not run, not run in this column
- omp x step-3.7-flash: not run, not run in this column
- opencode x claude-fable-5-1: not run, not run in this column
- opencode x claude-opus-5.5: not run, not run in this column
- opencode x claude-sonnet-5.5: not run, not run in this column
- opencode x gemini-3-flash-preview: not run, not run in this column
- opencode x gemini-3.1-flash-lite: not run, not run in this column
- opencode x gemini-3.1-pro-preview: not run, not run in this column
- opencode x gemini-3.5-flash: not run, not run in this column
- opencode x gemini-3.5-flash-lite: not run, not run in this column
- opencode x gemini-3.7-flash: not run, not run in this column
- opencode x gemini-3.8-flash: not run, not run in this column
- opencode x gemini-4-argon: not run, not run in this column
- opencode x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x muse-wonder-1.0: not run, not run in this column
- opencode x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x claude-fable-5: not run, not run in this column
- openhands x claude-fable-5-1: not run, not run in this column
- openhands x claude-haiku-4.5: not run, not run in this column
- openhands x claude-opus-4.7: not run, not run in this column
- openhands x claude-opus-4.8: not run, not run in this column
- openhands x claude-opus-5: not run, not run in this column
- openhands x claude-opus-5.5: not run, not run in this column
- openhands x claude-sonnet-4.6: not run, not run in this column
- openhands x claude-sonnet-5: not run, not run in this column
- openhands x claude-sonnet-5.5: not run, not run in this column
- openhands x deepseek-v4-flash: not run, not run in this column
- openhands x deepseek-v4-pro: not run, not run in this column
- openhands x deepseek-v4.1-flash: not run, not run in this column
- openhands x gemini-3-flash-preview: not run, not run in this column
- openhands x gemini-3.1-flash-lite: not run, not run in this column
- openhands x gemini-3.1-pro-preview: not run, not run in this column
- openhands x gemini-3.5-flash: not run, not run in this column
- openhands x gemini-3.5-flash-lite: not run, not run in this column
- openhands x gemini-3.6-flash: not run, not run in this column
- openhands x gemini-3.7-flash: not run, not run in this column
- openhands x gemini-3.8-flash: not run, not run in this column
- openhands x gemini-4-argon: not run, not run in this column
- openhands x glm-5.3: not run, not run in this column
- openhands x glm-5.3-flash: not run, not run in this column
- openhands x gpt-5.2: not run, not run in this column
- openhands x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- openhands x gpt-5.4: not run, not run in this column
- openhands x gpt-5.4-mini: not run, not run in this column
- openhands x gpt-5.5: not run, not run in this column
- openhands x gpt-5.6-luna: not run, not run in this column
- openhands x gpt-5.6-sol: not run, not run in this column
- openhands x gpt-5.6-terra: not run, not run in this column
- openhands x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- openhands x grok-4.20: not run, not run in this column
- openhands x grok-4.3: not run, not run in this column
- openhands x grok-4.5: not run, not run in this column
- openhands x grok-4.6: not run, not run in this column
- openhands x grok-build-0.1: not run, not run in this column
- openhands x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x hunyuan-4-preview: not run, not run in this column
- openhands x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x kimi-k2.7-code: not run, not run in this column
- openhands x kimi-k3: not run, not run in this column
- openhands x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x llama-3.3-70b: not run, not run in this column
- openhands x llama-4-maverick: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x mistral-medium-3.5: not run, not run in this column
- openhands x muse-glimmer-30b: not run, not run in this column
- openhands x muse-spark-1.1: not run, not run in this column
- openhands x muse-spark-1.2: not run, not run in this column
- openhands x muse-spark-1.3: not run, not run in this column
- openhands x muse-wonder-1.0: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x nemotron-3-super: not run, not run in this column
- openhands x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x nemotron-3.5-lightning: not run, not run in this column
- openhands x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x qwen3.7-max: not run, not run in this column
- openhands x qwen3.7-plus: not run, not run in this column
- openhands x qwen3.8-27b: not run, not run in this column
- openhands x qwen3.8-flash: not run, not run in this column
- openhands x qwen3.8-max: not run, not run in this column
- openhands x step-3.7-flash: not run, not run in this column
- pi x claude-fable-5-1: not run, not run in this column
- pi x claude-opus-5.5: not run, not run in this column
- pi x claude-sonnet-5.5: not run, not run in this column
- pi x gemini-3-flash-preview: not run, not run in this column
- pi x gemini-3.1-flash-lite: not run, not run in this column
- pi x gemini-3.1-pro-preview: not run, not run in this column
- pi x gemini-3.5-flash: not run, not run in this column
- pi x gemini-3.5-flash-lite: not run, not run in this column
- pi x gemini-3.7-flash: not run, not run in this column
- pi x gemini-3.8-flash: not run, not run in this column
- pi x gemini-4-argon: not run, not run in this column
- pi x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x muse-wonder-1.0: not run, not run in this column
- pi x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x claude-fable-5-1: not run, not run in this column
- qwen x claude-opus-5.5: not run, not run in this column
- qwen x claude-sonnet-5.5: not run, not run in this column
- qwen x gemini-3-flash-preview: not run, not run in this column
- qwen x gemini-3.1-flash-lite: not run, not run in this column
- qwen x gemini-3.1-pro-preview: not run, not run in this column
- qwen x gemini-3.5-flash: not run, not run in this column
- qwen x gemini-3.5-flash-lite: not run, not run in this column
- qwen x gemini-3.7-flash: not run, not run in this column
- qwen x gemini-3.8-flash: not run, not run in this column
- qwen x gemini-4-argon: not run, not run in this column
- qwen x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x muse-wonder-1.0: not run, not run in this column
- qwen x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x deepseek-v4.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- vibe x gpt-5.6-luna: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- vibe x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x jev-1.13: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x jev-latest: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x llama-3.3-70b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x llama-4-maverick: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
Provider: tokenrouter
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| aider | claude-sonnet-5.5 | pass | pass | pass (claude-opus-5) | pass | pass | My TokenRouter, Vercel AI Gateway | served as claude-sonnet-5-5 (the provider's alias of the same model) ; served by another connection (finding below) |
| aider | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| aider | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | My TokenRouter | |
| aider | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 45% of input |
| cheetahclaws | claude-sonnet-5.5 | pass | pass | pass (claude-opus-5) | pass | pass | My TokenRouter, Vercel AI Gateway | served as claude-sonnet-5-5 (the provider's alias of the same model) ; served by another connection (finding below) |
| cheetahclaws | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| cheetahclaws | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | My TokenRouter | |
| cheetahclaws | gpt-6.1-sol | FAIL | n/a | n/a | n/a | n/a | My TokenRouter | 1 turn(s) unlabelled (finding below) ; first: The turn failed: Failed — BadRequestError: Error code: 400 - {'error': {'message': "Unsupported value: 'reasoning_effort' does not support ' |
| claude-code | claude-sonnet-4.6 | pass | pass | pass (claude-opus-5) | pass | pass | My TokenRouter | |
| claude-code | claude-sonnet-5.5 | pass | pass | pass (claude-opus-5) | pass | pass | My TokenRouter, Vercel AI Gateway | served by another connection (finding below) |
| cline | claude-sonnet-4.6 | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| cline | claude-sonnet-5.5 | pass | pass | pass (claude-opus-5) | pass | pass | My TokenRouter, Vercel AI Gateway | served by another connection (finding below) |
| cline | deepseek-v4.1-flash | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| cline | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | |
| cline | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| cline | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | My TokenRouter | |
| cline | gpt-6.1-sol | FAIL | n/a | n/a | n/a | n/a | My TokenRouter | 1 turn(s) unlabelled (finding below) ; first: The turn failed: Unsupported value: 'reasoning_effort' does not support 'none' with this model. Supported values are: 'low', 'medium', 'high |
| cline | grok-4.20 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | |
| cline | grok-4.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | |
| cline | grok-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | |
| cline | grok-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | |
| cline | grok-build-0.1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | |
| cline | hunyuan-4-preview | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| cline | nemotron-3-super | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| cline | nemotron-3.5-lightning | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| cline | qwen3.7-plus | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| cline | qwen3.8-flash | pass | pass | pass (gpt-5.4) | pass | pass | My TokenRouter | |
| codex | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| codex | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| codex | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| codex | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | 4 turn(s) unlabelled (finding below) ; cache 58% of input |
| dcode | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 63% of input |
| dcode | deepseek-v4.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 70% of input |
| dcode | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 50% of input |
| dcode | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 33% of input |
| dcode | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | 4 turn(s) unlabelled (finding below) ; cache 0% of input |
| dcode | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 60% of input |
| dcode | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 66% of input |
| dcode | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 66% of input |
| dcode | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 69% of input |
| dcode | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 66% of input |
| dcode | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 66% of input |
| dcode | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 66% of input |
| dcode | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 71% of input |
| dcode | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 66% of input |
| dcode | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 66% of input |
| dcode | grok-4.20 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 81% of input |
| dcode | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 67% of input |
| dcode | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 73% of input |
| dcode | grok-build-0.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 67% of input |
| dcode | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 68% of input |
| dcode | nemotron-3-super | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 0% of input |
| dcode | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 63% of input |
| dcode | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 66% of input |
| dcode | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 71% of input |
| dcode | qwen3.8-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 67% of input |
| dcode | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | My TokenRouter | cache 52% of input |
| docker-agent | claude-fable-5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| docker-agent | claude-haiku-4.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| docker-agent | claude-opus-4.8 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| docker-agent | claude-opus-5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| docker-agent | claude-opus-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| docker-agent | claude-sonnet-4.6 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| docker-agent | claude-sonnet-5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| docker-agent | claude-sonnet-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| docker-agent | deepseek-v4-pro | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 78% of input |
| docker-agent | deepseek-v4.1-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 84% of input |
| docker-agent | gemini-3-flash-preview | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 0% of input |
| docker-agent | gemini-3.1-pro-preview | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 0% of input |
| docker-agent | gemini-3.5-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 0% of input |
| docker-agent | gemini-3.5-flash-lite | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 0% of input |
| docker-agent | gemini-3.6-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 0% of input |
| docker-agent | gemini-3.7-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 0% of input |
| docker-agent | gemini-3.8-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 0% of input |
| docker-agent | glm-5.3 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 80% of input |
| docker-agent | glm-5.3-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 79% of input |
| docker-agent | gpt-5.2 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 89% of input |
| docker-agent | gpt-5.4 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 80% of input |
| docker-agent | gpt-5.4-mini | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 88% of input |
| docker-agent | gpt-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 74% of input |
| docker-agent | gpt-5.6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 100% of input |
| docker-agent | gpt-5.6-sol | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 100% of input |
| docker-agent | gpt-5.6-terra | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 100% of input |
| docker-agent | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 100% of input |
| docker-agent | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | My TokenRouter | cache 100% of input |
| docker-agent | grok-4.3 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 81% of input |
| docker-agent | grok-4.6 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 88% of input |
| docker-agent | hunyuan-4-preview | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 38% of input |
| docker-agent | kimi-k2.7-code | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 80% of input |
| docker-agent | mistral-medium-3.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 80% of input |
| docker-agent | nemotron-3-super | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 0% of input |
| docker-agent | qwen3.7-max | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 80% of input |
| docker-agent | qwen3.7-plus | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 80% of input |
| docker-agent | qwen3.8-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 79% of input |
| docker-agent | qwen3.8-max | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 78% of input |
| docker-agent | step-3.7-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 68% of input |
| dsh | claude-sonnet-5.5 | pass | pass | pass (claude-opus-5) | pass | pass | My TokenRouter, Vercel AI Gateway | served by another connection (finding below) |
| dsh | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| dsh | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| dsh | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| dsh | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | 4 turn(s) unlabelled (finding below) ; cache 100% of input |
| fast-agent | claude-fable-5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| fast-agent | claude-fable-5-1 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| fast-agent | claude-haiku-4.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| fast-agent | claude-opus-4.7 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| fast-agent | claude-opus-4.8 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| fast-agent | claude-opus-5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| fast-agent | claude-opus-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| fast-agent | claude-sonnet-4.6 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| fast-agent | claude-sonnet-5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| fast-agent | claude-sonnet-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| fast-agent | deepseek-v4-pro | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | deepseek-v4.1-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gemini-3-flash-preview | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gemini-3.1-pro-preview | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gemini-3.5-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gemini-3.5-flash-lite | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gemini-3.6-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gemini-3.7-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gemini-3.8-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | glm-5.3 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | glm-5.3-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gpt-5.2 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gpt-5.4 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gpt-5.4-mini | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gpt-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gpt-5.6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gpt-5.6-sol | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gpt-5.6-terra | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | My TokenRouter | |
| fast-agent | grok-4.20 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | grok-4.3 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | grok-4.6 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | grok-build-0.1 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | mistral-medium-3.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | qwen3.7-max | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | qwen3.7-plus | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | qwen3.8-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| fast-agent | step-3.7-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| goose | claude-sonnet-5.5 | pass | pass | pass (claude-sonnet-5) | pass | pass | My TokenRouter | served as claude-sonnet-5-5 (the provider's alias of the same model) ; served by another connection (finding below) |
| goose | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 52% of input |
| hermes | claude-sonnet-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| hermes | claude-sonnet-5.5 | pass | pass | pass (claude-opus-5) | pass | pass | My TokenRouter, Vercel AI Gateway | served by another connection (finding below) |
| hermes | deepseek-v4.1-flash | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| hermes | gpt-5.4 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| hermes | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| hermes | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| hermes | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| hermes | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 0% of input |
| hermes | grok-4.20 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| hermes | grok-4.3 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| hermes | grok-4.5 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| hermes | grok-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| hermes | grok-build-0.1 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| hermes | hunyuan-4-preview | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| hermes | nemotron-3-super | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| hermes | nemotron-3.5-lightning | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| hermes | qwen3.7-plus | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| hermes | qwen3.8-flash | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| kimi | claude-sonnet-5.5 | pass | pass | pass (claude-opus-5) | pass | pass | My TokenRouter, Vercel AI Gateway | served as claude-sonnet-5-5 (the provider's alias of the same model) ; served by another connection (finding below) |
| kimi | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| kimi | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | My TokenRouter | |
| kimi | gpt-6.1-sol | FAIL | n/a | n/a | n/a | n/a | My TokenRouter | 1 turn(s) unlabelled (finding below) ; first: The turn failed: error: failed to run prompt: provider.api_error: 400 Unsupported value: 'reasoning_effort' does not support 'none' with thi |
| omp | claude-sonnet-5.5 | pass | pass | pass (claude-sonnet-5) | pass | pass | My TokenRouter | served as anthropic/claude-sonnet-5.5 (the provider's alias of the same model) ; served by another connection (finding below) |
| omp | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | served as openai/gpt-6-astra (the provider's alias of the same model) |
| omp | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as openai/gpt-6-luna (the provider's alias of the same model) |
| omp | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as openai/gpt-6-sol (the provider's alias of the same model) |
| omp | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | 1 turn(s) unlabelled (finding below) ; served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 100% of input |
| opencode | claude-sonnet-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| opencode | claude-sonnet-5.5 | pass | pass | pass (claude-opus-5) | pass | pass | My TokenRouter, Vercel AI Gateway | served by another connection (finding below) |
| opencode | deepseek-v4.1-flash | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as deepseek-flash (the provider's alias of the same model) |
| opencode | gpt-5.4 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| opencode | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| opencode | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| opencode | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| opencode | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 100% of input |
| opencode | grok-4.20 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as x-ai/grok-4.20 (the provider's alias of the same model) |
| opencode | grok-4.3 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as x-ai/grok-4.3 (the provider's alias of the same model) |
| opencode | grok-4.5 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| opencode | grok-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| opencode | grok-build-0.1 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as x-ai/grok-build-0.1 (the provider's alias of the same model) |
| opencode | hunyuan-4-preview | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as tencent/hy4-preview (the provider's alias of the same model) |
| opencode | nemotron-3-super | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) |
| opencode | nemotron-3.5-lightning | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) |
| opencode | qwen3.7-plus | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as qwen/qwen3.7-plus (the provider's alias of the same model) |
| opencode | qwen3.8-flash | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | |
| openhands | claude-sonnet-5.5 | pass | pass | pass (claude-opus-5) | pass | pass | My TokenRouter, Vercel AI Gateway | served as claude-sonnet-5-5 (the provider's alias of the same model) ; served by another connection (finding below) |
| openhands | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| openhands | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | My TokenRouter | |
| openhands | gpt-6.1-sol | FAIL | n/a | n/a | n/a | n/a | My TokenRouter | first: The turn failed: the agent-server marked the conversation error: BadRequestError: Error code: 400 - {'error': {'message': "Unsupported value ; cache 0% of input |
| pi | claude-sonnet-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as anthropic/claude-sonnet-4.6 (the provider's alias of the same model) |
| pi | claude-sonnet-5.5 | pass | pass | pass (claude-opus-5) | pass | pass | My TokenRouter, Vercel AI Gateway | served as anthropic/claude-sonnet-5.5 (the provider's alias of the same model) ; served by another connection (finding below) |
| pi | deepseek-v4.1-flash | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as deepseek/deepseek-v4.1-flash (the provider's alias of the same model) |
| pi | gpt-5.4 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as openai/gpt-5.4 (the provider's alias of the same model) |
| pi | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | served as openai/gpt-6-astra (the provider's alias of the same model) |
| pi | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as openai/gpt-6-luna (the provider's alias of the same model) |
| pi | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as openai/gpt-6-sol (the provider's alias of the same model) |
| pi | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 100% of input |
| pi | grok-4.20 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as x-ai/grok-4.20-beta (the provider's alias of the same model) |
| pi | grok-4.3 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as x-ai/grok-4.3 (the provider's alias of the same model) |
| pi | grok-4.5 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as x-ai/grok-4.5 (the provider's alias of the same model) |
| pi | grok-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as x-ai/grok-4.6 (the provider's alias of the same model) |
| pi | grok-build-0.1 | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as x-ai/grok-build-0.1 (the provider's alias of the same model) |
| pi | hunyuan-4-preview | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as tencent/hy4-preview (the provider's alias of the same model) |
| pi | nemotron-3-super | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) |
| pi | nemotron-3.5-lightning | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) |
| pi | qwen3.7-plus | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as qwen/qwen3.7-plus (the provider's alias of the same model) |
| pi | qwen3.8-flash | pass | pass | pass (gpt-6-astra) | pass | pass | My TokenRouter | served as qwen/qwen3.8-flash (the provider's alias of the same model) |
| qwen | claude-sonnet-4.6 | pass | pass | pass (qwen3.7-max) | pass | pass | My TokenRouter | served as claude-sonnet-4-6 (the provider's alias of the same model) |
| qwen | claude-sonnet-5.5 | pass | pass | pass (claude-opus-5) | pass | pass | My TokenRouter, Vercel AI Gateway | served as claude-sonnet-5-5 (the provider's alias of the same model) ; served by another connection (finding below) |
| qwen | deepseek-v4.1-flash | pass | pass | pass (qwen3.7-max) | pass | pass | My TokenRouter | served as deepseek-flash (the provider's alias of the same model) |
| qwen | gpt-5.4 | pass | pass | pass (qwen3.7-max) | pass | pass | My TokenRouter | served as gpt-5.4-2026-03-05 (the provider's alias of the same model) |
| qwen | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | |
| qwen | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | My TokenRouter | |
| qwen | gpt-6.1-sol | FAIL | n/a | n/a | n/a | n/a | My TokenRouter | 1 turn(s) unlabelled (finding below) ; first: The turn failed: API Error: 400 Unsupported value: 'reasoning_effort' does not support 'none' with this model. Supported values are: 'low', |
| qwen | grok-4.20 | pass | pass | pass (gpt-5.6-sol) | FAIL | pass | My TokenRouter | served as x-ai/grok-4.20 (the provider's alias of the same model) ; artifact: no file card (files: none); Create a file named hello-qwen.txt containing exactly the word HELLO, then reply DONE. QWEN CODE DONE |
| qwen | grok-4.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as x-ai/grok-4.3 (the provider's alias of the same model) |
| qwen | grok-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | |
| qwen | grok-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | |
| qwen | grok-build-0.1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as x-ai/grok-build-0.1 (the provider's alias of the same model) |
| qwen | hunyuan-4-preview | pass | pass | pass (qwen3.7-max) | pass | pass | My TokenRouter | served as tencent/hy4-preview (the provider's alias of the same model) |
| qwen | nemotron-3-super | pass | pass | pass (qwen3.7-max) | pass | pass | My TokenRouter | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) |
| qwen | nemotron-3.5-lightning | pass | pass | pass (qwen3.7-max) | pass | pass | My TokenRouter | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) |
| qwen | qwen3.7-plus | pass | pass | pass (qwen3.7-max) | pass | pass | My TokenRouter | served as qwen/qwen3.7-plus (the provider's alias of the same model) |
| qwen | qwen3.8-flash | pass | pass | pass (qwen3.7-max) | pass | pass | My TokenRouter | |
| vibe | claude-fable-5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| vibe | claude-fable-5-1 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| vibe | claude-haiku-4.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| vibe | claude-opus-4.8 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| vibe | claude-opus-5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| vibe | claude-opus-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| vibe | claude-sonnet-4.6 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| vibe | claude-sonnet-5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| vibe | claude-sonnet-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | no prompt cache reads (finding below) |
| vibe | deepseek-v4-pro | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 33% of input |
| vibe | gemini-3-flash-preview | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 15% of input |
| vibe | gemini-3.1-pro-preview | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 14% of input |
| vibe | gemini-3.5-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 14% of input |
| vibe | gemini-3.5-flash-lite | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 18% of input |
| vibe | gemini-3.6-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 11% of input |
| vibe | gemini-3.7-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 0% of input |
| vibe | gemini-3.8-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 7% of input |
| vibe | glm-5.3 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 53% of input |
| vibe | glm-5.3-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 49% of input |
| vibe | gpt-5.2 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 30% of input |
| vibe | gpt-5.4 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 34% of input |
| vibe | gpt-5.4-mini | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 32% of input |
| vibe | gpt-5.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 31% of input |
| vibe | gpt-5.6-sol | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 17% of input |
| vibe | gpt-5.6-terra | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 17% of input |
| vibe | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 17% of input |
| vibe | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | My TokenRouter | cache 29% of input |
| vibe | grok-4.20 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 18% of input |
| vibe | grok-4.3 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 24% of input |
| vibe | grok-4.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 25% of input |
| vibe | grok-4.6 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 16% of input |
| vibe | grok-build-0.1 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 47% of input |
| vibe | hunyuan-4-preview | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 24% of input |
| vibe | kimi-k2.7-code | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 52% of input |
| vibe | mistral-medium-3.5 | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 36% of input |
| vibe | nemotron-3-super | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 0% of input |
| vibe | nemotron-3.5-lightning | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 0% of input |
| vibe | qwen3.7-max | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 36% of input |
| vibe | qwen3.7-plus | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 39% of input |
| vibe | qwen3.8-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 47% of input |
| vibe | qwen3.8-max | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 57% of input |
| vibe | step-3.7-flash | pass | pass | pass (gpt-6-sol) | pass | pass | My TokenRouter | cache 32% of input |
268 pairs, 1229 of 1231 scenario runs passed; 21 pairs served by another connection or as another model are findings, not counted.
Findings, pairs served by a connection other than the one under test or as a model other than the id asked for:
- aider x claude-sonnet-5.5: served by integration:My TokenRouter, integration:Vercel AI Gateway (turn records: integration:My TokenRouter, integration:Vercel AI Gateway)
- cheetahclaws x claude-sonnet-5.5: served by integration:My TokenRouter, integration:Vercel AI Gateway (turn records: integration:My TokenRouter, integration:Vercel AI Gateway)
- cheetahclaws x gpt-6.1-sol: 1 turn(s) report no served model, so rule 2 could not judge them
- claude-code x claude-sonnet-5.5: served by integration:My TokenRouter, integration:Vercel AI Gateway (turn records: integration:My TokenRouter, integration:Vercel AI Gateway)
- cline x claude-sonnet-5.5: served by integration:My TokenRouter, integration:Vercel AI Gateway (turn records: integration:My TokenRouter, integration:Vercel AI Gateway)
- cline x gpt-6.1-sol: 1 turn(s) report no served model, so rule 2 could not judge them
- codex x gpt-6.1-sol: 4 turn(s) report no served model, so rule 2 could not judge them
- dcode x glm-5.3: 4 turn(s) report no served model, so rule 2 could not judge them
- docker-agent x claude-fable-5: no prompt cache reads over 25128 input tokens (every call paid full price)
- docker-agent x claude-haiku-4.5: no prompt cache reads over 20272 input tokens (every call paid full price)
- docker-agent x claude-opus-4.8: no prompt cache reads over 25158 input tokens (every call paid full price)
- docker-agent x claude-opus-5: no prompt cache reads over 25088 input tokens (every call paid full price)
- docker-agent x claude-opus-5.5: no prompt cache reads over 25150 input tokens (every call paid full price)
- docker-agent x claude-sonnet-4.6: no prompt cache reads over 20215 input tokens (every call paid full price)
- docker-agent x claude-sonnet-5: no prompt cache reads over 25618 input tokens (every call paid full price)
- docker-agent x claude-sonnet-5.5: no prompt cache reads over 25196 input tokens (every call paid full price)
- dsh x claude-sonnet-5.5: served by integration:My TokenRouter, integration:Vercel AI Gateway (turn records: integration:My TokenRouter, integration:Vercel AI Gateway)
- dsh x gpt-6.1-sol: 4 turn(s) report no served model, so rule 2 could not judge them
- fast-agent x claude-fable-5: no prompt cache reads over 18454 input tokens (every call paid full price)
- fast-agent x claude-fable-5-1: no prompt cache reads over 16261 input tokens (every call paid full price)
- fast-agent x claude-haiku-4.5: no prompt cache reads over 14440 input tokens (every call paid full price)
- fast-agent x claude-opus-4.7: no prompt cache reads over 18667 input tokens (every call paid full price)
- fast-agent x claude-opus-4.8: no prompt cache reads over 16588 input tokens (every call paid full price)
- fast-agent x claude-opus-5: no prompt cache reads over 18228 input tokens (every call paid full price)
- fast-agent x claude-opus-5.5: no prompt cache reads over 16365 input tokens (every call paid full price)
- fast-agent x claude-sonnet-4.6: no prompt cache reads over 14463 input tokens (every call paid full price)
- fast-agent x claude-sonnet-5: no prompt cache reads over 18788 input tokens (every call paid full price)
- fast-agent x claude-sonnet-5.5: no prompt cache reads over 16313 input tokens (every call paid full price)
- goose x claude-sonnet-5.5: served by integration:My TokenRouter (turn records: integration:My TokenRouter)
- hermes x claude-sonnet-5.5: served by integration:My TokenRouter, integration:Vercel AI Gateway (turn records: integration:My TokenRouter, integration:Vercel AI Gateway)
- kimi x claude-sonnet-5.5: served by integration:My TokenRouter, integration:Vercel AI Gateway (turn records: integration:My TokenRouter, integration:Vercel AI Gateway)
- kimi x gpt-6.1-sol: 1 turn(s) report no served model, so rule 2 could not judge them
- omp x claude-sonnet-5.5: served by integration:My TokenRouter (turn records: integration:My TokenRouter)
- omp x gpt-6.1-sol: 1 turn(s) report no served model, so rule 2 could not judge them
- opencode x claude-sonnet-5.5: served by integration:My TokenRouter, integration:Vercel AI Gateway (turn records: integration:My TokenRouter, integration:Vercel AI Gateway)
- openhands x claude-sonnet-5.5: served by integration:My TokenRouter, integration:Vercel AI Gateway (turn records: integration:My TokenRouter, integration:Vercel AI Gateway)
- pi x claude-sonnet-5.5: served by integration:My TokenRouter, integration:Vercel AI Gateway (turn records: integration:My TokenRouter, integration:Vercel AI Gateway)
- qwen x claude-sonnet-5.5: served by integration:My TokenRouter, integration:Vercel AI Gateway (turn records: integration:My TokenRouter, integration:Vercel AI Gateway)
- qwen x gpt-6.1-sol: 1 turn(s) report no served model, so rule 2 could not judge them
- vibe x claude-fable-5: no prompt cache reads over 77249 input tokens (every call paid full price)
- vibe x claude-fable-5-1: no prompt cache reads over 77606 input tokens (every call paid full price)
- vibe x claude-haiku-4.5: no prompt cache reads over 61375 input tokens (every call paid full price)
- vibe x claude-opus-4.8: no prompt cache reads over 77462 input tokens (every call paid full price)
- vibe x claude-opus-5: no prompt cache reads over 76995 input tokens (every call paid full price)
- vibe x claude-opus-5.5: no prompt cache reads over 77334 input tokens (every call paid full price)
- vibe x claude-sonnet-4.6: no prompt cache reads over 61300 input tokens (every call paid full price)
- vibe x claude-sonnet-5: no prompt cache reads over 77995 input tokens (every call paid full price)
- vibe x claude-sonnet-5.5: no prompt cache reads over 77418 input tokens (every call paid full price)
Not run in this column, 614 pairs the provider serves that the harness did not run, with the reason:
- aider x claude-fable-5: not run, not run in this column
- aider x claude-fable-5-1: not run, not run in this column
- aider x claude-haiku-4.5: not run, not run in this column
- aider x claude-opus-4.7: not run, not run in this column
- aider x claude-opus-4.8: not run, not run in this column
- aider x claude-opus-5: not run, not run in this column
- aider x claude-opus-5.5: not run, not run in this column
- aider x claude-sonnet-4.6: not run, not run in this column
- aider x claude-sonnet-5: not run, not run in this column
- aider x deepseek-v4-pro: not run, not run in this column
- aider x deepseek-v4.1-flash: not run, not run in this column
- aider x gemini-3-flash-preview: not run, not run in this column
- aider x gemini-3.1-pro-preview: not run, not run in this column
- aider x gemini-3.5-flash: not run, not run in this column
- aider x gemini-3.5-flash-lite: not run, not run in this column
- aider x gemini-3.6-flash: not run, not run in this column
- aider x gemini-3.7-flash: not run, not run in this column
- aider x gemini-3.8-flash: not run, not run in this column
- aider x glm-5.3: not run, not run in this column
- aider x glm-5.3-flash: not run, not run in this column
- aider x gpt-5.2: not run, not run in this column
- aider x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- aider x gpt-5.4: not run, not run in this column
- aider x gpt-5.4-mini: not run, not run in this column
- aider x gpt-5.5: not run, not run in this column
- aider x gpt-5.6-luna: not run, not run in this column
- aider x gpt-5.6-sol: not run, not run in this column
- aider x gpt-5.6-terra: not run, not run in this column
- aider x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- aider x grok-4.20: not run, not run in this column
- aider x grok-4.3: not run, not run in this column
- aider x grok-4.5: not run, not run in this column
- aider x grok-4.6: not run, not run in this column
- aider x grok-build-0.1: not run, not run in this column
- aider x hunyuan-4-preview: not run, not run in this column
- aider x kimi-k2.7-code: not run, not run in this column
- aider x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x mistral-medium-3.5: not run, not run in this column
- aider x nemotron-3-super: not run, not run in this column
- aider x nemotron-3.5-lightning: not run, not run in this column
- aider x qwen3.7-max: not run, not run in this column
- aider x qwen3.7-plus: not run, not run in this column
- aider x qwen3.8-flash: not run, not run in this column
- aider x qwen3.8-max: not run, not run in this column
- aider x step-3.7-flash: not run, not run in this column
- cheetahclaws x claude-fable-5: not run, not run in this column
- cheetahclaws x claude-fable-5-1: not run, not run in this column
- cheetahclaws x claude-haiku-4.5: not run, not run in this column
- cheetahclaws x claude-opus-4.7: not run, not run in this column
- cheetahclaws x claude-opus-4.8: not run, not run in this column
- cheetahclaws x claude-opus-5: not run, not run in this column
- cheetahclaws x claude-opus-5.5: not run, not run in this column
- cheetahclaws x claude-sonnet-4.6: not run, not run in this column
- cheetahclaws x claude-sonnet-5: not run, not run in this column
- cheetahclaws x deepseek-v4-pro: not run, not run in this column
- cheetahclaws x deepseek-v4.1-flash: not run, not run in this column
- cheetahclaws x gemini-3-flash-preview: not run, not run in this column
- cheetahclaws x gemini-3.1-pro-preview: not run, not run in this column
- cheetahclaws x gemini-3.5-flash: not run, not run in this column
- cheetahclaws x gemini-3.5-flash-lite: not run, not run in this column
- cheetahclaws x gemini-3.6-flash: not run, not run in this column
- cheetahclaws x gemini-3.7-flash: not run, not run in this column
- cheetahclaws x gemini-3.8-flash: not run, not run in this column
- cheetahclaws x glm-5.3: not run, not run in this column
- cheetahclaws x glm-5.3-flash: not run, not run in this column
- cheetahclaws x gpt-5.2: not run, not run in this column
- cheetahclaws x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cheetahclaws x gpt-5.4: not run, not run in this column
- cheetahclaws x gpt-5.4-mini: not run, not run in this column
- cheetahclaws x gpt-5.5: not run, not run in this column
- cheetahclaws x gpt-5.6-luna: not run, not run in this column
- cheetahclaws x gpt-5.6-sol: not run, not run in this column
- cheetahclaws x gpt-5.6-terra: not run, not run in this column
- cheetahclaws x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cheetahclaws x grok-4.20: not run, not run in this column
- cheetahclaws x grok-4.3: not run, not run in this column
- cheetahclaws x grok-4.5: not run, not run in this column
- cheetahclaws x grok-4.6: not run, not run in this column
- cheetahclaws x grok-build-0.1: not run, not run in this column
- cheetahclaws x hunyuan-4-preview: not run, not run in this column
- cheetahclaws x kimi-k2.7-code: not run, not run in this column
- cheetahclaws x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x mistral-medium-3.5: not run, not run in this column
- cheetahclaws x nemotron-3-super: not run, not run in this column
- cheetahclaws x nemotron-3.5-lightning: not run, not run in this column
- cheetahclaws x qwen3.7-max: not run, not run in this column
- cheetahclaws x qwen3.7-plus: not run, not run in this column
- cheetahclaws x qwen3.8-flash: not run, not run in this column
- cheetahclaws x qwen3.8-max: not run, not run in this column
- cheetahclaws x step-3.7-flash: not run, not run in this column
- claude-code x claude-fable-5: not run, not run in this column
- claude-code x claude-fable-5-1: not run, not run in this column
- claude-code x claude-haiku-4.5: not run, not run in this column
- claude-code x claude-opus-4.7: not run, not run in this column
- claude-code x claude-opus-4.8: not run, not run in this column
- claude-code x claude-opus-5: not run, not run in this column
- claude-code x claude-opus-5.5: not run, not run in this column
- claude-code x claude-sonnet-5: not run, not run in this column
- claude-code x deepseek-v4-pro: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x deepseek-v4.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-3-flash-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-3.1-pro-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-3.5-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-3.5-flash-lite: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-3.6-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x glm-5.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x glm-5.3-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.2: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.3-codex: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.4: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.4-mini: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.6-luna: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.6-sol: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.6-terra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-6-astra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-6-luna: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-6-sol: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-6.1-sol: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x grok-4.20: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x grok-4.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x grok-4.6: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x grok-build-0.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x mistral-medium-3.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x nemotron-3-super: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x nemotron-3.5-lightning: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x qwen3.7-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x qwen3.7-plus: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x qwen3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x qwen3.8-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x step-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x claude-fable-5: not run, not run in this column
- cline x claude-fable-5-1: not run, not run in this column
- cline x claude-haiku-4.5: not run, not run in this column
- cline x claude-opus-4.7: not run, not run in this column
- cline x claude-opus-4.8: not run, not run in this column
- cline x claude-opus-5: not run, not run in this column
- cline x claude-opus-5.5: not run, not run in this column
- cline x claude-sonnet-5: not run, not run in this column
- cline x deepseek-v4-pro: not run, not run in this column
- cline x gemini-3-flash-preview: not run, not run in this column
- cline x gemini-3.1-pro-preview: not run, not run in this column
- cline x gemini-3.5-flash: not run, not run in this column
- cline x gemini-3.5-flash-lite: not run, not run in this column
- cline x gemini-3.6-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x gemini-3.7-flash: not run, not run in this column
- cline x gemini-3.8-flash: not run, not run in this column
- cline x glm-5.3: not run, not run in this column
- cline x glm-5.3-flash: not run, not run in this column
- cline x gpt-5.2: not run, not run in this column
- cline x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x gpt-5.4-mini: not run, not run in this column
- cline x gpt-5.5: not run, not run in this column
- cline x gpt-5.6-luna: not run, not run in this column
- cline x gpt-5.6-sol: not run, not run in this column
- cline x gpt-5.6-terra: not run, not run in this column
- cline x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x kimi-k2.7-code: not run, not run in this column
- cline x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x mistral-medium-3.5: not run, not run in this column
- cline x qwen3.7-max: not run, not run in this column
- cline x qwen3.8-max: not run, not run in this column
- cline x step-3.7-flash: not run, not run in this column
- codex x claude-fable-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-fable-5-1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-haiku-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-4.8: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-4.6: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4-pro: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3-flash-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.1-pro-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.5-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.5-flash-lite: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.6-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x glm-5.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x glm-5.3-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gpt-5.2: not run, not run in this column
- codex x gpt-5.3-codex: not run, not run in this column
- codex x gpt-5.4: not run, not run in this column
- codex x gpt-5.4-mini: not run, not run in this column
- codex x gpt-5.5: not run, not run in this column
- codex x gpt-5.6-luna: not run, not run in this column
- codex x gpt-5.6-sol: not run, not run in this column
- codex x gpt-5.6-terra: not run, not run in this column
- codex x grok-4.20: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.6: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-build-0.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x mistral-medium-3.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x nemotron-3-super: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x nemotron-3.5-lightning: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-plus: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x step-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x claude-fable-5: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x claude-fable-5-1: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x claude-haiku-4.5: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x claude-opus-4.7: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x claude-opus-4.8: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x claude-opus-5: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x claude-opus-5.5: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x claude-sonnet-4.6: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x claude-sonnet-5: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x claude-sonnet-5.5: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x gemini-3.5-flash: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x gemini-3.5-flash-lite: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x gemini-3.6-flash: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x gemini-3.7-flash: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x gemini-3.8-flash: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x glm-5.3-flash: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- dcode x gpt-5.3-codex: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x gpt-6-astra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x nemotron-3.5-lightning: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x claude-fable-5-1: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- docker-agent x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- docker-agent x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- docker-agent x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- docker-agent x grok-4.20: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x grok-build-0.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x nemotron-3.5-lightning: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x claude-fable-5: not run, not run in this column
- dsh x claude-fable-5-1: not run, not run in this column
- dsh x claude-haiku-4.5: not run, not run in this column
- dsh x claude-opus-4.7: not run, not run in this column
- dsh x claude-opus-4.8: not run, not run in this column
- dsh x claude-opus-5: not run, not run in this column
- dsh x claude-opus-5.5: not run, not run in this column
- dsh x claude-sonnet-4.6: not run, not run in this column
- dsh x claude-sonnet-5: not run, not run in this column
- dsh x deepseek-v4-pro: not run, not run in this column
- dsh x deepseek-v4.1-flash: not run, not run in this column
- dsh x gemini-3-flash-preview: not run, not run in this column
- dsh x gemini-3.1-pro-preview: not run, not run in this column
- dsh x gemini-3.5-flash: not run, not run in this column
- dsh x gemini-3.5-flash-lite: not run, not run in this column
- dsh x gemini-3.6-flash: not run, not run in this column
- dsh x gemini-3.7-flash: not run, not run in this column
- dsh x gemini-3.8-flash: not run, not run in this column
- dsh x glm-5.3: not run, not run in this column
- dsh x glm-5.3-flash: not run, not run in this column
- dsh x gpt-5.2: not run, not run in this column
- dsh x gpt-5.3-codex: not run, not run in this column
- dsh x gpt-5.4: not run, not run in this column
- dsh x gpt-5.4-mini: not run, not run in this column
- dsh x gpt-5.5: not run, not run in this column
- dsh x gpt-5.6-luna: not run, not run in this column
- dsh x gpt-5.6-sol: not run, not run in this column
- dsh x gpt-5.6-terra: not run, not run in this column
- dsh x grok-4.20: not run, not run in this column
- dsh x grok-4.3: not run, not run in this column
- dsh x grok-4.5: not run, not run in this column
- dsh x grok-4.6: not run, not run in this column
- dsh x grok-build-0.1: not run, not run in this column
- dsh x hunyuan-4-preview: not run, not run in this column
- dsh x kimi-k2.7-code: not run, not run in this column
- dsh x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x mistral-medium-3.5: not run, not run in this column
- dsh x nemotron-3-super: not run, not run in this column
- dsh x nemotron-3.5-lightning: not run, not run in this column
- dsh x qwen3.7-max: not run, not run in this column
- dsh x qwen3.7-plus: not run, not run in this column
- dsh x qwen3.8-flash: not run, not run in this column
- dsh x qwen3.8-max: not run, not run in this column
- dsh x step-3.7-flash: not run, not run in this column
- fast-agent x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- fast-agent x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- fast-agent x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- fast-agent x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x nemotron-3-super: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x nemotron-3.5-lightning: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x qwen3.8-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-fable-5: not run, not run in this column
- goose x claude-fable-5-1: not run, not run in this column
- goose x claude-haiku-4.5: not run, not run in this column
- goose x claude-opus-4.7: not run, not run in this column
- goose x claude-opus-4.8: not run, not run in this column
- goose x claude-opus-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-opus-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-sonnet-4.6: not run, not run in this column
- goose x claude-sonnet-5: not run, not run in this column
- goose x deepseek-v4-pro: not run, not run in this column
- goose x deepseek-v4.1-flash: not run, not run in this column
- goose x gemini-3-flash-preview: not run, not run in this column
- goose x gemini-3.1-pro-preview: not run, not run in this column
- goose x gemini-3.5-flash: not run, not run in this column
- goose x gemini-3.5-flash-lite: not run, not run in this column
- goose x gemini-3.6-flash: not run, not run in this column
- goose x gemini-3.7-flash: not run, not run in this column
- goose x gemini-3.8-flash: not run, not run in this column
- goose x glm-5.3: not run, not run in this column
- goose x glm-5.3-flash: not run, not run in this column
- goose x gpt-5.2: not run, not run in this column
- goose x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-5.4: not run, not run in this column
- goose x gpt-5.4-mini: not run, not run in this column
- goose x gpt-5.5: not run, not run in this column
- goose x gpt-5.6-luna: not run, not run in this column
- goose x gpt-5.6-sol: not run, not run in this column
- goose x gpt-5.6-terra: not run, not run in this column
- goose x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-luna: not run, not run in this column
- goose x gpt-6-sol: not run, not run in this column
- goose x grok-4.20: not run, not run in this column
- goose x grok-4.3: not run, not run in this column
- goose x grok-4.5: not run, not run in this column
- goose x grok-4.6: not run, not run in this column
- goose x grok-build-0.1: not run, not run in this column
- goose x hunyuan-4-preview: not run, not run in this column
- goose x kimi-k2.7-code: not run, not run in this column
- goose x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x mistral-medium-3.5: not run, not run in this column
- goose x nemotron-3-super: not run, not run in this column
- goose x nemotron-3.5-lightning: not run, not run in this column
- goose x qwen3.7-max: not run, not run in this column
- goose x qwen3.7-plus: not run, not run in this column
- goose x qwen3.8-flash: not run, not run in this column
- goose x qwen3.8-max: not run, not run in this column
- goose x step-3.7-flash: not run, not run in this column
- hermes x claude-fable-5: not run, not run in this column
- hermes x claude-fable-5-1: not run, not run in this column
- hermes x claude-haiku-4.5: not run, not run in this column
- hermes x claude-opus-4.7: not run, not run in this column
- hermes x claude-opus-4.8: not run, not run in this column
- hermes x claude-opus-5: not run, not run in this column
- hermes x claude-opus-5.5: not run, not run in this column
- hermes x claude-sonnet-5: not run, not run in this column
- hermes x deepseek-v4-pro: not run, not run in this column
- hermes x gemini-3-flash-preview: not run, not run in this column
- hermes x gemini-3.1-pro-preview: not run, not run in this column
- hermes x gemini-3.5-flash: not run, not run in this column
- hermes x gemini-3.5-flash-lite: not run, not run in this column
- hermes x gemini-3.6-flash: not run, not run in this column
- hermes x gemini-3.7-flash: not run, not run in this column
- hermes x gemini-3.8-flash: not run, not run in this column
- hermes x glm-5.3: not run, not run in this column
- hermes x glm-5.3-flash: not run, not run in this column
- hermes x gpt-5.2: not run, not run in this column
- hermes x gpt-5.3-codex: not run, not run in this column
- hermes x gpt-5.4-mini: not run, not run in this column
- hermes x gpt-5.5: not run, not run in this column
- hermes x gpt-5.6-luna: not run, not run in this column
- hermes x gpt-5.6-sol: not run, not run in this column
- hermes x gpt-5.6-terra: not run, not run in this column
- hermes x kimi-k2.7-code: not run, not run in this column
- hermes x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- hermes x mistral-medium-3.5: not run, not run in this column
- hermes x qwen3.7-max: not run, not run in this column
- hermes x qwen3.8-max: not run, not run in this column
- hermes x step-3.7-flash: not run, not run in this column
- kimi x claude-fable-5: not run, not run in this column
- kimi x claude-fable-5-1: not run, not run in this column
- kimi x claude-haiku-4.5: not run, not run in this column
- kimi x claude-opus-4.7: not run, not run in this column
- kimi x claude-opus-4.8: not run, not run in this column
- kimi x claude-opus-5: not run, not run in this column
- kimi x claude-opus-5.5: not run, not run in this column
- kimi x claude-sonnet-4.6: not run, not run in this column
- kimi x claude-sonnet-5: not run, not run in this column
- kimi x deepseek-v4-pro: not run, not run in this column
- kimi x deepseek-v4.1-flash: not run, not run in this column
- kimi x gemini-3-flash-preview: not run, not run in this column
- kimi x gemini-3.1-pro-preview: not run, not run in this column
- kimi x gemini-3.5-flash: not run, not run in this column
- kimi x gemini-3.5-flash-lite: not run, not run in this column
- kimi x gemini-3.6-flash: not run, not run in this column
- kimi x gemini-3.7-flash: not run, not run in this column
- kimi x gemini-3.8-flash: not run, not run in this column
- kimi x glm-5.3: not run, not run in this column
- kimi x glm-5.3-flash: not run, not run in this column
- kimi x gpt-5.2: not run, not run in this column
- kimi x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- kimi x gpt-5.4: not run, not run in this column
- kimi x gpt-5.4-mini: not run, not run in this column
- kimi x gpt-5.5: not run, not run in this column
- kimi x gpt-5.6-luna: not run, not run in this column
- kimi x gpt-5.6-sol: not run, not run in this column
- kimi x gpt-5.6-terra: not run, not run in this column
- kimi x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- kimi x grok-4.20: not run, not run in this column
- kimi x grok-4.3: not run, not run in this column
- kimi x grok-4.5: not run, not run in this column
- kimi x grok-4.6: not run, not run in this column
- kimi x grok-build-0.1: not run, not run in this column
- kimi x hunyuan-4-preview: not run, not run in this column
- kimi x kimi-k2.7-code: not run, not run in this column
- kimi x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x mistral-medium-3.5: not run, not run in this column
- kimi x nemotron-3-super: not run, not run in this column
- kimi x nemotron-3.5-lightning: not run, not run in this column
- kimi x qwen3.7-max: not run, not run in this column
- kimi x qwen3.7-plus: not run, not run in this column
- kimi x qwen3.8-flash: not run, not run in this column
- kimi x qwen3.8-max: not run, not run in this column
- kimi x step-3.7-flash: not run, not run in this column
- omp x claude-fable-5: not run, not run in this column
- omp x claude-fable-5-1: not run, not run in this column
- omp x claude-haiku-4.5: not run, not run in this column
- omp x claude-opus-4.7: not run, not run in this column
- omp x claude-opus-4.8: not run, not run in this column
- omp x claude-opus-5: not run, not run in this column
- omp x claude-opus-5.5: not run, not run in this column
- omp x claude-sonnet-4.6: not run, not run in this column
- omp x claude-sonnet-5: not run, not run in this column
- omp x deepseek-v4-pro: not run, not run in this column
- omp x deepseek-v4.1-flash: not run, not run in this column
- omp x gemini-3-flash-preview: not run, not run in this column
- omp x gemini-3.1-pro-preview: not run, not run in this column
- omp x gemini-3.5-flash: not run, not run in this column
- omp x gemini-3.5-flash-lite: not run, not run in this column
- omp x gemini-3.6-flash: not run, not run in this column
- omp x gemini-3.7-flash: not run, not run in this column
- omp x gemini-3.8-flash: not run, not run in this column
- omp x glm-5.3: not run, not run in this column
- omp x glm-5.3-flash: not run, not run in this column
- omp x gpt-5.2: not run, not run in this column
- omp x gpt-5.3-codex: not run, not run in this column
- omp x gpt-5.4: not run, not run in this column
- omp x gpt-5.4-mini: not run, not run in this column
- omp x gpt-5.5: not run, not run in this column
- omp x gpt-5.6-luna: not run, not run in this column
- omp x gpt-5.6-sol: not run, not run in this column
- omp x gpt-5.6-terra: not run, not run in this column
- omp x grok-4.20: not run, not run in this column
- omp x grok-4.3: not run, not run in this column
- omp x grok-4.5: not run, not run in this column
- omp x grok-4.6: not run, not run in this column
- omp x grok-build-0.1: not run, not run in this column
- omp x hunyuan-4-preview: not run, not run in this column
- omp x kimi-k2.7-code: not run, not run in this column
- omp x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x mistral-medium-3.5: not run, not run in this column
- omp x nemotron-3-super: not run, not run in this column
- omp x nemotron-3.5-lightning: not run, not run in this column
- omp x qwen3.7-max: not run, not run in this column
- omp x qwen3.7-plus: not run, not run in this column
- omp x qwen3.8-flash: not run, not run in this column
- omp x qwen3.8-max: not run, not run in this column
- omp x step-3.7-flash: not run, not run in this column
- opencode x claude-fable-5: not run, not run in this column
- opencode x claude-fable-5-1: not run, not run in this column
- opencode x claude-haiku-4.5: not run, not run in this column
- opencode x claude-opus-4.7: not run, not run in this column
- opencode x claude-opus-4.8: not run, not run in this column
- opencode x claude-opus-5: not run, not run in this column
- opencode x claude-opus-5.5: not run, not run in this column
- opencode x claude-sonnet-5: not run, not run in this column
- opencode x deepseek-v4-pro: not run, not run in this column
- opencode x gemini-3-flash-preview: not run, not run in this column
- opencode x gemini-3.1-pro-preview: not run, not run in this column
- opencode x gemini-3.5-flash: not run, not run in this column
- opencode x gemini-3.5-flash-lite: not run, not run in this column
- opencode x gemini-3.6-flash: not run, not run in this column
- opencode x gemini-3.7-flash: not run, not run in this column
- opencode x gemini-3.8-flash: not run, not run in this column
- opencode x glm-5.3: not run, not run in this column
- opencode x glm-5.3-flash: not run, not run in this column
- opencode x gpt-5.2: not run, not run in this column
- opencode x gpt-5.3-codex: not run, not run in this column
- opencode x gpt-5.4-mini: not run, not run in this column
- opencode x gpt-5.5: not run, not run in this column
- opencode x gpt-5.6-luna: not run, not run in this column
- opencode x gpt-5.6-sol: not run, not run in this column
- opencode x gpt-5.6-terra: not run, not run in this column
- opencode x kimi-k2.7-code: not run, not run in this column
- opencode x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x mistral-medium-3.5: not run, not run in this column
- opencode x qwen3.7-max: not run, not run in this column
- opencode x qwen3.8-max: not run, not run in this column
- opencode x step-3.7-flash: not run, not run in this column
- openhands x claude-fable-5: not run, not run in this column
- openhands x claude-fable-5-1: not run, not run in this column
- openhands x claude-haiku-4.5: not run, not run in this column
- openhands x claude-opus-4.7: not run, not run in this column
- openhands x claude-opus-4.8: not run, not run in this column
- openhands x claude-opus-5: not run, not run in this column
- openhands x claude-opus-5.5: not run, not run in this column
- openhands x claude-sonnet-4.6: not run, not run in this column
- openhands x claude-sonnet-5: not run, not run in this column
- openhands x deepseek-v4-pro: not run, not run in this column
- openhands x deepseek-v4.1-flash: not run, not run in this column
- openhands x gemini-3-flash-preview: not run, not run in this column
- openhands x gemini-3.1-pro-preview: not run, not run in this column
- openhands x gemini-3.5-flash: not run, not run in this column
- openhands x gemini-3.5-flash-lite: not run, not run in this column
- openhands x gemini-3.6-flash: not run, not run in this column
- openhands x gemini-3.7-flash: not run, not run in this column
- openhands x gemini-3.8-flash: not run, not run in this column
- openhands x glm-5.3: not run, not run in this column
- openhands x glm-5.3-flash: not run, not run in this column
- openhands x gpt-5.2: not run, not run in this column
- openhands x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- openhands x gpt-5.4: not run, not run in this column
- openhands x gpt-5.4-mini: not run, not run in this column
- openhands x gpt-5.5: not run, not run in this column
- openhands x gpt-5.6-luna: not run, not run in this column
- openhands x gpt-5.6-sol: not run, not run in this column
- openhands x gpt-5.6-terra: not run, not run in this column
- openhands x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- openhands x grok-4.20: not run, not run in this column
- openhands x grok-4.3: not run, not run in this column
- openhands x grok-4.5: not run, not run in this column
- openhands x grok-4.6: not run, not run in this column
- openhands x grok-build-0.1: not run, not run in this column
- openhands x hunyuan-4-preview: not run, not run in this column
- openhands x kimi-k2.7-code: not run, not run in this column
- openhands x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x mistral-medium-3.5: not run, not run in this column
- openhands x nemotron-3-super: not run, not run in this column
- openhands x nemotron-3.5-lightning: not run, not run in this column
- openhands x qwen3.7-max: not run, not run in this column
- openhands x qwen3.7-plus: not run, not run in this column
- openhands x qwen3.8-flash: not run, not run in this column
- openhands x qwen3.8-max: not run, not run in this column
- openhands x step-3.7-flash: not run, not run in this column
- pi x claude-fable-5: not run, not run in this column
- pi x claude-fable-5-1: not run, not run in this column
- pi x claude-haiku-4.5: not run, not run in this column
- pi x claude-opus-4.7: not run, not run in this column
- pi x claude-opus-4.8: not run, not run in this column
- pi x claude-opus-5: not run, not run in this column
- pi x claude-opus-5.5: not run, not run in this column
- pi x claude-sonnet-5: not run, not run in this column
- pi x deepseek-v4-pro: not run, not run in this column
- pi x gemini-3-flash-preview: not run, not run in this column
- pi x gemini-3.1-pro-preview: not run, not run in this column
- pi x gemini-3.5-flash: not run, not run in this column
- pi x gemini-3.5-flash-lite: not run, not run in this column
- pi x gemini-3.6-flash: not run, not run in this column
- pi x gemini-3.7-flash: not run, not run in this column
- pi x gemini-3.8-flash: not run, not run in this column
- pi x glm-5.3: not run, not run in this column
- pi x glm-5.3-flash: not run, not run in this column
- pi x gpt-5.2: not run, not run in this column
- pi x gpt-5.3-codex: not run, not run in this column
- pi x gpt-5.4-mini: not run, not run in this column
- pi x gpt-5.5: not run, not run in this column
- pi x gpt-5.6-luna: not run, not run in this column
- pi x gpt-5.6-sol: not run, not run in this column
- pi x gpt-5.6-terra: not run, not run in this column
- pi x kimi-k2.7-code: not run, not run in this column
- pi x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x mistral-medium-3.5: not run, not run in this column
- pi x qwen3.7-max: not run, not run in this column
- pi x qwen3.8-max: not run, not run in this column
- pi x step-3.7-flash: not run, not run in this column
- qwen x claude-fable-5: not run, not run in this column
- qwen x claude-fable-5-1: not run, not run in this column
- qwen x claude-haiku-4.5: not run, not run in this column
- qwen x claude-opus-4.7: not run, not run in this column
- qwen x claude-opus-4.8: not run, not run in this column
- qwen x claude-opus-5: not run, not run in this column
- qwen x claude-opus-5.5: not run, not run in this column
- qwen x claude-sonnet-5: not run, not run in this column
- qwen x deepseek-v4-pro: not run, not run in this column
- qwen x gemini-3-flash-preview: not run, not run in this column
- qwen x gemini-3.1-pro-preview: not run, not run in this column
- qwen x gemini-3.5-flash: not run, not run in this column
- qwen x gemini-3.5-flash-lite: not run, not run in this column
- qwen x gemini-3.6-flash: not run, not run in this column
- qwen x gemini-3.7-flash: not run, not run in this column
- qwen x gemini-3.8-flash: not run, not run in this column
- qwen x glm-5.3: not run, not run in this column
- qwen x glm-5.3-flash: not run, not run in this column
- qwen x gpt-5.2: not run, not run in this column
- qwen x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x gpt-5.4-mini: not run, not run in this column
- qwen x gpt-5.5: not run, not run in this column
- qwen x gpt-5.6-luna: not run, not run in this column
- qwen x gpt-5.6-sol: not run, not run in this column
- qwen x gpt-5.6-terra: not run, not run in this column
- qwen x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x kimi-k2.7-code: not run, not run in this column
- qwen x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x mistral-medium-3.5: not run, not run in this column
- qwen x qwen3.7-max: not run, not run in this column
- qwen x qwen3.8-max: not run, not run in this column
- qwen x step-3.7-flash: not run, not run in this column
- vibe x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x deepseek-v4.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- vibe x gpt-5.6-luna: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- vibe x gpt-6.1-sol: not run, this provider's connection refuses the pair for this harness (the gateway's channel rules)
- vibe x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
Provider: vercel
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
|---|---|---|---|---|---|---|---|---|
| agentzero | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) |
| aider | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) |
| aider | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-luna (the provider's alias of the same model) |
| aider | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-sol (the provider's alias of the same model) |
| aider | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 51% of input |
| cheetahclaws | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) |
| cheetahclaws | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-luna (the provider's alias of the same model) |
| cheetahclaws | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-sol (the provider's alias of the same model) |
| cheetahclaws | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 17% of input |
| claude-code | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) ; cache 99% of input |
| cline | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) ; 4 turn(s) unlabelled (finding below) |
| cline | deepseek-v4.1-flash | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | |
| cline | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | |
| cline | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | Vercel AI Gateway | |
| cline | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | 4 turn(s) unlabelled (finding below) ; cache 65% of input |
| cline | grok-4.1-fast | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | first: The turn failed: Stream error occurred |
| cline | grok-4.20 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | |
| cline | grok-4.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | |
| cline | grok-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | |
| cline | grok-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | |
| cline | grok-build-0.1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | |
| cline | hunyuan-4-preview | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | |
| cline | llama-3.3-70b | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | first: The turn failed: undefined: This model doesn't support tool use in streaming mode. |
| cline | llama-4-maverick | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | first: The turn failed: Tool calling is not supported for model: meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 |
| cline | llama-4-scout | pass | pass | pass (gpt-5.6-sol) | FAIL | pass | Vercel AI Gateway | artifact: no file card (files: none); ning exactly the word HELLO, then reply DONE. CLINE Used 2 tools editor("/data/workspaces/hsess42a22c26ecb546b9a |
| cline | muse-glimmer-30b | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | |
| cline | muse-spark-1.1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | |
| cline | muse-spark-1.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | |
| cline | muse-spark-1.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | |
| cline | nemotron-3-super | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | |
| cline | nemotron-3.5-lightning | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | |
| cline | qwen3.7-plus | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | |
| cline | qwen3.8-27b | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | |
| cline | qwen3.8-flash | pass | pass | pass (gpt-5.4) | pass | pass | Vercel AI Gateway | |
| codex | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | |
| codex | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| codex | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| codex | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | 4 turn(s) unlabelled (finding below) ; cache 52% of input |
| dcode | claude-fable-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) |
| dcode | claude-fable-5-1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) |
| dcode | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) |
| dcode | claude-haiku-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) |
| dcode | claude-opus-4.7 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) |
| dcode | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) |
| dcode | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) |
| dcode | claude-opus-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) |
| dcode | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) |
| dcode | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) |
| dcode | claude-sonnet-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) |
| dcode | deepseek-v4-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 63% of input |
| dcode | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 71% of input |
| dcode | deepseek-v4.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 75% of input |
| dcode | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 50% of input |
| dcode | gemini-3.1-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 46% of input |
| dcode | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| dcode | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 62% of input |
| dcode | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 46% of input |
| dcode | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 37% of input |
| dcode | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 49% of input |
| dcode | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 49% of input |
| dcode | gemini-4-argon | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 55% of input |
| dcode | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 67% of input |
| dcode | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 17% of input |
| dcode | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 63% of input |
| dcode | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 60% of input |
| dcode | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 66% of input |
| dcode | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 60% of input |
| dcode | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 66% of input |
| dcode | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 66% of input |
| dcode | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 71% of input |
| dcode | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 66% of input |
| dcode | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 66% of input |
| dcode | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | cache 66% of input |
| dcode | grok-4.20 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 67% of input |
| dcode | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 21% of input |
| dcode | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 51% of input |
| dcode | grok-build-0.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 61% of input |
| dcode | kimi-k3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 50% of input |
| dcode | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| dcode | muse-spark-1.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 50% of input |
| dcode | muse-spark-1.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 17% of input |
| dcode | muse-wonder-1.0 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 33% of input |
| dcode | nemotron-3-super | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| dcode | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 64% of input |
| dcode | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 66% of input |
| dcode | qwen3.8-27b | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 61% of input |
| dcode | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 66% of input |
| dcode | qwen3.8-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 73% of input |
| dcode | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 49% of input |
| docker-agent | claude-fable-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 73% of input |
| docker-agent | claude-fable-5-1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 73% of input |
| docker-agent | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 18% of input |
| docker-agent | claude-haiku-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 73% of input |
| docker-agent | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 73% of input |
| docker-agent | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 73% of input |
| docker-agent | claude-opus-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 73% of input |
| docker-agent | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 72% of input |
| docker-agent | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 73% of input |
| docker-agent | claude-sonnet-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 73% of input |
| docker-agent | deepseek-v4-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 68% of input |
| docker-agent | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 70% of input |
| docker-agent | deepseek-v4.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 68% of input |
| docker-agent | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| docker-agent | gemini-3.1-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| docker-agent | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| docker-agent | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| docker-agent | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| docker-agent | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| docker-agent | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| docker-agent | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| docker-agent | gemini-4-argon | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 13% of input |
| docker-agent | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 53% of input |
| docker-agent | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| docker-agent | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 80% of input |
| docker-agent | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 73% of input |
| docker-agent | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 75% of input |
| docker-agent | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 57% of input |
| docker-agent | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 67% of input |
| docker-agent | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 67% of input |
| docker-agent | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 67% of input |
| docker-agent | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 67% of input |
| docker-agent | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 72% of input |
| docker-agent | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | cache 72% of input |
| docker-agent | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 54% of input |
| docker-agent | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 66% of input |
| docker-agent | hunyuan-4-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 84% of input |
| docker-agent | kimi-k2.7-code | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 68% of input |
| docker-agent | kimi-k3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 17% of input |
| docker-agent | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| docker-agent | muse-glimmer-30b | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 78% of input |
| docker-agent | muse-spark-1.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 53% of input |
| docker-agent | muse-spark-1.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 80% of input |
| docker-agent | muse-spark-1.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 45% of input |
| docker-agent | muse-wonder-1.0 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 62% of input |
| docker-agent | nemotron-3-super | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| docker-agent | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 70% of input |
| docker-agent | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 70% of input |
| docker-agent | qwen3.8-27b | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 67% of input |
| docker-agent | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 70% of input |
| docker-agent | qwen3.8-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 88% of input |
| docker-agent | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 50% of input |
| dsh | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | 4 turn(s) unlabelled (finding below) ; cache 100% of input |
| dsh | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | |
| dsh | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| dsh | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| dsh | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | 4 turn(s) unlabelled (finding below) ; cache 100% of input |
| fast-agent | claude-fable-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 54% of input |
| fast-agent | claude-fable-5-1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 70% of input |
| fast-agent | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) |
| fast-agent | claude-haiku-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 70% of input |
| fast-agent | claude-opus-4.7 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 71% of input |
| fast-agent | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 70% of input |
| fast-agent | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 54% of input |
| fast-agent | claude-opus-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 70% of input |
| fast-agent | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 68% of input |
| fast-agent | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 54% of input |
| fast-agent | claude-sonnet-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 70% of input |
| fast-agent | deepseek-v4-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | deepseek-v4.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gemini-3.1-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gpt-5.6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | grok-4.20 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | grok-build-0.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | kimi-k3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | muse-glimmer-30b | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | muse-spark-1.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | muse-spark-1.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | muse-spark-1.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | muse-wonder-1.0 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| fast-agent | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | |
| goose | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) |
| goose | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 62% of input |
| grok | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) |
| hermes | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) |
| hermes | deepseek-v4.1-flash | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | |
| hermes | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 0% of input |
| hermes | grok-4.1-fast | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | grok-4.20 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | grok-4.3 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | grok-4.5 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | grok-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | grok-build-0.1 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | hunyuan-4-preview | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | kimi-k2.7-code | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | ling-3.0-flash | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | llama-3.3-70b | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | first: The turn failed: exit_code=0 |
| hermes | llama-4-maverick | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | first: The turn failed: exit_code=0 |
| hermes | llama-4-scout | pass | pass | pass (gpt-6-astra) | FAIL | FAIL | Vercel AI Gateway | artifact: no file card (files: none); Create a file named hello-hermes.txt containing exactly the word HELLO, then reply DONE. HERMES write_file(path= ; recycle: answered without M1-llama-4-scout: What exact word did I ask you to reply with in my very first message of this task? Reply with just that w |
| hermes | muse-glimmer-30b | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | muse-spark-1.1 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | muse-spark-1.2 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | muse-spark-1.3 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | nemotron-3-super | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | nemotron-3.5-lightning | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | qwen3.7-plus | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | qwen3.8-27b | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| hermes | qwen3.8-flash | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | |
| kilo | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) ; cache 69% of input |
| kimi | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) |
| kimi | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-luna (the provider's alias of the same model) |
| kimi | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-sol (the provider's alias of the same model) |
| kimi | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 50% of input |
| minimax | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) |
| minimax | minimax-m3.1-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 73% of input |
| omp | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) ; cache 100% of input |
| omp | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-astra (the provider's alias of the same model) |
| omp | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-luna (the provider's alias of the same model) |
| omp | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-sol (the provider's alias of the same model) |
| omp | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | 1 turn(s) unlabelled (finding below) ; served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 100% of input |
| opencode | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) ; cache 100% of input |
| opencode | deepseek-v4.1-flash | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as deepseek/deepseek-v4.1-flash (the provider's alias of the same model) |
| opencode | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-astra (the provider's alias of the same model) |
| opencode | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-luna (the provider's alias of the same model) |
| opencode | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-sol (the provider's alias of the same model) |
| opencode | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 100% of input |
| opencode | grok-4.1-fast | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.1-fast-reasoning (finding below) |
| opencode | grok-4.20 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.20-reasoning (the provider's alias of the same model) |
| opencode | grok-4.3 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.3 (the provider's alias of the same model) |
| opencode | grok-4.5 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.5 (the provider's alias of the same model) |
| opencode | grok-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.6 (the provider's alias of the same model) |
| opencode | grok-build-0.1 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-build-0.1 (the provider's alias of the same model) |
| opencode | hunyuan-4-preview | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as tencent/hy4-preview (the provider's alias of the same model) |
| opencode | llama-3.3-70b | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | first: The turn failed: undefined: The maximum tokens you requested exceeds the model limit of 8192. Try again with a maximum tokens value that is |
| opencode | llama-4-maverick | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | served as meta/llama-4-maverick (the provider's alias of the same model) ; first: The turn failed: Tool calling is not supported for model: meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 |
| opencode | llama-4-scout | pass | pass | pass (gpt-6-astra) | FAIL | FAIL | Vercel AI Gateway | served as meta/llama-4-scout (the provider's alias of the same model) ; artifact: no file card (files: none); e word HELLO, then reply DONE. OPENCODE Used a tool write(content="HELLO", filePath="/data/workspaces/hsess427c8 ; recycle: answered without M1-llama-4-scout: What exact word did I ask you to reply with in my very first message of this task? Reply with just that w |
| opencode | muse-glimmer-30b | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as meta/muse-glimmer-30b (the provider's alias of the same model) |
| opencode | muse-spark-1.1 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as meta/muse-spark-1.1 (the provider's alias of the same model) |
| opencode | muse-spark-1.2 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as meta/muse-spark-1.2 (the provider's alias of the same model) |
| opencode | muse-spark-1.3 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as meta/muse-spark-1.3 (the provider's alias of the same model) |
| opencode | nemotron-3-super | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) |
| opencode | nemotron-3.5-lightning | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) |
| opencode | qwen3.7-plus | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.7-plus (the provider's alias of the same model) |
| opencode | qwen3.8-27b | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.8-27b (the provider's alias of the same model) |
| opencode | qwen3.8-flash | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.8-flash (the provider's alias of the same model) |
| openhands | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) ; cache 69% of input |
| openhands | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-luna (the provider's alias of the same model) |
| openhands | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-sol (the provider's alias of the same model) |
| openhands | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 66% of input |
| pi | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) ; cache 83% of input |
| pi | deepseek-v4.1-flash | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as deepseek/deepseek-v4.1-flash (the provider's alias of the same model) |
| pi | gpt-6-astra | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-astra (the provider's alias of the same model) |
| pi | gpt-6-luna | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-luna (the provider's alias of the same model) |
| pi | gpt-6-sol | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-sol (the provider's alias of the same model) |
| pi | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 100% of input |
| pi | grok-4.1-fast | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.1-fast-reasoning (finding below) |
| pi | grok-4.20 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.20-reasoning (the provider's alias of the same model) |
| pi | grok-4.3 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.3 (the provider's alias of the same model) |
| pi | grok-4.5 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.5 (the provider's alias of the same model) |
| pi | grok-4.6 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.6 (the provider's alias of the same model) |
| pi | grok-build-0.1 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as spacexai/grok-build-0.1 (the provider's alias of the same model) |
| pi | hunyuan-4-preview | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as tencent/hy4-preview (the provider's alias of the same model) |
| pi | llama-3.3-70b | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | served as meta/llama-3.3-70b (the provider's alias of the same model) ; first: The turn failed: 400: {"message":"undefined: The maximum tokens you requested exceeds the model limit of 8192. Try again with a maximum toke |
| pi | llama-4-maverick | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | served as meta/llama-4-maverick (the provider's alias of the same model) ; first: The turn failed: 405: {"message":"Tool calling is not supported for model: meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8","type":"AI_API |
| pi | llama-4-scout | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as meta/llama-4-scout (the provider's alias of the same model) |
| pi | muse-glimmer-30b | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as meta/muse-glimmer-30b (the provider's alias of the same model) |
| pi | muse-spark-1.1 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as meta/muse-spark-1.1 (the provider's alias of the same model) |
| pi | muse-spark-1.2 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as meta/muse-spark-1.2 (the provider's alias of the same model) |
| pi | muse-spark-1.3 | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as meta/muse-spark-1.3 (the provider's alias of the same model) |
| pi | nemotron-3-super | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) |
| pi | nemotron-3.5-lightning | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) |
| pi | qwen3.7-plus | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.7-plus (the provider's alias of the same model) |
| pi | qwen3.8-27b | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.8-27b (the provider's alias of the same model) |
| pi | qwen3.8-flash | pass | pass | pass (gpt-6-astra) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.8-flash (the provider's alias of the same model) |
| qwen | claude-haiku-5.5 | pass | pass | pass (claude-haiku-4.5) | pass | pass | Vercel AI Gateway | no prompt cache reads (finding below) ; served as anthropic/claude-haiku-5.5 (the provider's alias of the same model) |
| qwen | deepseek-v4.1-flash | pass | pass | pass (qwen3.7-max) | pass | pass | Vercel AI Gateway | served as deepseek/deepseek-v4.1-flash (the provider's alias of the same model) |
| qwen | gpt-6-luna | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-luna (the provider's alias of the same model) |
| qwen | gpt-6-sol | pass | pass | pass (gpt-6-luna) | pass | pass | Vercel AI Gateway | served as openai/gpt-6-sol (the provider's alias of the same model) |
| qwen | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-6.1-sol (the provider's alias of the same model) ; cache 40% of input |
| qwen | grok-4.1-fast | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | served as spacexai/grok-4.1-fast-reasoning (finding below) ; first: The turn failed: API Error: Stream error occurred |
| qwen | grok-4.20 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.20-reasoning (the provider's alias of the same model) |
| qwen | grok-4.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.3 (the provider's alias of the same model) |
| qwen | grok-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.5 (the provider's alias of the same model) |
| qwen | grok-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as spacexai/grok-4.6 (the provider's alias of the same model) |
| qwen | grok-build-0.1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as spacexai/grok-build-0.1 (the provider's alias of the same model) |
| qwen | hunyuan-4-preview | pass | pass | pass (qwen3.7-max) | pass | pass | Vercel AI Gateway | served as tencent/hy4-preview (the provider's alias of the same model) |
| qwen | llama-3.3-70b | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | first: The turn failed: API Error: 400 undefined: The maximum tokens you requested exceeds the model limit of 8192. Try again with a maximum tokens |
| qwen | llama-4-maverick | FAIL | n/a | n/a | n/a | n/a | Vercel AI Gateway | first: The turn failed: API Error: 405 Tool calling is not supported for model: meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 |
| qwen | llama-4-scout | pass | pass | pass (gpt-5.6-sol) | FAIL | FAIL | Vercel AI Gateway | served as meta/llama-4-scout (the provider's alias of the same model) ; artifact: The turn failed: API Error: Model stream ended with empty response text. ; recycle: answered without M1-llama-4-scout: very first message of this task? Reply with just that word. QWEN CODE DONE <write_file for file_path '/d |
| qwen | muse-glimmer-30b | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as meta/muse-glimmer-30b (the provider's alias of the same model) |
| qwen | muse-spark-1.1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as meta/muse-spark-1.1 (the provider's alias of the same model) |
| qwen | muse-spark-1.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as meta/muse-spark-1.2 (the provider's alias of the same model) |
| qwen | muse-spark-1.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as meta/muse-spark-1.3 (the provider's alias of the same model) |
| qwen | nemotron-3-super | pass | pass | pass (qwen3.7-max) | pass | pass | Vercel AI Gateway | served as nvidia/nemotron-3-super-120b-a12b (the provider's alias of the same model) |
| qwen | nemotron-3.5-lightning | pass | pass | pass (qwen3.7-max) | pass | pass | Vercel AI Gateway | served as nvidia/nemotron-3.5-lightning (the provider's alias of the same model) |
| qwen | qwen3.7-plus | pass | pass | pass (qwen3.7-max) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.7-plus (the provider's alias of the same model) |
| qwen | qwen3.8-27b | pass | pass | pass (qwen3.7-max) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.8-27b (the provider's alias of the same model) |
| qwen | qwen3.8-flash | pass | pass | pass (qwen3.7-max) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.8-flash (the provider's alias of the same model) |
| vibe | claude-fable-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 71% of input |
| vibe | claude-fable-5-1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 71% of input |
| vibe | claude-haiku-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 69% of input |
| vibe | claude-haiku-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 71% of input |
| vibe | claude-opus-4.8 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 71% of input |
| vibe | claude-opus-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 71% of input |
| vibe | claude-opus-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 71% of input |
| vibe | claude-sonnet-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 69% of input |
| vibe | claude-sonnet-5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 71% of input |
| vibe | claude-sonnet-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 71% of input |
| vibe | deepseek-v4-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 33% of input |
| vibe | deepseek-v4-pro | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 30% of input |
| vibe | gemini-3-flash-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 15% of input |
| vibe | gemini-3.1-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| vibe | gemini-3.1-pro-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 7% of input |
| vibe | gemini-3.5-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 15% of input |
| vibe | gemini-3.5-flash-lite | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| vibe | gemini-3.6-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 11% of input |
| vibe | gemini-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 15% of input |
| vibe | gemini-3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 7% of input |
| vibe | gemini-4-argon | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 13% of input |
| vibe | glm-5.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 41% of input |
| vibe | glm-5.3-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| vibe | gpt-5.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 40% of input |
| vibe | gpt-5.4 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 41% of input |
| vibe | gpt-5.4-mini | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 37% of input |
| vibe | gpt-5.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 48% of input |
| vibe | gpt-5.6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 17% of input |
| vibe | gpt-5.6-terra | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 17% of input |
| vibe | gpt-6-luna | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 17% of input |
| vibe | gpt-6-sol | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 17% of input |
| vibe | gpt-6.1-sol | pass | pass | pass (gpt-6-sol) | pass | pass | Vercel AI Gateway | cache 17% of input |
| vibe | grok-4.20 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 30% of input |
| vibe | grok-4.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 7% of input |
| vibe | grok-4.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 25% of input |
| vibe | grok-4.6 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 24% of input |
| vibe | grok-build-0.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 53% of input |
| vibe | hunyuan-4-preview | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 68% of input |
| vibe | kimi-k2.7-code | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 40% of input |
| vibe | kimi-k3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 12% of input |
| vibe | mistral-medium-3.5 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 15% of input |
| vibe | muse-glimmer-30b | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 29% of input |
| vibe | muse-spark-1.1 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 1% of input |
| vibe | muse-spark-1.2 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 18% of input |
| vibe | muse-spark-1.3 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 1% of input |
| vibe | muse-wonder-1.0 | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| vibe | nemotron-3-super | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 0% of input |
| vibe | qwen3.7-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 30% of input |
| vibe | qwen3.7-plus | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 32% of input |
| vibe | qwen3.8-27b | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 14% of input |
| vibe | qwen3.8-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 48% of input |
| vibe | qwen3.8-max | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 17% of input |
| vibe | step-3.7-flash | pass | pass | pass (gpt-6.1-sol) | pass | pass | Vercel AI Gateway | cache 32% of input |
368 pairs, 1733 of 1751 scenario runs passed; 9 pairs served by another connection or as another model are findings, not counted.
Findings, pairs served by a connection other than the one under test or as a model other than the id asked for:
- agentzero x claude-haiku-5.5: no prompt cache reads over 56653 input tokens (every call paid full price)
- aider x claude-haiku-5.5: no prompt cache reads over 24423 input tokens (every call paid full price)
- cheetahclaws x claude-haiku-5.5: no prompt cache reads over 75392 input tokens (every call paid full price)
- cline x claude-haiku-5.5: 4 turn(s) report no served model, so rule 2 could not judge them
- cline x claude-haiku-5.5: no prompt cache reads over 51624 input tokens (every call paid full price)
- cline x gpt-6.1-sol: 4 turn(s) report no served model, so rule 2 could not judge them
- codex x gpt-6.1-sol: 4 turn(s) report no served model, so rule 2 could not judge them
- dcode x claude-fable-5: no prompt cache reads over 90026 input tokens (every call paid full price)
- dcode x claude-fable-5-1: no prompt cache reads over 89700 input tokens (every call paid full price)
- dcode x claude-haiku-4.5: no prompt cache reads over 69576 input tokens (every call paid full price)
- dcode x claude-haiku-5.5: no prompt cache reads over 90213 input tokens (every call paid full price)
- dcode x claude-opus-4.7: no prompt cache reads over 92080 input tokens (every call paid full price)
- dcode x claude-opus-4.8: no prompt cache reads over 89708 input tokens (every call paid full price)
- dcode x claude-opus-5: no prompt cache reads over 89610 input tokens (every call paid full price)
- dcode x claude-opus-5.5: no prompt cache reads over 90038 input tokens (every call paid full price)
- dcode x claude-sonnet-4.6: no prompt cache reads over 70411 input tokens (every call paid full price)
- dcode x claude-sonnet-5: no prompt cache reads over 90494 input tokens (every call paid full price)
- dcode x claude-sonnet-5.5: no prompt cache reads over 90264 input tokens (every call paid full price)
- dsh x claude-haiku-5.5: 4 turn(s) report no served model, so rule 2 could not judge them
- dsh x gpt-6.1-sol: 4 turn(s) report no served model, so rule 2 could not judge them
- fast-agent x claude-haiku-4.5: no prompt cache reads over 14387 input tokens (every call paid full price)
- goose x claude-haiku-5.5: no prompt cache reads over 179095 input tokens (every call paid full price)
- grok x claude-haiku-5.5: no prompt cache reads over 95311 input tokens (every call paid full price)
- hermes x claude-haiku-5.5: no prompt cache reads over 124056 input tokens (every call paid full price)
- kimi x claude-haiku-5.5: no prompt cache reads over 184677 input tokens (every call paid full price)
- minimax x claude-haiku-5.5: no prompt cache reads over 83965 input tokens (every call paid full price)
- omp x gpt-6.1-sol: 1 turn(s) report no served model, so rule 2 could not judge them
- opencode x grok-4.1-fast: served as spacexai/grok-4.1-fast-reasoning (the CLI reports the model it ran)
- pi x grok-4.1-fast: served as spacexai/grok-4.1-fast-reasoning (the CLI reports the model it ran)
- qwen x claude-haiku-5.5: no prompt cache reads over 569929 input tokens (every call paid full price)
- qwen x grok-4.1-fast: served as spacexai/grok-4.1-fast-reasoning (the CLI reports the model it ran)
Not run in this column, 1065 pairs the provider serves that the harness did not run, with the reason:
- agentzero x claude-fable-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x claude-fable-5-1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x claude-haiku-4.5: not run, not run in this column
- agentzero x claude-opus-4.7: not run, not run in this column
- agentzero x claude-opus-4.8: not run, not run in this column
- agentzero x claude-opus-5: not run, not run in this column
- agentzero x claude-opus-5.5: not run, not run in this column
- agentzero x claude-sonnet-4.6: not run, not run in this column
- agentzero x claude-sonnet-5: not run, not run in this column
- agentzero x claude-sonnet-5.5: not run, not run in this column
- agentzero x deepseek-v4-flash: not run, not run in this column
- agentzero x deepseek-v4-pro: not run, not run in this column
- agentzero x deepseek-v4.1-flash: not run, not run in this column
- agentzero x gemini-3-flash-preview: not run, not run in this column
- agentzero x gemini-3.1-flash-lite: not run, not run in this column
- agentzero x gemini-3.1-pro-preview: not run, not run in this column
- agentzero x gemini-3.5-flash: not run, not run in this column
- agentzero x gemini-3.5-flash-lite: not run, not run in this column
- agentzero x gemini-3.6-flash: not run, not run in this column
- agentzero x gemini-3.7-flash: not run, not run in this column
- agentzero x gemini-3.8-flash: not run, not run in this column
- agentzero x gemini-4-argon: not run, not run in this column
- agentzero x glm-5.3: not run, not run in this column
- agentzero x glm-5.3-flash: not run, not run in this column
- agentzero x gpt-5.2: not run, not run in this column
- agentzero x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- agentzero x gpt-5.4: not run, not run in this column
- agentzero x gpt-5.4-mini: not run, not run in this column
- agentzero x gpt-5.5: not run, not run in this column
- agentzero x gpt-5.6-luna: not run, not run in this column
- agentzero x gpt-5.6-sol: not run, not run in this column
- agentzero x gpt-5.6-terra: not run, not run in this column
- agentzero x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- agentzero x gpt-6-luna: not run, not run in this column
- agentzero x gpt-6-sol: not run, not run in this column
- agentzero x gpt-6.1-sol: not run, not run in this column
- agentzero x grok-4.20: not run, not run in this column
- agentzero x grok-4.3: not run, not run in this column
- agentzero x grok-4.5: not run, not run in this column
- agentzero x grok-4.6: not run, not run in this column
- agentzero x grok-build-0.1: not run, not run in this column
- agentzero x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x hunyuan-4-preview: not run, not run in this column
- agentzero x kimi-k2.7-code: not run, not run in this column
- agentzero x kimi-k3: not run, not run in this column
- agentzero x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x mistral-medium-3.5: not run, not run in this column
- agentzero x muse-glimmer-30b: not run, not run in this column
- agentzero x muse-spark-1.1: not run, not run in this column
- agentzero x muse-spark-1.2: not run, not run in this column
- agentzero x muse-spark-1.3: not run, not run in this column
- agentzero x muse-wonder-1.0: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x nemotron-3-super: not run, not run in this column
- agentzero x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- agentzero x qwen3.7-max: not run, not run in this column
- agentzero x qwen3.7-plus: not run, not run in this column
- agentzero x qwen3.8-27b: not run, not run in this column
- agentzero x qwen3.8-flash: not run, not run in this column
- agentzero x qwen3.8-max: not run, not run in this column
- agentzero x step-3.7-flash: not run, not run in this column
- aider x claude-fable-5: not run, not run in this column
- aider x claude-fable-5-1: not run, not run in this column
- aider x claude-haiku-4.5: not run, not run in this column
- aider x claude-opus-4.7: not run, not run in this column
- aider x claude-opus-4.8: not run, not run in this column
- aider x claude-opus-5: not run, not run in this column
- aider x claude-opus-5.5: not run, not run in this column
- aider x claude-sonnet-4.6: not run, not run in this column
- aider x claude-sonnet-5: not run, not run in this column
- aider x claude-sonnet-5.5: not run, not run in this column
- aider x deepseek-v4-flash: not run, not run in this column
- aider x deepseek-v4-pro: not run, not run in this column
- aider x deepseek-v4.1-flash: not run, not run in this column
- aider x gemini-3-flash-preview: not run, not run in this column
- aider x gemini-3.1-flash-lite: not run, not run in this column
- aider x gemini-3.1-pro-preview: not run, not run in this column
- aider x gemini-3.5-flash: not run, not run in this column
- aider x gemini-3.5-flash-lite: not run, not run in this column
- aider x gemini-3.6-flash: not run, not run in this column
- aider x gemini-3.7-flash: not run, not run in this column
- aider x gemini-3.8-flash: not run, not run in this column
- aider x gemini-4-argon: not run, not run in this column
- aider x glm-5.3: not run, not run in this column
- aider x glm-5.3-flash: not run, not run in this column
- aider x gpt-5.2: not run, not run in this column
- aider x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- aider x gpt-5.4: not run, not run in this column
- aider x gpt-5.4-mini: not run, not run in this column
- aider x gpt-5.5: not run, not run in this column
- aider x gpt-5.6-luna: not run, not run in this column
- aider x gpt-5.6-sol: not run, not run in this column
- aider x gpt-5.6-terra: not run, not run in this column
- aider x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- aider x grok-4.20: not run, not run in this column
- aider x grok-4.3: not run, not run in this column
- aider x grok-4.5: not run, not run in this column
- aider x grok-4.6: not run, not run in this column
- aider x grok-build-0.1: not run, not run in this column
- aider x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x hunyuan-4-preview: not run, not run in this column
- aider x kimi-k2.7-code: not run, not run in this column
- aider x kimi-k3: not run, not run in this column
- aider x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x mistral-medium-3.5: not run, not run in this column
- aider x muse-glimmer-30b: not run, not run in this column
- aider x muse-spark-1.1: not run, not run in this column
- aider x muse-spark-1.2: not run, not run in this column
- aider x muse-spark-1.3: not run, not run in this column
- aider x muse-wonder-1.0: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x nemotron-3-super: not run, not run in this column
- aider x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- aider x qwen3.7-max: not run, not run in this column
- aider x qwen3.7-plus: not run, not run in this column
- aider x qwen3.8-27b: not run, not run in this column
- aider x qwen3.8-flash: not run, not run in this column
- aider x qwen3.8-max: not run, not run in this column
- aider x step-3.7-flash: not run, not run in this column
- cheetahclaws x claude-fable-5: not run, not run in this column
- cheetahclaws x claude-fable-5-1: not run, not run in this column
- cheetahclaws x claude-haiku-4.5: not run, not run in this column
- cheetahclaws x claude-opus-4.7: not run, not run in this column
- cheetahclaws x claude-opus-4.8: not run, not run in this column
- cheetahclaws x claude-opus-5: not run, not run in this column
- cheetahclaws x claude-opus-5.5: not run, not run in this column
- cheetahclaws x claude-sonnet-4.6: not run, not run in this column
- cheetahclaws x claude-sonnet-5: not run, not run in this column
- cheetahclaws x claude-sonnet-5.5: not run, not run in this column
- cheetahclaws x deepseek-v4-flash: not run, not run in this column
- cheetahclaws x deepseek-v4-pro: not run, not run in this column
- cheetahclaws x deepseek-v4.1-flash: not run, not run in this column
- cheetahclaws x gemini-3-flash-preview: not run, not run in this column
- cheetahclaws x gemini-3.1-flash-lite: not run, not run in this column
- cheetahclaws x gemini-3.1-pro-preview: not run, not run in this column
- cheetahclaws x gemini-3.5-flash: not run, not run in this column
- cheetahclaws x gemini-3.5-flash-lite: not run, not run in this column
- cheetahclaws x gemini-3.6-flash: not run, not run in this column
- cheetahclaws x gemini-3.7-flash: not run, not run in this column
- cheetahclaws x gemini-3.8-flash: not run, not run in this column
- cheetahclaws x gemini-4-argon: not run, not run in this column
- cheetahclaws x glm-5.3: not run, not run in this column
- cheetahclaws x glm-5.3-flash: not run, not run in this column
- cheetahclaws x gpt-5.2: not run, not run in this column
- cheetahclaws x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cheetahclaws x gpt-5.4: not run, not run in this column
- cheetahclaws x gpt-5.4-mini: not run, not run in this column
- cheetahclaws x gpt-5.5: not run, not run in this column
- cheetahclaws x gpt-5.6-luna: not run, not run in this column
- cheetahclaws x gpt-5.6-sol: not run, not run in this column
- cheetahclaws x gpt-5.6-terra: not run, not run in this column
- cheetahclaws x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cheetahclaws x grok-4.20: not run, not run in this column
- cheetahclaws x grok-4.3: not run, not run in this column
- cheetahclaws x grok-4.5: not run, not run in this column
- cheetahclaws x grok-4.6: not run, not run in this column
- cheetahclaws x grok-build-0.1: not run, not run in this column
- cheetahclaws x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x hunyuan-4-preview: not run, not run in this column
- cheetahclaws x kimi-k2.7-code: not run, not run in this column
- cheetahclaws x kimi-k3: not run, not run in this column
- cheetahclaws x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x mistral-medium-3.5: not run, not run in this column
- cheetahclaws x muse-glimmer-30b: not run, not run in this column
- cheetahclaws x muse-spark-1.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x muse-spark-1.2: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x muse-spark-1.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x muse-wonder-1.0: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x nemotron-3-super: not run, not run in this column
- cheetahclaws x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cheetahclaws x qwen3.7-max: not run, not run in this column
- cheetahclaws x qwen3.7-plus: not run, not run in this column
- cheetahclaws x qwen3.8-27b: not run, not run in this column
- cheetahclaws x qwen3.8-flash: not run, not run in this column
- cheetahclaws x qwen3.8-max: not run, not run in this column
- cheetahclaws x step-3.7-flash: not run, not run in this column
- claude-code x claude-fable-5: not run, not run in this column
- claude-code x claude-fable-5-1: not run, not run in this column
- claude-code x claude-haiku-4.5: not run, not run in this column
- claude-code x claude-opus-4.7: not run, not run in this column
- claude-code x claude-opus-4.8: not run, not run in this column
- claude-code x claude-opus-5: not run, not run in this column
- claude-code x claude-opus-5.5: not run, not run in this column
- claude-code x claude-sonnet-4.6: not run, not run in this column
- claude-code x claude-sonnet-5: not run, not run in this column
- claude-code x claude-sonnet-5.5: not run, not run in this column
- claude-code x deepseek-v4-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x deepseek-v4-pro: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x deepseek-v4.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-3-flash-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-3.1-flash-lite: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-3.1-pro-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-3.5-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-3.5-flash-lite: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-3.6-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gemini-4-argon: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x glm-5.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x glm-5.3-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.2: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.3-codex: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.4: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.4-mini: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.6-luna: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.6-sol: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-5.6-terra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-6-astra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-6-luna: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-6-sol: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x gpt-6.1-sol: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x grok-4.20: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x grok-4.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x grok-4.6: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x grok-build-0.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x kimi-k3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x mistral-medium-3.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x muse-glimmer-30b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x muse-spark-1.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x muse-spark-1.2: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x muse-spark-1.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x muse-wonder-1.0: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x nemotron-3-super: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x qwen3.7-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x qwen3.7-plus: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x qwen3.8-27b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x qwen3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x qwen3.8-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- claude-code x step-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x claude-fable-5: not run, not run in this column
- cline x claude-fable-5-1: not run, not run in this column
- cline x claude-haiku-4.5: not run, not run in this column
- cline x claude-opus-4.7: not run, not run in this column
- cline x claude-opus-4.8: not run, not run in this column
- cline x claude-opus-5: not run, not run in this column
- cline x claude-opus-5.5: not run, not run in this column
- cline x claude-sonnet-4.6: not run, not run in this column
- cline x claude-sonnet-5: not run, not run in this column
- cline x claude-sonnet-5.5: not run, not run in this column
- cline x deepseek-v4-flash: not run, not run in this column
- cline x deepseek-v4-pro: not run, not run in this column
- cline x gemini-3-flash-preview: not run, not run in this column
- cline x gemini-3.1-flash-lite: not run, not run in this column
- cline x gemini-3.1-pro-preview: not run, not run in this column
- cline x gemini-3.5-flash: not run, not run in this column
- cline x gemini-3.5-flash-lite: not run, not run in this column
- cline x gemini-3.6-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x gemini-3.7-flash: not run, not run in this column
- cline x gemini-3.8-flash: not run, not run in this column
- cline x gemini-4-argon: not run, not run in this column
- cline x glm-5.3: not run, not run in this column
- cline x glm-5.3-flash: not run, not run in this column
- cline x gpt-5.2: not run, not run in this column
- cline x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x gpt-5.4: not run, not run in this column
- cline x gpt-5.4-mini: not run, not run in this column
- cline x gpt-5.5: not run, not run in this column
- cline x gpt-5.6-luna: not run, not run in this column
- cline x gpt-5.6-sol: not run, not run in this column
- cline x gpt-5.6-terra: not run, not run in this column
- cline x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- cline x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x kimi-k2.7-code: not run, not run in this column
- cline x kimi-k3: not run, not run in this column
- cline x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x mistral-medium-3.5: not run, not run in this column
- cline x muse-wonder-1.0: not run, not run in this column
- cline x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- cline x qwen3.7-max: not run, not run in this column
- cline x qwen3.8-max: not run, not run in this column
- cline x step-3.7-flash: not run, not run in this column
- codex x claude-fable-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-fable-5-1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-haiku-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-haiku-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-4.8: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-opus-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-4.6: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x claude-sonnet-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4-pro: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x deepseek-v4.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3-flash-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.1-flash-lite: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.1-pro-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.5-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.5-flash-lite: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.6-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gemini-4-argon: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x glm-5.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x glm-5.3-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x gpt-5.2: not run, not run in this column
- codex x gpt-5.3-codex: not run, not run in this column
- codex x gpt-5.4: not run, not run in this column
- codex x gpt-5.4-mini: not run, not run in this column
- codex x gpt-5.5: not run, not run in this column
- codex x gpt-5.6-luna: not run, not run in this column
- codex x gpt-5.6-sol: not run, not run in this column
- codex x gpt-5.6-terra: not run, not run in this column
- codex x grok-4.20: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-4.6: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x grok-build-0.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x kimi-k3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x mistral-medium-3.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-glimmer-30b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-spark-1.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-spark-1.2: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-spark-1.3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x muse-wonder-1.0: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x nemotron-3-super: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.7-plus: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-27b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x qwen3.8-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- codex x step-3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x gpt-5.3-codex: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x gpt-6-astra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x muse-glimmer-30b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x muse-spark-1.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dcode x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- docker-agent x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- docker-agent x grok-4.20: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x grok-build-0.1: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- docker-agent x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x claude-fable-5: not run, not run in this column
- dsh x claude-fable-5-1: not run, not run in this column
- dsh x claude-haiku-4.5: not run, not run in this column
- dsh x claude-opus-4.7: not run, not run in this column
- dsh x claude-opus-4.8: not run, not run in this column
- dsh x claude-opus-5: not run, not run in this column
- dsh x claude-opus-5.5: not run, not run in this column
- dsh x claude-sonnet-4.6: not run, not run in this column
- dsh x claude-sonnet-5: not run, not run in this column
- dsh x claude-sonnet-5.5: not run, not run in this column
- dsh x deepseek-v4-flash: not run, not run in this column
- dsh x deepseek-v4-pro: not run, not run in this column
- dsh x deepseek-v4.1-flash: not run, not run in this column
- dsh x gemini-3-flash-preview: not run, not run in this column
- dsh x gemini-3.1-flash-lite: not run, not run in this column
- dsh x gemini-3.1-pro-preview: not run, not run in this column
- dsh x gemini-3.5-flash: not run, not run in this column
- dsh x gemini-3.5-flash-lite: not run, not run in this column
- dsh x gemini-3.6-flash: not run, not run in this column
- dsh x gemini-3.7-flash: not run, not run in this column
- dsh x gemini-3.8-flash: not run, not run in this column
- dsh x gemini-4-argon: not run, not run in this column
- dsh x glm-5.3: not run, not run in this column
- dsh x glm-5.3-flash: not run, not run in this column
- dsh x gpt-5.2: not run, not run in this column
- dsh x gpt-5.3-codex: not run, not run in this column
- dsh x gpt-5.4: not run, not run in this column
- dsh x gpt-5.4-mini: not run, not run in this column
- dsh x gpt-5.5: not run, not run in this column
- dsh x gpt-5.6-luna: not run, not run in this column
- dsh x gpt-5.6-sol: not run, not run in this column
- dsh x gpt-5.6-terra: not run, not run in this column
- dsh x grok-4.20: not run, not run in this column
- dsh x grok-4.3: not run, not run in this column
- dsh x grok-4.5: not run, not run in this column
- dsh x grok-4.6: not run, not run in this column
- dsh x grok-build-0.1: not run, not run in this column
- dsh x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x hunyuan-4-preview: not run, not run in this column
- dsh x kimi-k2.7-code: not run, not run in this column
- dsh x kimi-k3: not run, not run in this column
- dsh x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x mistral-medium-3.5: not run, not run in this column
- dsh x muse-glimmer-30b: not run, not run in this column
- dsh x muse-spark-1.1: not run, not run in this column
- dsh x muse-spark-1.2: not run, not run in this column
- dsh x muse-spark-1.3: not run, not run in this column
- dsh x muse-wonder-1.0: not run, not run in this column
- dsh x nemotron-3-super: not run, not run in this column
- dsh x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- dsh x qwen3.7-max: not run, not run in this column
- dsh x qwen3.7-plus: not run, not run in this column
- dsh x qwen3.8-27b: not run, not run in this column
- dsh x qwen3.8-flash: not run, not run in this column
- dsh x qwen3.8-max: not run, not run in this column
- dsh x step-3.7-flash: not run, not run in this column
- fast-agent x gemini-4-argon: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- fast-agent x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- fast-agent x grok-4.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x hunyuan-4-preview: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x kimi-k2.7-code: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x nemotron-3-super: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x qwen3.8-27b: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- fast-agent x qwen3.8-max: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-fable-5: not run, not run in this column
- goose x claude-fable-5-1: not run, not run in this column
- goose x claude-haiku-4.5: not run, not run in this column
- goose x claude-opus-4.7: not run, not run in this column
- goose x claude-opus-4.8: not run, not run in this column
- goose x claude-opus-5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-opus-5.5: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x claude-sonnet-4.6: not run, not run in this column
- goose x claude-sonnet-5: not run, not run in this column
- goose x claude-sonnet-5.5: not run, not run in this column
- goose x deepseek-v4-flash: not run, not run in this column
- goose x deepseek-v4-pro: not run, not run in this column
- goose x deepseek-v4.1-flash: not run, not run in this column
- goose x gemini-3-flash-preview: not run, not run in this column
- goose x gemini-3.1-flash-lite: not run, not run in this column
- goose x gemini-3.1-pro-preview: not run, not run in this column
- goose x gemini-3.5-flash: not run, not run in this column
- goose x gemini-3.5-flash-lite: not run, not run in this column
- goose x gemini-3.6-flash: not run, not run in this column
- goose x gemini-3.7-flash: not run, not run in this column
- goose x gemini-3.8-flash: not run, not run in this column
- goose x gemini-4-argon: not run, not run in this column
- goose x glm-5.3: not run, not run in this column
- goose x glm-5.3-flash: not run, not run in this column
- goose x gpt-5.2: not run, not run in this column
- goose x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-5.4: not run, not run in this column
- goose x gpt-5.4-mini: not run, not run in this column
- goose x gpt-5.5: not run, not run in this column
- goose x gpt-5.6-luna: not run, not run in this column
- goose x gpt-5.6-sol: not run, not run in this column
- goose x gpt-5.6-terra: not run, not run in this column
- goose x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- goose x gpt-6-luna: not run, not run in this column
- goose x gpt-6-sol: not run, not run in this column
- goose x grok-4.20: not run, not run in this column
- goose x grok-4.3: not run, not run in this column
- goose x grok-4.5: not run, not run in this column
- goose x grok-4.6: not run, not run in this column
- goose x grok-build-0.1: not run, not run in this column
- goose x hunyuan-3: not run, not run in this column
- goose x hunyuan-4-preview: not run, not run in this column
- goose x kimi-k2.7-code: not run, not run in this column
- goose x kimi-k3: not run, not run in this column
- goose x ling-3.0-flash: not run, not run in this column
- goose x minimax-m3: not run, not run in this column
- goose x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- goose x mistral-medium-3.5: not run, not run in this column
- goose x muse-glimmer-30b: not run, not run in this column
- goose x muse-spark-1.1: not run, not run in this column
- goose x muse-spark-1.2: not run, not run in this column
- goose x muse-spark-1.3: not run, not run in this column
- goose x muse-wonder-1.0: not run, not run in this column
- goose x nemotron-3-super: not run, not run in this column
- goose x nemotron-3-ultra: not run, not run in this column
- goose x qwen3.7-flash: not run, not run in this column
- goose x qwen3.7-max: not run, not run in this column
- goose x qwen3.7-plus: not run, not run in this column
- goose x qwen3.8-27b: not run, not run in this column
- goose x qwen3.8-flash: not run, not run in this column
- goose x qwen3.8-max: not run, not run in this column
- goose x step-3.7-flash: not run, not run in this column
- grok x claude-fable-5: not run, not run in this column
- grok x claude-fable-5-1: not run, not run in this column
- grok x claude-haiku-4.5: not run, not run in this column
- grok x claude-opus-4.7: not run, not run in this column
- grok x claude-opus-4.8: not run, not run in this column
- grok x claude-opus-5: not run, not run in this column
- grok x claude-opus-5.5: not run, not run in this column
- grok x claude-sonnet-4.6: not run, not run in this column
- grok x claude-sonnet-5: not run, not run in this column
- grok x claude-sonnet-5.5: not run, not run in this column
- grok x deepseek-v4-flash: not run, not run in this column
- grok x deepseek-v4-pro: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x deepseek-v4.1-flash: not run, not run in this column
- grok x gemini-3-flash-preview: not run, not run in this column
- grok x gemini-3.1-flash-lite: not run, not run in this column
- grok x gemini-3.1-pro-preview: not run, not run in this column
- grok x gemini-3.5-flash: not run, not run in this column
- grok x gemini-3.5-flash-lite: not run, not run in this column
- grok x gemini-3.6-flash: not run, not run in this column
- grok x gemini-3.7-flash: not run, not run in this column
- grok x gemini-3.8-flash: not run, not run in this column
- grok x gemini-4-argon: not run, not run in this column
- grok x glm-5.3: not run, not run in this column
- grok x glm-5.3-flash: not run, not run in this column
- grok x gpt-5.2: not run, not run in this column
- grok x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- grok x gpt-5.4: not run, not run in this column
- grok x gpt-5.4-mini: not run, not run in this column
- grok x gpt-5.5: not run, not run in this column
- grok x gpt-5.6-luna: not run, not run in this column
- grok x gpt-5.6-sol: not run, not run in this column
- grok x gpt-5.6-terra: not run, not run in this column
- grok x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- grok x gpt-6-luna: not run, not run in this column
- grok x gpt-6-sol: not run, not run in this column
- grok x gpt-6.1-sol: not run, not run in this column
- grok x grok-4.20: not run, not run in this column
- grok x grok-4.3: not run, not run in this column
- grok x grok-4.5: not run, not run in this column
- grok x grok-4.6: not run, not run in this column
- grok x grok-build-0.1: not run, not run in this column
- grok x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x hunyuan-4-preview: not run, not run in this column
- grok x kimi-k2.7-code: not run, not run in this column
- grok x kimi-k3: not run, not run in this column
- grok x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x mistral-medium-3.5: not run, not run in this column
- grok x muse-glimmer-30b: not run, not run in this column
- grok x muse-spark-1.1: not run, not run in this column
- grok x muse-spark-1.2: not run, not run in this column
- grok x muse-spark-1.3: not run, not run in this column
- grok x muse-wonder-1.0: not run, not run in this column
- grok x nemotron-3-super: not run, not run in this column
- grok x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- grok x qwen3.7-max: not run, not run in this column
- grok x qwen3.7-plus: not run, not run in this column
- grok x qwen3.8-27b: not run, not run in this column
- grok x qwen3.8-flash: not run, not run in this column
- grok x qwen3.8-max: not run, not run in this column
- grok x step-3.7-flash: not run, not run in this column
- hermes x claude-fable-5: not run, not run in this column
- hermes x claude-fable-5-1: not run, not run in this column
- hermes x claude-haiku-4.5: not run, not run in this column
- hermes x claude-opus-4.7: not run, not run in this column
- hermes x claude-opus-4.8: not run, not run in this column
- hermes x claude-opus-5: not run, not run in this column
- hermes x claude-opus-5.5: not run, not run in this column
- hermes x claude-sonnet-4.6: not run, not run in this column
- hermes x claude-sonnet-5: not run, not run in this column
- hermes x claude-sonnet-5.5: not run, not run in this column
- hermes x deepseek-v4-flash: not run, not run in this column
- hermes x deepseek-v4-pro: not run, not run in this column
- hermes x gemini-3-flash-preview: not run, not run in this column
- hermes x gemini-3.1-flash-lite: not run, not run in this column
- hermes x gemini-3.1-pro-preview: not run, not run in this column
- hermes x gemini-3.5-flash: not run, not run in this column
- hermes x gemini-3.5-flash-lite: not run, not run in this column
- hermes x gemini-3.6-flash: not run, not run in this column
- hermes x gemini-3.7-flash: not run, not run in this column
- hermes x gemini-3.8-flash: not run, not run in this column
- hermes x gemini-4-argon: not run, not run in this column
- hermes x glm-5.3: not run, not run in this column
- hermes x glm-5.3-flash: not run, not run in this column
- hermes x gpt-5.2: not run, not run in this column
- hermes x gpt-5.3-codex: not run, not run in this column
- hermes x gpt-5.4: not run, not run in this column
- hermes x gpt-5.4-mini: not run, not run in this column
- hermes x gpt-5.5: not run, not run in this column
- hermes x gpt-5.6-luna: not run, not run in this column
- hermes x gpt-5.6-sol: not run, not run in this column
- hermes x gpt-5.6-terra: not run, not run in this column
- hermes x hunyuan-3: not run, not run in this column
- hermes x kimi-k3: not run, not run in this column
- hermes x minimax-m3: not run, not run in this column
- hermes x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- hermes x mistral-medium-3.5: not run, not run in this column
- hermes x muse-wonder-1.0: not run, not run in this column
- hermes x nemotron-3-ultra: not run, not run in this column
- hermes x qwen3.7-flash: not run, not run in this column
- hermes x qwen3.7-max: not run, not run in this column
- hermes x qwen3.8-max: not run, not run in this column
- hermes x step-3.7-flash: not run, not run in this column
- kilo x claude-fable-5: not run, not run in this column
- kilo x claude-fable-5-1: not run, not run in this column
- kilo x claude-haiku-4.5: not run, not run in this column
- kilo x claude-opus-4.7: not run, not run in this column
- kilo x claude-opus-4.8: not run, not run in this column
- kilo x claude-opus-5: not run, not run in this column
- kilo x claude-opus-5.5: not run, not run in this column
- kilo x claude-sonnet-4.6: not run, not run in this column
- kilo x claude-sonnet-5: not run, not run in this column
- kilo x claude-sonnet-5.5: not run, not run in this column
- kilo x deepseek-v4-flash: not run, not run in this column
- kilo x deepseek-v4-pro: not run, not run in this column
- kilo x deepseek-v4.1-flash: not run, not run in this column
- kilo x gemini-3-flash-preview: not run, not run in this column
- kilo x gemini-3.1-flash-lite: not run, not run in this column
- kilo x gemini-3.1-pro-preview: not run, not run in this column
- kilo x gemini-3.5-flash: not run, not run in this column
- kilo x gemini-3.5-flash-lite: not run, not run in this column
- kilo x gemini-3.6-flash: not run, not run in this column
- kilo x gemini-3.7-flash: not run, not run in this column
- kilo x gemini-3.8-flash: not run, not run in this column
- kilo x gemini-4-argon: not run, not run in this column
- kilo x glm-5.3: not run, not run in this column
- kilo x glm-5.3-flash: not run, not run in this column
- kilo x gpt-5.2: not run, not run in this column
- kilo x gpt-5.3-codex: not run, not run in this column
- kilo x gpt-5.4: not run, not run in this column
- kilo x gpt-5.4-mini: not run, not run in this column
- kilo x gpt-5.5: not run, not run in this column
- kilo x gpt-5.6-luna: not run, not run in this column
- kilo x gpt-5.6-sol: not run, not run in this column
- kilo x gpt-5.6-terra: not run, not run in this column
- kilo x gpt-6-astra: not run, not run in this column
- kilo x gpt-6-luna: not run, not run in this column
- kilo x gpt-6-sol: not run, not run in this column
- kilo x gpt-6.1-sol: not run, not run in this column
- kilo x grok-4.20: not run, not run in this column
- kilo x grok-4.3: not run, not run in this column
- kilo x grok-4.5: not run, not run in this column
- kilo x grok-4.6: not run, not run in this column
- kilo x grok-build-0.1: not run, not run in this column
- kilo x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kilo x hunyuan-4-preview: not run, not run in this column
- kilo x kimi-k2.7-code: not run, not run in this column
- kilo x kimi-k3: not run, not run in this column
- kilo x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kilo x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kilo x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kilo x mistral-medium-3.5: not run, not run in this column
- kilo x muse-glimmer-30b: not run, not run in this column
- kilo x muse-spark-1.1: not run, not run in this column
- kilo x muse-spark-1.2: not run, not run in this column
- kilo x muse-spark-1.3: not run, not run in this column
- kilo x muse-wonder-1.0: not run, not run in this column
- kilo x nemotron-3-super: not run, not run in this column
- kilo x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kilo x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kilo x qwen3.7-max: not run, not run in this column
- kilo x qwen3.7-plus: not run, not run in this column
- kilo x qwen3.8-27b: not run, not run in this column
- kilo x qwen3.8-flash: not run, not run in this column
- kilo x qwen3.8-max: not run, not run in this column
- kilo x step-3.7-flash: not run, not run in this column
- kimi x claude-fable-5: not run, not run in this column
- kimi x claude-fable-5-1: not run, not run in this column
- kimi x claude-haiku-4.5: not run, not run in this column
- kimi x claude-opus-4.7: not run, not run in this column
- kimi x claude-opus-4.8: not run, not run in this column
- kimi x claude-opus-5: not run, not run in this column
- kimi x claude-opus-5.5: not run, not run in this column
- kimi x claude-sonnet-4.6: not run, not run in this column
- kimi x claude-sonnet-5: not run, not run in this column
- kimi x claude-sonnet-5.5: not run, not run in this column
- kimi x deepseek-v4-flash: not run, not run in this column
- kimi x deepseek-v4-pro: not run, not run in this column
- kimi x deepseek-v4.1-flash: not run, not run in this column
- kimi x gemini-3-flash-preview: not run, not run in this column
- kimi x gemini-3.1-flash-lite: not run, not run in this column
- kimi x gemini-3.1-pro-preview: not run, not run in this column
- kimi x gemini-3.5-flash: not run, not run in this column
- kimi x gemini-3.5-flash-lite: not run, not run in this column
- kimi x gemini-3.6-flash: not run, not run in this column
- kimi x gemini-3.7-flash: not run, not run in this column
- kimi x gemini-3.8-flash: not run, not run in this column
- kimi x gemini-4-argon: not run, not run in this column
- kimi x glm-5.3: not run, not run in this column
- kimi x glm-5.3-flash: not run, not run in this column
- kimi x gpt-5.2: not run, not run in this column
- kimi x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- kimi x gpt-5.4: not run, not run in this column
- kimi x gpt-5.4-mini: not run, not run in this column
- kimi x gpt-5.5: not run, not run in this column
- kimi x gpt-5.6-luna: not run, not run in this column
- kimi x gpt-5.6-sol: not run, not run in this column
- kimi x gpt-5.6-terra: not run, not run in this column
- kimi x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- kimi x grok-4.20: not run, not run in this column
- kimi x grok-4.3: not run, not run in this column
- kimi x grok-4.5: not run, not run in this column
- kimi x grok-4.6: not run, not run in this column
- kimi x grok-build-0.1: not run, not run in this column
- kimi x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x hunyuan-4-preview: not run, not run in this column
- kimi x kimi-k2.7-code: not run, not run in this column
- kimi x kimi-k3: not run, not run in this column
- kimi x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x mistral-medium-3.5: not run, not run in this column
- kimi x muse-glimmer-30b: not run, not run in this column
- kimi x muse-spark-1.1: not run, not run in this column
- kimi x muse-spark-1.2: not run, not run in this column
- kimi x muse-spark-1.3: not run, not run in this column
- kimi x muse-wonder-1.0: not run, not run in this column
- kimi x nemotron-3-super: not run, not run in this column
- kimi x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- kimi x qwen3.7-max: not run, not run in this column
- kimi x qwen3.7-plus: not run, not run in this column
- kimi x qwen3.8-27b: not run, not run in this column
- kimi x qwen3.8-flash: not run, not run in this column
- kimi x qwen3.8-max: not run, not run in this column
- kimi x step-3.7-flash: not run, not run in this column
- minimax x claude-fable-5: not run, not run in this column
- minimax x claude-fable-5-1: not run, not run in this column
- minimax x claude-haiku-4.5: not run, not run in this column
- minimax x claude-opus-4.7: not run, not run in this column
- minimax x claude-opus-4.8: not run, not run in this column
- minimax x claude-opus-5: not run, not run in this column
- minimax x claude-opus-5.5: not run, not run in this column
- minimax x claude-sonnet-4.6: not run, not run in this column
- minimax x claude-sonnet-5: not run, not run in this column
- minimax x claude-sonnet-5.5: not run, not run in this column
- minimax x deepseek-v4-flash: not run, not run in this column
- minimax x deepseek-v4-pro: not run, not run in this column
- minimax x deepseek-v4.1-flash: not run, not run in this column
- minimax x gemini-3-flash-preview: not run, not run in this column
- minimax x gemini-3.1-flash-lite: not run, not run in this column
- minimax x gemini-3.1-pro-preview: not run, not run in this column
- minimax x gemini-3.5-flash: not run, not run in this column
- minimax x gemini-3.5-flash-lite: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- minimax x gemini-3.6-flash: not run, not run in this column
- minimax x gemini-3.7-flash: not run, not run in this column
- minimax x gemini-3.8-flash: not run, not run in this column
- minimax x gemini-4-argon: not run, not run in this column
- minimax x glm-5.3: not run, not run in this column
- minimax x glm-5.3-flash: not run, not run in this column
- minimax x gpt-5.2: not run, not run in this column
- minimax x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- minimax x gpt-5.4: not run, not run in this column
- minimax x gpt-5.4-mini: not run, not run in this column
- minimax x gpt-5.5: not run, not run in this column
- minimax x gpt-5.6-luna: not run, not run in this column
- minimax x gpt-5.6-sol: not run, not run in this column
- minimax x gpt-5.6-terra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- minimax x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- minimax x gpt-6-luna: not run, not run in this column
- minimax x gpt-6-sol: not run, not run in this column
- minimax x gpt-6.1-sol: not run, not run in this column
- minimax x grok-4.20: not run, not run in this column
- minimax x grok-4.3: not run, not run in this column
- minimax x grok-4.5: not run, not run in this column
- minimax x grok-4.6: not run, not run in this column
- minimax x grok-build-0.1: not run, not run in this column
- minimax x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- minimax x hunyuan-4-preview: not run, not run in this column
- minimax x kimi-k2.7-code: not run, not run in this column
- minimax x kimi-k3: not run, not run in this column
- minimax x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- minimax x minimax-m3: not run, not run in this column
- minimax x mistral-medium-3.5: not run, not run in this column
- minimax x muse-glimmer-30b: not run, not run in this column
- minimax x muse-spark-1.1: not run, not run in this column
- minimax x muse-spark-1.2: not run, not run in this column
- minimax x muse-spark-1.3: not run, not run in this column
- minimax x muse-wonder-1.0: not run, not run in this column
- minimax x nemotron-3-super: not run, not run in this column
- minimax x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- minimax x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- minimax x qwen3.7-max: not run, not run in this column
- minimax x qwen3.7-plus: not run, not run in this column
- minimax x qwen3.8-27b: not run, not run in this column
- minimax x qwen3.8-flash: not run, not run in this column
- minimax x qwen3.8-max: not run, not run in this column
- minimax x step-3.7-flash: not run, not run in this column
- omp x claude-fable-5: not run, not run in this column
- omp x claude-fable-5-1: not run, not run in this column
- omp x claude-haiku-4.5: not run, not run in this column
- omp x claude-opus-4.7: not run, not run in this column
- omp x claude-opus-4.8: not run, not run in this column
- omp x claude-opus-5: not run, not run in this column
- omp x claude-opus-5.5: not run, not run in this column
- omp x claude-sonnet-4.6: not run, not run in this column
- omp x claude-sonnet-5: not run, not run in this column
- omp x claude-sonnet-5.5: not run, not run in this column
- omp x deepseek-v4-flash: not run, not run in this column
- omp x deepseek-v4-pro: not run, not run in this column
- omp x deepseek-v4.1-flash: not run, not run in this column
- omp x gemini-3-flash-preview: not run, not run in this column
- omp x gemini-3.1-flash-lite: not run, not run in this column
- omp x gemini-3.1-pro-preview: not run, not run in this column
- omp x gemini-3.5-flash: not run, not run in this column
- omp x gemini-3.5-flash-lite: not run, not run in this column
- omp x gemini-3.6-flash: not run, not run in this column
- omp x gemini-3.7-flash: not run, not run in this column
- omp x gemini-3.8-flash: not run, not run in this column
- omp x gemini-4-argon: not run, not run in this column
- omp x glm-5.3: not run, not run in this column
- omp x glm-5.3-flash: not run, not run in this column
- omp x gpt-5.2: not run, not run in this column
- omp x gpt-5.3-codex: not run, not run in this column
- omp x gpt-5.4: not run, not run in this column
- omp x gpt-5.4-mini: not run, not run in this column
- omp x gpt-5.5: not run, not run in this column
- omp x gpt-5.6-luna: not run, not run in this column
- omp x gpt-5.6-sol: not run, not run in this column
- omp x gpt-5.6-terra: not run, not run in this column
- omp x grok-4.20: not run, not run in this column
- omp x grok-4.3: not run, not run in this column
- omp x grok-4.5: not run, not run in this column
- omp x grok-4.6: not run, not run in this column
- omp x grok-build-0.1: not run, not run in this column
- omp x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x hunyuan-4-preview: not run, not run in this column
- omp x kimi-k2.7-code: not run, not run in this column
- omp x kimi-k3: not run, not run in this column
- omp x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x mistral-medium-3.5: not run, not run in this column
- omp x muse-glimmer-30b: not run, not run in this column
- omp x muse-spark-1.1: not run, not run in this column
- omp x muse-spark-1.2: not run, not run in this column
- omp x muse-spark-1.3: not run, not run in this column
- omp x muse-wonder-1.0: not run, not run in this column
- omp x nemotron-3-super: not run, not run in this column
- omp x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- omp x qwen3.7-max: not run, not run in this column
- omp x qwen3.7-plus: not run, not run in this column
- omp x qwen3.8-27b: not run, not run in this column
- omp x qwen3.8-flash: not run, not run in this column
- omp x qwen3.8-max: not run, not run in this column
- omp x step-3.7-flash: not run, not run in this column
- opencode x claude-fable-5: not run, not run in this column
- opencode x claude-fable-5-1: not run, not run in this column
- opencode x claude-haiku-4.5: not run, not run in this column
- opencode x claude-opus-4.7: not run, not run in this column
- opencode x claude-opus-4.8: not run, not run in this column
- opencode x claude-opus-5: not run, not run in this column
- opencode x claude-opus-5.5: not run, not run in this column
- opencode x claude-sonnet-4.6: not run, not run in this column
- opencode x claude-sonnet-5: not run, not run in this column
- opencode x claude-sonnet-5.5: not run, not run in this column
- opencode x deepseek-v4-flash: not run, not run in this column
- opencode x deepseek-v4-pro: not run, not run in this column
- opencode x gemini-3-flash-preview: not run, not run in this column
- opencode x gemini-3.1-flash-lite: not run, not run in this column
- opencode x gemini-3.1-pro-preview: not run, not run in this column
- opencode x gemini-3.5-flash: not run, not run in this column
- opencode x gemini-3.5-flash-lite: not run, not run in this column
- opencode x gemini-3.6-flash: not run, not run in this column
- opencode x gemini-3.7-flash: not run, not run in this column
- opencode x gemini-3.8-flash: not run, not run in this column
- opencode x gemini-4-argon: not run, not run in this column
- opencode x glm-5.3: not run, not run in this column
- opencode x glm-5.3-flash: not run, not run in this column
- opencode x gpt-5.2: not run, not run in this column
- opencode x gpt-5.3-codex: not run, not run in this column
- opencode x gpt-5.4: not run, not run in this column
- opencode x gpt-5.4-mini: not run, not run in this column
- opencode x gpt-5.5: not run, not run in this column
- opencode x gpt-5.6-luna: not run, not run in this column
- opencode x gpt-5.6-sol: not run, not run in this column
- opencode x gpt-5.6-terra: not run, not run in this column
- opencode x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x kimi-k2.7-code: not run, not run in this column
- opencode x kimi-k3: not run, not run in this column
- opencode x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x mistral-medium-3.5: not run, not run in this column
- opencode x muse-wonder-1.0: not run, not run in this column
- opencode x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- opencode x qwen3.7-max: not run, not run in this column
- opencode x qwen3.8-max: not run, not run in this column
- opencode x step-3.7-flash: not run, not run in this column
- openhands x claude-fable-5: not run, not run in this column
- openhands x claude-fable-5-1: not run, not run in this column
- openhands x claude-haiku-4.5: not run, not run in this column
- openhands x claude-opus-4.7: not run, not run in this column
- openhands x claude-opus-4.8: not run, not run in this column
- openhands x claude-opus-5: not run, not run in this column
- openhands x claude-opus-5.5: not run, not run in this column
- openhands x claude-sonnet-4.6: not run, not run in this column
- openhands x claude-sonnet-5: not run, not run in this column
- openhands x claude-sonnet-5.5: not run, not run in this column
- openhands x deepseek-v4-flash: not run, not run in this column
- openhands x deepseek-v4-pro: not run, not run in this column
- openhands x deepseek-v4.1-flash: not run, not run in this column
- openhands x gemini-3-flash-preview: not run, not run in this column
- openhands x gemini-3.1-flash-lite: not run, not run in this column
- openhands x gemini-3.1-pro-preview: not run, not run in this column
- openhands x gemini-3.5-flash: not run, not run in this column
- openhands x gemini-3.5-flash-lite: not run, not run in this column
- openhands x gemini-3.6-flash: not run, not run in this column
- openhands x gemini-3.7-flash: not run, not run in this column
- openhands x gemini-3.8-flash: not run, not run in this column
- openhands x gemini-4-argon: not run, not run in this column
- openhands x glm-5.3: not run, not run in this column
- openhands x glm-5.3-flash: not run, not run in this column
- openhands x gpt-5.2: not run, not run in this column
- openhands x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- openhands x gpt-5.4: not run, not run in this column
- openhands x gpt-5.4-mini: not run, not run in this column
- openhands x gpt-5.5: not run, not run in this column
- openhands x gpt-5.6-luna: not run, not run in this column
- openhands x gpt-5.6-sol: not run, not run in this column
- openhands x gpt-5.6-terra: not run, not run in this column
- openhands x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- openhands x grok-4.20: not run, not run in this column
- openhands x grok-4.3: not run, not run in this column
- openhands x grok-4.5: not run, not run in this column
- openhands x grok-4.6: not run, not run in this column
- openhands x grok-build-0.1: not run, not run in this column
- openhands x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x hunyuan-4-preview: not run, not run in this column
- openhands x kimi-k2.7-code: not run, not run in this column
- openhands x kimi-k3: not run, not run in this column
- openhands x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x mistral-medium-3.5: not run, not run in this column
- openhands x muse-glimmer-30b: not run, not run in this column
- openhands x muse-spark-1.1: not run, not run in this column
- openhands x muse-spark-1.2: not run, not run in this column
- openhands x muse-spark-1.3: not run, not run in this column
- openhands x muse-wonder-1.0: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x nemotron-3-super: not run, not run in this column
- openhands x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- openhands x qwen3.7-max: not run, not run in this column
- openhands x qwen3.7-plus: not run, not run in this column
- openhands x qwen3.8-27b: not run, not run in this column
- openhands x qwen3.8-flash: not run, not run in this column
- openhands x qwen3.8-max: not run, not run in this column
- openhands x step-3.7-flash: not run, not run in this column
- pi x claude-fable-5: not run, not run in this column
- pi x claude-fable-5-1: not run, not run in this column
- pi x claude-haiku-4.5: not run, not run in this column
- pi x claude-opus-4.7: not run, not run in this column
- pi x claude-opus-4.8: not run, not run in this column
- pi x claude-opus-5: not run, not run in this column
- pi x claude-opus-5.5: not run, not run in this column
- pi x claude-sonnet-4.6: not run, not run in this column
- pi x claude-sonnet-5: not run, not run in this column
- pi x claude-sonnet-5.5: not run, not run in this column
- pi x deepseek-v4-flash: not run, not run in this column
- pi x deepseek-v4-pro: not run, not run in this column
- pi x gemini-3-flash-preview: not run, not run in this column
- pi x gemini-3.1-flash-lite: not run, not run in this column
- pi x gemini-3.1-pro-preview: not run, not run in this column
- pi x gemini-3.5-flash: not run, not run in this column
- pi x gemini-3.5-flash-lite: not run, not run in this column
- pi x gemini-3.6-flash: not run, not run in this column
- pi x gemini-3.7-flash: not run, not run in this column
- pi x gemini-3.8-flash: not run, not run in this column
- pi x gemini-4-argon: not run, not run in this column
- pi x glm-5.3: not run, not run in this column
- pi x glm-5.3-flash: not run, not run in this column
- pi x gpt-5.2: not run, not run in this column
- pi x gpt-5.3-codex: not run, not run in this column
- pi x gpt-5.4: not run, not run in this column
- pi x gpt-5.4-mini: not run, not run in this column
- pi x gpt-5.5: not run, not run in this column
- pi x gpt-5.6-luna: not run, not run in this column
- pi x gpt-5.6-sol: not run, not run in this column
- pi x gpt-5.6-terra: not run, not run in this column
- pi x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x kimi-k2.7-code: not run, not run in this column
- pi x kimi-k3: not run, not run in this column
- pi x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x mistral-medium-3.5: not run, not run in this column
- pi x muse-wonder-1.0: not run, not run in this column
- pi x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- pi x qwen3.7-max: not run, not run in this column
- pi x qwen3.8-max: not run, not run in this column
- pi x step-3.7-flash: not run, not run in this column
- qwen x claude-fable-5: not run, not run in this column
- qwen x claude-fable-5-1: not run, not run in this column
- qwen x claude-haiku-4.5: not run, not run in this column
- qwen x claude-opus-4.7: not run, not run in this column
- qwen x claude-opus-4.8: not run, not run in this column
- qwen x claude-opus-5: not run, not run in this column
- qwen x claude-opus-5.5: not run, not run in this column
- qwen x claude-sonnet-4.6: not run, not run in this column
- qwen x claude-sonnet-5: not run, not run in this column
- qwen x claude-sonnet-5.5: not run, not run in this column
- qwen x deepseek-v4-flash: not run, not run in this column
- qwen x deepseek-v4-pro: not run, not run in this column
- qwen x gemini-3-flash-preview: not run, not run in this column
- qwen x gemini-3.1-flash-lite: not run, not run in this column
- qwen x gemini-3.1-pro-preview: not run, not run in this column
- qwen x gemini-3.5-flash: not run, not run in this column
- qwen x gemini-3.5-flash-lite: not run, not run in this column
- qwen x gemini-3.6-flash: not run, not run in this column
- qwen x gemini-3.7-flash: not run, not run in this column
- qwen x gemini-3.8-flash: not run, not run in this column
- qwen x gemini-4-argon: not run, not run in this column
- qwen x glm-5.3: not run, not run in this column
- qwen x glm-5.3-flash: not run, not run in this column
- qwen x gpt-5.2: not run, not run in this column
- qwen x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x gpt-5.4: not run, not run in this column
- qwen x gpt-5.4-mini: not run, not run in this column
- qwen x gpt-5.5: not run, not run in this column
- qwen x gpt-5.6-luna: not run, not run in this column
- qwen x gpt-5.6-sol: not run, not run in this column
- qwen x gpt-5.6-terra: not run, not run in this column
- qwen x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- qwen x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x kimi-k2.7-code: not run, not run in this column
- qwen x kimi-k3: not run, not run in this column
- qwen x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x mistral-medium-3.5: not run, not run in this column
- qwen x muse-wonder-1.0: not run, not run in this column
- qwen x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- qwen x qwen3.7-max: not run, not run in this column
- qwen x qwen3.8-max: not run, not run in this column
- qwen x step-3.7-flash: not run, not run in this column
- vibe x claude-opus-4.7: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x deepseek-v4.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x gpt-5.3-codex: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- vibe x gpt-5.6-luna: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x gpt-6-astra: not run, the model answers on the Responses API only and this harness speaks chat/completions only
- vibe x hunyuan-3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x ling-3.0-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x minimax-m3: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x minimax-m3.1-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x nemotron-3-ultra: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
- vibe x qwen3.7-flash: not run, not in this harness's catalog (unmeasured or excluded, see the catalog's note)
Browser plugin, every base
Measured 2026-09-27 on hr-test (self-hosted, 0.25.7-rc.17). One task in plain words, "Open https://example.com/ in the browser, click the only link on that page, and reply with the URL and the title of the page you land on." A base passes when the task completed, the trace shows the browser navigating and clicking, the answer names the page the link leads to, and the browser session was stopped and billed; a failure is retested once. System One is not in the column: its models choose among offered actions and call no tools. The run's findings are in support-matrix-notes.md.
| base | model | browser | tools seen | seconds | notes |
|---|---|---|---|---|---|
| codex | gpt-5.4 | pass | click, navigate, open | 33 | |
| claude-code | claude-sonnet-4.6 | pass | click, navigate, open | 46 | |
| hermes | gpt-5.4 | pass | click, get_url, navigate, open | 52 | |
| pi | gpt-5.4 | pass | click, get_url, navigate, open | 34 | |
| omp | gpt-5.4 | pass | click, navigate, open | 40 | |
| dsh | deepseek-v4-pro | pass | click, navigate, open | 27 | |
| goose | gpt-5.4 | pass | click, get_url, navigate, open | 33 | |
| opencode | gpt-5.4 | pass | click, get_url, navigate, open | 27 | |
| aider | gpt-5.4 | pass | click, get_url, navigate, open, snapshot | 33 | |
| kimi | kimi-k3 | pass | click, navigate, open | 40 | |
| openhands | gpt-5.4 | pass | click, get_url, navigate, open, snapshot | 46 | |
| cheetahclaws | gpt-5.4 | pass | click, navigate, open | 27 | |
| qwen | qwen3.7-max | pass | click, get_url, navigate, open | 40 | |
| gemini | gemini-3.8-flash | pass | click, get_url, navigate, open | 33 | |
| cline | gpt-5.4 | pass | click, get_url, navigate, open, snapshot, wait_for | 33 |
15 of 15 bases drive the browser.