Skip to content

HarnessRouter Community Edition: the self-hosted, Apache-2.0 edition of the unified interface for agent harnesses. Run Codex, Claude Code, Hermes, PI, DSH, and more through one API, with sessions, streaming, files, cancellation, and failure handling. Implements the Unified Harness Protocol (UHP), an open standard. Your keys, your infrastructure.

The omp columns of the support matrix, and a served id judged as a model beside the exact rule (#119)

github-actions[bot]146cc0fc7710
diff --git a/docs/support-matrix-notes.md b/docs/support-matrix-notes.md
index 770547d4..0ec865b6 100644
--- a/docs/support-matrix-notes.md
+++ b/docs/support-matrix-notes.md
@@ -137,3 +137,31 @@ reason, never completes. The binary is pinned to 18.1.13 and verified against th
SHA256SUMS. Per-provider columns follow below as they are measured, one integration on the org at a
time, the served-model rule as judge.
+## omp (Oh My Pi), one provider at a time (2026-09-07)
+
+The omp harness (PR #68, omp 18.1.13, pi's lineage) was measured on the pre-release 0.14.3-rc.1 built from
+its branch, one column per provider with the org holding only that provider's integration, every id the
+provider serves that the omp catalog lists, all five scenarios, one retest for a failed row. 705 of 705
+scenario runs passed: Google AI Studio 55/55 (11 ids; gemini-3.1-flash-lite failed its first try and passed
+on retest), OpenRouter 184/184 (37), Vercel AI Gateway 184/184 (37), TokenRouter 164/164 (33), Anthropic
+40/40 (8), OpenAI 39/39 (8), Azure OpenAI E2 39/39 (8). Every turn ran on the column's own integration and
+no id was served as another model. The switch scenario has one run fewer per column because the switch
+partner, gpt-5.6-sol, is not switched to itself.
+
+omp reports the model it ran on every assistant message, so its columns are the first where a provider's
+own name for a model reached the judge: OpenRouter, Vercel and TokenRouter serve claude-fable-5 as
+anthropic/claude-fable-5, gemini-3.8-flash as google/gemini-3.8-flash, mistral-medium-3.5 as
+mistralai/mistral-medium-3-5, qwen3.8-max as qwen/qwen3.8-max-0902; Anthropic serves claude-haiku-4.5 as
+claude-haiku-4-5-20251001 and claude-opus-4.7 as claude-opus-4-7. The exact rule counts each as a served
+model other than the id asked for. Beside it the tables now apply the same-model rule the hosted gateway
+uses (scripts/support-matrix/samemodel.py, identical in both trees): a vendor prefix an aggregator adds
+or drops and a dated or versioned suffix a provider appends are the provider's alias of the same model,
+noted in the row and counted; a different family, number or tier under any prefix (google/gemini-3-flash-
+preview for gemini-3.8-flash, gemini-2.5-flash-lite for gemini-2.5-flash) stays a finding and is not
+counted. A turn that ran on several models is an alias only when every one of them is the same model.
+
+Two things the run itself taught. A snapshot that holds an integration whose key only its owner knows
+cannot be restored after a deletion-level column, because the instance masks stored keys; the restore
+now restores everything it can and names the rest instead of aborting whole. And a column's isolation
+deletes every other integration on the org, so nobody can test on the instance while a column runs;
+the org comes back between columns and at the end.
diff --git a/docs/support-matrix.md b/docs/support-matrix.md
index 04b6d18b..1bb2bfad 100644
--- a/docs/support-matrix.md
+++ b/docs/support-matrix.md
@@ -478,6 +478,36 @@ Scenarios: first turn, follow-up in the same session, switch model mid-session,
5 pairs, 6 of 8 scenario runs passed.
+## Provider: omp-anthropic
+
+| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
+|---|---|---|---|---|---|---|---|---|
+| omp | claude-fable-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
+| omp | claude-fable-5-1 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
+| omp | claude-haiku-4.5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | served as claude-haiku-4-5-20251001 (the provider's alias of the same model) |
+| omp | claude-opus-4.7 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | served as claude-opus-4-7 (the provider's alias of the same model) |
+| omp | claude-opus-4.8 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | served as claude-opus-4-8 (the provider's alias of the same model) |
+| omp | claude-opus-5 | pass | pass | pass (claude-fable-5) | pass | pass | Anthropic | |
+| omp | claude-sonnet-4.6 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | served as claude-sonnet-4-6 (the provider's alias of the same model) |
+| omp | claude-sonnet-5 | pass | pass | pass (claude-opus-5) | pass | pass | Anthropic | |
+
+8 pairs, 40 of 40 scenario runs passed.
+
+## Provider: omp-azure-e2
+
+| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
+|---|---|---|---|---|---|---|---|---|
+| omp | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | |
+| omp | gpt-5.3-codex | pass | pass | n/a | pass | pass | Azure OpenAI E2 | |
+| omp | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | |
+| omp | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | |
+| omp | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | |
+| omp | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | |
+| omp | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Azure OpenAI E2 | |
+| omp | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Azure OpenAI E2 | |
+
+8 pairs, 39 of 39 scenario runs passed.
+
## Provider: omp-google
| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
@@ -486,7 +516,7 @@ Scenarios: first turn, follow-up in the same session, switch model mid-session,
| omp | gemini-2.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| omp | gemini-2.5-pro | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| omp | gemini-3-flash-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
-| omp | gemini-3.1-flash-lite | n/a | n/a | n/a | n/a | n/a | ? | retested once; first try: artifact no file card (files: none); Create a file named hello-omp.txt containing exactly |
+| omp | gemini-3.1-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| omp | gemini-3.1-pro-preview | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| omp | gemini-3.5-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| omp | gemini-3.5-flash-lite | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
@@ -494,7 +524,150 @@ Scenarios: first turn, follow-up in the same session, switch model mid-session,
| omp | gemini-3.7-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
| omp | gemini-3.8-flash | pass | pass | pass (gemini-3.6-flash) | pass | pass | Google AI Studio | |
-11 pairs, 50 of 50 scenario runs passed.
+11 pairs, 55 of 55 scenario runs passed.
+
+## Provider: omp-openai
+
+| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
+|---|---|---|---|---|---|---|---|---|
+| omp | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
+| omp | gpt-5.3-codex | pass | pass | n/a | pass | pass | OpenAI | |
+| omp | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
+| omp | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
+| omp | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
+| omp | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
+| omp | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | OpenAI | |
+| omp | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenAI | |
+
+8 pairs, 39 of 39 scenario runs passed.
+
+## Provider: omp-openrouter
+
+| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
+|---|---|---|---|---|---|---|---|---|
+| omp | claude-fable-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-fable-5 (the provider's alias of the same model) |
+| omp | claude-fable-5-1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-fable-5.1 (the provider's alias of the same model) |
+| omp | claude-haiku-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-haiku-4.5 (the provider's alias of the same model) |
+| omp | claude-opus-4.7 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-opus-4.7 (the provider's alias of the same model) |
+| omp | claude-opus-4.8 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-opus-4.8 (the provider's alias of the same model) |
+| omp | claude-opus-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-opus-5 (the provider's alias of the same model) |
+| omp | claude-sonnet-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-sonnet-4.6 (the provider's alias of the same model) |
+| omp | claude-sonnet-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as anthropic/claude-sonnet-5 (the provider's alias of the same model) |
+| omp | deepseek-v4-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as deepseek/deepseek-v4-flash (the provider's alias of the same model) |
+| omp | deepseek-v4-pro | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as deepseek/deepseek-v4-pro (the provider's alias of the same model) |
+| omp | gemini-2.5-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-2.5-flash (the provider's alias of the same model) |
+| omp | gemini-2.5-flash-lite | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-2.5-flash-lite (the provider's alias of the same model) |
+| omp | gemini-2.5-pro | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-2.5-pro (the provider's alias of the same model) |
+| omp | gemini-3-flash-preview | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3-flash-preview (the provider's alias of the same model) |
+| omp | gemini-3.1-flash-lite | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3.1-flash-lite (the provider's alias of the same model) |
+| omp | gemini-3.1-pro-preview | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3.1-pro-preview (the provider's alias of the same model) |
+| omp | gemini-3.5-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3.5-flash (the provider's alias of the same model) |
+| omp | gemini-3.5-flash-lite | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3.5-flash-lite (the provider's alias of the same model) |
+| omp | gemini-3.6-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3.6-flash (the provider's alias of the same model) |
+| omp | gemini-3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3.7-flash (the provider's alias of the same model) |
+| omp | gemini-3.8-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as google/gemini-3.8-flash (the provider's alias of the same model) |
+| omp | glm-5.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as z-ai/glm-5.3 (the provider's alias of the same model) |
+| omp | glm-5.3-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as z-ai/glm-5.3-flash (the provider's alias of the same model) |
+| omp | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as openai/gpt-5.2 (the provider's alias of the same model) |
+| omp | gpt-5.3-codex | pass | pass | n/a | pass | pass | OpenRouter | served as openai/gpt-5.3-codex (the provider's alias of the same model) |
+| omp | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as openai/gpt-5.4 (the provider's alias of the same model) |
+| omp | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as openai/gpt-5.4-mini (the provider's alias of the same model) |
+| omp | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as openai/gpt-5.5 (the provider's alias of the same model) |
+| omp | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as openai/gpt-5.6-luna (the provider's alias of the same model) |
+| omp | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | OpenRouter | served as openai/gpt-5.6-sol (the provider's alias of the same model) |
+| omp | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as openai/gpt-5.6-terra (the provider's alias of the same model) |
+| omp | kimi-k2.7-code | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as moonshotai/kimi-k2.7-code (the provider's alias of the same model) |
+| omp | kimi-k3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as moonshotai/kimi-k3 (the provider's alias of the same model) |
+| omp | mistral-medium-3.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as mistralai/mistral-medium-3-5 (the provider's alias of the same model) |
+| omp | qwen3.7-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as qwen/qwen3.7-max (the provider's alias of the same model) |
+| omp | qwen3.8-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as qwen/qwen3.8-max-0902 (the provider's alias of the same model) |
+| omp | step-3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | OpenRouter | served as stepfun/step-3.7-flash (the provider's alias of the same model) |
+
+37 pairs, 184 of 184 scenario runs passed.
+
+## Provider: omp-tokenrouter
+
+| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
+|---|---|---|---|---|---|---|---|---|
+| omp | claude-fable-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as anthropic/claude-fable-5 (the provider's alias of the same model) |
+| omp | claude-fable-5-1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as anthropic/claude-fable-5.1 (the provider's alias of the same model) |
+| omp | claude-haiku-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as anthropic/claude-haiku-4.5 (the provider's alias of the same model) |
+| omp | claude-opus-4.7 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as anthropic/claude-opus-4.7 (the provider's alias of the same model) |
+| omp | claude-opus-4.8 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as anthropic/claude-opus-4.8 (the provider's alias of the same model) |
+| omp | claude-opus-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as anthropic/claude-opus-5 (the provider's alias of the same model) |
+| omp | claude-sonnet-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as anthropic/claude-sonnet-4.6 (the provider's alias of the same model) |
+| omp | claude-sonnet-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as anthropic/claude-sonnet-5 (the provider's alias of the same model) |
+| omp | deepseek-v4-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as deepseek/deepseek-v4-flash (the provider's alias of the same model) |
+| omp | deepseek-v4-pro | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as deepseek/deepseek-v4-pro (the provider's alias of the same model) |
+| omp | gemini-3-flash-preview | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as google/gemini-3-flash-preview (the provider's alias of the same model) |
+| omp | gemini-3.1-pro-preview | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as google/gemini-3.1-pro-preview (the provider's alias of the same model) |
+| omp | gemini-3.5-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as google/gemini-3.5-flash (the provider's alias of the same model) |
+| omp | gemini-3.5-flash-lite | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as google/gemini-3.5-flash-lite (the provider's alias of the same model) |
+| omp | gemini-3.6-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as google/gemini-3.6-flash (the provider's alias of the same model) |
+| omp | gemini-3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as google/gemini-3.7-flash (the provider's alias of the same model) |
+| omp | gemini-3.8-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as google/gemini-3.8-flash (the provider's alias of the same model) |
+| omp | glm-5.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as z-ai/glm-5.3 (the provider's alias of the same model) |
+| omp | glm-5.3-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as z-ai/glm-5.3-flash (the provider's alias of the same model) |
+| omp | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as openai/gpt-5.2 (the provider's alias of the same model) |
+| omp | gpt-5.3-codex | pass | pass | n/a | pass | pass | My TokenRouter | served as openai/gpt-5.3-codex (the provider's alias of the same model) |
+| omp | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as openai/gpt-5.4 (the provider's alias of the same model) |
+| omp | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as openai/gpt-5.4-mini (the provider's alias of the same model) |
+| omp | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as openai/gpt-5.5 (the provider's alias of the same model) |
+| omp | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as openai/gpt-5.6-luna (the provider's alias of the same model) |
+| omp | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | My TokenRouter | served as openai/gpt-5.6-sol (the provider's alias of the same model) |
+| omp | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as openai/gpt-5.6-terra (the provider's alias of the same model) |
+| omp | kimi-k2.7-code | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as moonshotai/kimi-k2.7-code (the provider's alias of the same model) |
+| omp | kimi-k3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as moonshotai/kimi-k3 (the provider's alias of the same model) |
+| omp | mistral-medium-3.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as mistralai/mistral-medium-3-5 (the provider's alias of the same model) |
+| omp | qwen3.7-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as qwen/qwen3.7-max (the provider's alias of the same model) |
+| omp | qwen3.8-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as qwen/qwen3.8-max (the provider's alias of the same model) |
+| omp | step-3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | My TokenRouter | served as stepfun/step-3.7-flash (the provider's alias of the same model) |
+
+33 pairs, 164 of 164 scenario runs passed.
+
+## Provider: omp-vercel
+
+| Harness | Model | First | Follow-up | Switch | Artifact | Recycle | Served by | Notes |
+|---|---|---|---|---|---|---|---|---|
+| omp | claude-fable-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-fable-5 (the provider's alias of the same model) |
+| omp | claude-fable-5-1 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-fable-5.1 (the provider's alias of the same model) |
+| omp | claude-haiku-4.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-haiku-4.5 (the provider's alias of the same model) |
+| omp | claude-opus-4.7 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-opus-4.7 (the provider's alias of the same model) |
+| omp | claude-opus-4.8 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-opus-4.8 (the provider's alias of the same model) |
+| omp | claude-opus-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-opus-5 (the provider's alias of the same model) |
+| omp | claude-sonnet-4.6 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-sonnet-4.6 (the provider's alias of the same model) |
+| omp | claude-sonnet-5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as anthropic/claude-sonnet-5 (the provider's alias of the same model) |
+| omp | deepseek-v4-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as deepseek/deepseek-v4-flash (the provider's alias of the same model) |
+| omp | deepseek-v4-pro | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as deepseek/deepseek-v4-pro (the provider's alias of the same model) |
+| omp | gemini-2.5-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-2.5-flash (the provider's alias of the same model) |
+| omp | gemini-2.5-flash-lite | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-2.5-flash-lite (the provider's alias of the same model) |
+| omp | gemini-2.5-pro | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-2.5-pro (the provider's alias of the same model) |
+| omp | gemini-3-flash-preview | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3-flash (the provider's alias of the same model) |
+| omp | gemini-3.1-flash-lite | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3.1-flash-lite (the provider's alias of the same model) |
+| omp | gemini-3.1-pro-preview | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3.1-pro-preview (the provider's alias of the same model) |
+| omp | gemini-3.5-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3.5-flash (the provider's alias of the same model) |
+| omp | gemini-3.5-flash-lite | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3.5-flash-lite (the provider's alias of the same model) |
+| omp | gemini-3.6-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3.6-flash (the provider's alias of the same model) |
+| omp | gemini-3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3.7-flash (the provider's alias of the same model) |
+| omp | gemini-3.8-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as google/gemini-3.8-flash (the provider's alias of the same model) |
+| omp | glm-5.3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as zai/glm-5.3 (the provider's alias of the same model) |
+| omp | glm-5.3-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as zai/glm-5.3-flash (the provider's alias of the same model) |
+| omp | gpt-5.2 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.2 (the provider's alias of the same model) |
+| omp | gpt-5.3-codex | pass | pass | n/a | pass | pass | Vercel AI Gateway | served as openai/gpt-5.3-codex (the provider's alias of the same model) |
+| omp | gpt-5.4 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.4 (the provider's alias of the same model) |
+| omp | gpt-5.4-mini | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.4-mini (the provider's alias of the same model) |
+| omp | gpt-5.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.5 (the provider's alias of the same model) |
+| omp | gpt-5.6-luna | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.6-luna (the provider's alias of the same model) |
+| omp | gpt-5.6-sol | pass | pass | pass (gpt-5.6-terra) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.6-sol (the provider's alias of the same model) |
+| omp | gpt-5.6-terra | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as openai/gpt-5.6-terra (the provider's alias of the same model) |
+| omp | kimi-k2.7-code | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as moonshotai/kimi-k2.7-code (the provider's alias of the same model) |
+| omp | kimi-k3 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as moonshotai/kimi-k3 (the provider's alias of the same model) |
+| omp | mistral-medium-3.5 | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as mistral/mistral-medium-3.5 (the provider's alias of the same model) |
+| omp | qwen3.7-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.7-max (the provider's alias of the same model) |
+| omp | qwen3.8-max | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as alibaba/qwen3.8-max (the provider's alias of the same model) |
+| omp | step-3.7-flash | pass | pass | pass (gpt-5.6-sol) | pass | pass | Vercel AI Gateway | served as stepfun/step-3.7-flash (the provider's alias of the same model) |
+
+37 pairs, 184 of 184 scenario runs passed.
## Provider: openai
diff --git a/scripts/support-matrix/render.py b/scripts/support-matrix/render.py
index 3299080c..a52eb4cc 100644
--- a/scripts/support-matrix/render.py
+++ b/scripts/support-matrix/render.py
@@ -1,5 +1,7 @@
"""Render the support matrix JSON as markdown: one table per provider, a row per harness x model."""
-import json, sys, collections
+import json, sys, collections, os
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from samemodel import alias_of # noqa: E402
res = json.load(open(sys.argv[1]))
mark = lambda r: 'n/a' if not r or r.get('ok') is None else ('pass' if r.get('ok') else 'FAIL')
by = collections.defaultdict(list)
@@ -23,12 +25,18 @@ for prov, rows in sorted(by.items()):
# a turn served by a connection other than the one under test is a finding, never a pass
findings.append(f"{r['harness']} x {r['model']}: served by {', '.join(r['foreign'])} (turn records: {', '.join(r.get('connections') or [])})")
notes.insert(0, "served by another connection (finding below)")
- if r.get('substituted'):
- # the CLI ran a model other than the id asked for: the id is not served as itself, a finding
- findings.append(f"{r['harness']} x {r['model']}: served as {', '.join(r['substituted'])} (the CLI reports the model it ran)")
- notes.insert(0, f"served as {', '.join(r['substituted'])} (finding below)")
+ # The runner records a served id other than the id asked for, exactly. An aggregator's
+ # vendor prefix or a provider's version suffix is the same model (the gateway's rule, in
+ # samemodel.py): noted and counted. Another family, number or tier is a finding, not counted.
+ aliases = [x for x in (r.get('substituted') or []) if alias_of(r['model'], x)]
+ others = [x for x in (r.get('substituted') or []) if not alias_of(r['model'], x)]
+ if aliases:
+ notes.insert(0, f"served as {', '.join(aliases)} (the provider's alias of the same model)")
+ if others:
+ findings.append(f"{r['harness']} x {r['model']}: served as {', '.join(others)} (the CLI reports the model it ran)")
+ notes.insert(0, f"served as {', '.join(others)} (finding below)")
out.append(f"| {r['harness']} | {r['model']} | {mark(r.get('first'))} | {mark(r.get('followup'))} | {mark(sw)}{(' ('+sw['to']+')') if sw.get('to') else ''} | {mark(r.get('artifact'))} | {mark(r.get('recycle'))} | {served} | {' ; '.join(notes).replace('|', '/')} |")
- clean = [r for r in rows if not r.get('foreign') and not r.get('substituted')]
+ clean = [r for r in rows if not r.get('foreign') and all(alias_of(r['model'], x) for x in (r.get('substituted') or []))]
ok = sum(1 for r in clean for sc in ('first','followup','switch','artifact','recycle') if (r.get(sc) or {}).get('ok') is True)
tot = sum(1 for r in clean for sc in ('first','followup','switch','artifact','recycle') if (r.get(sc) or {}).get('ok') is not None)
out += ["", f"{len(rows)} pairs, {ok} of {tot} scenario runs passed" + (f"; {len(rows) - len(clean)} pairs served by another connection or as another model are findings, not counted." if len(clean) < len(rows) else "."), ""]
diff --git a/scripts/support-matrix/samemodel.py b/scripts/support-matrix/samemodel.py
new file mode 100644
index 00000000..48f6aad5
--- /dev/null
+++ b/scripts/support-matrix/samemodel.py
@@ -0,0 +1,30 @@
+"""Whether a served model id names the model the pair asked for: the gateway's rule, verbatim.
+
+An aggregator adds or drops a vendor prefix (anthropic/claude-fable-5 is claude-fable-5) and a
+provider appends a dated or versioned suffix (claude-haiku-4-5-20251001, gemini-2.5-flash-001);
+those are the provider's alias of the same model. A different family or number under any prefix
+(google/gemini-3-flash-preview for gemini-3.8-flash), or a tier word such as -lite, is another
+model: a finding. The OSS tree carries the same file; keep them identical."""
+import re
+
+_VERSION_SUFFIX_RE = re.compile(r"^(-(\d{2,}|v\d+|preview|latest))+$") # -20251001, -001, -v2, -preview-05-20; never -lite
+
+
+def same_model(asked: str, served: str) -> bool:
+ def norm(m: str) -> str:
+ m = (m or "").strip().lower().split("/")[-1]
+ return m.replace(".", "-")
+ a, s = norm(asked), norm(served)
+ if not a or not s:
+ return False
+ if a == s:
+ return True
+ long, short = (s, a) if len(s) > len(a) else (a, s)
+ return long.startswith(short + "-") and _VERSION_SUFFIX_RE.match(long[len(short):]) is not None
+
+
+def alias_of(asked: str, served: str) -> bool:
+ """A served entry may join the models one turn ran on with commas; it is an alias only when
+ every id in it is the same model."""
+ ids = [x.strip() for x in str(served or "").split(",") if x.strip()]
+ return bool(ids) and all(same_model(asked, x) for x in ids)
diff --git a/scripts/support-matrix/test_samemodel.py b/scripts/support-matrix/test_samemodel.py
new file mode 100644
index 00000000..43728de6
--- /dev/null
+++ b/scripts/support-matrix/test_samemodel.py
@@ -0,0 +1,20 @@
+from samemodel import alias_of, same_model
+
+
+def test_the_providers_alias_of_the_same_model_is_the_same_model():
+ for asked, served in [("claude-fable-5", "anthropic/claude-fable-5"), ("gpt-5.6-sol", "openai/gpt-5.6-sol"),
+ ("mistral-medium-3.5", "mistralai/mistral-medium-3-5"), ("qwen3.8-max", "qwen/qwen3.8-max-0902"),
+ ("claude-haiku-4-5", "claude-haiku-4-5-20251001"), ("gemini-2.5-flash", "gemini-2.5-flash-001")]:
+ assert same_model(asked, served), (asked, served)
+
+
+def test_another_family_number_or_tier_is_a_finding():
+ for asked, served in [("gemini-3.8-flash", "google/gemini-3-flash-preview"), ("gemini-2.5-flash", "gemini-2.5-flash-lite"),
+ ("claude-sonnet-5", "claude-opus-5"), ("gemini-3.8-flash", "")]:
+ assert not same_model(asked, served), (asked, served)
+
+
+def test_a_turn_on_several_models_is_an_alias_only_when_all_are_the_same_model():
+ assert alias_of("gemini-3.8-flash", "google/gemini-3.8-flash, gemini-3.8-flash-001")
+ assert not alias_of("gemini-3.8-flash", "gemini-3.8-flash, gemini-3-flash-preview")
+ assert not alias_of("gemini-3.8-flash", "")