HarnessRouter Community Edition: the self-hosted, Apache-2.0 edition of the unified interface for agent harnesses. Run Codex, Claude Code, Hermes, PI, DSH, and more through one API, with sessions, streaming, files, cancellation, and failure handling. Implements the Unified Harness Protocol (UHP), an open standard. Your keys, your infrastructure.
Python agent-apiagent-frameworkagent-harnessai-agentsai-infrastructurecoding-agentconformance-testingdeepseek-harnessharness-engineeringhermesllmopen-protocolopen-sourceopen-standardpi-coding-agentself-hosteduhpunified-apiunified-harness-protocolunified-interface
relay: hermes and Codex on Azure name the model that served them and count cached input (#467)
github-actions[bot]fb6392a7846d
diff --git a/docs/support-matrix-notes.md b/docs/support-matrix-notes.mdindex 5c8e66e3..2bfc9d17 100644--- a/docs/support-matrix-notes.md+++ b/docs/support-matrix-notes.md@@ -6,6 +6,34 @@ The tables in [support-matrix.md](support-matrix.md) were produced by `scripts/sThe run started on v0.13.5 and finished on v0.13.13. Every release between them came out of a finding below and was deployed on the instance behind a live-turn gate before the next column: 0.13.5 (local blob store lists by prefix), 0.13.6 (read caches, the word "refused" is not a key refusal), 0.13.7 (a Codex history kept whole under the same account, finished turns release their process handle), 0.13.8 (a Google key can be saved, qwen drops gpt-5.3-codex, a task reopened by URL keeps its model, the broker resends a Google request without the refused field, a self-hosted sandbox reaches the broker on loopback), 0.13.9 (opencode's base carries /v1), 0.13.10 (the relay's base carries its API version), 0.13.11 (the claude CLI strips it), 0.13.12 (a checkpoint that cannot be restored aborts the turn), 0.13.13 (owner trust normalises an Azure base like the broker).+## hermes and Codex on Azure name the model that served them (2026-10-15)++The open item from 0.33.0-rc.15: hermes switching into gpt-5.5 on Azure finished with no served+model, four of four, and Codex the same, so the same-model rule could not judge those turns. Neither+CLI reports one; the relay reads it, and both bases called Azure directly (Codex also OpenAI's own+endpoint). Both now ride the relay at the connection's own base. Found on the way: the turn's route+was the first placeholder in the environment, and hermes registers its image question route first,+so a hermes turn whose image questions ride the relay read its served model, usage and finish+reason off the image model.++Measured on 0.34.0 in a side container on the test VM (codex and hermes only, a fresh volume), one+turn each straight to the runner, gpt-5.5 on the instance's Azure resource, "Reply with exactly the+word: pong", the hermes turn with its image questions routed through the relay. Then the same image+with this change's runner, then the stock one again with the provider's cache warm:++| Turn | 0.34.0 | with the change | 0.34.0, cache warm |+|---|---|---|---|+| codex (exec) | no model; 11,637 in | gpt-5.5; 1,525 in + 10,112 cached | no model; 11,637 in |+| codex (app-server) | no model; 12,372 in | gpt-5.5; 1,236 in + 11,136 cached | no model; 1,236 in + 11,136 cached |+| hermes | no model; 421 in | gpt-5.5; 421 in + 10,240 cached | no model; 421 in |++So the usage changes too. Codex's exec stream counts cached input as input (every cached token at+the full price) and hermes's counters leave cached input out; the provider's own split, read by the+relay, replaces both, as it already did on every relayed route. Codex's app-server already split+its own count and gains only the served model. Codex compacts a long thread locally behind the+relay; on an Azure URL it had used the provider's /responses/compact (0.154.0). It sends+`store: false` either way.+## Muse Wonder (2026-10-14 and 15)Meta announced two models. Muse Wonder Full has no API: Meta serves it only inside its own Muse Codediff --git a/runner/server.py b/runner/server.pyindex ce7bd031..a85a73da 100644--- a/runner/server.py+++ b/runner/server.py@@ -2714,17 +2714,22 @@ def _codex_prepare_env(provider: str, auth: Auth, model: str, cwd: str,marker.write_text(fp)except OSError:pass- # A connection that is not OpenAI's own or Azure's rides the relay, as it does for every other- # base. Codex groups a server's MCP tools in a namespace and adds an empty `properties` to each- # object node of their schemas; through Vercel the tool then received empty objects with- # gpt-6-luna and gpt-6.1-sol (two of two, 2026-10-05; not with gpt-5.4, whose tool definitions- # Codex defers behind a tool search), and the repair lives in the relay- # (_with_free_form_objects). The CLI is given a loopback URL and a placeholder, so the- # connection's key is no longer in its environment either. OpenAI's own endpoint and Azure's- # keep the direct route: both were intact as Codex sends it. The account fingerprint above is- # taken from the connection itself, never from the route, which is new on every turn.+ # Every connection rides the relay, as it does for every other base. Codex groups a server's MCP+ # tools in a namespace and adds an empty `properties` to each object node of their schemas;+ # through Vercel the tool then received empty objects with gpt-6-luna and gpt-6.1-sol (two of+ # two, 2026-10-05; not with gpt-5.4, whose tool definitions Codex defers behind a tool search),+ # and the repair lives in the relay (_with_free_form_objects). OpenAI's own endpoint and Azure's+ # were intact as Codex sends it and kept a direct route until 2026-10-15, but the relay is also the+ # one place a turn's served model is read: Codex reports none, so a turn on those two finished+ # unlabelled and the matrix could not tell which model had answered (gpt-5.5 on Azure). The+ # CLI is given a loopback URL and a placeholder, so the connection's key is not in its+ # environment either. One thing Codex decides from the URL: on an Azure host (or a provider+ # named "azure") it compacts a long thread through the provider's /responses/compact+ # (codex-rs/model-provider/src/provider.rs, 0.154.0); behind the loopback it compacts locally,+ # as it does on every other connection. The account fingerprint above is taken from the+ # connection itself, never from the route, which is new on every turn.cli_base, cli_key = base_url, auth.api_key- if p == "tokenrouter" and auth.api_key:+ if auth.api_key:cli_base, cli_key = _hermes_relay_route(base_url, auth.api_key, exact_base=True)# codex only speaks the OpenAI Responses API now — current releases removed# `wire_api = "chat"` entirely, so a custom chat-completions endpoint cannot be driven by@@ -5052,39 +5057,37 @@ def _finish_reason_in(data: bytes) -> str:return hits[-1].decode().replace("-", "_") if hits else ""+def _turn_relay_flags(env: dict) -> dict:+ """The flags of this turn's chat route, {} when the turn did not ride the relay. The route is+ found by the placeholder bearer the CLI was handed, whichever variable the backend keeps it in.++ A helper model's route is passed over. hermes registers one for its image questions+ (HR_VISION_API_KEY, marked `aux`), and registers it BEFORE the chat route, so the first bearer+ in the environment was the image route's: a hermes turn whose image questions ride the relay+ read its served model, usage and finish reason off the model it asks about pictures, "" until+ it asked one and the image model after. Rule 2 of docs/harness-verification.md reads the+ second as a substitution, a finding invented out of nothing."""+ for v in env.values():+ if isinstance(v, str) and v.startswith("hr-relay-"):+ route = _HERMES_RELAY["routes"].get(v)+ if route and not (route[2] or {}).get("aux"):+ return route[2] or {}+ return {}++def _relay_last_finish(env: dict) -> str:"""The finish_reason of the LAST provider answer on this turn's route, as the relay saw it.A "content_filter" here is the provider declining the request; a CLI that reports it as anempty answer (goose: "The model returned an empty response") has hidden a failure."""- for v in env.values():- if isinstance(v, str) and v.startswith("hr-relay-"):- route = _HERMES_RELAY["routes"].get(v)- if route:- # normalised the way _finish_reason_in normalises, so a caller compares one- # spelling whoever wrote the flag (Vercel says "content-filter", OpenAI- # "content_filter")- return str((route[2] or {}).get("last_finish") or "").replace("-", "_")- return ""+ # normalised the way _finish_reason_in normalises, so a caller compares one spelling whoever+ # wrote the flag (Vercel says "content-filter", OpenAI "content_filter")+ return str(_turn_relay_flags(env).get("last_finish") or "").replace("-", "_")def _relay_served_model(env: dict) -> str:"""What the relay saw the provider serve on this turn's route, "" when the turn did not ride- the relay or nothing answered yet. The route is found by the placeholder bearer the CLI was- handed, whichever variable the backend keeps it in.-- ONE relay token per env is the invariant this rests on — the first one found is taken as the- turn's chat route. It holds for every backend that reaches here: only _hermes_prepare_env- registers a second route (the auxiliary vision model, HR_VISION_API_KEY), and hermes runs on- _run_hermes_bg, which does not call this. A backend that ever hands its CLI two relay bearers- must distinguish them here first: reading the vision route would stamp the image model as the- turn's served model, which rule 2 of docs/harness-verification.md reads as a substitution —- a finding invented out of nothing."""- for v in env.values():- if isinstance(v, str) and v.startswith("hr-relay-"):- route = _HERMES_RELAY["routes"].get(v)- if route:- return str((route[2] or {}).get("served_model") or "")- return ""+ the relay or nothing answered yet (see _turn_relay_flags for which route that is)."""+ return str(_turn_relay_flags(env).get("served_model") or "")def _fill_relay_usage(ev: dict, env: dict) -> None:@@ -5136,20 +5139,15 @@ def _stamp_thinking(ev: dict, rec: dict) -> None:def _relay_usage(env: dict) -> dict:"""The tokens the provider reported on this turn's route, summed over its calls; {} when the- turn did not ride the relay or nothing carried usage. Found by the placeholder bearer, exactly- as _relay_served_model finds the served model, and subject to the same one-route invariant.+ turn did not ride the relay or nothing carried usage. Read off the same route as the served+ model (_turn_relay_flags).This is the usage of every backend whose own stream reports none (kimi, aider): the provider'sstatement of what it counted, read off the bytes as they pass, rather than a number the CLIestimated or nothing at all. A backend that reports its own usage keeps it; this only fills aresult event that arrived empty."""- for v in env.values():- if isinstance(v, str) and v.startswith("hr-relay-"):- route = _HERMES_RELAY["routes"].get(v)- if route:- u = (route[2] or {}).get("usage")- return dict(u) if isinstance(u, dict) and u else {}- return {}+ u = _turn_relay_flags(env).get("usage")+ return dict(u) if isinstance(u, dict) and u else {}# A local model server's own discovery endpoints (Ollama, llama.cpp, LM Studio, vLLM), as the path@@ -6102,9 +6100,10 @@ def _hermes_relay_route(base_url: str, api_key: str, drop_fields: tuple[str, ..._build_grok for the one client that needs it). `usage_no_nulls` rewrites a null token count in ananswer's usage to 0 for a client whose parser takes only a number there (_usage_without_nulls).`exact_base` joins the client's resource onto the connection's base as stored, for a client that- was calling that base directly before it had a route (Codex): the same URL, with the relay between.- `aux` marks a route for a helper model (hermes's image questions): the turn's thinking level is- the main model's and stays off it."""+ was calling that base directly before it had a route (Codex, and hermes on Azure): the same URL,+ with the relay between. `aux` marks a route for a helper model (hermes's image questions): the+ turn's thinking level is the main model's and stays off it, and what it serves is not the turn's+ (_turn_relay_flags passes it over)."""with _HERMES_RELAY["lock"]:if _HERMES_RELAY["server"] is None:srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), _HermesRelayHandler)@@ -6121,7 +6120,7 @@ def _hermes_relay_route(base_url: str, api_key: str, drop_fields: tuple[str, ...flags = {"rename_max_tokens": False, "drop_fields": tuple(drop_fields),"stream_usage": bool(stream_usage), "gemini_schemas": bool(gemini_schemas),"usage_no_nulls": bool(usage_no_nulls), "plain_stream": bool(plain_stream),- **({} if aux else _turn_effort())}+ **({"aux": True} if aux else _turn_effort())}_HERMES_RELAY["routes"][tok] = (upstream, api_key, flags if aux else _turn_route(flags))return f"http://127.0.0.1:{_HERMES_RELAY['port']}/v1", tok@@ -6202,9 +6201,20 @@ def _hermes_prepare_env(provider: str | None, auth: Auth, cwd: str, env: dict,elif auth.api_key:env["ANTHROPIC_API_KEY"] = auth.api_keyelif p == "azure-foundry": # Azure OpenAI (gpt family) — OpenAI-style endpoint + key- if auth.api_key:+ if auth.api_key and auth.base_url:+ # Through the relay, as every other hermes route. This one called Azure directly with the+ # connection's key in the CLI's environment, and the relay is the one place a turn's+ # served model is read: hermes reports none, so every hermes turn on Azure finished+ # unlabelled and the matrix could not tell which model had answered (gpt-5.5, four of+ # four on 0.33.0-rc.15). hermes's azure-foundry provider is the OpenAI SDK on the base it+ # is given, a bearer key; its own Azure rules read the URL for two things the runner+ # already settles by model (api_mode, written for the Responses family, and the+ # max_completion_tokens spelling, which hermes also infers from a gpt-5 name).+ env["AZURE_FOUNDRY_BASE_URL"], env["AZURE_FOUNDRY_API_KEY"] = _hermes_relay_route(+ auth.base_url, auth.api_key, exact_base=True)+ elif auth.api_key:env["AZURE_FOUNDRY_API_KEY"] = auth.api_key- if auth.base_url:+ elif auth.base_url:env["AZURE_FOUNDRY_BASE_URL"] = auth.base_urlelif p == "openrouter": # OpenRouter aggregator (vendor/model ids)if auth.api_key:@@ -12072,6 +12082,12 @@ def _run_codex_appserver_bg(turn_id: str, cwd: str, env: dict, model: str, prompres_txt = "\n\n".join(x for x in (state["final"].strip(), err_txt) if x)[:4000] or err_txtev = {"type": "result", "subtype": "success" if ok else "error", "is_error": not ok,"result": res_txt, "usage": usage} # surface the error in the trace+ # What the relay saw the provider say, as on the exec path: the app-server reports no served+ # model either.+ served = _relay_served_model(env)+ if served:+ ev["model"] = served+ _fill_relay_usage(ev, env)_stamp_thinking(ev, rec)append(ev)rec["result"] = state["final"]diff --git a/runner/tests/test_codex_aggregator_relay.py b/runner/tests/test_codex_aggregator_relay.pyindex e66dc47b..e12ab945 100644--- a/runner/tests/test_codex_aggregator_relay.py+++ b/runner/tests/test_codex_aggregator_relay.py@@ -57,12 +57,20 @@ def test_the_route_keeps_the_connection_base_exactly(tmp_path):rs._HERMES_RELAY["routes"].pop(tok, None)-def test_openai_and_azure_keep_their_direct_route(tmp_path):+def test_openai_and_azure_ride_the_relay_too(tmp_path):+ """They kept a direct route until 2026-10-15, and a turn on them finished with no served model:+ Codex reports none and the relay is where it is read (gpt-5.5 on Azure, 0.33.0-rc.15)."""for provider, var, base in (("openai", "OPENAI_API_KEY", "https://api.openai.com/v1"),("azure", "AZURE_OPENAI_API_KEY", "https://res.openai.azure.com/openai/v1")):env, cfg = _prepare(tmp_path, provider, rs.Auth(api_key="sk-direct", base_url=base))- assert env[var] == "sk-direct"- assert f'base_url = "{base}"' in cfg and "127.0.0.1" not in cfg+ tok = env[var]+ try:+ assert tok.startswith("hr-relay-")+ assert "sk-direct" not in json.dumps(env) and "sk-direct" not in cfg+ assert base not in cfg and 'base_url = "http://127.0.0.1:' in cfg+ assert rs._HERMES_RELAY["routes"][tok][:2] == (base, "sk-direct")+ finally:+ rs._HERMES_RELAY["routes"].pop(tok, None)def test_the_account_fingerprint_follows_the_connection_not_the_route(tmp_path):diff --git a/runner/tests/test_codex_reasoning_strip.py b/runner/tests/test_codex_reasoning_strip.pyindex aa414b93..7ba746ab 100644--- a/runner/tests/test_codex_reasoning_strip.py+++ b/runner/tests/test_codex_reasoning_strip.py@@ -145,10 +145,16 @@ def test_a_resumed_session_keeps_every_provider_id_it_ran_under(tmp_path, monkeycfg = (home / ".codex" / "config.toml").read_text()assert 'model_provider = "hr-openai"' in cfg and "[model_providers.hr-openai]" in cfgassert "[model_providers.hr-tokenrouter]" in cfg and "[model_providers.azure]" in cfg- assert cfg.count('base_url = "https://broker.example/v1/llm"') == 3, "every alias points at this turn's endpoint"+ # this turn's endpoint is its relay route (see _codex_prepare_env)+ route = cfg.split('base_url = "', 1)[1].split('"', 1)[0]+ assert route.startswith("http://127.0.0.1:") and "broker.example" not in cfg+ assert cfg.count(f'base_url = "{route}"') == 3, "every alias points at this turn's endpoint"fresh = {"HOME": str(tmp_path / "fresh")}_codex_prepare_env("openai", Auth(api_key="sk-x", base_url="https://broker.example/v1/llm"), "gpt-5.4-mini", str(tmp_path), fresh, resume=False)assert "[model_providers.hr-tokenrouter]" not in (tmp_path / "fresh" / ".codex" / "config.toml").read_text(), "a fresh session declares only its own"+ from server import _HERMES_RELAY+ for e in (env, fresh):+ _HERMES_RELAY["routes"].pop(e.get("OPENAI_API_KEY"), None)def test_a_failure_names_the_providers_refusal_over_the_clis_retries():diff --git a/runner/tests/test_relay_turn_route.py b/runner/tests/test_relay_turn_route.pynew file mode 100644index 00000000..1f5844b3--- /dev/null+++ b/runner/tests/test_relay_turn_route.py@@ -0,0 +1,74 @@+"""Which relay route is the turn's, and the two bases that did not ride one on Azure.++The relay is where a turn's served model is read (hermes and Codex report none of their own), so a+turn whose CLI called the provider directly finished unlabelled: hermes and Codex on Azure, where the+matrix could not tell which model answered gpt-5.5 (four of four hermes switches on 0.33.0-rc.15).+And a hermes turn whose image questions ride the relay registers that route FIRST, so the lookup+that takes the first placeholder in the environment read the image model's route instead.+"""+import json+import pathlib+import sys++sys.path.insert(0, str(pathlib.Path(__file__).resolve().parents[1]))+import server as rs # noqa: E402+++def _drop(env):+ for v in env.values():+ if isinstance(v, str) and v.startswith("hr-relay-"):+ rs._HERMES_RELAY["routes"].pop(v, None)+++def test_the_image_route_is_not_the_turns_even_when_it_comes_first(tmp_path):+ env = {"HOME": str(tmp_path)}+ rs._hermes_prepare_env("openrouter", rs.Auth(api_key="k-chat", base_url="https://openrouter.ai/api/v1"),+ str(tmp_path), env, model="openai/gpt-5.5",+ vision_auth={"provider": "openai-api", "model": "gpt-5.4-mini",+ "base_url": "https://vision.example/v1", "api_key": "k-vision"})+ try:+ tokens = [v for v in env.values() if isinstance(v, str) and v.startswith("hr-relay-")]+ assert tokens[0] == env["HR_VISION_API_KEY"], "the case this guards: the image route is first"+ vision = rs._HERMES_RELAY["routes"][env["HR_VISION_API_KEY"]][2]+ chat = rs._HERMES_RELAY["routes"][env["OPENROUTER_API_KEY"]][2]+ # before either answered+ assert rs._relay_served_model(env) == "" and rs._relay_usage(env) == {}+ vision.update(served_model="gpt-5.4-mini-2026-03-17", usage={"input_tokens": 900, "output_tokens": 40},+ last_finish="stop")+ assert rs._relay_served_model(env) == "", "the image model is not what served the turn"+ assert rs._relay_usage(env) == {} and rs._relay_last_finish(env) == ""+ chat.update(served_model="openai/gpt-5.5", usage={"input_tokens": 12, "output_tokens": 3},+ last_finish="content-filter")+ assert rs._relay_served_model(env) == "openai/gpt-5.5"+ assert rs._relay_usage(env) == {"input_tokens": 12, "output_tokens": 3}+ assert rs._relay_last_finish(env) == "content_filter"+ finally:+ _drop(env)+++def test_hermes_on_azure_rides_the_relay_at_the_connections_own_base(tmp_path):+ base = "https://res.openai.azure.com/openai/v1"+ env = {"HOME": str(tmp_path)}+ rs._hermes_prepare_env("azure-foundry", rs.Auth(api_key="sk-azure", base_url=base),+ str(tmp_path), env, model="gpt-5.5")+ try:+ tok = env["AZURE_FOUNDRY_API_KEY"]+ assert tok.startswith("hr-relay-")+ assert env["AZURE_FOUNDRY_BASE_URL"].startswith("http://127.0.0.1:")+ assert "sk-azure" not in json.dumps(env)+ assert "sk-azure" not in (tmp_path / ".hermes" / "config.yaml").read_text()+ upstream, key, flags = rs._HERMES_RELAY["routes"][tok]+ assert (upstream, key) == (base, "sk-azure") and not flags.get("aux")+ flags["served_model"] = "gpt-5.5-2026-04-23"+ assert rs._relay_served_model(env) == "gpt-5.5-2026-04-23"+ finally:+ _drop(env)+++def test_the_codex_app_server_result_carries_the_served_model_and_the_relays_usage():+ """The exec path stamped both; the app-server path stamped neither."""+ import inspect+ src = inspect.getsource(rs._run_codex_appserver_bg)+ tail = src[src.index('ev = {"type": "result"'):]+ assert tail.index("_relay_served_model(env)") < tail.index("append(ev)")+ assert tail.index("_fill_relay_usage(ev, env)") < tail.index("append(ev)")diff --git a/runner/tests/test_relay_usage.py b/runner/tests/test_relay_usage.pyindex 7f7f585f..86a8134f 100644--- a/runner/tests/test_relay_usage.py+++ b/runner/tests/test_relay_usage.py@@ -144,5 +144,5 @@ def test_a_backend_that_reports_its_own_usage_keeps_it():split count (codex, claude) is the record. Pinned on the source of the run loop, which needsa live process to exercise: both result sites go through the one helper."""src = pathlib.Path(__file__).resolve().parents[1].joinpath("server.py").read_text()- assert src.count("_fill_relay_usage(ev, env)") == 3 # the run loop, its eof path, the hermes driver+ assert src.count("_fill_relay_usage(ev, env)") == 4 # the run loop, its eof path, the hermes and codex app-server driversassert src.count('if ev.get("type") == "result" and not ev.get("usage"):') == 0