Skip to content

HarnessRouter Community Edition: the self-hosted, Apache-2.0 edition of the unified interface for agent harnesses. Run Codex, Claude Code, Hermes, PI, DSH, and more through one API, with sessions, streaming, files, cancellation, and failure handling. Implements the Unified Harness Protocol (UHP), an open standard. Your keys, your infrastructure.

A turn can say how much the model thinks, on every base (#400)

github-actions[bot]915c83471bb7
diff --git a/docs/self-hosting-guide.md b/docs/self-hosting-guide.md
index 90e1dfeb..8937cdf6 100644
--- a/docs/self-hosting-guide.md
+++ b/docs/self-hosting-guide.md
@@ -975,7 +975,7 @@ To add a plugin to a harness that already exists, remember that `PUT /v1/harness
the whole mutable configuration: any field you leave out is cleared, and an omitted `mcp_servers`
also removes the hosted tools behind it. Read the harness first, then send back its current
`system_prompt`, `default_model`, `mcp_servers`, `skills`, `disabled_tools`, `additional_headers`,
-`max_step` and `timeout_seconds` (the write body uses these snake_case names; the record you read
+`max_step`, `timeout_seconds` and `reasoning_effort` (the write body uses these snake_case names; the record you read
uses camelCase) with the new package appended to `plugins`. Installed plugins round-trip as
`{name, enabled, blob}`, so send those back unchanged and add the new one beside them.
@@ -1044,6 +1044,33 @@ back at once with the task's id and `status: in_progress`, and `GET /v1/response
it until it ends. Attached files land in the task's working directory under the name you give;
a relative path in `filename` (`inputs/report.pdf`) puts the file in that folder.
+**How much the model thinks.** A task can say it, with the Responses API's own field:
+
+```bash
+curl -sS "$HR/api/harness/v1/responses" -H "authorization: Bearer $KEY" -H 'content-type: application/json' \
+ -d '{"input": "...", "metadata": {"harness_id": "hrn_..."}, "reasoning": {"effort": "low"}}'
+```
+
+The levels are `none`, `minimal`, `low`, `medium`, `high` and `xhigh`. A harness can keep a default
+for its tasks (`reasoning_effort` on the harness, the Thinking control on its settings page) and a
+task's own level wins. Models do not all have the same levels: `GET /v1/bases` lists, per base and
+model, the ones each has (`models[].reasoning`), and a model asked for a level it lacks gets the
+nearest one it has. A model that cannot be turned off gets its lowest level for `none`; a task
+that asked for some thinking is never given none. The response says what happened:
+
+```json
+"reasoning": {"effort": "minimal", "applied": "low"},
+"usage": {"output_tokens": 262, "output_tokens_details": {"reasoning_tokens": 248}}
+```
+
+`applied` is `"default"` when nothing could be set: the model has no measured level on the
+connection that served it, or the provider refused the level, in which case the call was made
+again without it rather than failed. `reasoning_tokens` is the provider's own count and is absent
+where the provider gives none. A task and a harness that set no level send what they sent before
+levels existed. Which setting moves which model on which provider was measured, not assumed; the
+table and the method are in [support-matrix-notes.md](support-matrix-notes.md), "How much a model
+thinks".
+
<details>
<summary>The rest of the surface</summary>
diff --git a/docs/support-matrix-notes.md b/docs/support-matrix-notes.md
index 492afab4..f12aebf0 100644
--- a/docs/support-matrix-notes.md
+++ b/docs/support-matrix-notes.md
@@ -2585,3 +2585,176 @@ columns 5 of 5.
**Not measured.** A Codex turn whose provider stays silent past the relay's wait: the relay ends such a
call with its reason, as for every other base, and Codex was not driven into that case here.
+
+## How much a model thinks (2026-10-05)
+
+A turn can ask for a thinking level: `reasoning: {"effort": ...}` on a task, `reasoning_effort` on a
+harness, one of none, minimal, low, medium, high, xhigh. This section is what was measured to make
+that true, and it is the source of the table in `runner/reasoning.py`.
+
+**Method.** One question with a short answer, sent with no setting and then once per setting, straight
+at each provider through a connection of hr-test (`scripts/support-matrix/thinking/probe.py`, run as a
+shell command of a turn so no key leaves the instance). Judged by the provider's own usage:
+`reasoning_tokens`, with output tokens and elapsed time where a provider gives no such count. Seven
+routes (Vercel, TokenRouter, OpenRouter, OpenAI, Azure, Google AI Studio, and Anthropic's Messages API
+as TokenRouter passes it through), 66 model ids, three API shapes, about 900 calls. One sample per
+cell: the direction is the finding, not the number.
+
+**What it showed.** No field means the same thing everywhere, and a level a model lacks is refused
+with a 400, not ignored.
+
+`reasoning_effort` on Chat Completions:
+
+| models | Vercel | TokenRouter | OpenRouter | the vendor directly |
+|---|---|---|---|---|
+| gpt-5.2, 5.3-codex, 5.4, 5.4-mini | none, low to xhigh; `minimal` 400 | the same | all six | OpenAI and Azure: none, low to xhigh |
+| gpt-5.5, 5.6 line, 6-luna, 6-sol | all six | all six | all six | none, low to xhigh |
+| gpt-6.1-sol, gpt-6-astra | `none` accepted, not honoured | `none` 400 | `none` 400 | `none` 400 |
+| Claude, 4.5 to 5.5 | all six move it; opus-5.5 ignores `none` | accepted; the usage carries no count | the 5.5 line: `none` 400 | (Messages, below) |
+| Gemini 3 line | only `low` differs; `none` and `minimal` read as MORE | ignored | minimal, low, medium, high; `none` 400 | Google: none to high; `xhigh` 400 |
+| Grok 4.3, 4.5, 4.6 | two tiers; `none` reads as low | minimal to xhigh, in order; `none` 400 | the same; `none` 400 | |
+| DeepSeek v4 flash, v4.1 flash, v4 pro | `none` is off, the rest is on | the same; v4-pro `none` 400 | the same | |
+| Kimi k3 | `none` off; levels | `none` off | `none` off | |
+| Qwen 3.7-max, 3.8-flash | `none` off, the rest on | the same; low and medium 400 against the token cap | the same | |
+| Mistral medium 3.5 | none and high; minimal, low, medium 400 | none and every level | the same | |
+| Step 3.7 flash | minimal to xhigh, in order; `none` reads as more | the same | `none` 400 | |
+| MiniMax m3, Ling 3.0 flash, Nemotron 3.5 lightning, Hunyuan 3 | `none` off, the rest on | | the same | |
+| GLM 5.3, Kimi k2.7-code, Llama, Nemotron 3 super | nothing moved it, or it does not think | | | |
+
+The Responses API (`reasoning.effort`, OpenAI models only): as the first three rows, and `minimal` is
+a 400 on every route for every model tried, so `minimal` is never sent to an OpenAI model.
+
+Gemini needs Google's own setting where the field fails, and each aggregator keeps it under its own
+key. Through Vercel, `providerOptions.google.thinkingConfig` with `thinkingBudget: 0` for off and
+`thinkingLevel` for minimal, low, medium and high: gemini-3.5-flash 1,302 thinking tokens with no
+setting, 0 at a zero budget, about 600 at low, about 1,900 at high. Through TokenRouter the same
+setting under `extra_body.google.thinking_config` moves gemini-3.5-flash (0, 605, 1,636, 2,001) and
+nothing else: 3.6, 3.7, 3.8 flash and the lite models answered to no setting there. Per model: the
+pro model and the 3.7 and 3.8 flash models refuse a zero budget and `minimal` ("only works in
+thinking mode", "Thinking level MINIMAL is not supported"); the lite models do not think unless
+asked, and Google refuses `none` for them, so `none` there sends nothing.
+
+Anthropic's Messages API, read through TokenRouter's pass-through (the API's own refusals came back
+with their request ids):
+
+| model | off | a level |
+|---|---|---|
+| claude-haiku-4.5 | `thinking: disabled` | a budget (`enabled`, `budget_tokens`); adaptive and effort are 400 |
+| claude-sonnet-4.6 | `disabled` | adaptive with `output_config.effort` low, medium, high, max; `xhigh` 400 |
+| claude-opus-4.8 | `disabled` | adaptive with an effort; `enabled` 400 |
+| claude-sonnet-5, claude-opus-5 | `disabled` | adaptive with an effort |
+| claude-sonnet-5.5 | `between_tools`, the API's own word; `disabled` 400 | adaptive with an effort, xhigh and max included |
+| claude-fable-5.1 | cannot be turned off (both spellings 400) | adaptive with an effort |
+
+**The rules that follow from it** (`runner/reasoning.py`):
+
+- A level is sent only where a row above showed it accepted. A model that lacks the level asked for
+ gets the nearest one it has; `none` is given only to a turn that asked for it, and a model that
+ cannot be turned off gets its lowest level instead.
+- A model or a route that was not measured gets nothing sent and the turn's record says
+ `"applied": "default"`. A route is known by its host; a custom endpoint is not measured.
+- A level must never be what fails a call. If the provider answers 400 or 422 to a body the relay
+ wrote a level into, the body goes again as the client wrote it; if that is answered, the level is
+ dropped for that model for the rest of the turn.
+- One place by wire shape: the relay writes the level into Chat Completions, Responses, Messages and
+ Google's own bodies, for every base whose calls pass it. Three bases set it their own way: Claude
+ Code (`CLAUDE_CODE_EFFORT_LEVEL`, and `MAX_THINKING_TOKENS` for off and for the model that takes a
+ budget), Codex (`model_reasoning_effort`, which also holds on OpenAI's own endpoint and Azure's),
+ and the DeepSeek Harness driver, whose own relay calls the same functions.
+- The record: `reasoning: {"effort": asked, "applied": level}` on the response, and the provider's
+ count under `usage.output_tokens_details.reasoning_tokens`. The count is read where the relay reads
+ usage and kept apart from the usage that prices a turn, which is unchanged.
+- A turn and a harness that set no level send what they sent before: no route flag, no field, and
+ the tests compare the bytes.
+
+**Not measured, and said so in the table by absence:** Anthropic's API directly (the instance's key
+was refused that day; its rows come from the pass-through and are taken to hold), Bedrock and Vertex,
+any custom endpoint, and these ids: grok-4.20, grok-build-0.1, the other Qwen and Muse models,
+hunyuan-4-preview, claude-fable-5 (taken to be as fable-5.1). The direct OpenAI and Azure Chat
+Completions cells gave erratic counts for some models at some levels (zero where the Responses API
+gave a normal figure for the same level); the levels are accepted there, and the bases that reach
+those endpoints with OpenAI models speak the Responses API.
+
+**The column, every base** (`scripts/support-matrix/thinking/run-column.py`): one harness per base, the
+same task with no level and at none, low and high, through hr-test's connections (TokenRouter for
+these models), judged from the turn's record and the provider's count of thinking tokens. gpt-5.4
+wherever the base lists it. Run in a side container on the branch's runner, five bases at a time.
+
+| base | model | no level | none | low | high |
+|---|---|---:|---:|---:|---:|
+| codex | gpt-5.4 | 270 | 0 | 248 | 516 |
+| hermes | gpt-5.4 | 63 | 0 | 40 | 516 |
+| opencode | gpt-5.4 | 393 | 0 | 313 | 921 |
+| kilo | gpt-5.4 | 478 | 0 | 215 | 513 |
+| openhands | gpt-5.4 | 677 | 0 | 287 | 650 |
+| pi | gpt-5.4 | 0 | 0 | 230 | 485 |
+| omp | gpt-5.4 | 0 | 0 | 262 | 640 |
+| qwen | gpt-5.4 | 0 | 0 | 276 | 1,019 |
+| cline | gpt-5.4 | 0 | 0 | 298 | 778 |
+| kimi | gpt-5.4 | 0 | 0 | 242 | 666 |
+| minimax | gpt-5.4 | 0 | 0 | 192 | 512 |
+| grok | gpt-5.4 | 0 | 0 | 44 | 100 |
+| aider | gpt-5.4 | 0 | 0 | 313 | 512 |
+| agentzero | gpt-5.4 | 0 | 0 | 314 | 638 |
+| cheetahclaws | gpt-5.4 | 0 | 0 | 402 | 797 |
+| gemini | gemini-3.5-flash | 3,158 | no count | 2,531 | 4,125 |
+| goose | gpt-5.4 | 12 out | 12 out | 302 out | 652 out |
+| dsh | gpt-5.4 | 12 out | 12 out | 321 out | 698 out |
+| claude-code | claude-haiku-4.5 | 7,678 out | 859 out | 2,145 out | 7,757 out |
+
+19 of 19 bases that have a model with levels pass: every level asked is recorded as applied, none
+spends no thinking tokens, low spends fewer than high, and a turn with no level carries no
+`reasoning` on its record. System One has no model with levels and offers none. "out" is output
+tokens, where the turn has no thinking count: goose's and the DeepSeek Harness driver's calls do not
+give the relay one, and Anthropic gives none. On Gemini the provider left the count out of the answer
+that spent none (605 output tokens against 13 at the other levels: the answer written out).
+
+Claude Code's row is the weakest judge in the table and is read that way. Anthropic gives no thinking
+count, the level reaches the model as a budget (a ceiling, not a target) and the answer is written
+out at length, so one sample per level can cross: on Anthropic's own endpoint, three samples per level
+gave 791, 810 and 622 output tokens at none, 2,402, 1,700 and 2,128 at low, and 5,195, 2,838 and
+1,863 at high. Off is unmistakable; low below high holds on the sums (6,230 against 9,896) and not on
+every pair. The row above is the first run, through TokenRouter.
+
+**The same build, checked through the console and the API** (the candidate's image with the branch's
+runner and gateway, the day's last code):
+
+- The Thinking control on a harness's settings page, at 1440, 1024, 768 and 390 wide: in view, no
+ sideways scroll. Its options are the model's own (gpt-5.4: Model default, Off, Low, Medium, High,
+ Extra high; gpt-6.1-sol: the same without Off). Saved, reopened, still High. A level kept from
+ another model stays selected and the form says its tasks get the nearest one.
+- A task on that harness with no level of its own: `{"effort": "high", "applied": "high"}`, 15
+ thinking tokens. The same task asking for `none` itself: applied none. A value that is not a level:
+ 400 "reasoning.effort must be one of: none, minimal, low, medium, high, xhigh".
+- hermes with gpt-5.4, asked to write a 9,000 word file as its first action: completed after 619 s
+ with 13,340 output tokens, where 0.29.4 stopped it at 93 s.
+- CheetahClaws: 4 of 4 plain turns carry their usage.
+- A task sent with `backend: "claude-code"` completes, as with `backend: "claude"`.
+
+What "no level" means differs by base on the same model, which the column shows for the first time:
+gpt-5.4 does not think unless asked, and eleven bases leave it so; Codex asks for medium itself (its
+own config default), OpenHands asks for high for every model, and hermes, opencode and kilo ask for
+something of their own. A level set on the harness or the turn replaces all of these.
+
+**Three defects the column and the hosted service found, fixed with it:**
+
+- *A turn's record read "default" for a level that had been applied.* pi and omp keep their relay
+ placeholder in a file, not the environment, and the record was read through the environment: pi with
+ gpt-5.4 spent 12, 252 and 441 output tokens at none, low and high and recorded default three times.
+ A turn now keeps the routes it registered and reads its record off them.
+- *CheetahClaws turns had no usage at all.* The relay added a streamed call's usage to the route when
+ the stream ended, which is when the provider closes it, not when its last event passes; a client
+ that is done at `[DONE]` read the route before its own call was on it. On the published 0.29.4,
+ 8 of 8 plain CheetahClaws turns came back with no usage; with each chunk's figures folded in before
+ the chunk is forwarded, 0 of 6. When this began was not measured (the relay has forwarded streams
+ as they arrive since 0.29.1, which is the likely start).
+- *hermes stopped a turn whose first answer was long.* hermes writes a message into its database only
+ when it is complete, so a first answer that streamed for more than 90 s looked like a hung call and
+ was stopped (found on the hosted service; on 0.29.4 here, hermes with gpt-5.4 asked to write a
+ 9,000 word file as its first action ended `incomplete` after 93 s with nothing written). The guard
+ now also asks when a provider last sent an event on the turn's route; the CLI that hangs after its
+ provider answered is still caught.
+
+Also from the hosted service: `backend` on POST /v1/responses took a base's id ("claude-code") as the
+backend, matched no model, and refused every model as having no provider there. It names the base's
+backend now.
diff --git a/gateway/app.py b/gateway/app.py
index f67cbecb..75f4e67d 100644
--- a/gateway/app.py
+++ b/gateway/app.py
@@ -313,6 +313,33 @@ _TRACE_NONCE = uuid.uuid4().hex[:8]
# Streaming poll cadence for the /v1/responses path — much faster than the background driver's 30s
# because an open SSE connection wants timely deltas (and the fast poll also keeps the sandbox warm).
RESP_POLL_S = float(os.environ.get("HARNESS_RESP_POLL_S", "1.2"))
+
+
+def _load_reasoning():
+ """runner/reasoning.py, by path: the levels a model has are kept with the functions that apply
+ them, and the two services share a tree, not an import path."""
+ import importlib.util
+ path = pathlib.Path(__file__).resolve().parents[1] / "runner" / "reasoning.py"
+ spec = importlib.util.spec_from_file_location("hr_reasoning", path)
+ mod = importlib.util.module_from_spec(spec)
+ spec.loader.exec_module(mod)
+ return mod
+
+
+# How much a model thinks on a turn: a level on the request (`reasoning.effort`), else the harness's
+# (`reasoning_effort`), else nothing, which leaves every request exactly as it was before levels.
+reasoning = _load_reasoning()
+
+
+def _reasoning_level(value, where: str = "") -> str:
+ """A level as the API takes it, "" for none given. A value that is not a level is refused when
+ it came with the request (`where` names the field) and read as none when it was stored."""
+ level = str(value or "").strip().lower()
+ if level and level not in reasoning.LEVELS:
+ if where:
+ raise HTTPException(400, f"{where} must be one of: {', '.join(reasoning.LEVELS)}")
+ return ""
+ return level
# Output (container) files produced by a turn are capped to keep a single response bounded.
RESP_MAX_FILES = int(os.environ.get("HARNESS_RESP_MAX_FILES", "25"))
RESP_MAX_FILE_BYTES = int(os.environ.get("HARNESS_RESP_MAX_FILE_BYTES", str(25 * 1024 * 1024)))
@@ -513,7 +540,8 @@ def _req_hash(body: "CreateResponseBody") -> str:
canon = {"model": body.model, "input": body.input, "instructions": body.instructions,
"previous_response_id": body.previous_response_id, "backend": body.backend,
"tools": body.tools, "max_output_tokens": body.max_output_tokens,
- "max_step": body.max_step, "metadata": meta}
+ "max_step": body.max_step, "metadata": meta,
+ **({"reasoning": body.reasoning} if body.reasoning else {})}
return hashlib.sha256(json.dumps(canon, sort_keys=True, default=str).encode()).hexdigest()
REDIS_URL = os.environ.get("HARNESS_REDIS_URL", "")
@@ -5545,6 +5573,9 @@ def _blocks_from_canonical(ev: dict) -> list[tuple[str, object]]:
elif t == "result":
out.append(("result", {"text": ev.get("result") or "", "usage": ev.get("usage"),
"is_error": bool(ev.get("is_error")),
+ # the thinking level asked and the one applied (runner: _stamp_thinking);
+ # absent for a turn that asked for none
+ **({"reasoning": ev["reasoning"]} if ev.get("reasoning") else {}),
# the model the CLI reports it actually used, when it says (gemini-cli
# keys its stats by served model, and rewrites some ids on the way)
"model": str(ev.get("model") or "")}))
@@ -5601,6 +5632,9 @@ class _RespTranslator:
# matrix read it per turn (a report that said "every turn record" was reading the session).
self.connection = ""
self.served_model = "" # what the CLI reports it ran, when it reports; "" = unknown
+ # {"effort": asked, "applied": level | "default" | None}: None for a turn that asked for no
+ # level; "applied" is None until the runner's result says what the model was actually given.
+ self.reasoning: dict | None = None
self.seq = 0
self.out_index = -1
self.output: list[dict] = []
@@ -5630,6 +5664,7 @@ class _RespTranslator:
"output": self.output, "store": self.store, "usage": self.usage,
"connection": self.connection,
"served_model": self.served_model,
+ "reasoning": self.reasoning,
"metadata": meta}
def start(self) -> list[dict]:
@@ -5728,11 +5763,19 @@ class _RespTranslator:
elif kind == "result":
if payload.get("model"):
self.served_model = str(payload["model"])
+ r = payload.get("reasoning")
+ if isinstance(r, dict) and self.reasoning is not None:
+ self.reasoning = {"effort": self.reasoning.get("effort") or r.get("effort"),
+ "applied": r.get("applied") or "default"}
u = payload.get("usage")
if u:
self.usage = {"input_tokens": u.get("input_tokens", 0),
"output_tokens": u.get("output_tokens", 0),
"total_tokens": u.get("input_tokens", 0) + u.get("output_tokens", 0)}
+ # The provider's own count of thinking tokens, where it gave one (part of the
+ # output count, in the Responses API's place for it).
+ if u.get("reasoning_tokens") is not None:
+ self.usage["output_tokens_details"] = {"reasoning_tokens": int(u["reasoning_tokens"] or 0)}
# Cache tokens, priced separately as cache_read/write. Accept both the claude CLI's
# raw names AND the runner's already-normalized cache_read_tokens/cache_write_tokens.
for src, dst in (("cache_read_input_tokens", "cache_read_tokens"),
@@ -5836,7 +5879,11 @@ def _parse_input(inp) -> tuple[str, list[dict], list[dict]]:
def _route_backend(model: str | None, explicit: str | None) -> str:
if explicit:
- return explicit.lower()
+ # A base's id as GET /v1/bases lists it ("claude-code") or its backend ("claude"): the id
+ # was taken as the backend, matched no model, and every model was refused as having no
+ # provider there (found on the hosted service, 2026-10-05).
+ e = explicit.strip().lower()
+ return str((_BASE_CATALOG.get(e) or {}).get("backend") or e)
m = (model or "").lower()
if "hermes" in m:
return "hermes"
@@ -7369,6 +7416,7 @@ async def _resp_execute(translator: _RespTranslator, *, org: str, member: str, s
resume: str | None, emit, model_req: str = "", user_text: str = "",
harness_id: str = "", max_step: int = 40,
timeout_s: int | None = None,
+ effort: str = "",
hdr_vals: dict[str, str] | None = None,
partial_messages: bool = False, probe: dict | None = None,
codex_appserver: bool = False,
@@ -7560,6 +7608,12 @@ async def _resp_execute(translator: _RespTranslator, *, org: str, member: str, s
if backend == "gemini" and model_req else None),
"prompt": runner_prompt, "max_turns": max_step,
"timeout_seconds": timeout_s,
+ # How much the model thinks this turn; None asks nothing and changes nothing.
+ "reasoning_effort": effort or None,
+ # Which measured provider this connection is, read off its own base. The runner
+ # reads the base it is handed, and in broker trust that is this gateway's.
+ "reasoning_route": (reasoning.route_of(str((_with_provider_base(conn) or {}).get("base_url") or ""))
+ or None) if effort else None,
"auth": sandbox_auth, "resume_session_id": resume, "files": files_in,
"mcp_servers": mcp_servers, "skills": skills, "plugins": plugin_pkgs, "agent_doc": agent_doc,
"skills_suppressed": skills_suppressed, "tools_disabled": turn_tools_off,
@@ -8113,6 +8167,7 @@ class CreateResponseBody(BaseModel):
backend: str | None = None # non-OpenAI convenience: force codex|claude
max_step: int | None = None # per-request agent step budget (claude --max-turns)
timeout_seconds: int | None = None # per-request wall-clock cap for the turn
+ reasoning: dict | None = None # {"effort": level}: how much the model thinks this turn (the Responses API's field)
_IDEM_TERMINAL = {"completed", "failed", "incomplete", "cancelled", "error"}
@@ -8245,6 +8300,10 @@ async def create_response(body: CreateResponseBody, request: Request):
or _num((hv or {}).get("max_step")) or DEFAULT_MAX_STEP)
timeout_s = (_num(body.timeout_seconds) or _num(meta.get("timeout_seconds"))
or _num((hv or {}).get("timeout_seconds")) or DEFAULT_TIMEOUT_S)
+ # The turn's own level wins over the harness's; neither set, nothing is asked of the model.
+ effort = (_reasoning_level((body.reasoning or {}).get("effort") if isinstance(body.reasoning, dict) else body.reasoning,
+ "reasoning.effort")
+ or _reasoning_level((hv or {}).get("reasoning_effort")))
# Pin the backend to the harness's base (a custom harness always runs on its own backend);
# only fall back to inferring it from the model name when there's no harness. This makes the
# model permission check below meaningful — the requested model is validated against the
@@ -8384,6 +8443,8 @@ async def create_response(body: CreateResponseBody, request: Request):
_ignored = [f for f in ("tools", "include") if getattr(body, f, None) is not None]
tr = _RespTranslator(resp_id, model_req or body.model or backend, body.previous_response_id, body.store, created_at, sid=sid, ignored=_ignored)
tr.environment = str((environment or {}).get("id") or "")
+ if effort: # what the turn asked of the model's thinking; "applied" arrives with the result
+ tr.reasoning = {"effort": effort, "applied": None}
# Broadcast a synthetic turn-start so the bus alone can render a conversation turn from
# scratch (the native Responses events don't echo the user's prompt).
@@ -8431,7 +8492,7 @@ async def create_response(body: CreateResponseBody, request: Request):
tr, org=org, member=member, sid=sid, backend=backend, chain=chain,
prompt=prompt, files_in=files_in, resume=resume, emit=bus_emit_bg,
model_req=model_req, user_text=user_text, harness_id=harness_id,
- max_step=max_step, timeout_s=timeout_s, hdr_vals=hdr_vals,
+ max_step=max_step, timeout_s=timeout_s, effort=effort, hdr_vals=hdr_vals,
partial_messages=want_partial, codex_appserver=want_appserver, hv=hv, probe=probe, environment=environment)
# A failed turn says why in the transcript, not only in the response record: fail()
# carries the message as an error event, which the console prints under the answer.
@@ -8481,7 +8542,7 @@ async def create_response(body: CreateResponseBody, request: Request):
tr, org=org, member=member, sid=sid, backend=backend, chain=chain,
prompt=prompt, files_in=files_in, resume=resume, emit=emit, model_req=model_req,
user_text=user_text, harness_id=harness_id, max_step=max_step,
- timeout_s=timeout_s, hdr_vals=hdr_vals, partial_messages=want_partial, codex_appserver=want_appserver, hv=hv, probe=probe, environment=environment)
+ timeout_s=timeout_s, effort=effort, hdr_vals=hdr_vals, partial_messages=want_partial, codex_appserver=want_appserver, hv=hv, probe=probe, environment=environment)
# A failed turn says why in the transcript, not only in the response record: fail()
# carries the message as an error event, which the console prints under the answer.
for ev in (tr.fail(_turn_failure_message(rec)) if status == "failed" else tr.complete(status, produced)):
@@ -8539,7 +8600,7 @@ async def create_response(body: CreateResponseBody, request: Request):
tr, org=org, member=member, sid=sid, backend=backend, chain=chain,
prompt=prompt, files_in=files_in, resume=resume, emit=bus_emit, model_req=model_req,
user_text=user_text, harness_id=harness_id, max_step=max_step,
- timeout_s=timeout_s, hdr_vals=hdr_vals, partial_messages=want_partial, codex_appserver=want_appserver, hv=hv, probe=probe, environment=environment)
+ timeout_s=timeout_s, effort=effort, hdr_vals=hdr_vals, partial_messages=want_partial, codex_appserver=want_appserver, hv=hv, probe=probe, environment=environment)
# A failed turn says why in the transcript, not only in the response record: fail()
# carries the message as an error event, which the console prints under the answer.
for ev in (tr.fail(_turn_failure_message(rec)) if status == "failed" else tr.complete(status, produced)):
@@ -8814,6 +8875,8 @@ async def _session_turns_data(sid: str, limit: int = 0) -> dict:
"model": rec.get("model") or None,
# the model the CLI reported it ran, when it reported one
"served_model": rec.get("served_model") or None,
+ # the thinking level the turn asked for and the one the model was given
+ "reasoning": rec.get("reasoning") or None,
# WHY an incomplete turn is incomplete ("max_steps" | "timeout" |
# "interrupted"), so the console can say what actually happened instead of
# one banner for every cause. Absent on records from before the field.
@@ -15787,6 +15850,7 @@ class HarnessBody(BaseModel):
disabled_tools: list | None = _either("disabled_tools") # built-in tool names the harness disabled
max_step: int | None = _either("max_step") # default agent step budget for this harness's turns
timeout_seconds: int | None = _either("timeout_seconds") # default per-turn wall-clock cap
+ reasoning_effort: str | None = _either("reasoning_effort") # how much the model thinks on this harness's turns; a turn's own level wins
additional_headers: list | None = _either("additional_headers") # header NAMES callers may pass per request
env: dict | None = None # variables every turn's shell and tools start with: name -> literal,
# $headers.X-Name (a declared request header) or vault:ref
@@ -15832,6 +15896,7 @@ def _harness_out(v: dict) -> dict:
"env": _parse_env(v.get("env")),
"maxStep": int(v.get("max_step")) if str(v.get("max_step") or "").isdigit() else None,
"timeoutSeconds": int(v.get("timeout_seconds")) if str(v.get("timeout_seconds") or "").isdigit() else None,
+ "reasoningEffort": _reasoning_level(v.get("reasoning_effort")),
"calibrates": str(v.get("calibrates") or ""),
"environment": str(v.get("environment") or ""),
"member": v.get("member") or "", "workspace": v.get("workspace") or "", "createdAt": created}
@@ -15854,7 +15919,8 @@ def _harness_props(body: HarnessBody) -> dict:
"calibrates": str(body.calibrates or "").strip(),
"environment": str(body.environment or "").strip(),
"max_step": str(body.max_step) if body.max_step else "",
- "timeout_seconds": str(body.timeout_seconds) if body.timeout_seconds else ""}
+ "timeout_seconds": str(body.timeout_seconds) if body.timeout_seconds else "",
+ "reasoning_effort": _reasoning_level(body.reasoning_effort, "reasoning_effort")}
_WS_WRITE_MAX = 4 * 1024 * 1024 # an app writing its own state, not an upload path
@@ -17719,6 +17785,7 @@ async def _cloud_harness_body(org: str, hid: str, v: dict, plugins_ok: bool = Fa
**({"plugins": plugins} if plugins_ok else {}),
"disabled_tools": out.get("disabledTools") or [], "max_step": out.get("maxStep"),
"timeout_seconds": out.get("timeoutSeconds"), "additional_headers": out.get("additionalHeaders") or [],
+ "reasoning_effort": out.get("reasoningEffort") or None,
"kit": out.get("kit") or None,
"source": "selfhost", "source_instance": socket.gethostname()}
@@ -18055,7 +18122,10 @@ async def list_bases(request: Request) -> dict:
"id": bid, "object": "harness.base", "label": b["label"], "backend": backend,
"status": b["status"], "systemPrompt": b["system_prompt"],
"defaultModel": cat.get("default", ""),
- "models": [{"id": m["id"], "available": m["available"], "default": m["default"]}
+ # `reasoning`: the thinking levels the model has, lowest first; empty where none was
+ # measured, and the console offers no level there (runner/reasoning.py has the table).
+ "models": [{"id": m["id"], "available": m["available"], "default": m["default"],
+ "reasoning": list(reasoning.levels_for(m["id"]))}
for m in view["models"]],
"tools": [{"name": n, "label": lbl, "enforcement": b["tool_enforcement"]}
for n, lbl in b["tools"]],
diff --git a/gateway/tests/test_reasoning_effort.py b/gateway/tests/test_reasoning_effort.py
new file mode 100644
index 00000000..93100a9b
--- /dev/null
+++ b/gateway/tests/test_reasoning_effort.py
@@ -0,0 +1,109 @@
+"""How much a model thinks on a turn, on the gateway side: the request's `reasoning.effort`, the
+harness's `reasoning_effort`, what /v1/bases offers per model, and the turn's record of what was
+asked and what was applied. A turn and a harness that set neither read exactly as they did before."""
+import asyncio
+import os
+import sys
+from pathlib import Path
+
+import pytest
+from fastapi import HTTPException
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+import app as gw # noqa: E402
+
+
+def test_a_level_is_one_of_six_and_anything_else_is_refused_where_it_was_sent():
+ assert [gw._reasoning_level(v, "reasoning.effort") for v in ("low", " High ", "", None)] == ["low", "high", "", ""]
+ with pytest.raises(HTTPException) as e:
+ gw._reasoning_level("maximum", "reasoning.effort")
+ assert e.value.status_code == 400 and "none, minimal, low, medium, high, xhigh" in e.value.detail
+ assert gw._reasoning_level("maximum") == "" # a stored value that is not a level asks for nothing
+
+
+def test_the_request_takes_the_responses_apis_own_field_and_it_is_part_of_what_the_request_is():
+ plain = gw.CreateResponseBody(model="gpt-5.4", input="hi")
+ low = gw.CreateResponseBody(model="gpt-5.4", input="hi", reasoning={"effort": "low"})
+ high = gw.CreateResponseBody(model="gpt-5.4", input="hi", reasoning={"effort": "high"})
+ assert low.reasoning == {"effort": "low"}
+ # an Idempotency-Key reused with another level is another request...
+ assert len({gw._req_hash(plain), gw._req_hash(low), gw._req_hash(high)}) == 3
+ # ...and a request without a level hashes as it did before the field existed
+ src = open(os.path.join(os.path.dirname(__file__), "..", "app.py")).read()
+ assert '**({"reasoning": body.reasoning} if body.reasoning else {})' in src
+
+
+def test_a_harness_keeps_a_default_level_under_either_spelling():
+ body = gw.HarnessBody(name="h", base="codex", reasoning_effort="low")
+ assert gw._harness_props(body)["reasoning_effort"] == "low"
+ assert gw._harness_props(gw.HarnessBody(name="h", base="codex", reasoningEffort="High"))["reasoning_effort"] == "high"
+ assert gw._harness_props(gw.HarnessBody(name="h", base="codex"))["reasoning_effort"] == ""
+ with pytest.raises(HTTPException) as e:
+ gw._harness_props(gw.HarnessBody(name="h", base="codex", reasoning_effort="lots"))
+ assert e.value.status_code == 400 and "reasoning_effort must be one of" in e.value.detail
+ assert gw._harness_out({"id": "hrn_1", "name": "h", "base": "codex", "reasoning_effort": "low"})["reasoningEffort"] == "low"
+ assert gw._harness_out({"id": "hrn_1", "name": "h", "base": "codex"})["reasoningEffort"] == ""
+
+
+def test_the_turns_own_level_wins_over_the_harnesss_and_neither_asks_nothing():
+ src = open(os.path.join(os.path.dirname(__file__), "..", "app.py")).read()
+ i = src.index("effort = (_reasoning_level((body.reasoning or {}).get(\"effort\")")
+ rule = src[i:src.index("\n", src.index("or _reasoning_level((hv or {}).get(\"reasoning_effort\")))", i))]
+ assert rule.index("body.reasoning") < rule.index('(hv or {}).get("reasoning_effort")')
+ # the runner is told a level or nothing: None leaves its request as it was
+ assert '"reasoning_effort": effort or None,' in src
+ # ...and which measured provider the connection is, read off the connection's own base, since
+ # in broker trust the base the runner is handed is this gateway's
+ assert '"reasoning_route": (reasoning.route_of(str((_with_provider_base(conn) or {}).get("base_url") or ""))' in src
+ assert gw.reasoning.route_of(gw._with_provider_base({"provider": "google"})["base_url"]) == "google"
+ assert gw.reasoning.route_of(gw._with_provider_base({"provider": "openai"})["base_url"]) == "openai"
+
+
+def test_the_record_says_what_was_asked_at_once_and_what_was_applied_when_the_result_comes():
+ tr = gw._RespTranslator("resp_1", "gpt-5.4", None, True, 0.0)
+ assert tr._response_obj("in_progress")["reasoning"] is None # a turn that asked for no level
+ tr.reasoning = {"effort": "minimal", "applied": None} # set when the turn is created
+ assert tr._response_obj("in_progress")["reasoning"] == {"effort": "minimal", "applied": None}
+ tr.feed({"type": "result", "result": "ok", "reasoning": {"effort": "minimal", "applied": "low"},
+ "usage": {"input_tokens": 5, "output_tokens": 9, "reasoning_tokens": 7}})
+ done = tr._response_obj("completed")
+ assert done["reasoning"] == {"effort": "minimal", "applied": "low"}
+ # the provider's own count of thinking tokens, where the Responses API keeps it
+ assert done["usage"]["output_tokens_details"] == {"reasoning_tokens": 7}
+ assert done["usage"]["output_tokens"] == 9
+
+
+def test_a_turn_that_asked_for_no_level_reads_as_it_did_before():
+ ev = {"type": "result", "result": "PONG", "usage": {"input_tokens": 1, "output_tokens": 1}}
+ kinds = gw._blocks_from_canonical(ev)
+ assert kinds and all("reasoning" not in payload for _, payload in kinds)
+ tr = gw._RespTranslator("resp_2", "gpt-5.4", None, True, 0.0)
+ tr.feed(ev)
+ done = tr._response_obj("completed")
+ assert done["reasoning"] is None and "output_tokens_details" not in done["usage"]
+
+
+def test_bases_say_which_levels_each_model_has_and_none_where_none_was_measured(monkeypatch):
+ async def servable(org, backend):
+ return set(gw._MODEL_CATALOG.get(backend, {}).get("models") or [])
+
+ async def who(request):
+ return "org1", "m1"
+
+ monkeypatch.setattr(gw, "_servable_models", servable)
+ monkeypatch.setattr(gw, "_pub_org_member", who)
+ out = asyncio.run(gw.list_bases(None))
+ by_base = {b["id"]: {m["id"]: m["reasoning"] for m in b["models"]} for b in out["bases"]}
+ codex = by_base["codex"]
+ assert codex["gpt-5.4"] == ["none", "low", "medium", "high", "xhigh"]
+ assert codex["gpt-6.1-sol"] == ["low", "medium", "high", "xhigh"] # cannot be turned off
+ listed = {m: lv for models in by_base.values() for m, lv in models.items()}
+ assert listed["claude-haiku-4.5"][0] == "none" and listed["gemini-3.8-flash"] == ["low", "medium", "high"]
+ assert listed["llama-3.3-70b"] == [] and listed["glm-5.3"] == []
+ for model, levels in listed.items():
+ assert all(lv in gw.reasoning.LEVELS for lv in levels), model
+
+
+def test_the_levels_come_from_the_runners_table_not_a_second_one():
+ assert gw.reasoning.__file__.replace(os.sep, "/").endswith("runner/reasoning.py")
+ assert gw.reasoning.levels_for("openai/gpt-5.4") == ("none", "low", "medium", "high", "xhigh")
diff --git a/gateway/tests/test_route_backend.py b/gateway/tests/test_route_backend.py
new file mode 100644
index 00000000..9f8994db
--- /dev/null
+++ b/gateway/tests/test_route_backend.py
@@ -0,0 +1,23 @@
+"""`backend` on POST /v1/responses takes a base's id as GET /v1/bases lists it, or its backend.
+
+The id was returned as the backend ("claude-code"), matched no model, and every model was refused as
+"no connected provider serves it there" while "claude" worked (found on the hosted service, 2026-10-05).
+"""
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+import app as gw # noqa: E402
+
+
+def test_a_bases_id_names_its_backend():
+ assert gw._route_backend("claude-sonnet-5.5", "claude-code") == "claude"
+ assert gw._route_backend(None, " Claude-Code ") == "claude"
+ for base_id, base in gw._BASE_CATALOG.items():
+ assert gw._route_backend(None, base_id) == base["backend"], base_id
+
+
+def test_a_backend_named_directly_and_a_name_that_is_neither_are_as_before():
+ assert gw._route_backend(None, "claude") == "claude" and gw._route_backend(None, "CODEX") == "codex"
+ assert gw._route_backend(None, "nosuchbase") == "nosuchbase" # refused later, with its name
+ assert gw._route_backend("gpt-5.4", None) == "codex" and gw._route_backend("claude-opus-5", "") == "claude"
diff --git a/runner/dsh_driver.py b/runner/dsh_driver.py
index 49efb6c0..c0925df4 100644
--- a/runner/dsh_driver.py
+++ b/runner/dsh_driver.py
@@ -29,7 +29,12 @@ import threading
import time
import urllib.request
+import reasoning # how much a model thinks on a turn; the runner's relay applies the same functions
+
UPSTREAM_BASE = "" # set in main() from HR_DSH_BASE_URL, then scrubbed from the env
+EFFORT = "" # the thinking level the turn asked for (HR_DSH_REASONING_EFFORT), "" for none
+EFFORT_ROUTE = "" # the provider the gateway named for the connection (HR_DSH_REASONING_ROUTE), "" = read the base
+_effort_refused: set = set() # models whose provider refused the level on this turn
def _upstream_wait() -> float:
@@ -313,15 +318,34 @@ class _Relay(http.server.BaseHTTPRequestHandler):
_model = ""
if "gemini" in _model.lower():
body = _with_gemini_schemas(body) # TokenRouter's Gemini channels forward tool schemas to Google's validator as sent
+ # The turn's thinking level, as the runner's relay writes it (runner/reasoning.py): last of
+ # the body's changes, and never what fails the call.
+ plain, effort_model, effort_back = None, "", False
+ if EFFORT and reasoning.shape_of(tail):
+ try:
+ effort_model = str((json.loads(body) or {}).get("model") or "")
+ except (ValueError, AttributeError):
+ effort_model = ""
+ if effort_model not in _effort_refused:
+ leveled, applied = reasoning.apply(body, reasoning.shape_of(tail), UPSTREAM_BASE, EFFORT,
+ route=EFFORT_ROUTE)
+ if applied:
+ plain, body = body, leveled
resp = None
for attempt in (0, 1, 2):
req = urllib.request.Request(UPSTREAM_BASE.rstrip("/") + tail,
data=body, method="POST", headers=headers)
try:
resp = urllib.request.urlopen(req, timeout=UPSTREAM_WAIT_S)
+ if effort_back:
+ _effort_refused.add(effort_model) # answered without the level: the level was the cause
break
except urllib.error.HTTPError as e:
data = e.read()
+ if plain is not None and attempt < 2 and e.code in (400, 422):
+ body, plain, effort_back = plain, None, True
+ continue
+ effort_back = False
stripped = _drop_reasoning_effort(body)
if attempt < 2 and b"reasoning_effort" in data and stripped != body:
_strip_reasoning_effort = True
@@ -519,10 +543,12 @@ def _compose_patch(home: pathlib.Path, servers: list[dict], llm: dict | None = N
def main() -> int:
- global UPSTREAM_BASE, UPSTREAM_KEY
+ global UPSTREAM_BASE, UPSTREAM_KEY, EFFORT, EFFORT_ROUTE
job = json.loads(sys.argv[1])
UPSTREAM_BASE = os.environ.pop("HR_DSH_BASE_URL", "")
UPSTREAM_KEY = os.environ.pop("HR_DSH_API_KEY", "")
+ EFFORT = os.environ.pop("HR_DSH_REASONING_EFFORT", "")
+ EFFORT_ROUTE = os.environ.pop("HR_DSH_REASONING_ROUTE", "")
home = pathlib.Path(os.environ.get("HOME") or ".")
cwd = job.get("cwd") or os.getcwd()
diff --git a/runner/reasoning.py b/runner/reasoning.py
new file mode 100644
index 00000000..56e8aefc
--- /dev/null
+++ b/runner/reasoning.py
@@ -0,0 +1,272 @@
+"""How much a model thinks on a turn: which levels a model has, and what turns each one on.
+
+A turn may ask for a level (none, minimal, low, medium, high, xhigh). Nothing here is a guess: every
+entry below was measured on 2026-10-05 by sending one question at every level through each route and
+reading the provider's own reasoning-token count (the table and the method are in
+docs/support-matrix-notes.md, "How much a model thinks"). Three things the measurement taught:
+
+- No field means the same thing everywhere. `reasoning_effort` is honoured for most families on most
+ routes, but through Vercel a Gemini model takes `none` as "think more", through TokenRouter it
+ ignores the field entirely, and what works instead differs per route (a `providerOptions` block on
+ one, an `extra_body` block on the other).
+- A level a model lacks is refused, not ignored: `minimal` on gpt-5.4, `none` on gpt-6.1-sol or on
+ any Gemini through OpenRouter ("Reasoning is mandatory for this endpoint"), `low` on Mistral
+ through Vercel. So a level is only ever sent where it was seen to be accepted, and a model that
+ lacks the one asked for gets the nearest one it has.
+- A model and route that were not measured get NOTHING sent, and the turn says so ("default").
+
+Pure functions over bytes, no imports from the runner: the relay applies them (runner/server.py), the
+DeepSeek Harness driver's own relay applies them (runner/dsh_driver.py), and the gateway reads
+`levels_for` for what /v1/bases offers (gateway/app.py loads this file by path).
+"""
+from __future__ import annotations
+
+import json
+import re
+
+LEVELS = ("none", "minimal", "low", "medium", "high", "xhigh")
+# The providers that were measured. A route is one of these or "", and "" gets nothing sent.
+ROUTES = ("vercel", "tokenrouter", "openrouter", "openai", "azure", "google", "anthropic")
+
+# The levels a model has, by its bare id (the part after the vendor's prefix). First match wins.
+# Only ids that were measured are named; a family's unmeasured members are left out on purpose.
+_MODEL_LEVELS: tuple[tuple[str, tuple[str, ...]], ...] = (
+ # OpenAI. `minimal` is refused by half the line and by every route's Responses API, so it is
+ # never sent; gpt-6.1-sol and gpt-6-astra cannot be turned off.
+ (r"^gpt-6\.1-sol|^gpt-6-astra", ("low", "medium", "high", "xhigh")),
+ (r"^gpt-(5\.[2-6]|6)(-|$)", ("none", "low", "medium", "high", "xhigh")),
+ # Anthropic. fable-5.1 refuses both of the API's ways of turning thinking off (fable-5 is taken
+ # to be the same; it was not measured).
+ (r"^claude-fable-5", ("low", "medium", "high", "xhigh")),
+ (r"^claude-(haiku-4[.-]5|sonnet-4[.-]6|opus-4[.-][78]|sonnet-5|opus-5)", ("none", "low", "medium", "high", "xhigh")),
+ # Google. The pro model and the 3.7 and 3.8 flash models only work thinking, and refuse `minimal`.
+ (r"^gemini-3\.1-pro|^gemini-3\.[78]-flash$", ("low", "medium", "high")),
+ (r"^gemini-3(\.[156])?-flash(-lite|-preview)?$", ("none", "minimal", "low", "medium", "high")),
+ (r"^grok-4\.[356]$", ("minimal", "low", "medium", "high", "xhigh")), # cannot be turned off
+ (r"^deepseek-v4(\.1)?-(flash|pro)$", ("none", "low", "medium", "high", "xhigh")),
+ (r"^kimi-k3$", ("none", "low", "medium", "high")),
+ (r"^qwen3\.7-max$|^qwen3\.8-flash$", ("none", "high")), # on or off
+ (r"^mistral-medium-3[.-]5$", ("none", "high")),
+ (r"^step-3\.7-flash$", ("minimal", "low", "medium", "high", "xhigh")), # cannot be turned off
+ (r"^muse-spark-1\.3$", ("low", "high")),
+ (r"^minimax-m3$|^ling-3\.0-flash$|^nemotron-3\.5-lightning$", ("none", "high")),
+ (r"^hy3$|^hunyuan-3$", ("none", "low", "high")),
+)
+
+# A provider's own name for a model where its bare id differs from the catalog's.
+_ALIASES = {"gemini-3-flash": "gemini-3-flash-preview", "grok-4.20-reasoning": "grok-4.20"}
+
+# What a route cannot do for a model that has the level elsewhere: (route, bare-id pattern, levels).
+_ROUTE_LACKS: tuple[tuple[str, str, tuple[str, ...]], ...] = (
+ ("openrouter", r"^claude-(opus|sonnet)-5[.-]5", ("none",)), # "Reasoning is mandatory for this endpoint"
+ ("openrouter", r"^gemini-", ("none",)), # the same refusal, every Gemini
+ ("vercel", r"^claude-opus-5[.-]5", ("none",)), # accepted and not honoured
+ ("tokenrouter", r"^deepseek-v4-pro$", ("none",)), # "must be one of: low, medium, high, xhigh, max"
+ ("tokenrouter", r"^gemini-3\.1-pro", ("medium",)), # low and high were measured, medium was not
+ ("google", r"^gemini-.*-flash-lite", ("minimal",)),
+)
+# Gemini through TokenRouter's chat door: only these two answered to any setting.
+_TOKENROUTER_GEMINI = re.compile(r"^gemini-3\.5-flash$|^gemini-3\.1-pro")
+
+_BUDGET = {"low": 1024, "medium": 4096, "high": 16384, "xhigh": 32000} # claude-haiku-4.5 takes a budget, not an effort
+
+
+def bare(model: str) -> str:
+ """A model id without its vendor prefix and without a dated suffix, lower case."""
+ m = str(model or "").strip().lower().lstrip("~").rsplit("/", 1)[-1]
+ m = re.sub(r"-20\d{6}$|-\d{4}-\d{2}-\d{2}$", "", m)
+ return _ALIASES.get(m, m)
+
+
+def levels_for(model: str) -> tuple[str, ...]:
+ """The levels this model has, lowest first; () when none was measured for it."""
+ b = bare(model)
+ for pattern, levels in _MODEL_LEVELS:
+ if re.search(pattern, b):
+ return levels
+ return ()
+
+
+def route_of(base_url: str) -> str:
+ """Which provider a base URL belongs to, as far as the measurements go; "" for any other."""
+ host = re.sub(r"^[a-z]+://", "", str(base_url or "").lower()).split("/", 1)[0]
+ if host.endswith("ai-gateway.vercel.sh"):
+ return "vercel"
+ if host.endswith("openrouter.ai"):
+ return "openrouter"
+ if "tokenrouter" in host:
+ return "tokenrouter"
+ if host.endswith("api.openai.com"):
+ return "openai"
+ if host.endswith((".openai.azure.com", ".cognitiveservices.azure.com", ".services.ai.azure.com")):
+ return "azure"
+ if host.endswith("generativelanguage.googleapis.com"):
+ return "google"
+ if host.endswith("api.anthropic.com"):
+ return "anthropic"
+ return ""
+
+
+def nearest(asked: str, have: tuple[str, ...] | list[str]) -> str:
+ """The level in `have` closest to the one asked for, "" when there is none to give.
+
+ `none` is only ever given to a turn that asked for it. A model that cannot be turned off gets
+ its lowest level for `none`; a turn that asked for some thinking is never answered with none.
+ A tie goes to the lower level."""
+ if asked not in LEVELS or not have:
+ return ""
+ if asked in have:
+ return asked
+ on = [lv for lv in have if lv != "none"]
+ if not on:
+ return ""
+ want = LEVELS.index(asked)
+ return min(on, key=lambda lv: (abs(LEVELS.index(lv) - want), LEVELS.index(lv)))
+
+
+def level_on(route: str, model: str, asked: str) -> str:
+ """The level a model gets on a route for the one asked; "" when nothing is to be sent."""
+ b = bare(model)
+ have = list(levels_for(b))
+ for r, pattern, lacks in _ROUTE_LACKS:
+ if r == route and re.search(pattern, b):
+ have = [lv for lv in have if lv not in lacks]
+ if route == "tokenrouter" and b.startswith("gemini-") and not _TOKENROUTER_GEMINI.search(b):
+ return ""
+ return nearest(asked, have)
+
+
+def _family(b: str) -> str:
+ return "gpt" if b.startswith("gpt-") else "claude" if b.startswith("claude-") else \
+ "gemini" if b.startswith("gemini-") else "other"
+
+
+def _chat(doc: dict, route: str, b: str, level: str) -> bool:
+ if _family(b) != "gemini" or route in ("openrouter", "google"):
+ if route == "google" and level == "none" and "flash-lite" in b:
+ return True # this model does not think unless asked to; "none" is refused there
+ doc["reasoning_effort"] = level
+ return True
+ # A Gemini model through an aggregator that does not honour the field takes Google's own setting,
+ # each aggregator under its own key.
+ cfg = {"thinkingBudget": 0} if level == "none" else {"thinkingLevel": level}
+ doc.pop("reasoning_effort", None)
+ if route == "vercel":
+ key, inner, value = "providerOptions", "thinkingConfig", cfg
+ elif route == "tokenrouter":
+ key, inner = "extra_body", "thinking_config"
+ value = {"thinking_budget": 0} if level == "none" else {"thinking_level": level}
+ else:
+ return False
+ outer = dict(doc[key]) if isinstance(doc.get(key), dict) else {}
+ google = dict(outer["google"]) if isinstance(outer.get("google"), dict) else {}
+ google[inner] = value
+ outer["google"] = google
+ doc[key] = outer
+ return True
+
+
+def _responses(doc: dict, b: str, level: str) -> bool:
+ if _family(b) != "gpt":
+ return False
+ cur = doc.get("reasoning")
+ doc["reasoning"] = {**(cur if isinstance(cur, dict) else {}), "effort": level}
+ return True
+
+
+def _messages(doc: dict, b: str, level: str) -> bool:
+ """Anthropic's Messages API, as the API itself answered through a pass-through door: the older
+ model takes a budget and refuses an effort, the newer ones the reverse, and the 5.5 line names
+ its own way of turning thinking off."""
+ if _family(b) != "claude":
+ return False
+ if level == "none":
+ doc["thinking"] = {"type": "between_tools" if re.search(r"-5[.-]5", b) else "disabled"}
+ return True
+ if re.search(r"^claude-haiku-4", b):
+ cap = doc.get("max_tokens")
+ budget = _BUDGET.get(level, 1024)
+ if isinstance(cap, int):
+ budget = min(budget, cap // 2) # the budget comes out of max_tokens; half is left to answer with
+ if budget < 1024: # the API's floor
+ return False
+ doc["thinking"] = {"type": "enabled", "budget_tokens": budget}
+ return True
+ doc["thinking"] = {"type": "adaptive"}
+ cur = doc.get("output_config")
+ doc["output_config"] = {**(cur if isinstance(cur, dict) else {}), "effort": "max" if level == "xhigh" else level}
+ return True
+
+
+def _google(doc: dict, level: str) -> bool:
+ """Google's own generateContent body: a level, or a budget of zero for off, never both."""
+ gen = doc.setdefault("generationConfig", {})
+ if not isinstance(gen, dict):
+ return False
+ cfg = gen.get("thinkingConfig")
+ cfg = dict(cfg) if isinstance(cfg, dict) else {}
+ cfg.pop("thinkingBudget", None)
+ cfg.pop("thinkingLevel", None)
+ if level == "none":
+ cfg["thinkingBudget"] = 0
+ else:
+ cfg["thinkingLevel"] = level
+ gen["thinkingConfig"] = cfg
+ return True
+
+
+def apply(body: bytes, shape: str, base_url: str, asked: str, model: str = "", route: str = "") -> tuple[bytes, str]:
+ """One request body with the level a turn asked for → (the body to send, the level applied).
+
+ `shape` is the API the body is written for: "chat" (Chat Completions), "responses", "messages"
+ (Anthropic) or "google" (generateContent, whose model is in the URL and comes as `model`). The
+ level applied is "" when nothing was sent: the model or the route was not measured, or the shape
+ has no setting for the family. The body comes back as the same object in that case.
+
+ `route` names the provider when the caller knows it and the base does not say: behind a broker
+ every base is the broker's, and the gateway, which knows the connection, names the route. A
+ name that is not one of ROUTES is no route."""
+ if asked not in LEVELS or not body:
+ return body, ""
+ try:
+ doc = json.loads(body)
+ except (ValueError, UnicodeDecodeError):
+ return body, ""
+ if not isinstance(doc, dict):
+ return body, ""
+ b = bare(model or doc.get("model") or "")
+ route = route if route in ROUTES else route_of(base_url)
+ if not route:
+ return body, ""
+ if shape == "messages" and route not in ("anthropic", "tokenrouter", "vercel", "openrouter"):
+ return body, ""
+ level = level_on(route, b, asked)
+ if not level:
+ return body, ""
+ if shape == "chat":
+ done = _chat(doc, route, b, level)
+ elif shape == "responses":
+ done = _responses(doc, b, level)
+ elif shape == "messages":
+ done = _messages(doc, b, level)
+ elif shape == "google":
+ done = _family(b) == "gemini" and route in ("google", "tokenrouter") and _google(doc, level)
+ else:
+ done = False
+ if not done:
+ return body, ""
+ return json.dumps(doc, separators=(",", ":")).encode(), level
+
+
+def shape_of(path: str) -> str:
+ """The API a request path belongs to; "" for anything that carries no model call."""
+ p = str(path or "").split("?", 1)[0]
+ if p.endswith("/chat/completions"):
+ return "chat"
+ if p.endswith("/responses"):
+ return "responses"
+ if p.endswith("/messages"):
+ return "messages"
+ if p.endswith((":generateContent", ":streamGenerateContent")):
+ return "google"
+ return ""
diff --git a/runner/server.py b/runner/server.py
index 6ce591e7..45e2705e 100644
--- a/runner/server.py
+++ b/runner/server.py
@@ -43,6 +43,7 @@ Yjs sidecar layer on next.
from __future__ import annotations
import base64
+import contextvars
import hashlib
import http.client
import http.server
@@ -76,6 +77,7 @@ from fastapi.responses import FileResponse, JSONResponse
from pydantic import BaseModel
import environments # the read-only project layer a turn may name (runner/environments.py)
+import reasoning # how much a model thinks on a turn: levels and what sets them (runner/reasoning.py)
app = FastAPI(title="harness-runner")
@@ -1928,6 +1930,20 @@ def _claude_thinking_env(model: str) -> dict:
env = {"CLAUDE_CODE_EFFORT_LEVEL": "auto"}
if _CLAUDE_NO_THINKING.search(model or ""):
env["MAX_THINKING_TOKENS"] = "0"
+ # A level the turn asked for (runner/reasoning.py has the levels and the measurements). The CLI
+ # has two switches and each model takes one of them: a token budget for the model that refuses
+ # an effort (haiku-4.5), an effort for the rest, and a budget of zero for "none" on any.
+ thinking = _TURN_THINKING.get() or {}
+ level = reasoning.nearest(thinking.get("asked") or "", reasoning.levels_for(model))
+ if level == "none":
+ env["MAX_THINKING_TOKENS"] = "0"
+ elif level and re.search(r"haiku-4", model or "", re.I):
+ env["MAX_THINKING_TOKENS"] = str(reasoning._BUDGET.get(level, 1024))
+ elif level:
+ env.pop("MAX_THINKING_TOKENS", None)
+ env["CLAUDE_CODE_EFFORT_LEVEL"] = level
+ if level:
+ thinking["applied"] = level
return env
@@ -2201,6 +2217,17 @@ def _codex_insert_top_level(cfg: str, value: str) -> str:
return head + "\n" + value + sep + tail if sep else cfg + "\n" + value + "\n"
+def _codex_effort(model: str) -> str:
+ """Codex's `model_reasoning_effort` for this turn: the level the turn asked for, as the model
+ has it, else the instance's default (CODEX_REASONING_EFFORT). Codex writes it into every
+ request itself, so it holds on OpenAI's own endpoint and Azure's, which no relay sees."""
+ thinking = _TURN_THINKING.get() or {}
+ level = reasoning.nearest(thinking.get("asked") or "", reasoning.levels_for(model))
+ if level:
+ thinking["applied"] = level
+ return level or CODEX_REASONING_EFFORT
+
+
def _codex_prepare_env(provider: str, auth: Auth, model: str, cwd: str,
env: dict, mcp_toml: str = "", resume: bool = False,
tools_disabled: list[str] | None = None) -> "pathlib.Path":
@@ -2251,7 +2278,7 @@ def _codex_prepare_env(provider: str, auth: Auth, model: str, cwd: str,
# "responses" value; a custom-endpoint turn against chat-only would 404 clearly rather than
# crash on config load.
cfg = _CODEX_CONFIG_TMPL.format(
- model=model, provider=f"hr-{p}", effort=CODEX_REASONING_EFFORT, ctx=CODEX_CONTEXT_WINDOW,
+ model=model, provider=f"hr-{p}", effort=_codex_effort(model), ctx=CODEX_CONTEXT_WINDOW,
name=spec["name"], base_url=cli_base, env_key=spec["env_key"],
wire_api=auth.wire_api or "responses")
if _codex_namespace_tools_off(auth):
@@ -2464,6 +2491,15 @@ def _build_dsh(provider: str, auth: Auth, model: str, prompt: str, cwd: str, env
# a loopback relay URL and a placeholder key — the credential never enters its environment.
env["HR_DSH_BASE_URL"] = base
env["HR_DSH_API_KEY"] = auth.api_key or ""
+ # The turn's thinking level: the driver's relay writes it into each call with the same function
+ # this relay uses (runner/reasoning.py), so what is recorded here is what it will apply.
+ thinking = _TURN_THINKING.get() or {}
+ if thinking.get("asked"):
+ env["HR_DSH_REASONING_EFFORT"] = thinking["asked"]
+ if thinking.get("route"): # scrubbed by the driver with the rest of HR_DSH_*, before the runtime starts
+ env["HR_DSH_REASONING_ROUTE"] = thinking["route"]
+ thinking["applied"] = reasoning.level_on(thinking.get("route") or reasoning.route_of(base), model,
+ thinking["asked"])
# The driver's own relay waits on the provider as long as this one does (dsh_driver._upstream_wait).
env["HR_RELAY_UPSTREAM_TIMEOUT_S"] = f"{HR_RELAY_UPSTREAM_TIMEOUT_S:g}"
# No system_prompt in the job: harness instructions land in AGENTS.md (dsh reads it via
@@ -3136,6 +3172,34 @@ def _hermes_mcp_section(servers: list[dict] | None) -> dict:
# turn (a few dozen bytes per turn, process lifetime — the sandbox recycles long before this
# matters).
_HERMES_RELAY: dict = {"server": None, "port": 0, "routes": {}, "lock": threading.Lock()}
+# The thinking of the turn being built, while /turn runs a backend's builder: {"asked": the level
+# the turn asked for ("" for none), "applied": what a backend whose calls never pass the relay set
+# for its CLI (Claude Code, Codex on OpenAI's own endpoint), "routes": the flags of every route the
+# turn registered, "route": the provider the gateway named for the turn's connection, "" when it
+# named none and the base says}. A route registered meanwhile carries the level, and the turn's result reads its
+# routes back for what was applied and what the provider counted, whichever file or variable the
+# backend keeps its placeholder in. A context variable, so two turns being built at once never read
+# each other's.
+_TURN_THINKING: contextvars.ContextVar[dict | None] = contextvars.ContextVar("hr_turn_thinking", default=None)
+
+
+def _turn_route(flags: dict) -> dict:
+ """A route's flags, remembered as one of the routes of the turn being built."""
+ t = _TURN_THINKING.get()
+ if t is not None:
+ t["routes"].append(flags)
+ return flags
+
+
+def _turn_effort() -> dict:
+ """The route flag that carries the turn's level, {} for a turn that asked for none: such a
+ turn's routes are what they were before a level existed."""
+ t = _TURN_THINKING.get()
+ if not (t and t.get("asked")):
+ return {}
+ # effort_route stays in the relay's memory: which provider serves a model is not something the
+ # agent behind a broker is to read, so it is in no environment and no file.
+ return {"effort": t["asked"], **({"effort_route": t["route"]} if t.get("route") else {})}
def _with_anthropic_cache(body: bytes) -> bytes:
@@ -4067,6 +4131,66 @@ def _usage_in_sse_line(line: bytes) -> dict:
return {}
+def _thinking_tokens_in(doc) -> int | None:
+ """The provider's own count of thinking tokens in a response document or a stream event, None
+ where it gives none: `reasoning_tokens` under the usage's details on Chat Completions and on the
+ Responses API (whose stream carries the usage inside `response`), `thoughtsTokenCount` on
+ Google's own API. Anthropic's Messages API has no such figure. Kept apart from the usage that
+ prices a turn (_usage_fields), which this does not touch."""
+ if not isinstance(doc, dict):
+ return None
+ inner = doc.get("response")
+ for u in (doc.get("usage"), inner.get("usage") if isinstance(inner, dict) else None):
+ if isinstance(u, dict):
+ for key in ("completion_tokens_details", "output_tokens_details"):
+ d = u.get(key)
+ if isinstance(d, dict) and d.get("reasoning_tokens") is not None:
+ return max(int(d["reasoning_tokens"] or 0), 0)
+ meta = doc.get("usageMetadata")
+ if isinstance(meta, dict) and meta.get("thoughtsTokenCount") is not None:
+ return max(int(meta["thoughtsTokenCount"] or 0), 0)
+ return None
+
+
+def _thinking_in_sse_line(line: bytes) -> int | None:
+ if not line.startswith(b"data:") or not (b"reasoning_tokens" in line or b"thoughtsTokenCount" in line):
+ return None
+ try:
+ return _thinking_tokens_in(json.loads(line[5:].strip()))
+ except (ValueError, TypeError):
+ return None
+
+
+def _thinking_add(flags: dict, count: int | None) -> None:
+ """Fold one call's thinking count into the route's total; the key exists only once a provider gave one."""
+ if count is not None:
+ flags["reasoning_tokens"] = int(flags.get("reasoning_tokens") or 0) + count
+
+
+def _stream_fold(flags: dict, call_usage: dict, call_thinking: int | None, folded: dict) -> None:
+ """Bring the route's totals up to what a streamed call has said SO FAR, before its bytes go on to
+ the client. `folded` is what this call has already contributed (a call's usage arrives over
+ several events, later values replacing earlier ones), so each figure is counted once.
+
+ The totals used to be updated when the stream ended, which is when the provider closes it, not
+ when its last event passes: a client that is done at `[DONE]` read the route before its own call
+ was on it. CheetahClaws's in-process driver is such a client, and its turns came back with no
+ usage at all (the thinking column on a candidate, 2026-10-05: the relay had read 329 output
+ tokens and 312 thinking tokens for a call whose turn recorded none)."""
+ if call_usage:
+ total = flags.setdefault("usage", {})
+ for k, v in call_usage.items():
+ delta = int(v) - int(folded.get(k, 0))
+ if delta or k not in total: # a zero is still a figure the provider gave
+ total[k] = int(total.get(k, 0)) + delta
+ folded[k] = int(v)
+ if call_thinking is not None:
+ delta = call_thinking - int(folded.get("_thinking", 0))
+ if delta or "_thinking" not in folded:
+ flags["reasoning_tokens"] = int(flags.get("reasoning_tokens") or 0) + delta
+ folded["_thinking"] = call_thinking
+
+
def _usage_add(flags: dict, call_usage: dict) -> None:
"""Fold one call's usage into the route's running total: a turn is many provider calls."""
if not call_usage:
@@ -4154,6 +4278,38 @@ def _fill_relay_usage(ev: dict, env: dict) -> None:
ev["usage"] = relay
+def _stamp_thinking(ev: dict, rec: dict) -> None:
+ """On a turn's result: the provider's own count of thinking tokens where it gave one, and, for a
+ turn that asked for a level, what was asked and what was applied.
+
+ Both are read off the routes the turn registered (what the relay wrote into the calls of the
+ turn's own model, "" when the model or the route has no measured setting or the provider refused
+ it), then from what a backend that never passes the relay set for its CLI. "default" says nothing
+ was applied and the model ran as it does without a level. A turn that asked for no level gets no
+ `reasoning` on its result."""
+ thinking = rec.get("reasoning")
+ if not isinstance(thinking, dict):
+ return
+ routes = [f for f in thinking.get("routes") or [] if isinstance(f, dict)]
+ counts = [int(f["reasoning_tokens"]) for f in routes if f.get("reasoning_tokens") is not None]
+ if counts:
+ usage = ev.get("usage") if isinstance(ev.get("usage"), dict) else None
+ if usage is None:
+ usage = ev["usage"] = {}
+ usage.setdefault("reasoning_tokens", sum(counts))
+ if not thinking.get("asked"):
+ return
+ applied = str(thinking.get("applied") or "")
+ model = str(rec.get("model") or "")
+ for f in routes:
+ seen = f.get("effort_applied")
+ if isinstance(seen, dict) and seen:
+ mine = [lv for m, lv in seen.items() if m == model or reasoning.bare(m) == reasoning.bare(model)]
+ applied = (mine or list(seen.values()))[-1] or ""
+ break
+ ev["reasoning"] = {"effort": thinking["asked"], "applied": applied or "default"}
+
+
def _relay_usage(env: dict) -> dict:
"""The tokens the provider reported on this turn's route, summed over its calls; {} when the
turn did not ride the relay or nothing carried usage. Found by the placeholder bearer, exactly
@@ -4285,6 +4441,22 @@ class _HermesRelayHandler(http.server.BaseHTTPRequestHandler):
if fixed != body:
body = fixed
headers["content-length"] = str(len(body))
+ # The thinking level the turn asked for, written the way this route and this model take it
+ # (runner/reasoning.py). Last of the body's changes, so it is what the provider reads. A
+ # turn that asked for none never reaches here: its body leaves as the client wrote it.
+ plain = None # the body without the level, kept while the level is on trial
+ effort_model = str(flags.get("model") or "") if flags.get("google_native") else str(_body_model)
+ if (flags.get("effort") and body is not None and self.command == "POST"
+ and not flags.get(f"effort_refused:{effort_model}")):
+ shape = reasoning.shape_of(tail)
+ if shape:
+ leveled, applied = reasoning.apply(body, shape, base, flags["effort"], model=effort_model,
+ route=str(flags.get("effort_route") or ""))
+ flags.setdefault("effort_applied", {})[effort_model] = applied
+ if applied:
+ plain, body = body, leveled
+ headers["content-length"] = str(len(body))
+ effort_back = False # the level was taken out after a refusal; a success now means it was the cause
resp = None
tried_slim = False
dropped_once = False
@@ -4293,12 +4465,27 @@ class _HermesRelayHandler(http.server.BaseHTTPRequestHandler):
method=self.command, headers=headers)
try:
resp = urllib.request.urlopen(req, timeout=HR_RELAY_UPSTREAM_TIMEOUT_S)
+ if effort_back:
+ # the provider took the call without the level: not offered again for this model
+ flags[f"effort_refused:{effort_model}"] = True
+ flags.setdefault("effort_applied", {})[effort_model] = ""
+ print(f"[relay] provider refused the thinking level {flags.get('effort')!r} for "
+ f"model={effort_model}; the call went at the model's default", flush=True)
if attempt > 0 and tried_slim:
# the blind no-stream_options retry is part of what made this route work
flags["drop_stream_options"] = True
break
except urllib.error.HTTPError as e:
data = e.read()
+ if plain is not None and attempt < 2 and e.code in (400, 422):
+ # A level must never be what fails a call. The body goes again as the client
+ # wrote it; if that is answered, the level was the cause and is dropped for this
+ # model (above); if it is refused too, the cause is elsewhere and the repairs
+ # below see the client's own body, as they would have without a level.
+ body, plain, effort_back = plain, None, True
+ headers["content-length"] = str(len(body))
+ continue
+ effort_back = False
renamed = _rename_max_tokens(body) if body is not None else None
if (attempt < 2 and e.code == 400 and b"max_completion_tokens" in data
and renamed is not None and renamed != body):
@@ -4465,6 +4652,8 @@ class _HermesRelayHandler(http.server.BaseHTTPRequestHandler):
carry = b"" # tail of the previous chunk, so a split "model":"…" is still seen
fcarry = b"" # tail of the previous chunk, for the finish_reason field
call_usage: dict = {} # what this call's events said about tokens, unioned
+ call_thinking: int | None = None # ...and about thinking tokens, the last figure given
+ folded: dict = {} # what of the two is already on the route's totals (_stream_fold)
def _stopped(why: str) -> None:
# The answer began and then stopped: the provider went silent past the socket's
@@ -4485,7 +4674,7 @@ class _HermesRelayHandler(http.server.BaseHTTPRequestHandler):
"message": msg, "type": "upstream_unavailable",
"code": "upstream_unavailable"}}) + "\n\n"
call_usage.update(_usage_in_sse_line(pending.strip()))
- _usage_add(flags, call_usage)
+ _stream_fold(flags, call_usage, call_thinking, folded)
try:
raw = ev.encode()
self.wfile.write(f"{len(raw):x}\r\n".encode() + raw + b"\r\n")
@@ -4544,12 +4733,17 @@ class _HermesRelayHandler(http.server.BaseHTTPRequestHandler):
line = line.strip()
if line and not line.startswith(b":"):
last_event = time.monotonic() # an event's line; a comment is not one
+ flags["last_data"] = time.time() # ...and the route's own note of it (_hermes_startup_hung)
call_usage.update(_usage_in_sse_line(line))
+ seen_thinking = _thinking_in_sse_line(line)
+ if seen_thinking is not None:
+ call_thinking = seen_thinking
if sigs is not None:
for cid, sig in _google_signatures_in_line(line):
sigs[cid] = sig
if out_lines is not None:
chunk = b"".join(out_lines)
+ _stream_fold(flags, call_usage, call_thinking, folded) # on the route before the client has it
if chunk:
self.wfile.write(f"{len(chunk):x}\r\n".encode() + chunk + b"\r\n")
self.wfile.flush()
@@ -4559,7 +4753,8 @@ class _HermesRelayHandler(http.server.BaseHTTPRequestHandler):
_stopped(f"no data for {HR_RELAY_UPSTREAM_TIMEOUT_S:g} s, only keep-alive lines")
return
call_usage.update(_usage_in_sse_line(pending.strip()))
- _usage_add(flags, call_usage)
+ seen_thinking = _thinking_in_sse_line(pending.strip())
+ _stream_fold(flags, call_usage, seen_thinking if seen_thinking is not None else call_thinking, folded)
if flags.get("usage_no_nulls") and pending:
tail_bytes = _usage_without_nulls(pending) # a last line with no newline after it
self.wfile.write(f"{len(tail_bytes):x}\r\n".encode() + tail_bytes + b"\r\n")
@@ -4597,7 +4792,9 @@ class _HermesRelayHandler(http.server.BaseHTTPRequestHandler):
flags["last_finish"] = fr
if body is not None and b"sage" in data:
try:
- _usage_add(flags, _usage_in_doc(json.loads(data)))
+ whole = json.loads(data)
+ _usage_add(flags, _usage_in_doc(whole))
+ _thinking_add(flags, _thinking_tokens_in(whole))
except ValueError:
pass
if body is None and "/models" in tail.split("?", 1)[0]:
@@ -4802,13 +4999,15 @@ def _gemini_relay_route(host_root: str, api_key: str, model: str = "", native_mo
_HERMES_RELAY["server"], _HERMES_RELAY["port"] = srv, srv.server_address[1]
tok = "hr-relay-" + uuid.uuid4().hex
_HERMES_RELAY["routes"][tok] = (host_root.rstrip("/"), api_key,
- {"google_native": True, "model": model, "native_model": native_model})
+ _turn_route({"google_native": True, "model": model,
+ "native_model": native_model, **_turn_effort()}))
return f"http://127.0.0.1:{_HERMES_RELAY['port']}/v1", tok
def _hermes_relay_route(base_url: str, api_key: str, drop_fields: tuple[str, ...] = (),
stream_usage: bool = False, gemini_schemas: bool = False,
- usage_no_nulls: bool = False, exact_base: bool = False) -> tuple[str, str]:
+ usage_no_nulls: bool = False, exact_base: bool = False,
+ aux: bool = False) -> tuple[str, str]:
"""Register one turn's upstream; → (relay base_url, placeholder bearer for the CLI).
`drop_fields` names top-level request fields this route's client sends on its own initiative and
@@ -4819,7 +5018,9 @@ def _hermes_relay_route(base_url: str, api_key: str, drop_fields: tuple[str, ...
_build_grok for the one client that needs it). `usage_no_nulls` rewrites a null token count in an
answer's usage to 0 for a client whose parser takes only a number there (_usage_without_nulls).
`exact_base` joins the client's resource onto the connection's base as stored, for a client that
- was calling that base directly before it had a route (Codex): the same URL, with the relay between."""
+ was calling that base directly before it had a route (Codex): the same URL, with the relay between.
+ `aux` marks a route for a helper model (hermes's image questions): the turn's thinking level is
+ the main model's and stays off it."""
with _HERMES_RELAY["lock"]:
if _HERMES_RELAY["server"] is None:
srv = http.server.ThreadingHTTPServer(("127.0.0.1", 0), _HermesRelayHandler)
@@ -4833,11 +5034,10 @@ def _hermes_relay_route(base_url: str, api_key: str, drop_fields: tuple[str, ...
# matrix; Anthropic's OpenAI-compatible surface lives under /v1). Bedrock keeps its host
# (its own path is built in _bedrock_anthropic).
upstream = (base_url or "").rstrip("/") if exact_base else _relay_base_with_version(base_url)
- _HERMES_RELAY["routes"][tok] = (upstream, api_key,
- {"rename_max_tokens": False, "drop_fields": tuple(drop_fields),
- "stream_usage": bool(stream_usage),
- "gemini_schemas": bool(gemini_schemas),
- "usage_no_nulls": bool(usage_no_nulls)})
+ flags = {"rename_max_tokens": False, "drop_fields": tuple(drop_fields),
+ "stream_usage": bool(stream_usage), "gemini_schemas": bool(gemini_schemas),
+ "usage_no_nulls": bool(usage_no_nulls), **({} if aux else _turn_effort())}
+ _HERMES_RELAY["routes"][tok] = (upstream, api_key, flags if aux else _turn_route(flags))
return f"http://127.0.0.1:{_HERMES_RELAY['port']}/v1", tok
@@ -4887,7 +5087,7 @@ def _hermes_prepare_env(provider: str | None, auth: Auth, cwd: str, env: dict,
vision: dict = {"provider": vp, "model": str(vision_auth["model"])}
vkey, vbase = vision_auth.get("api_key") or "", vision_auth.get("base_url") or ""
if vp == "openai-api" and vkey and vbase:
- vbase, vkey = _hermes_relay_route(vbase, vkey)
+ vbase, vkey = _hermes_relay_route(vbase, vkey, aux=True)
if vbase:
vision["base_url"] = vbase
if vkey:
@@ -8712,6 +8912,7 @@ def _run_turn_bg(turn_id: str, cmd: list[str], env: dict, cwd: str, normalize, m
ev["model"] = served
if ev.get("type") == "result":
_fill_relay_usage(ev, env)
+ _stamp_thinking(ev, rec)
with _turns_lock:
rec["events"].append(ev)
if ev.get("type") == "system" and ev.get("subtype") == "init" and ev.get("session_id"):
@@ -8741,6 +8942,7 @@ def _run_turn_bg(turn_id: str, cmd: list[str], env: dict, cwd: str, normalize, m
ev["model"] = served
if ev.get("type") == "result":
_fill_relay_usage(ev, env)
+ _stamp_thinking(ev, rec)
with _turns_lock:
rec["events"].append(ev)
if ev.get("type") == "result":
@@ -9058,8 +9260,10 @@ def _run_codex_appserver_bg(turn_id: str, cwd: str, env: dict, model: str, promp
res_txt = state["final"]
else:
res_txt = "\n\n".join(x for x in (state["final"].strip(), err_txt) if x)[:4000] or err_txt
- append({"type": "result", "subtype": "success" if ok else "error", "is_error": not ok,
- "result": res_txt, "usage": usage}) # surface the error in the trace
+ ev = {"type": "result", "subtype": "success" if ok else "error", "is_error": not ok,
+ "result": res_txt, "usage": usage} # surface the error in the trace
+ _stamp_thinking(ev, rec)
+ append(ev)
rec["result"] = state["final"]
rec["status"] = ("cancelled" if rec.get("cancelled") else "timeout" if rec.get("capped")
else "done" if ok else "failed")
@@ -9091,9 +9295,32 @@ _HERMES_POLL_S = 0.8
#
# A tool call that legitimately runs for minutes is NOT affected: hermes writes the assistant
# message carrying the tool call before executing it, so output exists and the guard is disarmed.
+#
+# A first answer that is LONG is not affected either, since 2026-10-05: hermes writes a message
+# into its database only when the message is complete, so an answer that streams for more than the
+# limit looked exactly like a hung call and was stopped mid-stream (found on the hosted service:
+# gpt-6-luna and gpt-5.4 asked to write an 18,000 word file as their first step, both stopped at
+# 90 s; the same answers completed after 293 s and 676 s when a tool call came first). The relay
+# sees that call's events as they pass, so the guard also asks when a provider last sent one on any
+# of the turn's routes. The CLI that hangs AFTER its provider answered is still caught: the last
+# event is then older than the limit.
_HERMES_STARTUP_TIMEOUT_S = float(os.environ.get("HERMES_STARTUP_TIMEOUT_S", "90"))
+def _hermes_startup_hung(produced: bool, started: float, last_data: float, now: float) -> bool:
+ """Whether a hermes turn that has written no message yet is hung, not merely slow to finish its
+ first one: nothing produced within the limit, and no provider event within it either."""
+ return (not produced and now - started > _HERMES_STARTUP_TIMEOUT_S
+ and now - last_data > _HERMES_STARTUP_TIMEOUT_S)
+
+
+def _routes_last_data(rec: dict) -> float:
+ """When a provider last sent an event on any route of this turn; 0.0 when none has, or the turn
+ has no route (its calls do not pass the relay)."""
+ return max((float(f.get("last_data") or 0.0) for f in rec.get("routes") or [] if isinstance(f, dict)),
+ default=0.0)
+
+
def _hermes_db_ro(db_path: str) -> sqlite3.Connection | None:
try:
db = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True, timeout=2)
@@ -9287,7 +9514,7 @@ def _run_hermes_bg(turn_id: str, cwd: str, env: dict, model: str, provider: str,
# Fires whether the session row never appeared or appeared and then produced nothing;
# both mean the provider call hung before any output, and both used to be survivable
# only by the six-hour cap.
- if not produced and (time.time() - t0) > _HERMES_STARTUP_TIMEOUT_S:
+ if _hermes_startup_hung(produced, t0, _routes_last_data(rec), time.time()):
rec["capped"] = True
rec["startup_timeout"] = True
_kill_proc_tree(proc)
@@ -9355,6 +9582,7 @@ def _run_hermes_bg(turn_id: str, cwd: str, env: dict, model: str, provider: str,
if served:
ev["model"] = served
_fill_relay_usage(ev, env)
+ _stamp_thinking(ev, rec)
append(ev)
rec["exit_code"] = rc
rec["result"] = final
@@ -9747,6 +9975,9 @@ class TurnReq(BaseModel):
vision_auth: dict | None = None # hermes: {provider, model, base_url, api_key} for its image questions
codex_appserver: bool = False # codex: run via app-server (streams item/agentMessage/delta)
environment: dict | None = None # {id, slug, entry}: the project layer at /env/<slug>, read-only (environments.py)
+ reasoning_effort: str | None = None # how much the model thinks this turn: one of reasoning.LEVELS; unset = the model's default
+ reasoning_route: str | None = None # which measured provider the connection is (reasoning.ROUTES), from the gateway,
+ # which knows it when the base here is a broker's; unset = read it off the base
@app.post("/turn")
@@ -9827,6 +10058,13 @@ def turn(req: TurnReq, identifier: str = "") -> dict:
# The harness's own variables under the runner's, so nothing a caller names shadows the
# runner's credentials or paths; the platform's HR_ names come after and win (below).
env = {**_caller_env(req.env), **_child_env()}
+ # The thinking level, for whatever the builder below registers or configures. Set for every
+ # turn, so a turn that asked for none also clears what an earlier one left in this context.
+ effort = str(req.reasoning_effort or "").strip().lower()
+ named = str(req.reasoning_route or "").strip().lower()
+ turn_thinking = {"asked": effort if effort in reasoning.LEVELS else "", "applied": "", "routes": [],
+ "route": named if named in reasoning.ROUTES else ""}
+ _TURN_THINKING.set(turn_thinking)
# Image generation. Deliberately NOT the OPENAI_* names: on a codex harness those already
# point at the CHAT connection, which is often a different provider, and one env pair can
# only carry one credential. The imagegen skill's wrapper reads these and passes them to the
@@ -10035,6 +10273,9 @@ def turn(req: TurnReq, identifier: str = "") -> dict:
"host": socket.gethostname(), "deduplicated": True, "max_seconds": MAX_TURN_SECONDS}
_turns[turn_id] = {"status": "running", "events": [], "result": "", "done": False,
"backend": backend, "model": model, "started": time.time(),
+ "reasoning": turn_thinking,
+ # the flags of every route the turn registered (see _turn_route)
+ "routes": turn_thinking["routes"],
# the values the record may not carry (see get_turn)
"secrets": _turn_secrets(_caller_env(req.env), req.env_secret)}
if codex_note: # the follow-up's Codex history was not here: the transcript says so first
diff --git a/runner/tests/test_hermes_startup_guard.py b/runner/tests/test_hermes_startup_guard.py
new file mode 100644
index 00000000..8ec75591
--- /dev/null
+++ b/runner/tests/test_hermes_startup_guard.py
@@ -0,0 +1,70 @@
+"""Hermes's startup guard stops a turn that is hung, not one whose first answer is long.
+
+hermes writes a message into its database only when the message is complete. A first answer that
+streams for longer than the guard's limit therefore looked exactly like a provider call that never
+answered, and was stopped mid-stream: on the hosted service, 2026-10-05, gpt-6-luna and gpt-5.4 asked
+to write an 18,000 word file as their first step were both stopped at 90 s, and completed after 293 s
+and 676 s of the same silence when a tool call came first. The relay sees that call's events as they
+pass, so the guard also asks when a provider last sent one.
+"""
+import http.client
+import http.server
+import json
+import sys
+import threading
+import time
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+import server as rs # noqa: E402
+
+LIMIT = rs._HERMES_STARTUP_TIMEOUT_S
+
+
+def test_a_first_answer_that_is_still_arriving_is_not_a_hung_call():
+ now = 10_000.0
+ started = now - LIMIT - 200 # well past the limit, nothing written yet
+ assert rs._hermes_startup_hung(False, started, last_data=now - 2, now=now) is False # events 2 s ago
+ assert rs._hermes_startup_hung(False, started, last_data=now - LIMIT - 1, now=now) is True
+ # the case the guard was written for: the provider answered at once, then the CLI hung
+ assert rs._hermes_startup_hung(False, started, last_data=started + 2.2, now=now) is True
+ # no route at all (the calls do not pass the relay): the guard is what it was
+ assert rs._hermes_startup_hung(False, started, last_data=0.0, now=now) is True
+ assert rs._hermes_startup_hung(False, now - LIMIT + 5, last_data=0.0, now=now) is False
+ assert rs._hermes_startup_hung(True, started, last_data=0.0, now=now) is False # it has produced
+
+
+def test_the_last_event_is_read_off_the_turns_own_routes():
+ assert rs._routes_last_data({}) == 0.0 and rs._routes_last_data({"routes": [{}]}) == 0.0
+ assert rs._routes_last_data({"routes": [{"last_data": 5.0}, {"last_data": 9.5}, {}]}) == 9.5
+
+
+def test_the_relay_notes_each_event_of_a_stream_on_the_route():
+ class Up(http.server.BaseHTTPRequestHandler):
+ def do_POST(self):
+ self.rfile.read(int(self.headers["content-length"]))
+ self.send_response(200); self.send_header("content-type", "text/event-stream"); self.end_headers()
+ for chunk in (b': keep-alive\n\n', b'data: {"choices":[{"delta":{"content":"a"}}]}\n\n',
+ b'data: {"choices":[{"delta":{},"finish_reason":"stop"}]}\n\n', b'data: [DONE]\n\n'):
+ self.wfile.write(chunk); self.wfile.flush()
+
+ def log_message(self, *a):
+ pass
+
+ up = http.server.HTTPServer(("127.0.0.1", 0), Up)
+ threading.Thread(target=up.serve_forever, daemon=True).start()
+ token = rs._TURN_THINKING.set({"asked": "", "applied": "", "routes": []})
+ try:
+ relay, tok = rs._hermes_relay_route(f"http://127.0.0.1:{up.server_port}/v1", "sk-real")
+ routes = rs._TURN_THINKING.get()["routes"]
+ finally:
+ rs._TURN_THINKING.reset(token)
+ conn = http.client.HTTPConnection(relay.removeprefix("http://").removesuffix("/v1"), timeout=10)
+ try:
+ before = time.time()
+ conn.request("POST", "/v1/chat/completions", body=json.dumps({"model": "m", "stream": True, "messages": []}),
+ headers={"authorization": f"Bearer {tok}", "content-type": "application/json"})
+ conn.getresponse().read()
+ assert rs._routes_last_data({"routes": routes}) >= before
+ finally:
+ conn.close(); up.shutdown(); rs._HERMES_RELAY["routes"].pop(tok, None)
diff --git a/runner/tests/test_reasoning.py b/runner/tests/test_reasoning.py
new file mode 100644
index 00000000..67d3d06b
--- /dev/null
+++ b/runner/tests/test_reasoning.py
@@ -0,0 +1,412 @@
+"""How much a model thinks on a turn (runner/reasoning.py and where the runner applies it).
+
+Every expectation here is a cell of the measurement of 2026-10-05 (docs/support-matrix-notes.md, "How
+much a model thinks"): one question at every level through each route, judged by the provider's own
+reasoning-token count. The traps are the tests: a level a model lacks is REFUSED by the provider, and
+the field that works is not the same on every route.
+"""
+import http.client
+import http.server
+import json
+import sys
+import threading
+from pathlib import Path
+
+import pytest
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
+import reasoning # noqa: E402
+import server as rs # noqa: E402
+
+VERCEL = "https://ai-gateway.vercel.sh/v1"
+TOKENROUTER = "https://api.tokenrouter.com/v1"
+OPENROUTER = "https://openrouter.ai/api/v1"
+GOOGLE = "https://generativelanguage.googleapis.com/v1beta/openai"
+OPENAI = "https://api.openai.com/v1"
+
+
+def _apply(shape, base, model, asked, **extra):
+ body = json.dumps({"model": model, "messages": [{"role": "user", "content": "hi"}], **extra}).encode()
+ out, applied = reasoning.apply(body, shape, base, asked)
+ doc = json.loads(out)
+ return applied, doc, out is body
+
+
[email protected]("shape,base,model,asked,applied,sent", [
+ # OpenAI: `minimal` is refused by gpt-5.4 ("Unsupported value") and by every route's Responses API
+ ("chat", VERCEL, "openai/gpt-5.4", "minimal", "low", {"reasoning_effort": "low"}),
+ ("chat", OPENAI, "gpt-5.4-2026-03-05", "none", "none", {"reasoning_effort": "none"}),
+ ("responses", OPENAI, "gpt-5.4", "xhigh", "xhigh", {"reasoning": {"effort": "xhigh"}}),
+ # gpt-6.1-sol and gpt-6-astra cannot be turned off: "does not support 'none' with this model"
+ ("chat", TOKENROUTER, "openai/gpt-6.1-sol", "none", "low", {"reasoning_effort": "low"}),
+ ("responses", OPENROUTER, "openai/gpt-6-astra", "none", "low", {"reasoning": {"effort": "low"}}),
+ # Gemini through Vercel takes `reasoning_effort: none` as "think more"; Google's own setting works
+ ("chat", VERCEL, "google/gemini-3.5-flash", "none", "none",
+ {"providerOptions": {"google": {"thinkingConfig": {"thinkingBudget": 0}}}}),
+ ("chat", VERCEL, "google/gemini-3-flash", "minimal", "minimal",
+ {"providerOptions": {"google": {"thinkingConfig": {"thinkingLevel": "minimal"}}}}),
+ # ...and the 3.8 flash model refuses both a zero budget and `minimal`: its lowest level is low
+ ("chat", VERCEL, "google/gemini-3.8-flash", "none", "low",
+ {"providerOptions": {"google": {"thinkingConfig": {"thinkingLevel": "low"}}}}),
+ # through TokenRouter the field is ignored and the setting rides `extra_body`
+ ("chat", TOKENROUTER, "google/gemini-3.5-flash", "high", "high",
+ {"extra_body": {"google": {"thinking_config": {"thinking_level": "high"}}}}),
+ # OpenRouter honours the field and refuses `none` for every Gemini; `minimal` is its zero
+ ("chat", OPENROUTER, "google/gemini-3.5-flash", "none", "minimal", {"reasoning_effort": "minimal"}),
+ ("chat", GOOGLE, "gemini-3.1-pro-preview", "none", "low", {"reasoning_effort": "low"}),
+ # Claude over Chat Completions; the 5.5 line cannot be turned off through OpenRouter
+ ("chat", VERCEL, "anthropic/claude-haiku-4.5", "medium", "medium", {"reasoning_effort": "medium"}),
+ ("chat", OPENROUTER, "anthropic/claude-sonnet-5.5", "none", "low", {"reasoning_effort": "low"}),
+ # a model that cannot be turned off gets its lowest level for `none`
+ ("chat", TOKENROUTER, "x-ai/grok-4.6", "none", "minimal", {"reasoning_effort": "minimal"}),
+ ("chat", TOKENROUTER, "deepseek/deepseek-v4-pro", "none", "low", {"reasoning_effort": "low"}),
+ # on or off only: any level that asks for thinking is "on"; Mistral through Vercel refuses `low`
+ ("chat", VERCEL, "alibaba/qwen3.7-max", "low", "high", {"reasoning_effort": "high"}),
+ ("chat", VERCEL, "mistral/mistral-medium-3.5", "low", "high", {"reasoning_effort": "high"}),
+])
+def test_a_level_is_written_the_way_the_route_and_the_model_take_it(shape, base, model, asked, applied, sent):
+ got, doc, same = _apply(shape, base, model, asked)
+ assert got == applied and not same
+ for key, value in sent.items():
+ assert doc[key] == value
+ if "providerOptions" in sent or "extra_body" in sent:
+ assert "reasoning_effort" not in doc
+
+
[email protected]("shape,base,model", [
+ ("chat", VERCEL, "zai/glm-5.3"), # no setting changed what it did
+ ("chat", "https://llm.example/v1", "gpt-5.4"), # a route that was not measured
+ ("chat", TOKENROUTER, "google/gemini-3.8-flash"), # answered to nothing through this door
+ ("responses", VERCEL, "anthropic/claude-haiku-4.5"), # that shape has no setting for the family
+ ("chat", OPENROUTER, "meta-llama/llama-3.3-70b-instruct"),
+])
+def test_an_unmeasured_model_or_route_gets_nothing_and_the_same_bytes(shape, base, model):
+ body = json.dumps({"model": model, "messages": []}).encode()
+ out, applied = reasoning.apply(body, shape, base, "high")
+ assert out is body and applied == ""
+
+
+def test_a_model_that_does_not_think_unless_asked_is_left_alone_for_none():
+ """Google refuses `none` for the flash-lite models ("invalid argument") and they do not think by
+ default, so nothing is sent and the turn still reads none."""
+ got, doc, _ = _apply("chat", GOOGLE, "gemini-3.5-flash-lite", "none")
+ assert got == "none" and "reasoning_effort" not in doc
+
+
+def test_anthropic_messages_take_a_budget_or_an_effort_by_model():
+ # the older model: a budget, out of max_tokens with half left to answer; an effort is refused
+ got, doc, _ = _apply("messages", TOKENROUTER, "anthropic/claude-haiku-4.5", "high", max_tokens=8000)
+ assert got == "high" and doc["thinking"] == {"type": "enabled", "budget_tokens": 4000}
+ got, doc, same = _apply("messages", TOKENROUTER, "anthropic/claude-haiku-4.5", "low", max_tokens=1500)
+ assert got == "" and same # below the API's floor of 1024
+ # the newer ones: adaptive with an effort ("...enabled is not supported for this model")
+ got, doc, _ = _apply("messages", "https://api.anthropic.com", "claude-sonnet-4-6", "xhigh",
+ output_config={"format": "x"})
+ assert doc["thinking"] == {"type": "adaptive"} and doc["output_config"] == {"format": "x", "effort": "max"}
+ # off: `disabled`, except the 5.5 line, whose API names its own word; fable has no off at all
+ assert _apply("messages", TOKENROUTER, "anthropic/claude-opus-5", "none")[1]["thinking"] == {"type": "disabled"}
+ assert _apply("messages", TOKENROUTER, "anthropic/claude-sonnet-5.5", "none")[1]["thinking"] == {"type": "between_tools"}
+ got, doc, _ = _apply("messages", TOKENROUTER, "anthropic/claude-fable-5.1", "none")
+ assert got == "low" and doc["output_config"] == {"effort": "low"}
+
+
+def test_googles_own_body_takes_a_level_or_a_zero_budget_never_both():
+ body = json.dumps({"contents": [], "generationConfig": {"thinkingConfig": {"includeThoughts": True, "thinkingBudget": -1}}}).encode()
+ out, applied = reasoning.apply(body, "google", GOOGLE, "low", model="gemini-3.5-flash")
+ assert applied == "low"
+ assert json.loads(out)["generationConfig"]["thinkingConfig"] == {"includeThoughts": True, "thinkingLevel": "low"}
+ out, applied = reasoning.apply(body, "google", GOOGLE, "none", model="gemini-3.5-flash")
+ assert json.loads(out)["generationConfig"]["thinkingConfig"] == {"includeThoughts": True, "thinkingBudget": 0}
+
+
+def test_nearest_never_turns_thinking_off_for_a_turn_that_asked_for_some():
+ assert reasoning.nearest("low", ("none", "high")) == "high"
+ assert reasoning.nearest("none", ("low", "high")) == "low"
+ assert reasoning.nearest("medium", ("low", "high")) == "low" # a tie goes down
+ assert reasoning.nearest("xhigh", ("none", "low", "medium")) == "medium"
+ assert reasoning.nearest("low", ("none",)) == "" and reasoning.nearest("bogus", ("low",)) == ""
+
+
+def test_levels_are_named_only_for_models_that_were_measured():
+ assert reasoning.levels_for("openai/gpt-5.4-mini") == ("none", "low", "medium", "high", "xhigh")
+ assert reasoning.levels_for("claude-haiku-4-5-20251001") == ("none", "low", "medium", "high", "xhigh")
+ assert "none" not in reasoning.levels_for("gpt-6.1-sol") and "none" not in reasoning.levels_for("grok-4.6")
+ assert reasoning.levels_for("gemini-3.8-flash") == ("low", "medium", "high")
+ for unmeasured in ("llama-3.3-70b", "glm-5.3", "grok-4.20", "qwen3.8-max", "nemotron-3-super", "jev-1.13", ""):
+ assert reasoning.levels_for(unmeasured) == ()
+ for model, levels in ((m, reasoning.levels_for(m)) for m in ("gpt-5.4", "grok-4.5", "kimi-k3", "hy3")):
+ assert levels and all(lv in reasoning.LEVELS for lv in levels), model
+
+
+# ── the relay ────────────────────────────────────────────────────────────────────────────────
+
+class _Upstream:
+ """A provider that records what it is sent and refuses whatever `refuse` says to."""
+
+ def __init__(self, refuse=lambda doc: False):
+ seen = self.seen = []
+
+ class H(http.server.BaseHTTPRequestHandler):
+ def do_POST(self):
+ raw = self.rfile.read(int(self.headers["content-length"]))
+ seen.append(raw)
+ if refuse(json.loads(raw)):
+ out, code = b'{"error":{"message":"Reasoning is mandatory for this endpoint and cannot be disabled."}}', 400
+ else:
+ out, code = (b'{"id":"c","model":"m","choices":[{"message":{"role":"assistant","content":"ok"},'
+ b'"finish_reason":"stop"}],"usage":{"prompt_tokens":5,"completion_tokens":9,'
+ b'"completion_tokens_details":{"reasoning_tokens":7}}}'), 200
+ self.send_response(code); self.send_header("content-type", "application/json")
+ self.send_header("content-length", str(len(out))); self.end_headers(); self.wfile.write(out)
+
+ def log_message(self, *a):
+ pass
+
+ self.srv = http.server.HTTPServer(("127.0.0.1", 0), H)
+ threading.Thread(target=self.srv.serve_forever, daemon=True).start()
+
+ def route(self, asked, monkeypatch):
+ """A route to this provider for a turn that asked for `asked`, measured as OpenRouter is."""
+ monkeypatch.setattr(reasoning, "route_of", lambda base: "openrouter")
+ token = rs._TURN_THINKING.set({"asked": asked, "applied": "", "routes": []})
+ try:
+ relay, tok = rs._hermes_relay_route(f"http://127.0.0.1:{self.srv.server_port}/v1", "sk-real")
+ finally:
+ rs._TURN_THINKING.reset(token)
+ return relay.removeprefix("http://").removesuffix("/v1"), tok
+
+
+def _post(host, tok, doc):
+ conn = http.client.HTTPConnection(host, timeout=10)
+ try:
+ raw = json.dumps(doc).encode()
+ conn.request("POST", "/v1/chat/completions", body=raw,
+ headers={"authorization": f"Bearer {tok}", "content-type": "application/json"})
+ resp = conn.getresponse()
+ return resp.status, resp.read(), raw
+ finally:
+ conn.close()
+
+
+def test_a_turn_that_asked_for_no_level_sends_what_the_client_wrote(monkeypatch):
+ up = _Upstream()
+ host, tok = up.route("", monkeypatch)
+ try:
+ assert "effort" not in rs._HERMES_RELAY["routes"][tok][2]
+ status, _, raw = _post(host, tok, {"model": "openai/gpt-5.4", "messages": [{"role": "user", "content": "hi"}]})
+ assert status == 200 and up.seen == [raw]
+ finally:
+ up.srv.shutdown(); rs._HERMES_RELAY["routes"].pop(tok, None)
+
+
+def test_the_relay_writes_the_level_and_remembers_what_it_applied(monkeypatch):
+ up = _Upstream()
+ host, tok = up.route("minimal", monkeypatch)
+ try:
+ status, _, _ = _post(host, tok, {"model": "openai/gpt-5.4", "messages": [{"role": "user", "content": "hi"}]})
+ assert status == 200 and json.loads(up.seen[0])["reasoning_effort"] == "low"
+ flags = rs._HERMES_RELAY["routes"][tok][2]
+ assert flags["effort_applied"] == {"openai/gpt-5.4": "low"}
+ # a helper model on the same route that has no measured level is left alone, and said to be
+ _post(host, tok, {"model": "meta-llama/llama-3.3-70b-instruct", "messages": []})
+ assert "reasoning_effort" not in json.loads(up.seen[1])
+ assert flags["effort_applied"]["meta-llama/llama-3.3-70b-instruct"] == ""
+ assert flags["reasoning_tokens"] == 14 # the provider's own count, both calls
+ assert "reasoning_tokens" not in flags["usage"] # the usage that prices a turn is untouched
+ finally:
+ up.srv.shutdown(); rs._HERMES_RELAY["routes"].pop(tok, None)
+
+
+def test_a_refused_level_never_fails_the_call_and_is_not_offered_again(monkeypatch):
+ up = _Upstream(refuse=lambda doc: "reasoning_effort" in doc)
+ host, tok = up.route("high", monkeypatch)
+ try:
+ doc = {"model": "openai/gpt-5.4", "messages": [{"role": "user", "content": "hi"}]}
+ status, _, raw = _post(host, tok, doc)
+ assert status == 200
+ assert "reasoning_effort" in json.loads(up.seen[0]) and up.seen[1] == raw # then as the client wrote it
+ flags = rs._HERMES_RELAY["routes"][tok][2]
+ assert flags["effort_applied"] == {"openai/gpt-5.4": ""} and flags["effort_refused:openai/gpt-5.4"]
+ _post(host, tok, doc)
+ assert len(up.seen) == 3 and up.seen[2] == raw # one call, no level
+ finally:
+ up.srv.shutdown(); rs._HERMES_RELAY["routes"].pop(tok, None)
+
+
+def test_a_refusal_that_is_not_about_the_level_is_the_clients_to_see_and_blames_nothing(monkeypatch):
+ up = _Upstream(refuse=lambda doc: True)
+ host, tok = up.route("high", monkeypatch)
+ try:
+ status, _, raw = _post(host, tok, {"model": "openai/gpt-5.4", "messages": []})
+ assert status == 400 and up.seen[1] == raw
+ flags = rs._HERMES_RELAY["routes"][tok][2]
+ assert "effort_refused:openai/gpt-5.4" not in flags and flags["effort_applied"] == {"openai/gpt-5.4": "high"}
+ finally:
+ up.srv.shutdown(); rs._HERMES_RELAY["routes"].pop(tok, None)
+
+
+# ── the turn's record, and the backends whose calls never pass the relay ──────────────────────
+
+def _thinking(asked):
+ return rs._TURN_THINKING.set({"asked": asked, "applied": "", "routes": []})
+
+
+def test_the_result_says_what_was_asked_and_what_was_applied():
+ """Read off the routes the turn registered, wherever the backend keeps its placeholder: pi and
+ omp write theirs into a file, and a lookup through the turn's environment recorded "default"
+ for a level that had been applied (the thinking column on a candidate, 2026-10-05)."""
+ flags = {"effort": "minimal", "effort_applied": {"aux/model": "", "openai/gpt-5.4": "low"},
+ "usage": {"input_tokens": 1, "output_tokens": 9}, "reasoning_tokens": 7}
+ rec = {"model": "openai/gpt-5.4", "reasoning": {"asked": "minimal", "applied": "", "routes": [flags]}}
+ ev = {"type": "result", "usage": {"input_tokens": 1, "output_tokens": 9}}
+ rs._stamp_thinking(ev, rec)
+ assert ev["reasoning"] == {"effort": "minimal", "applied": "low"} and ev["usage"]["reasoning_tokens"] == 7
+ # nothing was applied (an unmeasured model, or a refusal): the record says so
+ flags["effort_applied"] = {"openai/gpt-5.4": ""}
+ ev = {"type": "result"}
+ rs._stamp_thinking(ev, rec)
+ assert ev["reasoning"] == {"effort": "minimal", "applied": "default"}
+ # a backend that set its own CLI, with no route at all
+ ev = {"type": "result"}
+ rs._stamp_thinking(ev, {"model": "claude-haiku-4-5", "reasoning": {"asked": "high", "applied": "high", "routes": []}})
+ assert ev == {"type": "result", "reasoning": {"effort": "high", "applied": "high"}}
+ # a turn that asked for no level says nothing about one, and still carries the provider's count
+ ev = {"type": "result"}
+ rs._stamp_thinking(ev, {"model": "openai/gpt-5.4", "reasoning": {"asked": "", "applied": "", "routes": [flags]}})
+ assert ev == {"type": "result", "usage": {"reasoning_tokens": 7}}
+
+
+def test_a_turns_routes_are_remembered_without_its_environment():
+ token = _thinking("low")
+ tok = aux = ""
+ try:
+ _, tok = rs._hermes_relay_route("https://ai-gateway.vercel.sh/v1", "sk-real")
+ _, aux = rs._hermes_relay_route("https://ai-gateway.vercel.sh/v1", "sk-real", aux=True)
+ routes = rs._TURN_THINKING.get()["routes"]
+ assert len(routes) == 1 and routes[0] is rs._HERMES_RELAY["routes"][tok][2] and routes[0]["effort"] == "low"
+ assert "effort" not in rs._HERMES_RELAY["routes"][aux][2] # a helper model's route
+ finally:
+ rs._TURN_THINKING.reset(token)
+ rs._HERMES_RELAY["routes"].pop(tok, None); rs._HERMES_RELAY["routes"].pop(aux, None)
+
+
+def test_claude_code_gets_its_own_switch_by_model():
+ token = _thinking("high")
+ try:
+ assert rs._claude_thinking_env("claude-sonnet-5-5")["CLAUDE_CODE_EFFORT_LEVEL"] == "high"
+ env = rs._claude_thinking_env("claude-haiku-4-5-20251001") # refuses an effort; takes a budget
+ assert env["CLAUDE_CODE_EFFORT_LEVEL"] == "auto" and env["MAX_THINKING_TOKENS"] == "16384"
+ assert rs._TURN_THINKING.get()["applied"] == "high"
+ finally:
+ rs._TURN_THINKING.reset(token)
+ token = _thinking("none")
+ try:
+ assert rs._claude_thinking_env("claude-sonnet-5-5")["MAX_THINKING_TOKENS"] == "0"
+ finally:
+ rs._TURN_THINKING.reset(token)
+ token = _thinking("")
+ try: # no level asked: as before
+ assert rs._claude_thinking_env("claude-sonnet-5-5") == {"CLAUDE_CODE_EFFORT_LEVEL": "auto"}
+ assert rs._claude_thinking_env("claude-opus-4-8") == {"CLAUDE_CODE_EFFORT_LEVEL": "auto", "MAX_THINKING_TOKENS": "0"}
+ finally:
+ rs._TURN_THINKING.reset(token)
+
+
+def test_codex_writes_the_level_into_its_own_config(tmp_path):
+ def cfg(model, asked):
+ token = rs._TURN_THINKING.set({"asked": asked, "applied": "", "routes": []})
+ try:
+ home = tmp_path / f"{model}-{asked or 'unset'}"
+ home.mkdir()
+ d = rs._codex_prepare_env("openai", rs.Auth(api_key="k", base_url="https://api.openai.com/v1"),
+ model, str(home), {"HOME": str(home)})
+ return (d / "config.toml").read_text(), (rs._TURN_THINKING.get() or {}).get("applied")
+ finally:
+ rs._TURN_THINKING.reset(token)
+
+ assert f'model_reasoning_effort = "{rs.CODEX_REASONING_EFFORT}"' in cfg("gpt-5.4", "")[0]
+ text, applied = cfg("gpt-5.4", "minimal") # refused by the model; its nearest is low
+ assert 'model_reasoning_effort = "low"' in text and applied == "low"
+ text, applied = cfg("gpt-6.1-sol", "none") # cannot be turned off
+ assert 'model_reasoning_effort = "low"' in text and applied == "low"
+
+
+def test_the_providers_count_of_thinking_tokens_is_read_in_every_shape_and_apart_from_pricing():
+ chat = {"usage": {"prompt_tokens": 5, "completion_tokens": 9, "completion_tokens_details": {"reasoning_tokens": 7}}}
+ responses = {"usage": {"input_tokens": 5, "output_tokens": 9, "output_tokens_details": {"reasoning_tokens": 4}}}
+ stream_end = {"type": "response.completed", "response": responses} # the Responses API's stream
+ google = {"usageMetadata": {"promptTokenCount": 5, "candidatesTokenCount": 9, "thoughtsTokenCount": 3}}
+ assert [rs._thinking_tokens_in(d) for d in (chat, responses, stream_end, google)] == [7, 4, 4, 3]
+ assert rs._thinking_tokens_in({"usage": {"input_tokens": 5, "output_tokens": 9}}) is None # Anthropic gives none
+ assert rs._thinking_in_sse_line(b"data: " + json.dumps(stream_end).encode()) == 4
+ assert rs._thinking_in_sse_line(b"data: [DONE]") is None
+ # the usage that prices a turn carries what it carried before
+ assert rs._usage_fields(chat["usage"]) == {"input_tokens": 5, "output_tokens": 9, "cache_read_tokens": 0}
+ assert rs._usage_fields(google["usageMetadata"]) == {"input_tokens": 5, "output_tokens": 9, "cache_read_tokens": 0}
+
+
+def test_a_streamed_calls_usage_is_on_the_route_before_the_client_has_the_bytes():
+ """The provider's last events pass long before it closes the stream. A client that is done at
+ `[DONE]` (CheetahClaws's in-process driver) read the route's totals before its own call was on
+ them, and its turn came back with no usage."""
+ release = threading.Event()
+
+ class Up(http.server.BaseHTTPRequestHandler):
+ def do_POST(self):
+ self.rfile.read(int(self.headers["content-length"]))
+ self.send_response(200); self.send_header("content-type", "text/event-stream"); self.end_headers()
+ for chunk in (b'data: {"choices":[{"delta":{"content":"ok"}}]}\n\n',
+ b'data: {"choices":[],"usage":{"prompt_tokens":5,"completion_tokens":9,'
+ b'"completion_tokens_details":{"reasoning_tokens":7}}}\n\n', b'data: [DONE]\n\n'):
+ self.wfile.write(chunk); self.wfile.flush()
+ release.wait(10) # the provider has not closed the stream yet
+
+ def log_message(self, *a):
+ pass
+
+ up = http.server.ThreadingHTTPServer(("127.0.0.1", 0), Up)
+ threading.Thread(target=up.serve_forever, daemon=True).start()
+ relay, tok = rs._hermes_relay_route(f"http://127.0.0.1:{up.server_port}/v1", "sk-real")
+ conn = http.client.HTTPConnection(relay.removeprefix("http://").removesuffix("/v1"), timeout=10)
+ try:
+ conn.request("POST", "/v1/chat/completions", body=json.dumps({"model": "m", "stream": True, "messages": []}),
+ headers={"authorization": f"Bearer {tok}", "content-type": "application/json"})
+ resp = conn.getresponse()
+ seen = b""
+ while b"[DONE]" not in seen:
+ seen += resp.read1(65536)
+ flags = rs._HERMES_RELAY["routes"][tok][2] # what a client done at [DONE] reads
+ assert flags["usage"] == {"input_tokens": 5, "output_tokens": 9, "cache_read_tokens": 0}
+ assert flags["reasoning_tokens"] == 7
+ release.set()
+ resp.read()
+ assert flags["usage"]["output_tokens"] == 9 and flags["reasoning_tokens"] == 7 # counted once
+ finally:
+ release.set(); conn.close(); up.shutdown(); rs._HERMES_RELAY["routes"].pop(tok, None)
+
+
+def test_a_route_the_gateway_names_wins_over_the_base(monkeypatch):
+ """Behind a broker every base is the broker's, so the base says nothing about the provider. The
+ gateway, which knows the connection, names the route on the turn."""
+ body = json.dumps({"model": "google/gemini-3.5-flash", "messages": []}).encode()
+ broker = "https://hr.example/v1/llm"
+ assert reasoning.apply(body, "chat", broker, "none") == (body, "")
+ out, applied = reasoning.apply(body, "chat", broker, "none", route="vercel")
+ assert applied == "none" and json.loads(out)["providerOptions"]["google"]["thinkingConfig"] == {"thinkingBudget": 0}
+ assert reasoning.apply(body, "chat", broker, "none", route="not-a-route") == (body, "")
+ # the turn carries it to its routes, in the relay's memory and nowhere the agent can read
+ token = rs._TURN_THINKING.set({"asked": "low", "applied": "", "routes": [], "route": "openrouter"})
+ tok = ""
+ try:
+ _, tok = rs._hermes_relay_route(broker, "per-turn-credential")
+ flags = rs._HERMES_RELAY["routes"][tok][2]
+ assert flags["effort"] == "low" and flags["effort_route"] == "openrouter"
+ finally:
+ rs._TURN_THINKING.reset(token); rs._HERMES_RELAY["routes"].pop(tok, None)
+ # a turn with no named route reads the base, as before
+ token = rs._TURN_THINKING.set({"asked": "low", "applied": "", "routes": [], "route": ""})
+ try:
+ _, tok = rs._hermes_relay_route(VERCEL, "k")
+ assert "effort_route" not in rs._HERMES_RELAY["routes"][tok][2]
+ finally:
+ rs._TURN_THINKING.reset(token); rs._HERMES_RELAY["routes"].pop(tok, None)
diff --git a/runner/tests/test_session_uids.py b/runner/tests/test_session_uids.py
index 07f09ed7..e5547293 100644
--- a/runner/tests/test_session_uids.py
+++ b/runner/tests/test_session_uids.py
@@ -112,7 +112,7 @@ def test_hermes_vision_auth_is_its_own_route(tmp_path, monkeypatch):
import yaml
env = {"HOME": str(tmp_path)}
monkeypatch.setattr(server, "_hermes_relay_route",
- lambda base, key: ("http://127.0.0.1:1/v1", "hr-relay-placeholder"))
+ lambda base, key, **kw: ("http://127.0.0.1:1/v1", "hr-relay-placeholder"))
server._hermes_prepare_env("openai-api", server.Auth(api_key="k-chat", base_url="https://x/v1"),
str(tmp_path), env, model="qwen/qwen3.8-max",
vision_auth={"provider": "openai-api", "model": "gpt-5.4-mini",
diff --git a/scripts/support-matrix/README.md b/scripts/support-matrix/README.md
index 05cf40ba..9d0d71a7 100644
--- a/scripts/support-matrix/README.md
+++ b/scripts/support-matrix/README.md
@@ -79,6 +79,25 @@ the harness's Browser section in the agent's instructions file is what the agent
The table (`--md`) is the Browser section of docs/support-matrix.md; the run's findings go to
docs/support-matrix-notes.md like every other column's.
+## The thinking column
+
+`thinking/run-column.py` asks whether a level a turn asks for changes how much the model thinks, on
+every base: one harness per base on a model that has levels, the same task with no level and at
+`none`, `low` and `high`, judged from the turn's own record (`reasoning.applied`) and the provider's
+count of thinking tokens (`usage.output_tokens_details.reasoning_tokens`), with output tokens
+standing in where a provider gives no count. What the agent says about itself is not read.
+
+ python3 thinking/run-column.py --base-url https://<instance>/api/harness --api-key "$KEY" \
+ --out thinking.json # every base, its first model with levels
+ python3 thinking/run-column.py ... --bases codex,pi --model gpt-5.4 --repeat 2
+
+`thinking/probe.py` is how the levels in runner/reasoning.py were found: one question at every
+level, and at each provider's own switches, straight at a provider's API, reading its usage. Run it
+again when a model is added; a level goes into the table only where it was seen to be accepted.
+
+ THINKING_PROBE_BASE=https://openrouter.ai/api/v1 THINKING_PROBE_KEY=... \
+ python3 thinking/probe.py chat:openai/gpt-5.4 responses:openai/gpt-5.4 messages:anthropic/claude-haiku-4.5
+
## Environments column
`python3 environments/column.py --base-url ... --api-key ... --environment henv_...` runs three tasks per base (pip, npm, apt) on one built environment and judges each by the trace (no install), the answer (the package's path under the environment) and the session record. See docs/support-matrix-notes.md.
diff --git a/scripts/support-matrix/thinking/probe.py b/scripts/support-matrix/thinking/probe.py
new file mode 100644
index 00000000..48dde3a7
--- /dev/null
+++ b/scripts/support-matrix/thinking/probe.py
@@ -0,0 +1,139 @@
+#!/usr/bin/env python3
+"""What turns a model's thinking up, down and off, on one provider.
+
+Sends one question to each model named, once with no setting and once per variant (every level of
+`reasoning_effort`, and each provider's own switches), and prints what the provider's usage says the
+model spent thinking. This is how the table in runner/reasoning.py was measured: a level is in it
+only where a row of this output showed it accepted and moving the count.
+
+ THINKING_PROBE_BASE=https://openrouter.ai/api/v1 THINKING_PROBE_KEY=... \
+ python3 probe.py chat:openai/gpt-5.4 responses:openai/gpt-5.4 messages:anthropic/claude-haiku-4.5 \
+ chat:google/gemini-3.5-flash:unset,e:none,e:low,e:high,g:0,g:lo,x:0
+
+An argument is shape:model[:variants]; shape is chat (Chat Completions), responses or messages
+(Anthropic). The variants are the keys of VARIANTS below; without a list the shape's default set
+runs. `@file` reads the arguments from a file. Without the two variables it reads a Grok Build
+turn's own route (GROK_HOME's config and HR_GROK_API_KEY), which is how to reach a connection of an
+instance without holding its key: run it as a shell command of a turn on that connection.
+
+Output: one line per model, variant=thinking tokens/output tokens@seconds, `+rc` when the answer
+carried reasoning text, `!status message` for a refusal. Results also go to thinking_result.json.
+One sample per cell: read the direction, not the number.
+"""
+import concurrent.futures, json, os, re, sys, time, urllib.request, urllib.error
+base, key = os.environ.get("THINKING_PROBE_BASE", "").rstrip("/"), os.environ.get("THINKING_PROBE_KEY", "")
+if not (base and key):
+ home = os.environ["GROK_HOME"]; key = os.environ["HR_GROK_API_KEY"]
+ base = re.search(r'^base_url = "([^"]+)"', open(home + "/config.toml").read(), re.M).group(1)
+Q = "How many integers between 1 and 1000 inclusive are divisible by none of 6, 10 and 15? Reply with the number only."
+CAP = 6000
+VARIANTS = {
+ "chat": {
+ "unset": {},
+ "e:none": {"reasoning_effort": "none"}, "e:minimal": {"reasoning_effort": "minimal"},
+ "e:low": {"reasoning_effort": "low"}, "e:medium": {"reasoning_effort": "medium"},
+ "e:high": {"reasoning_effort": "high"}, "e:xhigh": {"reasoning_effort": "xhigh"},
+ "r:off": {"reasoning": {"enabled": False}}, "r:low": {"reasoning": {"effort": "low"}},
+ "r:high": {"reasoning": {"effort": "high"}}, "r:none": {"reasoning": {"effort": "none"}},
+ "g:0": {"providerOptions": {"google": {"thinkingConfig": {"thinkingBudget": 0}}}},
+ "g:512": {"providerOptions": {"google": {"thinkingConfig": {"thinkingBudget": 512}}}},
+ "x:0": {"extra_body": {"google": {"thinking_config": {"thinking_budget": 0}}}},
+ "g:med": {"providerOptions": {"google": {"thinkingConfig": {"thinkingLevel": "medium"}}}},
+ "x:hi": {"extra_body": {"google": {"thinking_config": {"thinking_level": "high"}}}},
+ "x:med": {"extra_body": {"google": {"thinking_config": {"thinking_level": "medium"}}}},
+ "x:min": {"extra_body": {"google": {"thinking_config": {"thinking_level": "minimal"}}}},
+ "g:min": {"providerOptions": {"google": {"thinkingConfig": {"thinkingLevel": "minimal"}}}},
+ "g:lo": {"providerOptions": {"google": {"thinkingConfig": {"thinkingLevel": "low"}}}},
+ "g:hi": {"providerOptions": {"google": {"thinkingConfig": {"thinkingLevel": "high"}}}},
+ "x:lo": {"extra_body": {"google": {"thinking_config": {"thinking_level": "low"}}}},
+ "x:512": {"extra_body": {"google": {"thinking_config": {"thinking_budget": 512}}}},
+ "r:minimal": {"reasoning": {"effort": "minimal"}}, "r:max0": {"reasoning": {"max_tokens": 0}},
+ "r:max512": {"reasoning": {"max_tokens": 512}},
+ "pa:off": {"providerOptions": {"anthropic": {"thinking": {"type": "disabled"}}}},
+ "t:off": {"thinking": {"type": "disabled"}}, "t:on": {"thinking": {"type": "enabled"}},
+ "q:off": {"enable_thinking": False}, "q:on": {"enable_thinking": True},
+ },
+ "responses": {
+ "unset": {}, "e:none": {"reasoning": {"effort": "none"}}, "e:minimal": {"reasoning": {"effort": "minimal"}},
+ "e:low": {"reasoning": {"effort": "low"}}, "e:medium": {"reasoning": {"effort": "medium"}},
+ "e:high": {"reasoning": {"effort": "high"}}, "e:xhigh": {"reasoning": {"effort": "xhigh"}},
+ },
+ "messages": {
+ "unset": {}, "t:off": {"thinking": {"type": "disabled"}},
+ "t:1024": {"thinking": {"type": "enabled", "budget_tokens": 1024}},
+ "t:4000": {"thinking": {"type": "enabled", "budget_tokens": 4000}},
+ "a": {"thinking": {"type": "adaptive"}},
+ "bt": {"thinking": {"type": "between_tools"}},
+ "a:medium": {"thinking": {"type": "adaptive"}, "output_config": {"effort": "medium"}},
+ "a:xhigh": {"thinking": {"type": "adaptive"}, "output_config": {"effort": "xhigh"}},
+ "a:max": {"thinking": {"type": "adaptive"}, "output_config": {"effort": "max"}},
+ "a:low": {"thinking": {"type": "adaptive"}, "output_config": {"effort": "low"}},
+ "a:high": {"thinking": {"type": "adaptive"}, "output_config": {"effort": "high"}},
+ "o:low": {"output_config": {"effort": "low"}}, "o:high": {"output_config": {"effort": "high"}},
+ },
+}
+DEFAULT = {"chat": "unset,e:none,e:minimal,e:low,e:medium,e:high,e:xhigh", "responses": "unset,e:none,e:minimal,e:low,e:medium,e:high,e:xhigh",
+ "messages": "unset,t:off,t:1024,t:4000,a,a:low,a:high,o:low,o:high"}
+
+def one(shape, model, vname):
+ extra = VARIANTS[shape][vname]
+ if shape == "chat":
+ path, body = "/chat/completions", {"model": model, "stream": False, "max_tokens": CAP, "messages": [{"role": "user", "content": Q}]}
+ elif shape == "responses":
+ path, body = "/responses", {"model": model, "stream": False, "max_output_tokens": CAP, "input": Q}
+ else:
+ path, body = "/messages", {"model": model, "stream": False, "max_tokens": CAP, "messages": [{"role": "user", "content": Q}]}
+ body.update(extra)
+ hdr = {"authorization": "Bearer " + key, "content-type": "application/json"}
+ if shape == "messages":
+ hdr["anthropic-version"] = "2023-06-01"
+ rec = {"shape": shape, "model": model, "variant": vname}
+ t0 = time.time()
+ try:
+ d = json.loads(urllib.request.urlopen(urllib.request.Request(base + path, data=json.dumps(body).encode(), headers=hdr), timeout=240).read())
+ u = d.get("usage") or {}
+ det = u.get("completion_tokens_details") or u.get("output_tokens_details") or {}
+ rec["rt"] = det.get("reasoning_tokens")
+ rec["out"] = u.get("completion_tokens", u.get("output_tokens"))
+ rec["served"] = d.get("model")
+ if shape == "chat":
+ m = (d.get("choices") or [{}])[0].get("message") or {}
+ rec["ans"] = str(m.get("content") or "")[:24]
+ rec["rc"] = len(str(m.get("reasoning") or m.get("reasoning_content") or ""))
+ elif shape == "responses":
+ rec["ans"] = "".join(c.get("text", "") for it in d.get("output") or [] if it.get("type") == "message" for c in it.get("content") or [])[:24]
+ rec["rc"] = sum(1 for it in d.get("output") or [] if it.get("type") == "reasoning")
+ else:
+ cs = d.get("content") or []
+ rec["ans"] = "".join(c.get("text", "") for c in cs if c.get("type") == "text")[:24]
+ rec["rc"] = sum(len(c.get("thinking") or "") for c in cs if c.get("type") == "thinking") or sum(1 for c in cs if c.get("type") == "redacted_thinking")
+ except urllib.error.HTTPError as e:
+ rec["err"] = str(e.code) + " " + e.read()[:260].decode("utf-8", "replace")
+ except Exception as e: # noqa: BLE001
+ rec["err"] = repr(e)[:160]
+ rec["s"] = round(time.time() - t0, 1)
+ return rec
+
+jobs = []
+ARGS = open(sys.argv[1][1:]).read().split() if sys.argv[1].startswith('@') else sys.argv[1:]
+for a in ARGS:
+ parts = a.split(":", 2)
+ shape, model = parts[0], parts[1]
+ for v in (parts[2] if len(parts) > 2 else DEFAULT[shape]).split(","):
+ jobs.append((shape, model, v))
+with concurrent.futures.ThreadPoolExecutor(8) as ex:
+ out = list(ex.map(lambda j: one(*j), jobs))
+json.dump(out, open("thinking_result.json", "w"), indent=0)
+by = {}
+for r in out:
+ by.setdefault((r["shape"], r["model"]), []).append(r)
+for (shape, model), rs in by.items():
+ cells = []
+ for r in rs:
+ if "err" in r:
+ m = re.search(r'"message"\s*:\s*"([^"]{0,90})', r["err"])
+ cells.append(f"{r['variant']}=!{r['err'].split(' ', 1)[0]} {m.group(1) if m else r['err'][4:60]}")
+ else:
+ cells.append(f"{r['variant']}={r.get('rt')}/{r.get('out')}{'+rc' if r.get('rc') else ''}@{r.get('s')}")
+ print(f"{shape[:4]} {model}: " + " | ".join(cells))
+print("wrote", len(out))
diff --git a/scripts/support-matrix/thinking/run-column.py b/scripts/support-matrix/thinking/run-column.py
new file mode 100644
index 00000000..f927a37b
--- /dev/null
+++ b/scripts/support-matrix/thinking/run-column.py
@@ -0,0 +1,179 @@
+#!/usr/bin/env python3
+"""The thinking column: does a level a turn asks for change how much the model thinks, on every base.
+
+For each base it creates a harness on one model that has levels, runs the same task at each level
+(and once with no level), and reads the turn's own record: what was applied, and the provider's count
+of thinking tokens. The judge is the provider's figure, never what the agent says about itself:
+
+ unset no level asked: the record carries no `reasoning`, as before levels existed
+ applied each level asked is recorded with the level the model was given
+ off `none` spends no thinking tokens, where the model has `none`
+ order the low level spends fewer thinking tokens than the high one
+
+Where a provider reports no thinking count for the model (Anthropic through some doors), output tokens
+stand in, since thinking is billed as output, and the row says so.
+
+ python3 thinking/run-column.py --base-url https://your-instance/api/harness --api-key "$KEY" \
+ --bases codex,hermes --out thinking.json
+
+--model pins one model for every base; without it each base gets the first model of PREFERRED it
+lists as available with levels. --repeat N runs every level N times and compares the sums.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+import time
+import urllib.error
+import urllib.request
+
+# A question worth thinking about, with a one-line answer, and no reason to reach for a tool.
+PROMPT = ("Answer from your own reasoning; do not run any tool or command. A regular clock's hour and "
+ "minute hands overlap 11 times in every 12 hours. At what time, to the nearest second, does the "
+ "7th overlap after 12:00:00 happen? Then, how many integers between 1 and 5000 inclusive are "
+ "divisible by none of 6, 10 and 15? Reply with exactly two lines: the time as HH:MM:SS, then the count.")
+PREFERRED = ["gpt-5.4", "claude-haiku-4.5", "gemini-3.5-flash", "grok-4.6", "kimi-k3", "deepseek-v4-flash",
+ "minimax-m3", "qwen3.7-max"]
+
+
+class Api:
+ def __init__(self, base_url: str, key: str):
+ self.base, self.key = base_url.rstrip("/"), key
+
+ def call(self, method: str, path: str, body=None, timeout: float = 60.0):
+ data = json.dumps(body).encode() if body is not None else None
+ req = urllib.request.Request(self.base + path, data=data, method=method, headers={
+ "authorization": f"Bearer {self.key}", "accept": "application/json",
+ **({"content-type": "application/json"} if data else {})})
+ try:
+ with urllib.request.urlopen(req, timeout=timeout) as r:
+ return r.status, json.loads(r.read() or b"{}")
+ except urllib.error.HTTPError as e:
+ raw = e.read()
+ try:
+ return e.code, json.loads(raw or b"{}")
+ except ValueError:
+ return e.code, {"raw": raw[:300].decode(errors="replace")}
+ except Exception as e: # noqa: BLE001
+ return 0, {"raw": repr(e)[:200]}
+
+
+def one_turn(api: Api, harness: str, level: str, timeout: float) -> dict:
+ body = {"input": PROMPT, "metadata": {"harness_id": harness}, "stream": False, "max_step": 6}
+ if level:
+ body["reasoning"] = {"effort": level}
+ t0 = time.time()
+ status, resp = api.call("POST", "/v1/responses", body, timeout=timeout)
+ usage = resp.get("usage") or {}
+ return {"level": level or "unset", "http": status, "status": resp.get("status"),
+ "reasoning": resp.get("reasoning"), "connection": resp.get("connection"),
+ "thinking_tokens": (usage.get("output_tokens_details") or {}).get("reasoning_tokens"),
+ "output_tokens": usage.get("output_tokens"), "seconds": round(time.time() - t0, 1),
+ "error": ((resp.get("error") or {}).get("message") or resp.get("raw") or "")[:200] if status != 200 or resp.get("status") != "completed" else ""}
+
+
+def judge(levels: list[str], turns: list[dict]) -> tuple[str, list[str]]:
+ """→ (verdict, findings). A finding names the claim that did not hold, with its numbers."""
+ by: dict[str, list[dict]] = {}
+ for t in turns:
+ by.setdefault(t["level"], []).append(t)
+ findings = []
+ bad = [f"{t['level']}: {t['status'] or t['http']} {t['error']}" for t in turns if t["status"] != "completed"]
+ if bad:
+ return "not run", bad
+ if any(t["reasoning"] is not None for t in by.get("unset", [])):
+ findings.append(f"unset: the record carries a level {by['unset'][0]['reasoning']}")
+ for lv in levels:
+ for t in by.get(lv, []):
+ r = t["reasoning"] or {}
+ if r.get("effort") != lv or r.get("applied") in (None, "", "default"):
+ findings.append(f"{lv}: recorded {t['reasoning']}")
+ # A provider may leave the count out of an answer that spent none, so a missing count on the
+ # `none` turn is a zero; on a turn that was asked to think it means the provider gives no count.
+ counted = all(t["thinking_tokens"] is not None for lv in levels if lv != "none" for t in by.get(lv, []))
+ key = "thinking_tokens" if counted else "output_tokens"
+
+ def total(lv):
+ return sum(int(t[key] or 0) for t in by.get(lv, []))
+ if "none" in levels and counted and total("none") != 0:
+ findings.append(f"none: {total('none')} thinking tokens")
+ low, high = ("low" if "low" in levels else ""), ("high" if "high" in levels else "")
+ if low and high and not total(low) < total(high):
+ findings.append(f"order: {low} {total(low)} is not below {high} {total(high)} ({key})")
+ if not low and "none" in levels and high and not total("none") < total(high):
+ findings.append(f"order: none {total('none')} is not below {high} {total(high)} ({key})")
+ return ("pass" if not findings else "fail"), ([] if not findings else findings) + ([] if counted else ["judged on output tokens: no thinking count from the provider"])
+
+
+def main() -> int:
+ p = argparse.ArgumentParser()
+ p.add_argument("--base-url", required=True)
+ p.add_argument("--api-key", required=True)
+ p.add_argument("--bases", default="all")
+ p.add_argument("--model", default="")
+ p.add_argument("--repeat", type=int, default=1)
+ p.add_argument("--task-timeout", type=float, default=420.0)
+ p.add_argument("--keep", action="store_true")
+ p.add_argument("--out", default="")
+ a = p.parse_args()
+
+ api = Api(a.base_url, a.api_key)
+ status, doc = api.call("GET", "/v1/bases")
+ if status != 200:
+ print(f"GET /v1/bases answered {status}: {doc}", file=sys.stderr)
+ return 2
+ bases = {b["id"]: b for b in doc.get("bases") or []}
+ if not any("reasoning" in m for b in bases.values() for m in b.get("models") or []):
+ print("this server lists no thinking levels on /v1/bases", file=sys.stderr)
+ return 2
+ wanted = list(bases) if a.bases == "all" else [b.strip() for b in a.bases.split(",") if b.strip()]
+ rows = []
+ for base in wanted:
+ info = bases.get(base)
+ row = {"base": base, "model": "", "levels": [], "turns": [], "verdict": "", "findings": []}
+ rows.append(row)
+ if not info:
+ row["verdict"], row["findings"] = "not run", ["no such base on this server"]
+ continue
+ have = {m["id"]: m.get("reasoning") or [] for m in info.get("models") or [] if m.get("available")}
+ model = a.model if a.model else next((m for m in PREFERRED if have.get(m)), "") or \
+ next((m for m, lv in have.items() if lv), "")
+ if not model or not have.get(model):
+ row["verdict"], row["findings"] = "no levels", [f"no available model with levels ({a.model or 'none listed'})"]
+ print(f"{base:13s} no levels {row['findings'][0]}", flush=True)
+ continue
+ row["model"] = model
+ offered = have[model]
+ levels = [lv for lv in ("none", "low", "high") if lv in offered]
+ row["levels"] = levels
+ status, created = api.call("POST", "/v1/harnesses", {"name": f"thinking-column-{base}", "base": base,
+ "default_model": model})
+ if status != 200:
+ row["verdict"], row["findings"] = "not run", [f"create harness: HTTP {status} {str(created)[:160]}"]
+ print(f"{base:13s} not run {row['findings'][0]}", flush=True)
+ continue
+ try:
+ for _ in range(max(a.repeat, 1)):
+ for lv in [""] + levels:
+ row["turns"].append(one_turn(api, created["id"], lv, a.task_timeout))
+ finally:
+ if not a.keep:
+ api.call("DELETE", f"/v1/harnesses/{created['id']}")
+ row["verdict"], row["findings"] = judge(levels, row["turns"])
+ cells = " ".join(f"{t['level']}={t['thinking_tokens']}/{t['output_tokens']}"
+ f"{'>' + str((t['reasoning'] or {}).get('applied')) if t['reasoning'] else ''}" for t in row["turns"])
+ print(f"{base:13s} {row['verdict']:8s} {model:20s} {cells}", flush=True)
+ for f in row["findings"]:
+ print(f"{'':13s} {f}", flush=True)
+ if a.out:
+ json.dump({"target": a.base_url, "prompt": PROMPT, "rows": rows}, open(a.out, "w"), indent=2)
+ ran = [r for r in rows if r["verdict"] in ("pass", "fail")]
+ print(f"\n thinking: {sum(r['verdict'] == 'pass' for r in ran)}/{len(ran)} bases pass; "
+ f"{sum(r['verdict'] == 'no levels' for r in rows)} with no levels; "
+ f"{sum(r['verdict'] == 'not run' for r in rows)} not run")
+ return 0 if all(r["verdict"] in ("pass", "no levels") for r in rows) else 1
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/ui/src/components/ApiReference.tsx b/ui/src/components/ApiReference.tsx
index ccc50b90..707fe803 100644
--- a/ui/src/components/ApiReference.tsx
+++ b/ui/src/components/ApiReference.tsx
@@ -121,6 +121,7 @@ ${addHdrs.map((h) => ` -H "${h}: <value>" \\\n`).join('')} -d '{ "model": "${m
<Param name="metadata" type="object">Up to 16 key/value string pairs attached to the response. <code>metadata.session_id</code> doubles as a continuation fallback (send it together with <code>previous_response_id</code>).</Param>
<Param name="max_step" type="integer">Per-request override of the agent&apos;s step budget (harness default, else 400).</Param>
<Param name="timeout_seconds" type="integer">Per-request override of the per-task wall-clock cap (harness default, else 1800).</Param>
+ <Param name="reasoning" type="object">How much the model thinks on this turn: <code>{'{"effort": "low"}'}</code>, one of <code>none</code>, <code>minimal</code>, <code>low</code>, <code>medium</code>, <code>high</code>, <code>xhigh</code>. Overrides the harness default. A model that lacks the level gets the nearest one it has, and the response&apos;s <code>reasoning</code> field says what was asked and what was applied. Unset leaves the model at its own default.</Param>
</ParamTable>
<div className="apidoc-sub">Response (200)</div>
<CodeBlock lang="json" code={`{
@@ -301,6 +302,7 @@ curl "${mgmtBase}/sessions/{session_id}/files?changed=true" \\
<Param name="additional_headers" type="array">Header NAMES your product passes per request for app-level auth (see Additional headers above).</Param>
<Param name="max_step" type="integer">Default agent step budget (default 400).</Param>
<Param name="timeout_seconds" type="integer">Default per-task wall-clock cap (default 1800).</Param>
+ <Param name="reasoning_effort" type="string">Default thinking level for this harness&apos;s tasks, one of the levels above. <code>GET /v1/bases</code> lists the levels each model has. Unset leaves each model at its own default.</Param>
</ParamTable>
<p className="hr-meta">Casing note: request bodies are snake_case (<code>default_model</code>,
<code> mcp_servers</code>, <code>max_step</code>); responses come back camelCase
diff --git a/ui/src/components/HarnessSettings.tsx b/ui/src/components/HarnessSettings.tsx
index c8815554..fa2959ce 100644
--- a/ui/src/components/HarnessSettings.tsx
+++ b/ui/src/components/HarnessSettings.tsx
@@ -29,6 +29,11 @@ const isOwnSkill = (s: Skill) => Boolean((s.files && s.files.length) || (s as {
/** Harness settings. `embedded`: rendered inside the Agent harnesses page, which owns the title
* row and the way back; navigation asks the host instead of pushing routes. */
+/** What each thinking level is called here. The levels themselves come from the server. */
+const THINKING_LABELS: Record<string, string> = {
+ none: 'Off', minimal: 'Minimal', low: 'Low', medium: 'Medium', high: 'High', xhigh: 'Extra high',
+};
+
export function HarnessSettings({ id, embedded = false, onNavigate }: {
id: string; embedded?: boolean; onNavigate?: (to: 'tasks' | 'list') => void;
}) {
@@ -200,6 +205,11 @@ export function HarnessSettings({ id, embedded = false, onNavigate }: {
const srvBase = bases?.[base?.id || draft?.base || ''] || null;
const baseTools = srvBase?.tools || [];
const models = oobModels(base);
+ // The thinking levels of the model this Harness runs on, as the server lists them for it.
+ const thinkingModel = oob ? oobDefaultModel(oob) : (draft?.defaultModel || oobDefaultModel(oobById(draft?.base || '')) || '');
+ const thinkingLevels = bases?.[base?.id || '']?.models.find((m) => m.id === thinkingModel)?.reasoning || [];
+ const thinkingStored = draft?.reasoningEffort || '';
+ const thinkingKept = Boolean(thinkingStored) && !thinkingLevels.includes(thinkingStored);
const upd = (p: Partial<CustomHarness>) => setDraft((d) => (d ? { ...d, ...p } : d));
async function save() {
@@ -319,6 +329,19 @@ export function HarnessSettings({ id, embedded = false, onNavigate }: {
))}
</select>
<span className="field-help">Tasks may choose another compatible model at runtime.</span></div>
+ {!oob && (thinkingLevels.length > 0 || thinkingStored) && (
+ <div className="field"><label htmlFor="hsThinking">Thinking</label>
+ <select id="hsThinking" disabled={readOnly} value={thinkingStored}
+ onChange={(e) => upd({ reasoningEffort: e.target.value })}>
+ <option value="">Model default</option>
+ {(thinkingKept ? [...thinkingLevels, thinkingStored] : thinkingLevels).map((lv) => (
+ <option key={lv} value={lv}>{THINKING_LABELS[lv] || lv}</option>
+ ))}
+ </select>
+ <span className="field-help">{thinkingKept
+ ? 'This model does not have that level. Its Tasks get the nearest one it has.'
+ : 'How much the model thinks before it answers. A Task may set its own.'}</span></div>
+ )}
</div>
</section>
diff --git a/ui/src/lib/harness.ts b/ui/src/lib/harness.ts
index 06fa1758..d0903670 100644
--- a/ui/src/lib/harness.ts
+++ b/ui/src/lib/harness.ts
@@ -48,6 +48,7 @@ export interface CustomHarness {
disabledTools?: string[]; // inherited/built-in tool names disabled for this harness
maxStep?: number | null; // default agent step budget per turn (blank = 40)
timeoutSeconds?: number | null; // default per-turn wall-clock cap (blank = server default)
+ reasoningEffort?: string; // how much the model thinks on this harness's Tasks ('' = the model's default)
additionalHeaders?: string[]; // declared header NAMES callers pass per request (app-level auth)
// Variables every Task's shell and tools start with. A value is `$headers.X-Name` (a declared
// request header), `vault:ref` (a stored secret) or a literal. Resolved by the service per turn.
@@ -313,7 +314,9 @@ export interface BuiltinSkill { name: string; title: string; description: string
export interface BaseInfo {
id: string; label: string; backend: string; status: string; systemPrompt: string;
defaultModel: string;
- models: { id: string; available: boolean; default: boolean }[];
+ /** `reasoning`: the thinking levels the model has, lowest first; empty or absent where it has
+ * none to offer. The control is drawn from this list and never from one kept here. */
+ models: { id: string; available: boolean; default: boolean; reasoning?: string[] }[];
tools: BaseTool[];
/** Skills bundled into the image, which any harness can use. `defaultEnabled` is what a NEW
* harness starts with; a harness that stored its own answer overrides it. */
@@ -453,6 +456,7 @@ function harnessBody(input: Partial<CustomHarness> & { name: string; base: strin
plugins: input.plugins || [],
max_step: input.maxStep || null,
timeout_seconds: input.timeoutSeconds || null,
+ reasoning_effort: input.reasoningEffort || null,
additional_headers: input.additionalHeaders || [],
env: input.env || {},
environment: input.environment || '',