Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
docs(routing): document model selection and routing checks (#998)
eric-liu-nvidia6f4e7749083a
diff --git a/README.md b/README.mdindex 67291e800..f1f64d5db 100644--- a/README.md+++ b/README.md@@ -35,7 +35,7 @@ LiteLLM and NeMo Relay run in your own deployment. Follow each gateway's guidanc### Try routing locally-[Try routing locally](docs/getting_started.md#server-path) with the standalone proxy for demos and evaluation.+[Try routing locally](docs/getting_started.md#server-path) with the standalone proxy for demos and evaluation, then [verify routing is active](docs/getting_started.md#verify-routing-is-active).Agent-specific guides are available for [pi](docs/integrations/pi.md) and[Oh My Pi](docs/integrations/oh_my_pi.md).diff --git a/docs/getting_started.md b/docs/getting_started.mdindex 41c1a0d23..7d5d91cff 100644--- a/docs/getting_started.md+++ b/docs/getting_started.md@@ -102,6 +102,9 @@ capable_target = "strong"efficient_target = "weak"```+The two targets are placeholders. See [Model Selection](routing_algorithms/model_selection.md)+to choose models for the capable and efficient tiers.+`format` selects the upstream protocol and must be `openai_chat`,`openai_responses`, or `anthropic_messages`. `api_key_env` names the environmentvariable the server reads; the secret does not belong in the TOML file.@@ -146,6 +149,100 @@ curl http://localhost:4000/v1/chat/completions \-d '{"model":"switchyard","messages":[{"role":"user","content":"hello"}]}'```+### Verify routing is active++The first request usually lands on the efficient model. That does not mean+routing is off. Auto decides from tool results, and a request without tool+history has none, so the efficient tier serves it by design. Use these checks+to see routing work. Your model IDs and token counts will differ from the+examples.++1. **Confirm the configuration loads.** The dry run prints the route IDs the+ server will serve:++ ```bash+ switchyard-server --config routes.toml --dry-run+ ```++ ```text+ server OK: switchyard+ ```++2. **Read the served model from the response header.** Every response carries+ `x-model-router-selected-model`. A chat-only request is served by the+ efficient model:++ ```bash+ curl -si http://localhost:4000/v1/chat/completions \+ -H "Content-Type: application/json" \+ -d '{"model":"switchyard","messages":[{"role":"user","content":"hello"}]}' \+ | grep -i x-model-router-selected-model+ ```++ ```text+ x-model-router-selected-model: openai/gpt-4o-mini+ ```++3. **Send a request that escalates.** A tool result that reports a critical+ error forces the capable tier. Save this request as `escalate.json`:++ ```json+ {+ "model": "switchyard",+ "messages": [+ {"role": "user", "content": "Run the test suite."},+ {"role": "assistant", "content": null, "tool_calls": [+ {"id": "call_1", "type": "function",+ "function": {"name": "bash", "arguments": "{\"command\":\"pytest -q\"}"}}+ ]},+ {"role": "tool", "tool_call_id": "call_1",+ "content": "MemoryError: cannot allocate memory\nKilled"},+ {"role": "user", "content": "Fix it."}+ ]+ }+ ```++ ```bash+ curl -si http://localhost:4000/v1/chat/completions \+ -H "Content-Type: application/json" -d @escalate.json \+ | grep -i x-model-router-selected-model+ ```++ ```text+ x-model-router-selected-model: openai/gpt-4o+ ```++ The same hard failure appearing twice in recent tool results, or a+ context-compaction marker, also forces the capable tier. See+ [decision sources](routing_algorithms/stage_router_routing.md#decision-sources).++4. **Count decisions by source.** `GET /v1/stats` reports calls per model and+ routing decisions by source:++ ```bash+ curl -s http://localhost:4000/v1/stats | jq -c '.models | map_values(.calls)'+ curl -s http://localhost:4000/v1/stats | jq -c '.algorithm_stats.stage_router.routing_decisions'+ ```++ ```json+ {"openai/gpt-4o":1,"openai/gpt-4o-mini":1}+ {"fall_open":{"total":1,"targets":{"openai/gpt-4o-mini":1}},"override":{"total":1,"targets":{"openai/gpt-4o":1}}}+ ```++ `fall_open` counts turns with no usable signals that landed on the picker's+ default tier. `override` counts turns a hard error forced to the capable+ tier. Counts reset when the server restarts or on `POST /v1/stats/reset`.++5. **Keep a record of every request.** Start the server with+ `--routing-log-file routing.jsonl` to append one JSON line per served request:++ ```json+ {"ts":"2026-10-15T06:16:22.643Z","route_id":"switchyard","algorithm":"stage_router","origin":null,"task":null,"trial_id":null,"session_id":null,"model":"openai/gpt-4o","tier":"","prompt_tokens":42,"cached_tokens":0,"cache_creation_tokens":0,"completion_tokens":5,"reasoning_tokens":0,"total_tokens":47}+ ```++ The record names the served model and its token counts. It does not record+ why that model was chosen; use `GET /v1/stats` for the decision counts.+### Routing algorithms#### Choose a route typediff --git a/docs/integrations/pi.md b/docs/integrations/pi.mdindex 8560fa2b5..75a2853b3 100644--- a/docs/integrations/pi.md+++ b/docs/integrations/pi.md@@ -68,7 +68,9 @@ pi -p --provider switchyard --model switchyard "List the files in this directory## Check the routingThe command `curl -s localhost:4000/v1/stats | jq '.models | map_values({calls, prompt_tokens})'`-lists each target model with its call count. Every response carries the header+lists each target model with its call count. For a step-by-step check with an+escalating request, see+[Verify routing is active](../getting_started.md#verify-routing-is-active). Every response carries the header`x-model-router-selected-model`. With `--routing-log-file PATH`, the server writes onerecord per call. In the record below, `session_id` is the value of pi's `x-session-id`header:diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.mdindex b3160a4c7..b70ba9531 100644--- a/docs/reference/toml_schema.md+++ b/docs/reference/toml_schema.md@@ -376,19 +376,44 @@ optional `handoff_notes` and `classifier` tables and for tuning.### `auto`-Uses Switchyard's recommended default routing strategy instead of one you pick-yourself: a `stage_router` preset with `picker = "efficient_first"` and-`confidence_threshold = 0.5`, no classifier. See-[Stage-Router Routing](../routing_algorithms/stage_router_routing.md) for a-deeper dive on the current default, or the-[strategy table](../routing_algorithms/overview.md#choose-a-strategy) to pick-one manually.+`auto` is a preset. It builds a `stage_router` route with fixed settings and no+classifier. Use it when you want the recommended starting point without tuning.+See [Auto](../routing_algorithms/overview.md#auto) for what the preset does and+when it does not route.| Key | Required | Default | Meaning ||---|:---:|---|---|| `capable_target` | Yes | — | Capable tier. || `efficient_target` | Yes | — | Efficient tier. |+`auto` accepts no other keys. Adding one is a configuration error. For example,+adding `confidence_threshold = 0.5` to an `auto` route fails `--dry-run` with:++```text+unknown field `confidence_threshold`, expected `capable_target` or `efficient_target`+```++An `auto` route behaves the same as this `stage_router` route. It has no+classifier, no handoff notes, and no per-tier system prompts:++```toml+[routes.smart]+id = "switchyard"+type = "stage_router"+capable_target = "strong"+efficient_target = "weak"+picker = "efficient_first"+confidence_threshold = 0.5+recent_turn_window = 3+capable_hold_turns = 2+```++To tune the preset, replace the `auto` route with this block and edit the keys.+The [stage_router](#stage_router) table lists every key. See+[Stage-Router Routing](../routing_algorithms/stage_router_routing.md) for+tuning guidance and the [model selection guide](../routing_algorithms/model_selection.md)+for choosing the two targets.+### `composite`Composes other algorithms, letting one set another'sdiff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.mdindex 994359a08..5a1209780 100644--- a/docs/routing_algorithms/escalation_router_routing.md+++ b/docs/routing_algorithms/escalation_router_routing.md@@ -252,6 +252,8 @@ retained nor logged.## Related- [Routing Overview](overview.md): compare all supported routing strategies.+- [Model Selection](model_selection.md): choose the weak, strong, and judge+ models, with measured pairs.- [LLM Classifier Routing](llm_classifier_routing.md): pick a tier up frontinstead of judging the run.- [Architecture](../architecture.md): the end-to-end request lifecycle and systemdiff --git a/docs/routing_algorithms/model_selection.md b/docs/routing_algorithms/model_selection.mdnew file mode 100644index 000000000..bb4c83d5c--- /dev/null+++ b/docs/routing_algorithms/model_selection.md@@ -0,0 +1,306 @@+# Model Selection++Switchyard routes each request to one of the models you configure. The models+you put in each role change results more than any threshold does. This page says+what each role needs, how to check a model before you route to it, which pairs+have measured results, and the mistakes we have seen in real runs.++## Roles++Each strategy names its models by role. The TOML keys differ by route type:++| Route `type` | Serves most turns | Takes the hard turns | Decides |+|---|---|---|---|+| `auto`, `stage_router`, `composite` | `efficient_target` | `capable_target` | Tool-result signals. `composite` adds a classifier. |+| `plan_execute` | `efficient_target`, after the first edit | `capable_target`, while planning | The first file edit. |+| `llm_classifier` (capability, escalation) | `weak_target` | `strong_target` | The judge at `classifier_target`. |+| `llm_classifier` (custom) | Your target names | Your target names | The judge in `models.judge`. |+| `advisor` | `executor_target` | `advisor_target` reviews. It does not serve turns. | The gate rules. |++A role is not a property of a model. The same model can be the capable tier in+one route and the efficient tier in another. See+[Core Concepts](../core_concepts.md#routing-algorithms).++What each role needs:++- **Capable, strong, or advisor.** Your best model on this workload. Every+ escalation lands here. Routing cannot solve a task this model cannot solve.+- **Efficient, weak, or executor.** Serves most turns. It must call the agent's+ tools correctly, follow the output format, hold the agent's context, and still+ complete tasks. Judge it by the tasks it completes on your workload, not by its+ price.+- **Judge or classifier.** Returns schema-valid JSON on every call, and does so+ quickly. See+ [Judge model compatibility](llm_classifier_routing.md#judge-model-compatibility).++## Check the efficient model before you route++Run these checks before you measure routing. Add a `passthrough` route for the+efficient model to the file from+[Getting Started](../getting_started.md#server-path), start the server with+`--routing-log-file routing.jsonl`, and send the checks to that route:++```toml+[routes.efficient_only]+id = "efficient-only"+type = "passthrough"+target = "weak"+```++### 1. Tool calling++Send a request with one tool and read the call back:++```bash+curl -s http://localhost:4000/v1/chat/completions \+ -H "Content-Type: application/json" \+ -d '{+ "model": "efficient-only",+ "messages": [{"role": "user", "content": "What is the weather in Paris? Use the tool."}],+ "tools": [{"type": "function", "function": {+ "name": "get_weather",+ "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}]+ }' | jq '.choices[0].message.tool_calls[0].function.arguments | fromjson'+```++Pass: the output is a JSON object with a `city` key. Fail: `null`, or a+`fromjson` error. Then run your agent against this route on one short task.+Models that pass with one tool can produce malformed calls once the agent sends+its full tool list. The agent reports those as invalid tool calls, or it repeats+the same step.++### 2. Structured output++Judges, and some agents, need JSON that matches a schema:++```bash+curl -s http://localhost:4000/v1/chat/completions \+ -H "Content-Type: application/json" \+ -d '{+ "model": "efficient-only",+ "messages": [{"role": "user", "content": "Is 7 a prime number?"}],+ "response_format": {"type": "json_schema", "json_schema": {"name": "verdict", "strict": true,+ "schema": {"type": "object", "properties": {"prime": {"type": "boolean"}},+ "required": ["prime"], "additionalProperties": false}}}+ }' | jq '.choices[0].message.content | fromjson'+```++Pass: `{"prime": true}`. An HTTP 400 means the provider does not offer JSON+Schema mode for this model. For a judge, set+`response_format_type = "json_object"` on the `llm_classifier` route. See+[Judge model compatibility](llm_classifier_routing.md#judge-model-compatibility).++### 3. Context window++The efficient model must hold the agent's largest prompt. Find that size after+a run through the passthrough route:++```bash+jq -s 'map(.prompt_tokens) | max' routing.jsonl+```++Compare it with the context length your provider lists for the model.+OpenRouter reports it as `context_length` in `GET https://openrouter.ai/api/v1/models`.+Set the route's `context_window` to the smaller of the two tiers' limits so+clients that read it compact in time. The server does not enforce it. See the+[route keys](../reference/toml_schema.md#routesname).++### 4. Reasoning++A model that reasons at length on every turn is slow and costly in the+efficient role. Read the reasoning share from the log:++```bash+jq -c '{model, completion_tokens, reasoning_tokens}' routing.jsonl+```++Providers that do not report reasoning tokens leave the field at `0`. When+reasoning tokens dominate on simple turns, set `reasoning_effort` on the target+for `openai_chat` and `openai_responses` clients, or use the provider's own+control through `extra_body`. See [Reasoning effort](#reasoning-effort).++### 5. Wire format++`format` on the LLM client must match the endpoint. `--dry-run` checks the file+only. It does not contact the provider. Send one request to the passthrough+route and confirm an HTTP 200 with a `choices` array. The same model can also+behave differently on different endpoints. See+[Pin the provider and endpoint](#pin-the-provider-and-endpoint).++## Measured pairs++These pairs have results from checked-in profiles or a public report. Each row+describes one pair on one workload. The thresholds and results do not transfer+to another pair or another workload. The profile headers give the serving stack,+agent version, and scoring rule for each run, and absolute results vary by+serving stack.++| Efficient tier | Capable tier | Strategy | Workload | Result | Single-model anchors |+|---|---|---|---|---|---|+| GPT-5.6 Luna | GPT-5.6 Sol | [`stage_router`](../../benchmark/routing-profiles/deepswe-v11-stage-router-luna-sol.toml): `efficient_first`, threshold 0.5, window 5, hold 2, handoff note | DeepSWE v1.1, 113 tasks, Codex CLI, k=1 | 76/113 solved (67.3% strict) | None in the same run |+| GPT-5.6 Luna, executes | GPT-5.6 Sol, plans | [`plan_execute`](../../benchmark/routing-profiles/deepswe-v11-plan-execute-luna-sol.toml), both at effort `max` | DeepSWE v1.1, 113 tasks, Codex CLI, k=3 | 60.8% ± 5.3 at $180.51 ± 9.43 per run | Sol alone 62.8% ± 3.1 at $720.95. Luna alone 51.0% ± 3.7 at $50.02. |+| GPT-5.6 Luna, executes | GPT-5.6 Sol, reviews | [`advisor`](../../benchmark/routing-profiles/deepswe-v11-advisor-gate-luna-sol.toml), both at effort `max` | DeepSWE v1.1, 113 tasks, Codex CLI, k=3 | 57.5% ± 3.2 at $208 ± 22 per run | Sol alone 62.8% ± 3.1 at $743. Luna alone 46.9% and 52.2% at about $49. |+| Qwen3.6-35B-A3B, also the judge | Claude Opus 4.7 | `llm_classifier` custom, [balanced](../../benchmark/routing-profiles/tau2-telecom-custom-opus-qwen-balanced.toml) and [aggressive](../../benchmark/routing-profiles/tau2-telecom-custom-opus-qwen-aggressive.toml) | tau2-bench telecom, 114 tasks | 0.903 ± 0.071 solve rate with 45% of turns on the weak tier. 0.891 ± 0.029 with about 85%. | Within noise of the strong tier, per the profile headers |+| Nemotron 3.5 Lightning | Claude Opus 4.8, with Gemini 3.1 Flash-Lite as judge | `llm_classifier` escalation, `confirmations = 2` | LangChain Deep Agents suite, 145 tasks, 5 runs ([report](https://www.langchain.com/blog/switchyard-agent-routing-benchmark)) | 80.0% at $3.00 per run, with 7% of calls on Opus | Opus alone 86.0% at $11.45. Nemotron alone 77.7% at $0.72. |++What the rows show about model choice:++- The gap between the two single models sets the room for routing. On DeepSWE,+ Sol alone beat Luna alone by about 11 to 16 points. The plan/execute and+ advisor arms landed between them at about a quarter of Sol's cost.+- When the efficient model is close to the capable model, escalation keeps most+ of the accuracy at a fraction of the cost. On the LangChain suite the two+ single models were 8 points apart. Routing kept 80 of the 86 points at about+ a quarter of the cost.+- The judge's cost counts. In the LangChain runs the judge took 21.2% of spend.+ Choose a judge that is small and fast, and include it in the cost per run.++Other checked-in configurations:++- [`tb-lite-llm-classifier-opus-kimi-gemini.toml`](../../benchmark/server-configs/tb-lite-llm-classifier-opus-kimi-gemini.toml)+ (Claude Opus 4.7 strong, Kimi K2.7 code weak, Gemini 3.5 Flash judge) is a+ smoke test. It has no published result.+- [`tb21-escalation-opus-glm-deepseek.toml`](../../benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml)+ (Claude Opus 4.8 strong, GLM 5.2 weak, DeepSeek V4 Flash judge) is the+ escalation configuration used for the v0.2.0 Terminal-Bench 2.1 runs. Its+ header links the post about those runs.++## Counter-examples++These cases came from users' runs. Each describes one pair on one workload.++- **The efficient model was too weak.** A first stage-router run scored below+ the single capable model. The efficient tier failed tasks the capable model+ solves, and the signals escalated too late to recover them. Check: run the+ [fixed arms](#check-for-an-inverted-pair) first. When the efficient-only arm+ is far below the capable-only arm, routing has to escalate a large share of+ turns to close the gap, and the savings shrink.+- **The pair was inverted.** On one coding benchmark the model in the efficient+ tier solved more tasks than the model in the capable tier. Every escalation+ then cost more and solved less. Check:+ [the inversion check](#check-for-an-inverted-pair).+- **The tool list broke the efficient model.** A 4-billion-parameter local model+ called tools correctly in a small test and produced malformed calls once the+ agent sent its full list of 27 tools. Check: [tool calling](#1-tool-calling)+ with the agent itself, not only with a one-tool request.+- **Switching cost more than it saved.** A pair matched the single capable+ model's accuracy but cost more. Each switch between models lost the prompt+ cache, so the next turn paid full price for the whole prompt, and the+ efficient model kept too little of the traffic to make up for it. Check:+ `cached_tokens` in the routing log on the turn after a model change, and cost+ per run for the routed arm against the fixed arms, not per-token prices.++## Check for an inverted pair++Symptom: the routed run scores below the efficient-only run, or `GET /v1/stats`+shows most turns on the capable tier while results do not improve.++Detection: run both models alone on the same tasks. Two `passthrough` routes in+one file do this:++```toml+[routes.capable_only]+id = "capable-only"+type = "passthrough"+target = "strong"++[routes.efficient_only]+id = "efficient-only"+type = "passthrough"+target = "weak"+```++Run your agent against each route ID on the same set of tasks, at least 20.+Count solved tasks per arm. Read token use per arm from the routing log:++```bash+jq -s 'group_by(.route_id) | map({route: .[0].route_id, calls: length,+ prompt_tokens: (map(.prompt_tokens) | add),+ completion_tokens: (map(.completion_tokens) | add)})' routing.jsonl+```++Multiply the totals by each model's prices to get cost per arm. The pi+integration page shows [one way to do that](../integrations/pi.md#estimate-the-cost).++Fix: if the efficient-only arm solves at least as many tasks as the capable-only+arm, swap the two roles, or serve the efficient model alone with `passthrough`.+If both arms solve the same tasks, routing has little to gain on this workload.+The room for routing is the set of tasks only the capable model solves.++## Pin the provider and endpoint++A model ID does not fix how the model is served. Three things move results+between runs.++**Provider.** Through an aggregator, one model ID can be served by several+providers with different quantization, context length, and tool-call support.+Symptom: the same configuration gives different results on different days, and+the routing log's `model` field does not change. Detection: the aggregator's+response metadata or dashboard shows which provider served each call. Fix: pin+the provider in the target's `extra_body`. On OpenRouter, `order` names the+providers to try and `allow_fallbacks = false` stops it from trying others:++```toml+[targets.weak]+id = "qwen/qwen3-35b-a3b"+llm_client = "openrouter"+extra_body = { provider = { order = ["your-provider"], allow_fallbacks = false } }+```++Use the provider slug from the model's page on OpenRouter.++**Endpoint and wire format.** The same model can behave differently on+`openai_chat`, `openai_responses`, and `anthropic_messages`. One measured case+is Claude behind an LLM gateway. On `anthropic_messages` the thinking level+worked and the prompt cache was read. On the two OpenAI formats a thinking level+returned HTTP 400, and `openai_responses` never read the cache. See+[Claude through an LLM gateway](../integrations/pi.md#claude-through-an-llm-gateway).+Record `format` and `base_url` with every result.++**Streamed token counts on `/v1/messages`.** When a client uses Anthropic+Messages and the selected target is on an OpenAI format, the streamed+`message_start` event reports `input_tokens: 0`. The counts the upstream+reports arrive in the final `message_delta` event. Symptom: the client shows zero prompt tokens, or a+context meter that never moves, for turns served by those targets. Detection:+compare the client's numbers with `prompt_tokens` in the routing log, which+records the upstream's counts. Fix: read usage from `message_delta`, or use the+routing log and `GET /v1/stats` for accounting.++### Reasoning effort++Effort changes a model's results and cost as much as the model choice does.+Record it with every result. Three rules apply:++- On `openai_responses` the effort field is `reasoning.effort`. A top-level+ `reasoning_effort` key in `extra_body` does not set it. Use the target's+ `reasoning_effort` key instead. It writes `reasoning.effort` on+ `openai_responses` and `reasoning_effort` on `openai_chat`, replacing the value+ the client sent. The DeepSWE profiles above use it to run both tiers at `max`.+- `reasoning_effort` is rejected on `anthropic_messages` clients. The server does+ not start:++ ```text+ target weak reasoning_effort is only supported on openai_chat and openai_responses clients+ ```++ For those targets the effort the client sends is passed through in+ Anthropic's form.++- One route cannot hold the same model at two efforts. Two targets with the same+ model ID on one client but different `reasoning_effort` or `extra_body` fail+ to load:++ ```text+ targets strong and strong_max both name model openai/gpt-4o on llm client openrouter but with different reasoning_effort, extra_body, or omit_body_fields; one target per model id is kept, so give each its own model id or llm client+ ```++ Separate routes can use separate `llm_clients` entries for the same+ endpoint. See the [target keys](../reference/toml_schema.md#targetsname).++## Related++- [Routing Overview](overview.md): choose a strategy and read what Auto resolves to.+- [Verify routing is active](../getting_started.md#verify-routing-is-active): see the served model and the decision counts.+- [Stage Router](stage_router_routing.md#tuning-confidence_threshold): tune the threshold once the pair is right.+- [Escalation-Router Routing](escalation_router_routing.md#judge-model-compatibility): judge requirements for escalation routes.+- [Benchmarks](../../benchmark/README.md): how the measured runs were made.diff --git a/docs/routing_algorithms/overview.md b/docs/routing_algorithms/overview.mdindex 3878d7d3c..372de7db4 100644--- a/docs/routing_algorithms/overview.md+++ b/docs/routing_algorithms/overview.md@@ -24,15 +24,57 @@ controls whether the judge runs for each request, each user turn, or once persession. Execution uses the stage router, which reads tool-result history andcan optionally call an LLM judge. The TOML configuration keys are unchanged.+Whichever strategy you choose, the two models you put behind it matter more than+any threshold. See [Model Selection](model_selection.md).+### Auto-Auto is a preset, not a separate routing algorithm. It currently uses Execution-with `picker = "efficient_first"`, `confidence_threshold = 0.5`, and no LLM-judge. It does not compare strategies at runtime. Use `stage_router` directly-to tune these settings.+Auto is a preset, not a separate routing algorithm. An `auto` route builds a+stage router with these settings. It does not compare strategies at runtime.-See the [Auto configuration reference](../reference/toml_schema.md#auto) for-the required targets.+| Setting | Value | What it does |+|---|---|---|+| `picker` | `efficient_first` | The efficient tier serves a turn when the tool signals are not confident. |+| `confidence_threshold` | `0.5` | How strongly the tool signals must agree before they pick a tier on their own. |+| `recent_turn_window` | `3` | How many recent tool results the signals are computed over. |+| `capable_hold_turns` | `2` | How many turns stay on the capable tier after an escalation. A clean test pass ends the hold early. |+| Classifier | none | No LLM judge is called. |++Auto accepts no tuning keys. To change a setting, write the equivalent+`stage_router` route shown in the [TOML schema](../reference/toml_schema.md#auto)+and edit it there. Choose the two models with the+[model selection guide](model_selection.md).++Only the `efficient_first` picker has published results. The `capable_first`+picker is [experimental](stage_router_routing.md#pickers).++#### When Auto does not route++Auto decides from tool results in the request. In these cases every request goes+to one tier:++- **No tool results in the request.** Chat-only requests, and the first request+ of an agent session, carry no tool history. The signals abstain and the+ efficient tier serves the turn. This is by design. `GET /v1/stats` counts these+ turns under `algorithm_stats.stage_router.routing_decisions.fall_open`. See+ [Verify routing is active](../getting_started.md#verify-routing-is-active).+- **Both tiers name the same target.** The server accepts a route whose+ `capable_target` and `efficient_target` are the same target. Every request then+ goes to that one model, and `GET /v1/stats` lists a single entry under `.models`.+- **No session ID on the request.** The capable hold is kept per session. A+ request without a session ID gets fresh state, so a hold set by one request is+ gone on the next. Requests with a session ID but no agent ID share one hold+ within that session.++A critical error, a repeated identical failure, or a context-compaction marker in+the tool history still sends the turn to the capable tier. See+[decision sources](stage_router_routing.md#decision-sources).++Auto uses the settings of the published stage-router results, with two+differences: those runs set `recent_turn_window = 5` and added a handoff note.+Every published result comes from a coding agent whose requests carry tool+results. The [model selection guide](model_selection.md#measured-pairs) lists the+measured model pairs.## More optionsdiff --git a/docs/routing_algorithms/stage_router_routing.md b/docs/routing_algorithms/stage_router_routing.mdindex 5f71e322c..366ee9932 100644--- a/docs/routing_algorithms/stage_router_routing.md+++ b/docs/routing_algorithms/stage_router_routing.md@@ -336,6 +336,8 @@ paths through its cascade:## Related+- [Model Selection](model_selection.md): choose and check the two models before+ tuning the threshold.- [Architecture](../architecture.md): the end-to-end request lifecycle andsystem boundaries.- [TOML Schema](../reference/toml_schema.md#stage_router): the completediff --git a/mkdocs.yml b/mkdocs.ymlindex a291889d3..6c5309ffc 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -31,6 +31,7 @@ nav:- Single-provider coding agents: recipes/single_provider_coding_agents.md- Routing:- Overview: routing_algorithms/overview.md+ - Model Selection: routing_algorithms/model_selection.md- Task (LLM Classifier): routing_algorithms/llm_classifier_routing.md- Execution (Stage Router): routing_algorithms/stage_router_routing.md- Plan/Execute Routing: routing_algorithms/plan_execute_routing.mddiff --git a/tests/getting_started/test_getting_started.py b/tests/getting_started/test_getting_started.pyindex 1b8fe89f5..fb428324d 100644--- a/tests/getting_started/test_getting_started.py+++ b/tests/getting_started/test_getting_started.py@@ -16,6 +16,14 @@ def test_getting_started_documents_current_paths() -> None:assert "nemo-switchyard[cli]" not in guideassert "cargo install --locked switchyard-server" in guideassert "switchyard-server --config routes.toml --dry-run" in guide+ assert (+ guide.index("### Run the server")+ < guide.index("### Verify routing is active")+ < guide.index("### Routing algorithms")+ )+ assert "x-model-router-selected-model" in guide+ assert "/v1/stats" in guide+ assert "--routing-log-file" in guidefor legacy_command in ("switchyard configure",