Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

docs(nav): surface the tuning guides and trim Get Started (#1211)

eric-liu-nvidiaae44c600a5c0
diff --git a/docs/getting_started.md b/docs/getting_started.md
index 571170d6f..01580f584 100644
--- a/docs/getting_started.md
+++ b/docs/getting_started.md
@@ -264,6 +264,7 @@ Your client returns `switchyard_protocol::Response` and adapts errors to `libsy:
- **Explore the Rust API:** [libsy crate](../crates/libsy/README.md) and [generated API reference](reference/rust_api.md).
- **Use a gateway integration:** [LiteLLM](../examples/litellm/README.md) or [NeMo Relay](integrations/nemo_relay.md). Check each guide's supported algorithms and versions.
- **Choose another algorithm:** [routing overview](routing_algorithms/overview.md).
+- **Tune your configuration:** [Tuning](tuning/index.md) explains how to define an objective, compare candidate configurations on calibration tasks, and validate the selected one on held-out tasks.
- **Try the standalone proxy:** [local evaluation guide](operations/local_evaluation.md). The server is for demos, evaluation, and personal use, not a supported production path.
- **Use Python:** [Python bindings](reference/host_integration.md#python-bindings).
- **Use an agent directly:** [Claude Code example](../examples/claude/README.md), [pi](integrations/pi.md), or [Oh My Pi](integrations/oh_my_pi.md).
diff --git a/docs/index.md b/docs/index.md
index c714b2b01..19dc47041 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -29,10 +29,10 @@ Use it with [Claude Code or Codex](recipes/single_provider_coding_agents.md),
- [Mechanics of Libsy](mechanics_of_libsy.md): understand steps, calls, and outcomes
- [Switchyard Protocols](protocols/index.md): connect your model data to libsy
- [Routing Algorithms](routing_algorithms/overview.md): choose how requests select a model
+- [Tuning](tuning/index.md): find a configuration that meets your objective on your workload
- [Host Integration](reference/host_integration.md): connect your clients and application lifecycle
- [Rust API](reference/rust_api.md): browse the core library, protocols, translation, clients, and runner
- [Context-Window Handling](operations/context_window.md): configure eviction and retry behavior
-- [Tuning](tuning/index.md): foundations and worked configuration comparisons
- [Troubleshooting](operations/troubleshooting.md): symptoms, silent misconfigurations, and fixes
## Reference
diff --git a/docs/operations/local_evaluation.md b/docs/operations/local_evaluation.md
index ad3bbc1a3..57fc606d6 100644
--- a/docs/operations/local_evaluation.md
+++ b/docs/operations/local_evaluation.md
@@ -152,6 +152,7 @@ With an unreleased build from main, `GET /v1/routing/session-stats?session_id=..
- [Benchmark results](../benchmarks.md) link to model pools, results, and runnable profiles. For Terminal-Bench 2.1, the [escalation profile](../../benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml) includes its judge and reasoning settings. Its comments explain the endpoint differences from the measured runs.
- [Benchmark setup](../../benchmark/README.md) explains how to compare routing against single-model baselines on the same tasks.
+- [Tuning](../tuning/index.md) explains how to define an objective, compare candidate configurations on calibration tasks, and validate the selected one on held-out tasks.
- [Routing overview](../routing_algorithms/overview.md) explains Auto, Task, Execution, and Composite. A TOML file can expose multiple routes through different route IDs.
- [Server reference](../../crates/switchyard-server/README.md) covers configuration, TLS, logging, and metrics.
diff --git a/docs/operations/troubleshooting.md b/docs/operations/troubleshooting.md
index 4a912d469..b42eb0a49 100644
--- a/docs/operations/troubleshooting.md
+++ b/docs/operations/troubleshooting.md
@@ -11,6 +11,7 @@ log field, a log line, or a `--dry-run` message.
|---|---|---|---|
| The first request lands on the efficient model | `x-model-router-selected-model` on the response | Auto and Execution decide from tool results, and a request without tool history has none. This is by design. | Send a request with a critical tool error, as in [Verify routing is active](local_evaluation.md#verify-routing-is-active). |
| Every request lands on one model | `curl -s localhost:4000/v1/stats | jq -c '.models | map_values(.calls)'` and `.algorithm_stats.stage_router.routing_decisions` | Chat-only traffic (`fall_open` only); both tiers name the same target or the same model ID; `capable_first` with a threshold the signals cannot reach; one tier is down and the other serves its fallbacks. | See the matching rows in [Silent misconfigurations](#silent-misconfigurations) and [When Auto does not route](../routing_algorithms/overview.md#when-auto-does-not-route). |
+| Tasks fail on the efficient model and the route rarely escalates | `.models` call counts and `routing_decisions` in `/v1/stats` | The threshold is too high for the signals this workload produces; the capable hold is short; the efficient model is too weak for the tasks. | [Stage-router controls](../tuning/examples.md#choose-the-stage-router-controls-to-tune). |
| The judge never changes the outcome | `switchyard_classifier_fail_open_total` on `/metrics`, and the log line `judge verdict unavailable; routing without one` with its `reason` | The provider rejects JSON Schema mode (`json_object` needed); the verdict is truncated by `max_output_tokens`; the judge model returns prose instead of JSON. | [Judge model compatibility](../routing_algorithms/llm_classifier_routing.md#judge-model-compatibility). |
| Escalation never latches | The log line `stateful escalation has no session ID; routing state will not persist` | The client sends no session ID. No escalation state survives a request without one, so the route never latches at any `confirmations`. The warning fires only when `confirmations` is 2 or more. | Send `x-switchyard-session-id`. See [Tuning options](../routing_algorithms/escalation_router_routing.md#tuning-options). |
| Requests hang or take much longer than the model needs | Server log for retries and cooldown; `/v1/stats` call counts per model | A judge or tier without `timeout_ms`; a judge call on every tool step; a dead tier retried before fallback. | Give the judge its own `[llm_clients]` entry with `timeout_ms`. On a Task or Custom `llm_classifier` route, set `classify_trigger = "user_turn"` and send a session ID. See the dead-tier row below. |
@@ -20,7 +21,7 @@ log field, a log line, or a `--dry-run` message.
| The server does not start: `unknown field`, or `have no effect on the stage fallback judge` | The `--dry-run` message | An `auto` route with a tuning key; a typo; a key from another route type; `classifier.classify_trigger` other than the default, or `classifier.message_hash_fallback`, on a `stage_router` route. | [Auto](../reference/toml_schema.md#auto), the [stage_router keys](../reference/toml_schema.md#stage_router), and the [validation errors](../reference/toml_schema.md#validation-errors). |
| The server does not start: two targets `both name model` | The `--dry-run` message | Two targets share a model ID on one client with different `reasoning_effort`, `extra_body`, or `omit_body_fields`. | [Reasoning effort](../routing_algorithms/model_selection.md#reasoning-effort). |
| The client shows zero input tokens on `/v1/messages` | `prompt_tokens` in the routing log | The streamed `message_start` event reports zero usage when the target is on an OpenAI format. The real counts arrive in `message_delta`. | [Streamed token counts](../routing_algorithms/model_selection.md#pin-the-provider-and-endpoint). |
-| Cost per run is higher than the single capable model | `cached_tokens` in the routing log on the turn after a model change; cost per arm | Each switch loses the prompt cache; the judge adds a call per turn; the efficient model is too weak and most turns escalate. | [Counter-examples](../routing_algorithms/model_selection.md#counter-examples) and the [inverted-pair check](../routing_algorithms/model_selection.md#check-for-an-inverted-pair). |
+| Cost per run is higher than the single capable model | `cached_tokens` in the routing log on the turn after a model change; cost per arm | Each switch loses the prompt cache; the judge adds a call per turn; the efficient model is too weak and most turns escalate. | [Counter-examples](../routing_algorithms/model_selection.md#counter-examples) and the [inverted-pair check](../routing_algorithms/model_selection.md#check-for-an-inverted-pair). For the threshold and hold controls, see [Stage-router controls](../tuning/examples.md#choose-the-stage-router-controls-to-tune). |
| The prompt cache is never read | `cached_tokens` stays `0` on repeated prompts | The gateway does not cache on that endpoint, or the prompt is too short to cache. | [Prompt caching](../integrations/pi.md#prompt-caching). |
| `/v1/stats` has no `tiers` key | The `/v1/stats` body | The Rust server reports decisions under `algorithm_stats.<algorithm>`. | Read `algorithm_stats.stage_router.routing_decisions`, as in [Verify routing is active](local_evaluation.md#verify-routing-is-active). |
| The routing log does not say why a model was chosen | A routing-log record | The log records the served model and token counts only. | Use the decision counts in `/v1/stats`. For overrides, the `stage router override` log line also names the reason. See [Decision sources](../routing_algorithms/stage_router_routing.md#decision-sources). |
diff --git a/docs/routing_algorithms/advisor_gate_routing.md b/docs/routing_algorithms/advisor_gate_routing.md
index 6c78ad7e1..9be64c76e 100644
--- a/docs/routing_algorithms/advisor_gate_routing.md
+++ b/docs/routing_algorithms/advisor_gate_routing.md
@@ -132,3 +132,5 @@ already frontier-class, benchmark
[stage-router routing](stage_router_routing.md) first; reach for the advisor
gate when the executor is markedly weaker than the best model you can call, or
when you need a review trail rather than a model swap.
+
+To compare candidate values on your workload, see [Tuning](../tuning/index.md).
diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.md
index 6a3eecfdf..490c6cd64 100644
--- a/docs/routing_algorithms/escalation_router_routing.md
+++ b/docs/routing_algorithms/escalation_router_routing.md
@@ -259,3 +259,4 @@ retained nor logged.
- [LLM Classifier Routing](llm_classifier_routing.md): pick a tier up front
instead of judging the run.
- [Mechanics of Libsy](../mechanics_of_libsy.md): the execution model and host responsibilities.
+- [Tuning](../tuning/index.md): check your current route and compare candidate settings on your workload.
diff --git a/docs/routing_algorithms/llm_classifier_routing.md b/docs/routing_algorithms/llm_classifier_routing.md
index 25b86b3b4..9b77ba8ac 100644
--- a/docs/routing_algorithms/llm_classifier_routing.md
+++ b/docs/routing_algorithms/llm_classifier_routing.md
@@ -133,6 +133,8 @@ for the server merge behavior.
| `max_output_tokens` | `4096` | Maximum completion tokens available to the classifier verdict. Must be at least `1`. |
| `judge_deadline_ms` | unset | Bounds the whole judge consultation in milliseconds, the model call and the response drain included. On expiry the judge is treated as unavailable and follows `fail_open`: the route either falls back to the capable tier (default) or stops with a 504. Must be at least `1`. Unset leaves the consultation unbounded. |
+To compare candidate values on your workload, see [Tuning](../tuning/index.md).
+
### Override the classifier prompt
Set `prompt` on the route when the packaged capability rubric does not describe
diff --git a/docs/routing_algorithms/overview.md b/docs/routing_algorithms/overview.md
index bbda63f3e..24dc89766 100644
--- a/docs/routing_algorithms/overview.md
+++ b/docs/routing_algorithms/overview.md
@@ -160,7 +160,9 @@ stage router with these settings. It does not compare strategies at runtime.
Auto accepts no tuning keys. To change a setting, write the equivalent
`stage_router` route shown in the [TOML schema](../reference/toml_schema.md#auto)
and edit it there. Choose the two models with the
-[model selection guide](model_selection.md).
+[model selection guide](model_selection.md). To tune the rewritten route, start
+with [Before you start](../tuning/foundations.md#before-you-start) in Tuning
+Foundations.
Only the `efficient_first` picker has published results. The `capable_first`
picker is [experimental](stage_router_routing.md#pickers).
diff --git a/docs/routing_algorithms/stage_router_routing.md b/docs/routing_algorithms/stage_router_routing.md
index f18dfce9a..652e6b34b 100644
--- a/docs/routing_algorithms/stage_router_routing.md
+++ b/docs/routing_algorithms/stage_router_routing.md
@@ -454,3 +454,4 @@ the `stage router override` log line carries it for overrides:
- [Mechanics of Libsy](../mechanics_of_libsy.md): the execution model and host responsibilities.
- [TOML Schema](../reference/toml_schema.md#stage_router): the complete
configuration reference.
+- [Tuning](../tuning/index.md): check your current route and compare candidate settings on your workload.
diff --git a/docs/tuning/examples.md b/docs/tuning/examples.md
index 4fc030a2b..bb1a635ff 100644
--- a/docs/tuning/examples.md
+++ b/docs/tuning/examples.md
@@ -123,6 +123,45 @@ The frontier describes the calibration sample. Repeat promising runs, then
freeze the selection. Run E and all three baselines on the untouched 400-task
validation split to check whether the cost advantage and success target hold.
+## Check the efficient model when overrides dominate
+
+This example keeps the objective and the route from the cost comparison. The
+capable model has most of the calls in `.models`, and the first instinct is to
+raise `confidence_threshold`. Read `routing_decisions` in `/v1/stats` first,
+after a reset and a run of the calibration tasks, as in
+[Before you start](foundations.md#before-you-start).
+
+When `override` is a large share of the counts, the threshold is the wrong
+control. Each `override` means a repeated failure, a critical error, or a
+context-compaction marker in the recent tool results forced the capable tier.
+These decisions skip the score, so `confidence_threshold` and the holds do not
+change them. Raising the threshold moves only the `dimensions` decisions and
+leaves the `override` share where it is.
+
+The `stage router override` log line names the cause in `override_reason`:
+`repeated_failure`, `critical_error`, or `compaction`. Count them. `compaction`
+comes from a context-compaction marker in the history, so leave it out of this
+comparison. Repeated failures and critical errors point at the efficient model:
+
+1. **Measure the efficient model alone.** Add the `passthrough` route from
+ [Check the efficient model before you route](../routing_algorithms/model_selection.md#check-the-efficient-model-before-you-route)
+ and run the same calibration tasks. Compare it with the capable-only
+ baseline. When the efficient-only arm is far below the capable-only arm,
+ routing has to escalate a large share of turns to close the gap, and the
+ savings shrink.
+2. **Run the compatibility checks on that page.** Tool calling, structured
+ output, context window, reasoning, and wire format. A model that passes a
+ one-tool test can still produce malformed calls with the agent's full tool
+ list.
+3. **Change the efficient model.** Compare pairs as in
+ [Model-pair selection](#model-pair-selection-with-fixed-algorithm-settings).
+ If no cheaper model clears the tasks, the capable share reflects the tasks,
+ and the cost objective needs a new target.
+
+After a change, reset the counters, rerun the calibration tasks, and read the
+`override` share again before you compare cost. Then validate on the held-out
+split as in [final validation](foundations.md#4-check-the-final-choice).
+
## Model-pair selection with fixed algorithm settings
This independent example fixes the algorithm settings and changes the
diff --git a/docs/tuning/foundations.md b/docs/tuning/foundations.md
index 9cdc938ce..0de213ffe 100644
--- a/docs/tuning/foundations.md
+++ b/docs/tuning/foundations.md
@@ -8,6 +8,47 @@ before adopting it as your baseline.
See [Examples](examples.md) for configuration and model-pair comparisons
with concrete data.
+## Before you start
+
+Most routes begin as `type = "auto"`. Auto is a stage-router preset:
+`efficient_first`, `confidence_threshold = 0.5`, `recent_turn_window = 3`,
+`capable_hold_turns = 2`, and no classifier. It accepts no tuning keys. To
+change a setting, rewrite the route as the equivalent `stage_router` route from
+the [TOML schema](../reference/toml_schema.md#auto). Record the starting values;
+they are the current-routing baseline in step 2. The overview's
+[Start here](../routing_algorithms/overview.md#start-here) list names the first
+dial to move for each strategy.
+
+Then read the decision counts, because three results mean no setting will help.
+The counters keep every request since the server started, so clear them first
+and run a sample of the workload you want to tune:
+
+```bash
+curl -s -X POST localhost:4000/v1/stats/reset
+# Run the workload sample, then read the counts.
+curl -s localhost:4000/v1/stats | jq '.algorithm_stats.stage_router.routing_decisions, (.models | map_values(.calls))'
+```
+
+- **`fall_open` is most of the total.** The tool-signal scorer abstained, which
+ happens on chat-only traffic and on the first request of a session. The
+ threshold and holds do not affect these turns. Add the
+ [classifier fallback](../routing_algorithms/stage_router_routing.md#optional-llm-classifier-fallback),
+ or use Task or Composite routing, which judge the request itself.
+- **`.models` lists one entry.** Every sampled request went to one model. Check
+ `capable_target` and `efficient_target` in the route. If they name the same
+ target, fix them, then run the
+ [inverted-pair check](../routing_algorithms/model_selection.md#check-for-an-inverted-pair).
+ If they differ, the decision counts show which decision sent every request to
+ that tier.
+- **Requests carry no session ID.** Holds and escalation state live in the
+ session. Send `x-switchyard-session-id` before tuning `capable_hold_turns` on
+ a stage-router route or `confirmations` on an escalation route.
+
+[Decision sources](../routing_algorithms/stage_router_routing.md#decision-sources)
+explains each count, and
+[When routing will not help](../routing_algorithms/overview.md#when-routing-will-not-help)
+lists the remaining cases.
+
## 1. Define the objective
Define what the user wants to improve, on which workload, and what makes a result
@@ -43,6 +84,16 @@ a baseline and a small set of changes you expect to help:
4. **Keep runs comparable.** Use the same tasks, evaluation procedure, and initial
cache setup. Pin model and provider versions, and save each configuration.
+Collect these for every run:
+
+| Measure | Source |
+|---|---|
+| Routing split and decision reasons | `GET /v1/stats`: `.models` call counts and `routing_decisions`. `POST /v1/stats/reset` clears them between runs. |
+| Cost | The routing log (`--routing-log-file routing.jsonl`): the selected model and token usage per request, including `cached_tokens`. |
+| Served model per response | The `x-model-router-selected-model` response header. |
+| Metrics in your own host | [OpenTelemetry](../reference/opentelemetry.md#metrics) routing and client metrics. |
+| Task success | Your agent harness. For Terminal-Bench and DeepSWE, the [benchmark setup](../../benchmark/README.md#run-with-switchyard-routing) runs a profile and [inspects the result](../../benchmark/README.md#inspect-a-run). |
+
Run the same tasks for every candidate. The following examples show how to use
the results to choose between them.
@@ -54,6 +105,9 @@ Start with the comparison that matches what you want to change:
keep the algorithm and model pair fixed, vary routing settings, and find the
lowest-cost option that meets the task-success target. A Pareto chart shows
the cost and success tradeoffs.
+- **[Check the efficient model when overrides dominate](examples.md#check-the-efficient-model-when-overrides-dominate):**
+ when `override` is most of the decision counts, the threshold and holds cannot
+ change those turns; measure the efficient model alone and change it if needed.
- **[Select a model pair](examples.md#model-pair-selection-with-fixed-algorithm-settings):**
keep algorithm settings fixed, swap the efficient/capable models, and compare
execution cost plus a penalty for failed tasks.
diff --git a/docs/tuning/index.md b/docs/tuning/index.md
index 89b4b62a2..f0db9f4e8 100644
--- a/docs/tuning/index.md
+++ b/docs/tuning/index.md
@@ -1,14 +1,27 @@
# Tuning
Tuning helps you find a configuration that meets your goals on your workload.
-Start with Foundations to plan the comparison, then use Examples to
-see how to evaluate candidates and choose a configuration.
+Start with Foundations to check your current route and plan the comparison, then
+use Examples to see how to evaluate candidates and choose a configuration.
-- [Tuning Foundations](foundations.md): define your objective, candidates, and evaluation splits.
-- [Examples](examples.md): compare routing settings and model pairs with worked examples.
+- [Tuning Foundations](foundations.md): check your current route, then define your objective, candidates, and evaluation splits.
+- [Examples](examples.md): compare routing settings and model pairs with worked examples, and check the efficient model when overrides dominate.
+
+## Controls by algorithm
+
+Examples walks through the stage router. Each algorithm has its own controls:
+
+- [Stage router](../routing_algorithms/stage_router_routing.md#knob-reference): every route key, [how `confidence_threshold` works](../routing_algorithms/stage_router_routing.md#tuning-confidence_threshold), and [how to calibrate it from run data](../routing_algorithms/stage_router_routing.md#calibrating-the-threshold-from-run-data).
+- [LLM classifier](../routing_algorithms/llm_classifier_routing.md#tuning-options): thresholds, judge context, and when the judge runs.
+- [Escalation router](../routing_algorithms/escalation_router_routing.md#tuning-options): confirmations and judge window.
+- [Advisor gate](../routing_algorithms/advisor_gate_routing.md#tuning): review count and stall limits, with the benchmarked settings.
+- [Composite](../routing_algorithms/composite_routing.md#independent-stage-profiles): separate stage-router settings per profile.
+- [Plan/Execute](../routing_algorithms/plan_execute_routing.md#optional-settings): handoff triggers and prompts.
+- [Verification-gated](../routing_algorithms/vgr_routing.md#serving-modes): serving mode and [execution limits](../routing_algorithms/vgr_routing.md#execution-limits).
+- [Random](../routing_algorithms/random_routing.md#control-the-traffic-split): target weights.
## Related
-- [Routing Overview](../routing_algorithms/overview.md): choose a strategy.
+- [Start here](../routing_algorithms/overview.md#start-here): a strategy, a profile to copy, and the first dial to move.
- [Model Selection](../routing_algorithms/model_selection.md): choose and check the models.
- [Troubleshooting](../operations/troubleshooting.md): symptoms and silent misconfigurations.
diff --git a/mkdocs.yml b/mkdocs.yml
index d50e697c9..cf3b91b56 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -19,13 +19,12 @@ nav:
- Home: index.md
- Get Started:
- Getting Started: getting_started.md
+ - Integrate Switchyard:
- Mechanics of Libsy: mechanics_of_libsy.md
- Switchyard Protocols:
- - Overview: protocols/index.md
+ - protocols/index.md
- LLM-Related Protocols: protocols/llm.md
- Decision Model-Related Protocols: protocols/decisions.md
- - Known Issues: known_issues.md
- - Integrate Switchyard:
- Embed in Your Gateway or Harness: reference/host_integration.md
- Gateways:
- NeMo Relay: integrations/nemo_relay.md
@@ -42,11 +41,12 @@ nav:
- Advisor-Gate Routing: routing_algorithms/advisor_gate_routing.md
- Verification-Gated Routing: routing_algorithms/vgr_routing.md
- Choose Models: routing_algorithms/model_selection.md
- - Evaluate Routing:
+ - Evaluate and Tune:
- Benchmarks: benchmarks.md
- - Tuning: tuning/index.md
- - Tuning Foundations: tuning/foundations.md
- - Examples: tuning/examples.md
+ - Tuning:
+ - tuning/index.md
+ - Foundations: tuning/foundations.md
+ - Examples: tuning/examples.md
- Local Demo:
- Run the Reference Server: operations/local_evaluation.md
- Local Agents:
@@ -68,6 +68,7 @@ nav:
- Rust API: reference/rust_api.md
- Context-Window Handling: operations/context_window.md
- Troubleshooting: operations/troubleshooting.md
+ - Known Issues: known_issues.md
- Demo Server:
- CLI: cli_reference.md
- Soak Testing: operations/soak_test.md
@@ -100,6 +101,7 @@ theme:
features:
- navigation.top
- navigation.sections
+ - navigation.indexes
- search.suggest
- search.highlight
- content.code.copy