Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
docs(nav): surface the tuning guides and trim Get Started (#1211)
eric-liu-nvidiaae44c600a5c0
diff --git a/docs/getting_started.md b/docs/getting_started.mdindex 571170d6f..01580f584 100644--- a/docs/getting_started.md+++ b/docs/getting_started.md@@ -264,6 +264,7 @@ Your client returns `switchyard_protocol::Response` and adapts errors to `libsy:- **Explore the Rust API:** [libsy crate](../crates/libsy/README.md) and [generated API reference](reference/rust_api.md).- **Use a gateway integration:** [LiteLLM](../examples/litellm/README.md) or [NeMo Relay](integrations/nemo_relay.md). Check each guide's supported algorithms and versions.- **Choose another algorithm:** [routing overview](routing_algorithms/overview.md).+- **Tune your configuration:** [Tuning](tuning/index.md) explains how to define an objective, compare candidate configurations on calibration tasks, and validate the selected one on held-out tasks.- **Try the standalone proxy:** [local evaluation guide](operations/local_evaluation.md). The server is for demos, evaluation, and personal use, not a supported production path.- **Use Python:** [Python bindings](reference/host_integration.md#python-bindings).- **Use an agent directly:** [Claude Code example](../examples/claude/README.md), [pi](integrations/pi.md), or [Oh My Pi](integrations/oh_my_pi.md).diff --git a/docs/index.md b/docs/index.mdindex c714b2b01..19dc47041 100644--- a/docs/index.md+++ b/docs/index.md@@ -29,10 +29,10 @@ Use it with [Claude Code or Codex](recipes/single_provider_coding_agents.md),- [Mechanics of Libsy](mechanics_of_libsy.md): understand steps, calls, and outcomes- [Switchyard Protocols](protocols/index.md): connect your model data to libsy- [Routing Algorithms](routing_algorithms/overview.md): choose how requests select a model+- [Tuning](tuning/index.md): find a configuration that meets your objective on your workload- [Host Integration](reference/host_integration.md): connect your clients and application lifecycle- [Rust API](reference/rust_api.md): browse the core library, protocols, translation, clients, and runner- [Context-Window Handling](operations/context_window.md): configure eviction and retry behavior-- [Tuning](tuning/index.md): foundations and worked configuration comparisons- [Troubleshooting](operations/troubleshooting.md): symptoms, silent misconfigurations, and fixes## Referencediff --git a/docs/operations/local_evaluation.md b/docs/operations/local_evaluation.mdindex ad3bbc1a3..57fc606d6 100644--- a/docs/operations/local_evaluation.md+++ b/docs/operations/local_evaluation.md@@ -152,6 +152,7 @@ With an unreleased build from main, `GET /v1/routing/session-stats?session_id=..- [Benchmark results](../benchmarks.md) link to model pools, results, and runnable profiles. For Terminal-Bench 2.1, the [escalation profile](../../benchmark/routing-profiles/tb21-escalation-opus-glm-deepseek.toml) includes its judge and reasoning settings. Its comments explain the endpoint differences from the measured runs.- [Benchmark setup](../../benchmark/README.md) explains how to compare routing against single-model baselines on the same tasks.+- [Tuning](../tuning/index.md) explains how to define an objective, compare candidate configurations on calibration tasks, and validate the selected one on held-out tasks.- [Routing overview](../routing_algorithms/overview.md) explains Auto, Task, Execution, and Composite. A TOML file can expose multiple routes through different route IDs.- [Server reference](../../crates/switchyard-server/README.md) covers configuration, TLS, logging, and metrics.diff --git a/docs/operations/troubleshooting.md b/docs/operations/troubleshooting.mdindex 4a912d469..b42eb0a49 100644--- a/docs/operations/troubleshooting.md+++ b/docs/operations/troubleshooting.md@@ -11,6 +11,7 @@ log field, a log line, or a `--dry-run` message.|---|---|---|---|| The first request lands on the efficient model | `x-model-router-selected-model` on the response | Auto and Execution decide from tool results, and a request without tool history has none. This is by design. | Send a request with a critical tool error, as in [Verify routing is active](local_evaluation.md#verify-routing-is-active). || Every request lands on one model | `curl -s localhost:4000/v1/stats | jq -c '.models | map_values(.calls)'` and `.algorithm_stats.stage_router.routing_decisions` | Chat-only traffic (`fall_open` only); both tiers name the same target or the same model ID; `capable_first` with a threshold the signals cannot reach; one tier is down and the other serves its fallbacks. | See the matching rows in [Silent misconfigurations](#silent-misconfigurations) and [When Auto does not route](../routing_algorithms/overview.md#when-auto-does-not-route). |+| Tasks fail on the efficient model and the route rarely escalates | `.models` call counts and `routing_decisions` in `/v1/stats` | The threshold is too high for the signals this workload produces; the capable hold is short; the efficient model is too weak for the tasks. | [Stage-router controls](../tuning/examples.md#choose-the-stage-router-controls-to-tune). || The judge never changes the outcome | `switchyard_classifier_fail_open_total` on `/metrics`, and the log line `judge verdict unavailable; routing without one` with its `reason` | The provider rejects JSON Schema mode (`json_object` needed); the verdict is truncated by `max_output_tokens`; the judge model returns prose instead of JSON. | [Judge model compatibility](../routing_algorithms/llm_classifier_routing.md#judge-model-compatibility). || Escalation never latches | The log line `stateful escalation has no session ID; routing state will not persist` | The client sends no session ID. No escalation state survives a request without one, so the route never latches at any `confirmations`. The warning fires only when `confirmations` is 2 or more. | Send `x-switchyard-session-id`. See [Tuning options](../routing_algorithms/escalation_router_routing.md#tuning-options). || Requests hang or take much longer than the model needs | Server log for retries and cooldown; `/v1/stats` call counts per model | A judge or tier without `timeout_ms`; a judge call on every tool step; a dead tier retried before fallback. | Give the judge its own `[llm_clients]` entry with `timeout_ms`. On a Task or Custom `llm_classifier` route, set `classify_trigger = "user_turn"` and send a session ID. See the dead-tier row below. |@@ -20,7 +21,7 @@ log field, a log line, or a `--dry-run` message.| The server does not start: `unknown field`, or `have no effect on the stage fallback judge` | The `--dry-run` message | An `auto` route with a tuning key; a typo; a key from another route type; `classifier.classify_trigger` other than the default, or `classifier.message_hash_fallback`, on a `stage_router` route. | [Auto](../reference/toml_schema.md#auto), the [stage_router keys](../reference/toml_schema.md#stage_router), and the [validation errors](../reference/toml_schema.md#validation-errors). || The server does not start: two targets `both name model` | The `--dry-run` message | Two targets share a model ID on one client with different `reasoning_effort`, `extra_body`, or `omit_body_fields`. | [Reasoning effort](../routing_algorithms/model_selection.md#reasoning-effort). || The client shows zero input tokens on `/v1/messages` | `prompt_tokens` in the routing log | The streamed `message_start` event reports zero usage when the target is on an OpenAI format. The real counts arrive in `message_delta`. | [Streamed token counts](../routing_algorithms/model_selection.md#pin-the-provider-and-endpoint). |-| Cost per run is higher than the single capable model | `cached_tokens` in the routing log on the turn after a model change; cost per arm | Each switch loses the prompt cache; the judge adds a call per turn; the efficient model is too weak and most turns escalate. | [Counter-examples](../routing_algorithms/model_selection.md#counter-examples) and the [inverted-pair check](../routing_algorithms/model_selection.md#check-for-an-inverted-pair). |+| Cost per run is higher than the single capable model | `cached_tokens` in the routing log on the turn after a model change; cost per arm | Each switch loses the prompt cache; the judge adds a call per turn; the efficient model is too weak and most turns escalate. | [Counter-examples](../routing_algorithms/model_selection.md#counter-examples) and the [inverted-pair check](../routing_algorithms/model_selection.md#check-for-an-inverted-pair). For the threshold and hold controls, see [Stage-router controls](../tuning/examples.md#choose-the-stage-router-controls-to-tune). || The prompt cache is never read | `cached_tokens` stays `0` on repeated prompts | The gateway does not cache on that endpoint, or the prompt is too short to cache. | [Prompt caching](../integrations/pi.md#prompt-caching). || `/v1/stats` has no `tiers` key | The `/v1/stats` body | The Rust server reports decisions under `algorithm_stats.<algorithm>`. | Read `algorithm_stats.stage_router.routing_decisions`, as in [Verify routing is active](local_evaluation.md#verify-routing-is-active). || The routing log does not say why a model was chosen | A routing-log record | The log records the served model and token counts only. | Use the decision counts in `/v1/stats`. For overrides, the `stage router override` log line also names the reason. See [Decision sources](../routing_algorithms/stage_router_routing.md#decision-sources). |diff --git a/docs/routing_algorithms/advisor_gate_routing.md b/docs/routing_algorithms/advisor_gate_routing.mdindex 6c78ad7e1..9be64c76e 100644--- a/docs/routing_algorithms/advisor_gate_routing.md+++ b/docs/routing_algorithms/advisor_gate_routing.md@@ -132,3 +132,5 @@ already frontier-class, benchmark[stage-router routing](stage_router_routing.md) first; reach for the advisorgate when the executor is markedly weaker than the best model you can call, orwhen you need a review trail rather than a model swap.++To compare candidate values on your workload, see [Tuning](../tuning/index.md).diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.mdindex 6a3eecfdf..490c6cd64 100644--- a/docs/routing_algorithms/escalation_router_routing.md+++ b/docs/routing_algorithms/escalation_router_routing.md@@ -259,3 +259,4 @@ retained nor logged.- [LLM Classifier Routing](llm_classifier_routing.md): pick a tier up frontinstead of judging the run.- [Mechanics of Libsy](../mechanics_of_libsy.md): the execution model and host responsibilities.+- [Tuning](../tuning/index.md): check your current route and compare candidate settings on your workload.diff --git a/docs/routing_algorithms/llm_classifier_routing.md b/docs/routing_algorithms/llm_classifier_routing.mdindex 25b86b3b4..9b77ba8ac 100644--- a/docs/routing_algorithms/llm_classifier_routing.md+++ b/docs/routing_algorithms/llm_classifier_routing.md@@ -133,6 +133,8 @@ for the server merge behavior.| `max_output_tokens` | `4096` | Maximum completion tokens available to the classifier verdict. Must be at least `1`. || `judge_deadline_ms` | unset | Bounds the whole judge consultation in milliseconds, the model call and the response drain included. On expiry the judge is treated as unavailable and follows `fail_open`: the route either falls back to the capable tier (default) or stops with a 504. Must be at least `1`. Unset leaves the consultation unbounded. |+To compare candidate values on your workload, see [Tuning](../tuning/index.md).+### Override the classifier promptSet `prompt` on the route when the packaged capability rubric does not describediff --git a/docs/routing_algorithms/overview.md b/docs/routing_algorithms/overview.mdindex bbda63f3e..24dc89766 100644--- a/docs/routing_algorithms/overview.md+++ b/docs/routing_algorithms/overview.md@@ -160,7 +160,9 @@ stage router with these settings. It does not compare strategies at runtime.Auto accepts no tuning keys. To change a setting, write the equivalent`stage_router` route shown in the [TOML schema](../reference/toml_schema.md#auto)and edit it there. Choose the two models with the-[model selection guide](model_selection.md).+[model selection guide](model_selection.md). To tune the rewritten route, start+with [Before you start](../tuning/foundations.md#before-you-start) in Tuning+Foundations.Only the `efficient_first` picker has published results. The `capable_first`picker is [experimental](stage_router_routing.md#pickers).diff --git a/docs/routing_algorithms/stage_router_routing.md b/docs/routing_algorithms/stage_router_routing.mdindex f18dfce9a..652e6b34b 100644--- a/docs/routing_algorithms/stage_router_routing.md+++ b/docs/routing_algorithms/stage_router_routing.md@@ -454,3 +454,4 @@ the `stage router override` log line carries it for overrides:- [Mechanics of Libsy](../mechanics_of_libsy.md): the execution model and host responsibilities.- [TOML Schema](../reference/toml_schema.md#stage_router): the completeconfiguration reference.+- [Tuning](../tuning/index.md): check your current route and compare candidate settings on your workload.diff --git a/docs/tuning/examples.md b/docs/tuning/examples.mdindex 4fc030a2b..bb1a635ff 100644--- a/docs/tuning/examples.md+++ b/docs/tuning/examples.md@@ -123,6 +123,45 @@ The frontier describes the calibration sample. Repeat promising runs, thenfreeze the selection. Run E and all three baselines on the untouched 400-taskvalidation split to check whether the cost advantage and success target hold.+## Check the efficient model when overrides dominate++This example keeps the objective and the route from the cost comparison. The+capable model has most of the calls in `.models`, and the first instinct is to+raise `confidence_threshold`. Read `routing_decisions` in `/v1/stats` first,+after a reset and a run of the calibration tasks, as in+[Before you start](foundations.md#before-you-start).++When `override` is a large share of the counts, the threshold is the wrong+control. Each `override` means a repeated failure, a critical error, or a+context-compaction marker in the recent tool results forced the capable tier.+These decisions skip the score, so `confidence_threshold` and the holds do not+change them. Raising the threshold moves only the `dimensions` decisions and+leaves the `override` share where it is.++The `stage router override` log line names the cause in `override_reason`:+`repeated_failure`, `critical_error`, or `compaction`. Count them. `compaction`+comes from a context-compaction marker in the history, so leave it out of this+comparison. Repeated failures and critical errors point at the efficient model:++1. **Measure the efficient model alone.** Add the `passthrough` route from+ [Check the efficient model before you route](../routing_algorithms/model_selection.md#check-the-efficient-model-before-you-route)+ and run the same calibration tasks. Compare it with the capable-only+ baseline. When the efficient-only arm is far below the capable-only arm,+ routing has to escalate a large share of turns to close the gap, and the+ savings shrink.+2. **Run the compatibility checks on that page.** Tool calling, structured+ output, context window, reasoning, and wire format. A model that passes a+ one-tool test can still produce malformed calls with the agent's full tool+ list.+3. **Change the efficient model.** Compare pairs as in+ [Model-pair selection](#model-pair-selection-with-fixed-algorithm-settings).+ If no cheaper model clears the tasks, the capable share reflects the tasks,+ and the cost objective needs a new target.++After a change, reset the counters, rerun the calibration tasks, and read the+`override` share again before you compare cost. Then validate on the held-out+split as in [final validation](foundations.md#4-check-the-final-choice).+## Model-pair selection with fixed algorithm settingsThis independent example fixes the algorithm settings and changes thediff --git a/docs/tuning/foundations.md b/docs/tuning/foundations.mdindex 9cdc938ce..0de213ffe 100644--- a/docs/tuning/foundations.md+++ b/docs/tuning/foundations.md@@ -8,6 +8,47 @@ before adopting it as your baseline.See [Examples](examples.md) for configuration and model-pair comparisonswith concrete data.+## Before you start++Most routes begin as `type = "auto"`. Auto is a stage-router preset:+`efficient_first`, `confidence_threshold = 0.5`, `recent_turn_window = 3`,+`capable_hold_turns = 2`, and no classifier. It accepts no tuning keys. To+change a setting, rewrite the route as the equivalent `stage_router` route from+the [TOML schema](../reference/toml_schema.md#auto). Record the starting values;+they are the current-routing baseline in step 2. The overview's+[Start here](../routing_algorithms/overview.md#start-here) list names the first+dial to move for each strategy.++Then read the decision counts, because three results mean no setting will help.+The counters keep every request since the server started, so clear them first+and run a sample of the workload you want to tune:++```bash+curl -s -X POST localhost:4000/v1/stats/reset+# Run the workload sample, then read the counts.+curl -s localhost:4000/v1/stats | jq '.algorithm_stats.stage_router.routing_decisions, (.models | map_values(.calls))'+```++- **`fall_open` is most of the total.** The tool-signal scorer abstained, which+ happens on chat-only traffic and on the first request of a session. The+ threshold and holds do not affect these turns. Add the+ [classifier fallback](../routing_algorithms/stage_router_routing.md#optional-llm-classifier-fallback),+ or use Task or Composite routing, which judge the request itself.+- **`.models` lists one entry.** Every sampled request went to one model. Check+ `capable_target` and `efficient_target` in the route. If they name the same+ target, fix them, then run the+ [inverted-pair check](../routing_algorithms/model_selection.md#check-for-an-inverted-pair).+ If they differ, the decision counts show which decision sent every request to+ that tier.+- **Requests carry no session ID.** Holds and escalation state live in the+ session. Send `x-switchyard-session-id` before tuning `capable_hold_turns` on+ a stage-router route or `confirmations` on an escalation route.++[Decision sources](../routing_algorithms/stage_router_routing.md#decision-sources)+explains each count, and+[When routing will not help](../routing_algorithms/overview.md#when-routing-will-not-help)+lists the remaining cases.+## 1. Define the objectiveDefine what the user wants to improve, on which workload, and what makes a result@@ -43,6 +84,16 @@ a baseline and a small set of changes you expect to help:4. **Keep runs comparable.** Use the same tasks, evaluation procedure, and initialcache setup. Pin model and provider versions, and save each configuration.+Collect these for every run:++| Measure | Source |+|---|---|+| Routing split and decision reasons | `GET /v1/stats`: `.models` call counts and `routing_decisions`. `POST /v1/stats/reset` clears them between runs. |+| Cost | The routing log (`--routing-log-file routing.jsonl`): the selected model and token usage per request, including `cached_tokens`. |+| Served model per response | The `x-model-router-selected-model` response header. |+| Metrics in your own host | [OpenTelemetry](../reference/opentelemetry.md#metrics) routing and client metrics. |+| Task success | Your agent harness. For Terminal-Bench and DeepSWE, the [benchmark setup](../../benchmark/README.md#run-with-switchyard-routing) runs a profile and [inspects the result](../../benchmark/README.md#inspect-a-run). |+Run the same tasks for every candidate. The following examples show how to usethe results to choose between them.@@ -54,6 +105,9 @@ Start with the comparison that matches what you want to change:keep the algorithm and model pair fixed, vary routing settings, and find thelowest-cost option that meets the task-success target. A Pareto chart showsthe cost and success tradeoffs.+- **[Check the efficient model when overrides dominate](examples.md#check-the-efficient-model-when-overrides-dominate):**+ when `override` is most of the decision counts, the threshold and holds cannot+ change those turns; measure the efficient model alone and change it if needed.- **[Select a model pair](examples.md#model-pair-selection-with-fixed-algorithm-settings):**keep algorithm settings fixed, swap the efficient/capable models, and compareexecution cost plus a penalty for failed tasks.diff --git a/docs/tuning/index.md b/docs/tuning/index.mdindex 89b4b62a2..f0db9f4e8 100644--- a/docs/tuning/index.md+++ b/docs/tuning/index.md@@ -1,14 +1,27 @@# TuningTuning helps you find a configuration that meets your goals on your workload.-Start with Foundations to plan the comparison, then use Examples to-see how to evaluate candidates and choose a configuration.+Start with Foundations to check your current route and plan the comparison, then+use Examples to see how to evaluate candidates and choose a configuration.-- [Tuning Foundations](foundations.md): define your objective, candidates, and evaluation splits.-- [Examples](examples.md): compare routing settings and model pairs with worked examples.+- [Tuning Foundations](foundations.md): check your current route, then define your objective, candidates, and evaluation splits.+- [Examples](examples.md): compare routing settings and model pairs with worked examples, and check the efficient model when overrides dominate.++## Controls by algorithm++Examples walks through the stage router. Each algorithm has its own controls:++- [Stage router](../routing_algorithms/stage_router_routing.md#knob-reference): every route key, [how `confidence_threshold` works](../routing_algorithms/stage_router_routing.md#tuning-confidence_threshold), and [how to calibrate it from run data](../routing_algorithms/stage_router_routing.md#calibrating-the-threshold-from-run-data).+- [LLM classifier](../routing_algorithms/llm_classifier_routing.md#tuning-options): thresholds, judge context, and when the judge runs.+- [Escalation router](../routing_algorithms/escalation_router_routing.md#tuning-options): confirmations and judge window.+- [Advisor gate](../routing_algorithms/advisor_gate_routing.md#tuning): review count and stall limits, with the benchmarked settings.+- [Composite](../routing_algorithms/composite_routing.md#independent-stage-profiles): separate stage-router settings per profile.+- [Plan/Execute](../routing_algorithms/plan_execute_routing.md#optional-settings): handoff triggers and prompts.+- [Verification-gated](../routing_algorithms/vgr_routing.md#serving-modes): serving mode and [execution limits](../routing_algorithms/vgr_routing.md#execution-limits).+- [Random](../routing_algorithms/random_routing.md#control-the-traffic-split): target weights.## Related-- [Routing Overview](../routing_algorithms/overview.md): choose a strategy.+- [Start here](../routing_algorithms/overview.md#start-here): a strategy, a profile to copy, and the first dial to move.- [Model Selection](../routing_algorithms/model_selection.md): choose and check the models.- [Troubleshooting](../operations/troubleshooting.md): symptoms and silent misconfigurations.diff --git a/mkdocs.yml b/mkdocs.ymlindex d50e697c9..cf3b91b56 100644--- a/mkdocs.yml+++ b/mkdocs.yml@@ -19,13 +19,12 @@ nav:- Home: index.md- Get Started:- Getting Started: getting_started.md+ - Integrate Switchyard:- Mechanics of Libsy: mechanics_of_libsy.md- Switchyard Protocols:- - Overview: protocols/index.md+ - protocols/index.md- LLM-Related Protocols: protocols/llm.md- Decision Model-Related Protocols: protocols/decisions.md- - Known Issues: known_issues.md- - Integrate Switchyard:- Embed in Your Gateway or Harness: reference/host_integration.md- Gateways:- NeMo Relay: integrations/nemo_relay.md@@ -42,11 +41,12 @@ nav:- Advisor-Gate Routing: routing_algorithms/advisor_gate_routing.md- Verification-Gated Routing: routing_algorithms/vgr_routing.md- Choose Models: routing_algorithms/model_selection.md- - Evaluate Routing:+ - Evaluate and Tune:- Benchmarks: benchmarks.md- - Tuning: tuning/index.md- - Tuning Foundations: tuning/foundations.md- - Examples: tuning/examples.md+ - Tuning:+ - tuning/index.md+ - Foundations: tuning/foundations.md+ - Examples: tuning/examples.md- Local Demo:- Run the Reference Server: operations/local_evaluation.md- Local Agents:@@ -68,6 +68,7 @@ nav:- Rust API: reference/rust_api.md- Context-Window Handling: operations/context_window.md- Troubleshooting: operations/troubleshooting.md+ - Known Issues: known_issues.md- Demo Server:- CLI: cli_reference.md- Soak Testing: operations/soak_test.md@@ -100,6 +101,7 @@ theme:features:- navigation.top- navigation.sections+ - navigation.indexes- search.suggest- search.highlight- content.code.copy