Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
docs(stage-router): clarify routing configuration (#822)
sabhatinasbec0cb58305f
diff --git a/docs/assets/stage-router-threshold.svg b/docs/assets/stage-router-threshold.svgnew file mode 100644index 000000000..ffd209789--- /dev/null+++ b/docs/assets/stage-router-threshold.svg@@ -0,0 +1,53 @@+<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 880 330" role="img" aria-labelledby="title desc">+ <title id="title">Stage score distribution at confidence threshold 0.5</title>+ <desc id="desc">An illustrative histogram of signed Stage scores. Scores below negative 0.5 select efficient, scores from negative 0.5 through positive 0.5 form the ambiguous band and fall back to efficient in efficient-first mode, and scores above positive 0.5 select capable.</desc>+ <rect width="880" height="330" rx="12" fill="#ffffff"/>+ <text x="440" y="30" text-anchor="middle" font-family="sans-serif" font-size="17" font-weight="700" fill="#1f2937">Illustrative score distribution · efficient_first · threshold = 0.5</text>++ <rect x="70" y="58" width="185" height="190" fill="#dbeafe"/>+ <rect x="255" y="58" width="370" height="190" fill="#fef3c7"/>+ <rect x="625" y="58" width="185" height="190" fill="#ede9fe"/>++ <text x="162" y="82" text-anchor="middle" font-family="sans-serif" font-size="14" font-weight="700" fill="#1e40af">signal → efficient</text>+ <text x="440" y="82" text-anchor="middle" font-family="sans-serif" font-size="14" font-weight="700" fill="#92400e">ambiguous band → default efficient</text>+ <text x="717" y="82" text-anchor="middle" font-family="sans-serif" font-size="14" font-weight="700" fill="#5b21b6">signal → capable</text>++ <g fill="#4b5563" opacity="0.82">+ <rect x="85" y="220" width="24" height="20" rx="2"/>+ <rect x="121" y="207" width="24" height="33" rx="2"/>+ <rect x="157" y="188" width="24" height="52" rx="2"/>+ <rect x="193" y="171" width="24" height="69" rx="2"/>+ <rect x="229" y="151" width="24" height="89" rx="2"/>+ <rect x="265" y="130" width="24" height="110" rx="2"/>+ <rect x="301" y="116" width="24" height="124" rx="2"/>+ <rect x="337" y="102" width="24" height="138" rx="2"/>+ <rect x="373" y="94" width="24" height="146" rx="2"/>+ <rect x="409" y="89" width="24" height="151" rx="2"/>+ <rect x="445" y="96" width="24" height="144" rx="2"/>+ <rect x="481" y="108" width="24" height="132" rx="2"/>+ <rect x="517" y="123" width="24" height="117" rx="2"/>+ <rect x="553" y="142" width="24" height="98" rx="2"/>+ <rect x="589" y="163" width="24" height="77" rx="2"/>+ <rect x="625" y="180" width="24" height="60" rx="2"/>+ <rect x="661" y="197" width="24" height="43" rx="2"/>+ <rect x="697" y="211" width="24" height="29" rx="2"/>+ <rect x="733" y="220" width="24" height="20" rx="2"/>+ <rect x="769" y="228" width="24" height="12" rx="2"/>+ </g>++ <line x1="70" y1="240" x2="810" y2="240" stroke="#111827" stroke-width="2"/>+ <line x1="255" y1="58" x2="255" y2="248" stroke="#92400e" stroke-width="2" stroke-dasharray="6 5"/>+ <line x1="625" y1="58" x2="625" y2="248" stroke="#92400e" stroke-width="2" stroke-dasharray="6 5"/>+ <line x1="440" y1="235" x2="440" y2="248" stroke="#111827" stroke-width="2"/>++ <g font-family="sans-serif" font-size="13" fill="#374151" text-anchor="middle">+ <text x="70" y="265">-1.0</text>+ <text x="255" y="265">-0.5</text>+ <text x="440" y="265">0</text>+ <text x="625" y="265">+0.5</text>+ <text x="810" y="265">+1.0</text>+ </g>+ <text x="440" y="292" text-anchor="middle" font-family="sans-serif" font-size="14" font-weight="700" fill="#1f2937">raw signed Stage score</text>+ <text x="70" y="313" font-family="sans-serif" font-size="12" fill="#6b7280">← efficient evidence</text>+ <text x="810" y="313" text-anchor="end" font-family="sans-serif" font-size="12" fill="#6b7280">capable evidence →</text>+</svg>diff --git a/docs/routing_algorithms/stage_router_routing.md b/docs/routing_algorithms/stage_router_routing.mdindex 7882b1258..84c815947 100644--- a/docs/routing_algorithms/stage_router_routing.md+++ b/docs/routing_algorithms/stage_router_routing.md@@ -1,19 +1,17 @@-# Stage-Router Routing+# Stage Router**Execution** routing uses the stage router to choose a model from tool resultsand agent progress. Configure it with `type = "stage_router"`.Stage-router routing sends each request to either a **capable** model or a-cheaper **efficient** one, depending on where the agent is in its run. The goal-is to spend the capable model on the turns that need it (exploration, error-recovery, hard reasoning) and let the efficient model carry the routine,-mechanical work. Which tier a turn defaults to depends on the picker you choose-(`capable_first` or `efficient_first`); the signals then move individual turns-off that default. You configure it with a single knob, `confidence_threshold`,-plus an optional LLM classifier.--If the selected target exceeds its context window, the client tries the route's-remaining targets in configured order for that request. See+typically cheaper **efficient** one, depending on where the agent is in its run.+The goal is to spend the capable model on turns that need exploration or error+recovery and let the efficient model carry routine, mechanical work. Choose a+required `picker` for ambiguous turns, tune the split with+`confidence_threshold`, and optionally add an LLM classifier.++Model-call fallback happens after the routing decision. Eligible context-window+or availability failures can try the route's other completion target. See[Context-Window Handling](../operations/context_window.md).## How it works@@ -22,47 +20,69 @@ A tool-using agent's run moves through stages that call for different amounts ofmodel capability. The built-in vocabulary is calibrated for coding agents: itrecognizes file observation, mutation, planning, shell activity, and test results.-For each LLM call, stage-router estimates which stage the agent is in from the-**tool-result history** on the conversation, scoring two axes:+For each LLM call, stage-router estimates the agent's state from the+conversation's **tool-result history**, scoring signals in two directions:-- **WRONG → capable**: `severity` (windowed error severity), `spinning` (deep+- **RECOVERY → capable**: `severity` (windowed error severity), `spinning` (deepchurn with no reads or writes), and `exploring` (reading or planning withoutproducing) push toward the capable tier.-- **PROGRESS → efficient**: `recent_production_intensity` (writes and edits- landing over the recent window) pushes toward the efficient tier.+- **PROGRESS → efficient**: `production_intensity` (the share of recent+ recognized operations that are writes or edits) pushes toward the efficient+ tier.The axes are **corroborative**: the signed score is `tanh`-squashed to a-confidence in `[0, 1]`, so one full signal alone scores ~`0.46` and a second-corroborating signal is what pushes it decisively past a `0.5` threshold. A-repeated failures and critical-error severity are hard overrides that escalate. The router-then routes:--- the **capable** tier for uncertain, exploratory, or error-recovery turns, and-- the **efficient** tier for settled, mechanical turns.+confidence in `[0, 1]`. One maxed scoring dimension produces about `0.46`;+corroborating evidence pushes confidence decisively past a `0.5` threshold.+Repeated failures, critical-error severity, and context compaction are hard+overrides to the capable tier. An active capable hold also bypasses the scorer.`confidence_threshold` sets how sure that estimate must be before the router acts-on the signal alone. Below it, the turn stays on the picker's default tier (or,-if you added the optional classifier, goes to it first). A turn with no-tool-result history yet has no stage to estimate, so it takes the default tier.+on the signal alone. Scores inside the ambiguous band go to the optional+classifier and then, if it cannot decide, to the picker's default tier. A turn+with no tool-result history follows the same fallback path.The routing decision for one turn:```mermaid-%%{init: {"flowchart": {"nodeSpacing": 18, "rankSpacing": 26}}}%%-flowchart LR- t["turn"] --> g{"confidence >= threshold?"}- g -->|yes| s["signals pick capable/efficient"]- g -->|no| c{"classifier set?"}- c -->|yes| k["classifier picks capable/efficient"]- c -->|no| d["use picker default tier"]-- classDef box font-family:monospace,fill:none,stroke:#9aa0a6,stroke-width:1px;- class t,s,k,d,g,c box;+%%{init: {"flowchart": {"curve": "linear", "nodeSpacing": 36, "rankSpacing": 38}}}%%+flowchart TB+ start(["New turn"])+ history[/"Read tool-result history"/]+ recovery{"Hard recovery signal<br/>or capable hold?"}+ scorer["Compute signed tool-signal score"]+ band{"Outside the<br/>ambiguous band?"}+ classifier{"Usable classifier verdict?"}+ capable(["Route to CAPABLE"])+ signalRoute(["Route by score sign<br/>negative → efficient · positive → capable"])+ classifierRoute(["Route by classifier verdict"])+ defaultRoute(["Route to picker default"])++ start --> history+ history --> recovery+ recovery -->|Yes| capable+ recovery -->|No| scorer+ scorer --> band+ band -->|Yes| signalRoute+ band -->|No · ambiguous| classifier+ classifier -->|Yes| classifierRoute+ classifier -->|No · absent or invalid| defaultRoute++ classDef input fill:#e0f2fe,stroke:#0284c7,color:#0c4a6e,stroke-width:2px;+ classDef process fill:#f8fafc,stroke:#64748b,color:#1e293b,stroke-width:1.5px;+ classDef decision fill:#fef3c7,stroke:#d97706,color:#78350f,stroke-width:2px;+ classDef capable fill:#ede9fe,stroke:#7c3aed,color:#4c1d95,stroke-width:3px;+ classDef outcome fill:#dcfce7,stroke:#16a34a,color:#14532d,stroke-width:2px;++ class start,history input;+ class scorer process;+ class recovery,band,classifier decision;+ class capable capable;+ class signalRoute,classifierRoute,defaultRoute outcome;```-With `capable_first`, the default is capable, so a turn only reaches the cheaper-efficient model on a confident efficient signal (or an efficient verdict from-the classifier). Raising the threshold shrinks that path; lowering it widens it.+After a signal-driven capable decision, `capable_hold_turns` keeps recovery turns+on that tier for up to two requests by default. A clean passing test clears the+hold early. Set the value to `0` to disable it.## Pickers@@ -87,99 +107,67 @@ Both pickers read the same signals; only the default tier differs.## Tuning `confidence_threshold`-The scorer rates each turn from `0` (signals are neutral) to `1` (signals point-hard at one tier). `confidence_threshold` is the bar that rating has to clear-before the router will switch off the picker's default tier. Clear it and the-router routes to the tier the signals indicate; fall short and the turn stays on-the default.--With the default `capable_first` picker, every turn starts on the capable tier-and only drops to the efficient tier when the signals say "efficient" and clear-the threshold. So the threshold sets how much evidence it takes to switch to the-cheaper tier:--- Raise it and only strong, decisive signals drop a turn to efficient, so the- router stays on capable longer (more quality, more cost).-- Lower it and weaker signals are enough to drop to efficient, so more turns go- cheap (more savings, more risk).+The tool-signal scorer gives each turn a signed score in `(-1, 1)`: negative+scores point to the efficient tier, positive scores point to the capable tier,+and the absolute value is the confidence. `confidence_threshold` creates a+closed **ambiguous band** from `-threshold` to `+threshold`. Scores outside the+band make a signal-based decision; scores inside it go to the optional+classifier or fall back to the picker's default tier.-`efficient_first` is the mirror: turns start on efficient and need a signal that-clears the threshold to escalate to capable.--(If you add the optional classifier, sub-threshold turns go to it instead of-staying on the default tier.)+The TOML schema requires you to choose `picker` explicitly; there is no implicit+default.**Set `0.5` explicitly.** `confidence_threshold` is required by the TOML schema;-`0.5` is the recommended starting point and what the example below uses.+`0.5` is the recommended starting point, derived from many coding benchmarks,+and what the example below uses.-| `confidence_threshold` | Include `classifier:` block? | Typical use |-|---|---|---|-| `0.0` | no | Cost/latency-sensitive. Every signal-based verdict is accepted; no per-turn LLM call. Critical-error signals still escalate to capable. |-| `0.5` | no | Recommended starting point. The scorer is corroborative — one full wrong signal scores ~`0.46`, just under `0.5` — so a decisive escalation takes a strong signal plus corroboration, while a critical error overrides regardless. Derived from SWE-Bench Pro Python-75 calibration. |-| `0.7` - `0.9` | yes | Classifier-assisted. Low-confidence turns go to the LLM classifier before falling back to the default tier. |-| `1.0` | yes (required) | Classifier-driven. Tool signals only apply hard overrides; other turns reach the classifier. |+### What `0.5` means with `efficient_first`-The signal-vs-classifier split is dataset-dependent. Measure it in-production: `/v1/stats` reports stage-router decisions by source and semantic-target, while response headers and structured decision logs explain individual selections.+-### Calibrating the threshold from run data+With `picker = "efficient_first"`, no classifier, and a threshold of `0.5`:-The recommended `0.5` starting point was derived from SWE-Bench Pro Python-75-calibration. To tune for a different task set or model pair, follow this-minimum-data path.+- scores below `-0.5` route to efficient from the tool signals;+- scores from `-0.5` through `+0.5` are ambiguous and fall back to efficient;+- scores above `+0.5` route to capable from the tool signals.-**What you need**+Hard overrides and capable-hold state can still select capable independently of+this score. Lowering the threshold narrows the ambiguous band, so more turns are+decided directly by the scorer. Raising it widens the band, so more turns use the+picker default (or the classifier, when configured). That movement changes the+efficient/capable routing split.-| Run | Coverage | Purpose |+| `confidence_threshold` | Behavior | Typical use ||---|---|---|-| Pure-capable | ~40–75 representative tasks | Baseline outcomes + signal features |-| Pure-efficient | ~20 tasks (sampled from capable results) | Counterfactual outcomes |--Neither run needs to cover the full task set. A few dozen capable tasks gives-enough outcome diversity; the efficient probe only needs to cover the interesting-quadrant candidates identified from those capable results.--**How to sample the efficient probe set**--Stratify the pure-capable results across four quadrant candidates before running efficient:+| `0.0` | Every non-zero score makes a signal-based decision. A neutral score or missing tool history still falls through. | Maximize decisions by the tool-signal scorer. |+| `0.5` | Scores from `-0.5` through `+0.5` are ambiguous. | Recommended starting point, derived from many coding benchmarks. |+| `1.0` | Every ordinary score is ambiguous; hard overrides and capable holds still apply. | Classifier-driven when a classifier is configured; picker-driven otherwise. |-| Category | Criterion | Count | Value |-|---|---|---|---|-| Easy + clean | Capable passes, small diff, clear spec | ~5 | Establishes SAFE floor |-| Easy + tricky | Capable passes, subtle logic | ~5 | Finds RESCUE candidates |-| Hard + structural | Capable fails, large multi-file diff | ~5 | HARD noise baseline |-| Hard + localized | Capable fails, small targeted fix | ~5 | Finds LOSS candidates |+A route with a `1.0` threshold remains valid without a classifier. In that case,+ordinary sub-threshold turns fall back to the picker's default tier; hard+overrides and capable-hold behavior still apply.-Sample across repos and diff sizes. Don't over-represent one project.+The signal-vs-classifier split is dataset-dependent. Measure it in production:+`/v1/stats` reports decision counts by source and selected target, plus aggregate+scorer metrics. Response headers and structured decision logs explain individual+selections.-**Building RESCUE / LOSS quadrants**--For efficient-to-capable escalation, group the overlap tasks (those with both-capable and efficient results):--- `RESCUE` = efficient-fail ∩ capable-pass, a candidate for beneficial escalation-- `LOSS` = efficient-pass ∩ capable-fail, a candidate for harmful escalation-- `SAFE` = both pass-- `HARD` = both fail--Sweep a few candidate thresholds in routed benchmark runs. For-`efficient_first`, choose the lowest threshold that improves outcomes on RESCUE-tasks without turning passing LOSS tasks into failures. Because the scorer is-corroborative, a `0.5` threshold takes-roughly 1.5 signals of agreement.+### Calibrating the threshold from run data-**Caveat on fixed-model outcomes vs. routed outcomes**+Use about 10% of your representative tasks. Replay their agent histories through+the Stage tool-signal scorer and collect the unthresholded signed score for each+ordinary turn. Track hard overrides and capable holds separately because changing+the threshold does not affect them. Plot the score distribution, then overlay+candidate ambiguous bands.-After a switch, the selected model inherits conversation history from the other-model. Fixed-model runs do not measure this effect. Inherited context may help-or hurt, so these quadrants identify candidates, not guaranteed outcomes.-Validate the threshold with routed runs on the same tasks.+For `efficient_first`, count how many turns fall below the band, inside it, and+above it. Without a classifier, those regions map to signal-selected efficient,+default-efficient, and signal-selected capable decisions. Choose the threshold+that gives the routing split you want, then validate it on the same sample before+running the full task set.## Route configuration-> Requires unreleased features. [Build from source](../getting_started.md#build-from-source) to run this example.-```tomlschema_version = 1@@ -188,12 +176,12 @@ format = "openai_chat"base_url = "https://openrouter.ai/api/v1"api_key_env = "OPENROUTER_API_KEY"-[targets.strong]+[targets.capable]id = "openai/gpt-4o"llm_client = "openrouter"# system_prompt = "diagnose before you edit" # optional-[targets.weak]+[targets.efficient]id = "openai/gpt-4o-mini"llm_client = "openrouter"# system_prompt = "follow the settled plan" # optional@@ -201,8 +189,8 @@ llm_client = "openrouter"[routes.stage]id = "switchyard/stage"type = "stage_router"-capable_target = "strong"-efficient_target = "weak"+capable_target = "capable"+efficient_target = "efficient"picker = "efficient_first"confidence_threshold = 0.5recent_turn_window = 3 # optional, defaults to 3@@ -238,8 +226,7 @@ The categories affect existing stage signals:- `plan` counts as investigation, like the built-in planning tools.- `new` records forward activity that suppresses false `spinning` and`exploring` signals, but does not otherwise favor either tier.-- `unknown` remains the fallback for tools that match neither the built-in- vocabulary nor configured semantics.+- Unmatched tools remain `unknown`; there is no `unknown` configuration key.Configuration cannot reclassify a built-in tool. Empty names, duplicate namesacross categories, and unknown category keys are rejected when the route is@@ -248,48 +235,54 @@ rules are outside this exact-name configuration.### Optional: handoff notes-Add a `[routes.stage.handoff_notes]` section to pass a contextual note to the-model the router switches to. The escalation note is sent to the capable tier on-a signal-driven escalation; the de-escalation note is sent when the scorer-decisively picks the efficient tier.+Add a `[routes.stage.handoff_notes]` section to append contextual guidance to a+request when tool signals decide its tier. Notes exist only in the forwarded+request; they do not accumulate in the caller's conversation. The escalation+note accompanies a signal-driven capable decision, and the optional de-escalation+note accompanies a signal-driven efficient decision.```toml[routes.stage.handoff_notes]escalation_note = "the previous model was stalling; pick up the diagnosis"# deescalation_note = "..." # optional-# only_on_wrong_signal_escalation = true # default; set false to always send+# only_on_wrong_signal_escalation = true # default; false also includes capable holds```+Classifier and picker-default decisions do not add handoff notes.+### Optional: LLM classifier fallback-The block is optional, and omitting it is the default. With no-`[routes.stage.classifier]` block there is no judge and no fallthrough to one:-a turn the signals leave undecided goes straight to the `picker` tier.+The block is optional. Without it, an ambiguous turn goes directly to the+`picker` tier.-To break ties on low-confidence turns with a model call, add the block and set-`confidence_threshold` above `0.0`. The classifier is consulted only for turns-that fall below the threshold:+To judge ambiguous turns with a model call, add the block. The classifier runs+when the scorer has no tool history or produces a score inside the closed+ambiguous band:```toml+[targets.stage_judge]+id = "openai/gpt-4o-mini"+llm_client = "openrouter"+[routes.stage.classifier]-target = "strong" # target the judge is called through (not a routing destination)+target = "stage_judge" # judge-only; never a completion destinationbase_threshold = 0.5 # p_solve floor to route efficient; below this → capable-threshold_step = 0.1 # adds 0.1 for uncertain and 0.2 for unsupported verdicts-recent_turn_window = 3 # conversation span the judge sees+threshold_step = 0.1 # +0.1 for uncertain/unmatched; +0.2 for unsupported+recent_turn_window = 3 # trailing turns added to the opening taskprompt = "Estimate whether the efficient target can complete this request."response_format_type = "json_object" # optional; default is "json_schema"````prompt` replaces the packaged capability-classifier prompt. In the default-`json_schema` mode, Switchyard sends the verdict schema through the structured-output-request. Set `response_format_type = "json_object"` for providers that only support-JSON Object mode; Switchyard then adds the schema to the judge prompt and validates-the returned object locally. The verdict schema and routing thresholds remain unchanged.+`json_schema` mode, Switchyard sends the verdict schema through the+structured-output request. Set `response_format_type = "json_object"` for+providers that only support JSON Object mode; Switchyard then adds the schema to+the judge prompt and validates the returned object locally. The verdict schema+and routing thresholds remain unchanged.-Leave the block out under-[Composite Routing](composite_routing.md). There the tier is already picked-at the user turn, so a fallthrough judge would sit between the signals and that-decision and overrule it.+The nested stage table in [Composite Routing](composite_routing.md) does not+accept a `classifier` block. Its outer classifier supplies the tier used when+the stage signals are ambiguous.Give the classifier its own LLM client or quota bucket where possible. Sharingone provider bucket with the efficient tier adds a request per classified turn@@ -297,11 +290,12 @@ and can cause sustained 429s at scale.## Observability-Each response carries two routing headers:+When a model serves the request, the response identifies it with this routing+header:| Header | Content ||---|---|-| `x-model-router-selected-model` | The model ID the turn was routed to. |+| `x-model-router-selected-model` | The model ID that served the response, including any model-call fallback. |### Decision sources@@ -312,9 +306,9 @@ paths through its cascade:|---|---|| `override` | A repeated failure, critical-error severity, or context-compaction marker forced the capable tier. Structured logs set `override_reason` to `repeated_failure`, `critical_error`, or `compaction`. || `capable_hold` | A recent escalation kept this recovery turn on the capable tier. |-| `dimensions` | The corroborative scorer crossed `confidence_threshold` and picked the tier by the sign of the score. |-| `llm-classifier` | The signals were ambiguous and the classifier returned a verdict. |-| `fall_open` | The signals were ambiguous and the classifier failed or wasn't configured; the default tier was used. |+| `dimensions` | The signed score fell outside the ambiguous band and selected a tier. |+| `llm-classifier` | The signals were ambiguous and the classifier returned a usable verdict. |+| `fall_open` | The signal scorer abstained and the classifier was absent or could not decide, so the picker default was used. |## When *not* to use stage-router@@ -323,10 +317,13 @@ paths through its cascade:[Random Routing](random_routing.md) (`type = "random"`).The stage-router's signals are wasted on a fixed traffic ratio.- **No tool-result history.** Stage-router needs meaningful tool-call traffic to- populate the tool-result signal. For pure chat-completion workloads every- ambiguous request lands on the picker's default tier.+ populate its signals. For pure chat-completion workloads, the signal scorer+ abstains; the optional classifier decides, or the request uses the picker+ default when no classifier is configured.## Related- [Architecture](../architecture.md): the end-to-end request lifecycle andsystem boundaries.+- [TOML Schema](../reference/toml_schema.md#stage_router): the complete+ configuration reference.