Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
fix(stage-router): reject knobs that have no effect on a standalone route (#1036)
eric-liu-nvidiafae5be0a3333
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex b79ec9d58..07c46f22c 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -31,6 +31,18 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).[host integration guide](docs/getting_started.md#drive-the-algorithm).([#948](https://github.com/NVIDIA-NeMo/Switchyard/pull/948))+### Changed++- **Stage fallback-judge knobs that had no effect are now rejected** — a+ `stage_router` route no longer loads with a `classifier.classify_trigger` other+ than `every_request` or with `classifier.message_hash_fallback`. The stage+ fallback judge runs on every ambiguous turn and retains no verdict, so these+ keys did nothing. Composite routes, which own the judge trigger, are+ unchanged. The server also warns at startup when+ a route's two tiers name the same target, and when an `llm_classifier` route+ that selects escalation mode through its `escalation` table carries capability+ keys that mode ignores.+## [0.3.0]Switchyard 0.3.0 builds on the native server and Rust library introduced indiff --git a/crates/libsy/src/algorithms/stage.rs b/crates/libsy/src/algorithms/stage.rsindex 6dfc5a135..ebfb4f374 100644--- a/crates/libsy/src/algorithms/stage.rs+++ b/crates/libsy/src/algorithms/stage.rs@@ -19,6 +19,7 @@ use async_trait::async_trait;use super::fall_through::FallThrough;use super::llm_class::{LlmClassifierConfig, LlmTaskClassifier, TaskClassifierConfig};+use super::util::affinity::ClassifyTrigger;use super::util::prompts::prepend_system_prompt;use super::util::stage::{DecisionSource, HandoffNoteConfig, PickerMode, StageClassifier, Tier, fall_open_tier,@@ -187,8 +188,21 @@ impl StageRouter {/// Routes between the runtime `capable` and `efficient` models. The judge,/// when configured, is called through the runtime `judge` model.///- /// Errors if either threshold in `config` is outside `[0.0, 1.0]`.+ /// Errors if either threshold in `config` is outside `[0.0, 1.0]`, or if the+ /// fallback judge is asked to retain verdicts: it runs on every ambiguous+ /// turn, so `classify_trigger` and `message_hash_fallback` would do nothing.pub fn new(config: StageRouterConfig) -> Result<Self> {+ if let Some(fallback) = &config.llm_fallback+ && (fallback.config.classify_trigger != ClassifyTrigger::EveryRequest+ || fallback.config.message_hash_fallback)+ {+ return Err(LibsyError::AlgorithmError {+ message: "classify_trigger and message_hash_fallback have no effect on the \+ stage fallback judge, which runs on every ambiguous turn; remove \+ them, or use a composite route to judge once per user turn"+ .to_string(),+ });+ }Ok(Self {route: build_stage_route(config)?,})@@ -273,7 +287,6 @@ mod tests {use parking_lot::Mutex;use super::*;- use crate::algorithms::util::affinity::ClassifyTrigger;use crate::algorithms::util::stage::{clear_fall_open, set_fall_open};use crate::algorithms::util::tier_fixtures::{JUDGE, Recorder, turn_request};use crate::core::testing::test_drive_with_models;@@ -406,6 +419,31 @@ mod tests {));}+ #[test]+ fn rejects_retaining_triggers_on_the_fallback_judge() {+ for (trigger, message_hash_fallback) in [+ (ClassifyTrigger::UserTurn, false),+ (ClassifyTrigger::NewSession, false),+ (ClassifyTrigger::UserTurn, true),+ ] {+ let mut config = config();+ config.llm_fallback = Some(LlmFallback {+ config: TaskClassifierConfig {+ classify_trigger: trigger,+ message_hash_fallback,+ ..Default::default()+ },+ });+ assert!(+ matches!(+ StageRouter::new(config),+ Err(LibsyError::AlgorithmError { .. })+ ),+ "{trigger:?} with message_hash_fallback = {message_hash_fallback}"+ );+ }+ }+#[test]fn builds() -> Result<()> {let router = StageRouter::new(config())?;@@ -628,37 +666,6 @@ mod tests {Ok(())}- #[tokio::test]- async fn the_fallback_judge_ignores_the_retaining_triggers() -> Result<()> {- for trigger in [ClassifyTrigger::UserTurn, ClassifyTrigger::NewSession] {- let recorder = Arc::new(Recorder::default());- let mut config = config_with_judge(&recorder, 0.9);- config- .llm_fallback- .as_mut()- .expect("config_with_judge sets a judge")- .config- .classify_trigger = trigger;- let router = recording_router(config)?;-- // Two tool continuations of one user turn in one session. A standalone- // capability route with this trigger would judge once and retain the- // target; inside the stage cascade the judge is a plain classifier.- for _ in 0..2 {- test_drive_with_models(- router.clone(),- turn_request(false),- runtime_models(),- recorder.serve(),- )- .await?;- }-- assert_eq!(recorder.judge_calls(), 2, "{trigger:?} retained a verdict");- }- Ok(())- }-#[tokio::test]async fn a_judge_that_cannot_tell_lands_on_the_picker_default() -> Result<()> {let recorder = Arc::new(Recorder::default());diff --git a/crates/switchyard-runner/src/algorithm.rs b/crates/switchyard-runner/src/algorithm.rsindex b2f141ae2..9729bcde3 100644--- a/crates/switchyard-runner/src/algorithm.rs+++ b/crates/switchyard-runner/src/algorithm.rs@@ -598,9 +598,9 @@ pub struct StageClassifierConfig {/// for an uncertain verdict and twice for unsupported.#[serde(default)]pub threshold_step: Option<f64>,- /// How often the judge runs. Composite honors it. On a standalone stage route- /// it has no effect: the fallback judge runs on every ambiguous turn and- /// retains no verdict.+ /// How often the judge runs. Composite honors it. A standalone stage route+ /// rejects any value other than `every_request`, because its fallback judge+ /// runs on every ambiguous turn and retains no verdict.#[serde(default)]pub classify_trigger: ClassifyTrigger,/// Reuses the session's target by hashing the first user message when no@@ -654,6 +654,13 @@ pub struct StageTierConfig {pub handoff_notes: Option<HandoffNoteConfig>,}+impl StageTierConfig {+ /// Warns when both tiers name one target; see [`warn_identical_tiers`].+ fn warn_identical_tiers(&self, route_name: &str) {+ warn_identical_tiers(route_name, &self.capable_target, &self.efficient_target);+ }+}+impl StageClassifierConfig {fn task_classifier_config(&self,@@ -1249,16 +1256,23 @@ impl LlmClassifierRouteConfig {"llm_classifier route {route_name} mode escalation cannot use classify_trigger")));}- if mode.is_some()- && (base_threshold.is_some()- || threshold_step.is_some()- || *message_hash_fallback- || recent_turn_window.is_some())- {+ let has_capability_settings = base_threshold.is_some()+ || threshold_step.is_some()+ || fail_open.is_some()+ || *message_hash_fallback+ || recent_turn_window.is_some();+ if mode.is_some() && has_capability_settings {return Err(AlgorithmConfigError::new(format!("llm_classifier route {route_name} mode escalation cannot use capability routing settings")));}+ if has_capability_settings {+ // Existing files that select escalation through the `escalation` table alone+ // keep loading, but their capability keys do nothing in this mode.+ tracing::warn!(+ "llm_classifier route {route_name} runs in escalation mode because it has an escalation table; base_threshold, threshold_step, fail_open, recent_turn_window and message_hash_fallback are ignored in this mode. Set mode = \"escalation\" and remove them."+ );+ }Ok(LlmClassifierModeConfig::Escalation(EscalationClassifierRouteConfig {classifier_target: classifier_target.clone(),@@ -1591,6 +1605,7 @@ fn build_algorithm("stage_router route {route_name} uses picker \"capable_first\", which is experimental: published thresholds and routing results all come from \"efficient_first\", so there is no calibrated confidence_threshold for it and no measured accuracy or cost. Use \"efficient_first\" unless you are running your own calibration.");}+ tiers.warn_identical_tiers(route_name);let threshold = confidence_threshold.ok_or_else(|| {AlgorithmConfigError::new(format!("stage_router route {route_name}: confidence_threshold is required"@@ -1623,7 +1638,11 @@ fn build_algorithm(let parent: Arc<dyn Algorithm> = Arc::new(algorithm);attach_subagent_router(route_name, parent, subagents.as_ref(), targets)}- AlgorithmSpec::Auto { .. } => {+ AlgorithmSpec::Auto {+ capable_target,+ efficient_target,+ } => {+ warn_identical_tiers(route_name, capable_target, efficient_target);let config = StageRouterConfig::new(PickerMode::EfficientFirst, 0.5);let algorithm = StageRouter::new(config).map_err(|error| {AlgorithmConfigError::with_source(@@ -1638,6 +1657,7 @@ fn build_algorithm(stage,subagents,} => {+ stage.warn_identical_tiers(route_name);let mut stage_config = StageRouterConfig::new(PickerMode::EfficientFirst,stage.confidence_threshold.unwrap_or(0.5),@@ -1846,6 +1866,19 @@ fn default_classifier_max_output_tokens() -> u64 {LlmCapabilityConfig::default().max_output_tokens}+/// Both tiers on one target leave the route nothing to choose between; every decision+/// lands on the same model. The file still loads, since a single-model deployment can+/// be a deliberate overhead measurement.+fn warn_identical_tiers(route_name: &str, capable_target: &str, efficient_target: &str) {+ if capable_target == efficient_target {+ tracing::warn!(+ route = route_name,+ target = capable_target,+ "capable_target and efficient_target name the same target, so every request goes to one model; use passthrough for a single model, or point the tiers at two targets"+ );+ }+}+fn warn_single_target_classifier(route_name: &str, models: &CategoryModelConfig) {if models.routing_names().len() < 2 {tracing::warn!(diff --git a/crates/switchyard-runner/tests/classifier_modes.rs b/crates/switchyard-runner/tests/classifier_modes.rsnew file mode 100644index 000000000..0a9b938e4--- /dev/null+++ b/crates/switchyard-runner/tests/classifier_modes.rs@@ -0,0 +1,62 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++//! Capability-only keys on an escalation route: rejected when the mode is spelled+//! out, accepted (and ignored, with a warning) when the escalation table selects+//! the mode on its own.++use switchyard_runner::Runner;++const CONFIG: &str = r#"+schema_version = 1+[llm_clients.local]+format = "openai_chat"+base_url = "http://127.0.0.1:1/v1"+[targets.fast]+id = "fast-model"+llm_client = "local"+[targets.strong]+id = "strong-model"+llm_client = "local"+[targets.judge]+id = "judge-model"+llm_client = "local"+[routes.agent]+id = "agent"+type = "llm_classifier"+classifier_target = "judge"+strong_target = "strong"+weak_target = "fast"+escalation = { confirmations = 2 }+"#;++const CAPABILITY_KEYS: [&str; 5] = [+ "base_threshold = 0.5",+ "threshold_step = 0.1",+ "fail_open = false",+ "recent_turn_window = 4",+ "message_hash_fallback = true",+];++#[test]+fn implicit_escalation_mode_keeps_loading_with_capability_keys() {+ for setting in CAPABILITY_KEYS {+ let config = format!("{CONFIG}{setting}\n");+ assert!(Runner::from_toml(&config).is_ok(), "{setting}");+ }+}++#[test]+fn explicit_escalation_mode_rejects_capability_keys() {+ for setting in CAPABILITY_KEYS {+ let config = format!("{CONFIG}mode = \"escalation\"\n{setting}\n");+ let error = match Runner::from_toml(&config) {+ Ok(_) => panic!("expected a load error for {setting}"),+ Err(error) => error.to_string(),+ };+ assert!(+ error.contains("mode escalation cannot use capability routing settings"),+ "{setting}: {error}"+ );+ }+}diff --git a/crates/switchyard-runner/tests/stage_knobs.rs b/crates/switchyard-runner/tests/stage_knobs.rsnew file mode 100644index 000000000..d660eab1c--- /dev/null+++ b/crates/switchyard-runner/tests/stage_knobs.rs@@ -0,0 +1,60 @@+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.+// SPDX-License-Identifier: Apache-2.0++//! Stage fallback-judge knobs that cannot take effect are rejected at load.++use switchyard_runner::Runner;++const CONFIG: &str = r#"+schema_version = 1+[llm_clients.local]+format = "openai_chat"+base_url = "http://127.0.0.1:1/v1"+[targets.fast]+id = "fast-model"+llm_client = "local"+[targets.strong]+id = "strong-model"+llm_client = "local"+[targets.judge]+id = "judge-model"+llm_client = "local"+[routes.stage]+id = "stage"+type = "stage_router"+capable_target = "strong"+efficient_target = "fast"+picker = "efficient_first"+confidence_threshold = 0.5+"#;++const CLASSIFIER: &str = "[routes.stage.classifier]\ntarget = \"judge\"\nbase_threshold = 0.5\n";++fn load_error(config: &str) -> String {+ match Runner::from_toml(config) {+ Ok(_) => panic!("expected a load error for:\n{config}"),+ Err(error) => error.to_string(),+ }+}++#[test]+fn accepts_the_default_classifier_trigger() {+ assert!(Runner::from_toml(&format!("{CONFIG}{CLASSIFIER}")).is_ok());+ let explicit = format!("{CONFIG}{CLASSIFIER}classify_trigger = \"every_request\"\n");+ assert!(Runner::from_toml(&explicit).is_ok());+}++#[test]+fn rejects_classifier_knobs_that_have_no_effect() {+ for setting in [+ "classify_trigger = \"user_turn\"",+ "classify_trigger = \"new_session\"",+ "message_hash_fallback = true",+ ] {+ let error = load_error(&format!("{CONFIG}{CLASSIFIER}{setting}\n"));+ assert!(+ error.contains("have no effect on the stage fallback judge"),+ "{setting}: {error}"+ );+ }+}diff --git a/docs/operations/troubleshooting.md b/docs/operations/troubleshooting.mdindex bac4a1955..3d54ae117 100644--- a/docs/operations/troubleshooting.md+++ b/docs/operations/troubleshooting.md@@ -17,7 +17,7 @@ log field, a log line, or a `--dry-run` message.| HTTP 400 from the upstream when the client sends a thinking level | The upstream error body in the response | A Claude target on an OpenAI wire format behind a gateway. | [Thinking](../integrations/pi.md#thinking) and `omit_body_fields`. || HTTP 400 with code `context_length_exceeded` | The response body | Every target on the route overflowed. | [Context-Window Handling](context_window.md). || HTTP 400 on requests that contain images | The response body | The route sets `vision = false`. | Remove it, or declare `true` only when every target accepts images. See the [route keys](../reference/toml_schema.md#routesname). |-| The server does not start: `unknown field` | The `--dry-run` message | An `auto` route with a tuning key; a typo; a key from another route type. | [Auto](../reference/toml_schema.md#auto) and the [validation errors](../reference/toml_schema.md#validation-errors). |+| The server does not start: `unknown field`, or `have no effect on the stage fallback judge` | The `--dry-run` message | An `auto` route with a tuning key; a typo; a key from another route type; `classifier.classify_trigger` other than the default, or `classifier.message_hash_fallback`, on a `stage_router` route. | [Auto](../reference/toml_schema.md#auto), the [stage_router keys](../reference/toml_schema.md#stage_router), and the [validation errors](../reference/toml_schema.md#validation-errors). || The server does not start: two targets `both name model` | The `--dry-run` message | Two targets share a model ID on one client with different `reasoning_effort`, `extra_body`, or `omit_body_fields`. | [Reasoning effort](../routing_algorithms/model_selection.md#reasoning-effort). || The client shows zero input tokens on `/v1/messages` | `prompt_tokens` in the routing log | The streamed `message_start` event reports zero usage when the target is on an OpenAI format. The real counts arrive in `message_delta`. | [Streamed token counts](../routing_algorithms/model_selection.md#pin-the-provider-and-endpoint). || Cost per run is higher than the single capable model | `cached_tokens` in the routing log on the turn after a model change; cost per arm | Each switch loses the prompt cache; the judge adds a call per turn; the efficient model is too weak and most turns escalate. | [Counter-examples](../routing_algorithms/model_selection.md#counter-examples) and the [inverted-pair check](../routing_algorithms/model_selection.md#check-for-an-inverted-pair). |@@ -43,7 +43,6 @@ rows that return an error or hold a request open say so.| A tier that refuses connections or returns HTTP 408, 429, or 5xx, with the default `max_retries = 2` | Each request retries that tier, then falls through to the route's other target, so one model serves everything, slowly. After the retries fail, a 5-second cooldown sends later requests straight to the other target. A tier that hangs behaves differently: without `timeout_ms` the request waits, and with it the request that reaches the deadline returns `504`. Only requests inside the cooldown that follows fall through. | Runtime log lines for the failed attempts and the fallback. | Fix the endpoint and watch `.models` call counts in `/v1/stats`. Set `timeout_ms` so a hang becomes a `504` instead of a wait. See [`[llm_clients.<name>]`](../reference/toml_schema.md#llm_clientsname). || A judge client without `timeout_ms` | A judge that accepts the request and never finishes holds the request open. | None. | Give the judge its own `[llm_clients]` entry with `timeout_ms`. See [Tuning options](../routing_algorithms/llm_classifier_routing.md#tuning-options). || `auto` or `stage_router` serving chat-only traffic | No tool results, so the signals abstain and every turn falls open to the efficient tier. | None. | Use Task or Composite. See [When routing will not help](../routing_algorithms/overview.md#when-routing-will-not-help). |-| `classifier.classify_trigger = "user_turn"` or `"new_session"`, with or without `classifier.message_hash_fallback`, on a `stage_router` route | The stage fallback judge runs on every ambiguous turn and retains nothing, so the trigger changes nothing. The keys are accepted for compatibility. `message_hash_fallback` with the default trigger is a load error instead. | None. | Use a [composite](../routing_algorithms/composite_routing.md) route for a judge that runs once per user turn. || `capable_first_threshold` on a standalone `stage_router` route | Only a composite classifier selects the tier that uses it. | None. | Use `confidence_threshold`, or a composite route. See the [knob reference](../routing_algorithms/stage_router_routing.md#knob-reference). || A custom `llm_classifier` with one routing target | Every judge call is paid for and no choice is made. | Startup: `custom classifier has only one routing target; judge calls add cost without a routing choice`. | Add a second target, or use `passthrough`. |diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.mdindex f5f0b8e54..9497adca6 100644--- a/docs/reference/toml_schema.md+++ b/docs/reference/toml_schema.md@@ -424,8 +424,8 @@ so it is consulted on every ambiguous turn and never retains a verdict.| `classifier.threshold_step` | No | `0.0` | Finite, non-negative amount added once for uncertain or unmatched verdicts and twice for unsupported verdicts. `base_threshold + 2 * threshold_step` must be at most `1`. || `classifier.fail_open` | No | `true` | When `true`, judge client failures and deadlines let the turn fall open to the picker default and record fail-open evidence. When `false`, they stop the request. || `classifier.decision` | No | unset | Use a decision model instead of an LLM judge. The classifier target must be in `decision_targets`. It cannot be combined with `base_threshold`, `threshold_step`, `prompt`, `response_format_type`, or `max_output_tokens`; the load error is `decision cannot use LLM judge settings`. See [Decision model classifier](../routing_algorithms/composite_routing.md#decision-model-classifier). |-| `classifier.classify_trigger` | No | `every_request` | Accepted for compatibility with `llm_classifier`. No effect on the stage fallback judge: it runs on every ambiguous turn and retains no verdict, so `user_turn` and `new_session` change nothing here. Use a [composite](#composite) route to judge once per user turn. |-| `classifier.message_hash_fallback` | No | `false` | Accepted for compatibility. No effect on the stage fallback judge. Setting it with the default trigger is a load error. |+| `classifier.classify_trigger` | No | `every_request` | Only the default is accepted here. The stage fallback judge runs on every ambiguous turn and retains no verdict, so `user_turn` and `new_session` are rejected at load. Use a [composite](#composite) route to judge once per user turn. |+| `classifier.message_hash_fallback` | No | `false` | Rejected at load on a `stage_router` route. The fallback judge retains nothing to key on. || `classifier.recent_turn_window` | No | unset | When unset, the judge sees the opening task and the latest user follow-up. When set, it also sees that many trailing messages. This counts messages, while the route's `recent_turn_window` counts tool results, so the two values are different units. || `classifier.prompt` | No | packaged prompt | Replaces the capability judge prompt. The verdict schema is unchanged. || `classifier.response_format_type` | No | `json_schema` | Structured-output mode for the LLM judge. Use `json_object` when the provider does not support JSON Schema; Switchyard adds the schema to the prompt and validates the verdict locally. Not with `classifier.decision`. |diff --git a/docs/routing_algorithms/stage_router_routing.md b/docs/routing_algorithms/stage_router_routing.mdindex 234b677fa..80edf1277 100644--- a/docs/routing_algorithms/stage_router_routing.md+++ b/docs/routing_algorithms/stage_router_routing.md@@ -243,8 +243,8 @@ required columns.| `classifier.threshold_step` | `0.0` | Added once for uncertain or unmatched verdicts, twice for unsupported. || `classifier.fail_open` | `true` | On judge failure, fall open to the picker default instead of failing the request. || `classifier.decision` | unset | Use a decision model instead of an LLM judge. Excludes the LLM judge keys below. |-| `classifier.classify_trigger` | `every_request` | No effect here. The judge runs on every ambiguous turn. |-| `classifier.message_hash_fallback` | `false` | No effect here. A load error with the default trigger. |+| `classifier.classify_trigger` | `every_request` | Only the default is accepted. The judge runs on every ambiguous turn. |+| `classifier.message_hash_fallback` | `false` | Rejected at load. The judge retains no verdict. || `classifier.recent_turn_window` | unset | Trailing messages the judge sees after the opening task. A different unit from the route's `recent_turn_window`, which counts tool results. || `classifier.prompt` | packaged | Replaces the judge prompt. || `classifier.response_format_type` | `json_schema` | `json_object` for providers without JSON Schema. |