Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

fix(stage-router): reject knobs that have no effect on a standalone route (#1036)

eric-liu-nvidiafae5be0a3333
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b79ec9d58..07c46f22c 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -31,6 +31,18 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
[host integration guide](docs/getting_started.md#drive-the-algorithm).
([#948](https://github.com/NVIDIA-NeMo/Switchyard/pull/948))
+### Changed
+
+- **Stage fallback-judge knobs that had no effect are now rejected** — a
+ `stage_router` route no longer loads with a `classifier.classify_trigger` other
+ than `every_request` or with `classifier.message_hash_fallback`. The stage
+ fallback judge runs on every ambiguous turn and retains no verdict, so these
+ keys did nothing. Composite routes, which own the judge trigger, are
+ unchanged. The server also warns at startup when
+ a route's two tiers name the same target, and when an `llm_classifier` route
+ that selects escalation mode through its `escalation` table carries capability
+ keys that mode ignores.
+
## [0.3.0]
Switchyard 0.3.0 builds on the native server and Rust library introduced in
diff --git a/crates/libsy/src/algorithms/stage.rs b/crates/libsy/src/algorithms/stage.rs
index 6dfc5a135..ebfb4f374 100644
--- a/crates/libsy/src/algorithms/stage.rs
+++ b/crates/libsy/src/algorithms/stage.rs
@@ -19,6 +19,7 @@ use async_trait::async_trait;
use super::fall_through::FallThrough;
use super::llm_class::{LlmClassifierConfig, LlmTaskClassifier, TaskClassifierConfig};
+use super::util::affinity::ClassifyTrigger;
use super::util::prompts::prepend_system_prompt;
use super::util::stage::{
DecisionSource, HandoffNoteConfig, PickerMode, StageClassifier, Tier, fall_open_tier,
@@ -187,8 +188,21 @@ impl StageRouter {
/// Routes between the runtime `capable` and `efficient` models. The judge,
/// when configured, is called through the runtime `judge` model.
///
- /// Errors if either threshold in `config` is outside `[0.0, 1.0]`.
+ /// Errors if either threshold in `config` is outside `[0.0, 1.0]`, or if the
+ /// fallback judge is asked to retain verdicts: it runs on every ambiguous
+ /// turn, so `classify_trigger` and `message_hash_fallback` would do nothing.
pub fn new(config: StageRouterConfig) -> Result<Self> {
+ if let Some(fallback) = &config.llm_fallback
+ && (fallback.config.classify_trigger != ClassifyTrigger::EveryRequest
+ || fallback.config.message_hash_fallback)
+ {
+ return Err(LibsyError::AlgorithmError {
+ message: "classify_trigger and message_hash_fallback have no effect on the \
+ stage fallback judge, which runs on every ambiguous turn; remove \
+ them, or use a composite route to judge once per user turn"
+ .to_string(),
+ });
+ }
Ok(Self {
route: build_stage_route(config)?,
})
@@ -273,7 +287,6 @@ mod tests {
use parking_lot::Mutex;
use super::*;
- use crate::algorithms::util::affinity::ClassifyTrigger;
use crate::algorithms::util::stage::{clear_fall_open, set_fall_open};
use crate::algorithms::util::tier_fixtures::{JUDGE, Recorder, turn_request};
use crate::core::testing::test_drive_with_models;
@@ -406,6 +419,31 @@ mod tests {
));
}
+ #[test]
+ fn rejects_retaining_triggers_on_the_fallback_judge() {
+ for (trigger, message_hash_fallback) in [
+ (ClassifyTrigger::UserTurn, false),
+ (ClassifyTrigger::NewSession, false),
+ (ClassifyTrigger::UserTurn, true),
+ ] {
+ let mut config = config();
+ config.llm_fallback = Some(LlmFallback {
+ config: TaskClassifierConfig {
+ classify_trigger: trigger,
+ message_hash_fallback,
+ ..Default::default()
+ },
+ });
+ assert!(
+ matches!(
+ StageRouter::new(config),
+ Err(LibsyError::AlgorithmError { .. })
+ ),
+ "{trigger:?} with message_hash_fallback = {message_hash_fallback}"
+ );
+ }
+ }
+
#[test]
fn builds() -> Result<()> {
let router = StageRouter::new(config())?;
@@ -628,37 +666,6 @@ mod tests {
Ok(())
}
- #[tokio::test]
- async fn the_fallback_judge_ignores_the_retaining_triggers() -> Result<()> {
- for trigger in [ClassifyTrigger::UserTurn, ClassifyTrigger::NewSession] {
- let recorder = Arc::new(Recorder::default());
- let mut config = config_with_judge(&recorder, 0.9);
- config
- .llm_fallback
- .as_mut()
- .expect("config_with_judge sets a judge")
- .config
- .classify_trigger = trigger;
- let router = recording_router(config)?;
-
- // Two tool continuations of one user turn in one session. A standalone
- // capability route with this trigger would judge once and retain the
- // target; inside the stage cascade the judge is a plain classifier.
- for _ in 0..2 {
- test_drive_with_models(
- router.clone(),
- turn_request(false),
- runtime_models(),
- recorder.serve(),
- )
- .await?;
- }
-
- assert_eq!(recorder.judge_calls(), 2, "{trigger:?} retained a verdict");
- }
- Ok(())
- }
-
#[tokio::test]
async fn a_judge_that_cannot_tell_lands_on_the_picker_default() -> Result<()> {
let recorder = Arc::new(Recorder::default());
diff --git a/crates/switchyard-runner/src/algorithm.rs b/crates/switchyard-runner/src/algorithm.rs
index b2f141ae2..9729bcde3 100644
--- a/crates/switchyard-runner/src/algorithm.rs
+++ b/crates/switchyard-runner/src/algorithm.rs
@@ -598,9 +598,9 @@ pub struct StageClassifierConfig {
/// for an uncertain verdict and twice for unsupported.
#[serde(default)]
pub threshold_step: Option<f64>,
- /// How often the judge runs. Composite honors it. On a standalone stage route
- /// it has no effect: the fallback judge runs on every ambiguous turn and
- /// retains no verdict.
+ /// How often the judge runs. Composite honors it. A standalone stage route
+ /// rejects any value other than `every_request`, because its fallback judge
+ /// runs on every ambiguous turn and retains no verdict.
#[serde(default)]
pub classify_trigger: ClassifyTrigger,
/// Reuses the session's target by hashing the first user message when no
@@ -654,6 +654,13 @@ pub struct StageTierConfig {
pub handoff_notes: Option<HandoffNoteConfig>,
}
+impl StageTierConfig {
+ /// Warns when both tiers name one target; see [`warn_identical_tiers`].
+ fn warn_identical_tiers(&self, route_name: &str) {
+ warn_identical_tiers(route_name, &self.capable_target, &self.efficient_target);
+ }
+}
+
impl StageClassifierConfig {
fn task_classifier_config(
&self,
@@ -1249,16 +1256,23 @@ impl LlmClassifierRouteConfig {
"llm_classifier route {route_name} mode escalation cannot use classify_trigger"
)));
}
- if mode.is_some()
- && (base_threshold.is_some()
- || threshold_step.is_some()
- || *message_hash_fallback
- || recent_turn_window.is_some())
- {
+ let has_capability_settings = base_threshold.is_some()
+ || threshold_step.is_some()
+ || fail_open.is_some()
+ || *message_hash_fallback
+ || recent_turn_window.is_some();
+ if mode.is_some() && has_capability_settings {
return Err(AlgorithmConfigError::new(format!(
"llm_classifier route {route_name} mode escalation cannot use capability routing settings"
)));
}
+ if has_capability_settings {
+ // Existing files that select escalation through the `escalation` table alone
+ // keep loading, but their capability keys do nothing in this mode.
+ tracing::warn!(
+ "llm_classifier route {route_name} runs in escalation mode because it has an escalation table; base_threshold, threshold_step, fail_open, recent_turn_window and message_hash_fallback are ignored in this mode. Set mode = \"escalation\" and remove them."
+ );
+ }
Ok(LlmClassifierModeConfig::Escalation(
EscalationClassifierRouteConfig {
classifier_target: classifier_target.clone(),
@@ -1591,6 +1605,7 @@ fn build_algorithm(
"stage_router route {route_name} uses picker \"capable_first\", which is experimental: published thresholds and routing results all come from \"efficient_first\", so there is no calibrated confidence_threshold for it and no measured accuracy or cost. Use \"efficient_first\" unless you are running your own calibration."
);
}
+ tiers.warn_identical_tiers(route_name);
let threshold = confidence_threshold.ok_or_else(|| {
AlgorithmConfigError::new(format!(
"stage_router route {route_name}: confidence_threshold is required"
@@ -1623,7 +1638,11 @@ fn build_algorithm(
let parent: Arc<dyn Algorithm> = Arc::new(algorithm);
attach_subagent_router(route_name, parent, subagents.as_ref(), targets)
}
- AlgorithmSpec::Auto { .. } => {
+ AlgorithmSpec::Auto {
+ capable_target,
+ efficient_target,
+ } => {
+ warn_identical_tiers(route_name, capable_target, efficient_target);
let config = StageRouterConfig::new(PickerMode::EfficientFirst, 0.5);
let algorithm = StageRouter::new(config).map_err(|error| {
AlgorithmConfigError::with_source(
@@ -1638,6 +1657,7 @@ fn build_algorithm(
stage,
subagents,
} => {
+ stage.warn_identical_tiers(route_name);
let mut stage_config = StageRouterConfig::new(
PickerMode::EfficientFirst,
stage.confidence_threshold.unwrap_or(0.5),
@@ -1846,6 +1866,19 @@ fn default_classifier_max_output_tokens() -> u64 {
LlmCapabilityConfig::default().max_output_tokens
}
+/// Both tiers on one target leave the route nothing to choose between; every decision
+/// lands on the same model. The file still loads, since a single-model deployment can
+/// be a deliberate overhead measurement.
+fn warn_identical_tiers(route_name: &str, capable_target: &str, efficient_target: &str) {
+ if capable_target == efficient_target {
+ tracing::warn!(
+ route = route_name,
+ target = capable_target,
+ "capable_target and efficient_target name the same target, so every request goes to one model; use passthrough for a single model, or point the tiers at two targets"
+ );
+ }
+}
+
fn warn_single_target_classifier(route_name: &str, models: &CategoryModelConfig) {
if models.routing_names().len() < 2 {
tracing::warn!(
diff --git a/crates/switchyard-runner/tests/classifier_modes.rs b/crates/switchyard-runner/tests/classifier_modes.rs
new file mode 100644
index 000000000..0a9b938e4
--- /dev/null
+++ b/crates/switchyard-runner/tests/classifier_modes.rs
@@ -0,0 +1,62 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+//! Capability-only keys on an escalation route: rejected when the mode is spelled
+//! out, accepted (and ignored, with a warning) when the escalation table selects
+//! the mode on its own.
+
+use switchyard_runner::Runner;
+
+const CONFIG: &str = r#"
+schema_version = 1
+[llm_clients.local]
+format = "openai_chat"
+base_url = "http://127.0.0.1:1/v1"
+[targets.fast]
+id = "fast-model"
+llm_client = "local"
+[targets.strong]
+id = "strong-model"
+llm_client = "local"
+[targets.judge]
+id = "judge-model"
+llm_client = "local"
+[routes.agent]
+id = "agent"
+type = "llm_classifier"
+classifier_target = "judge"
+strong_target = "strong"
+weak_target = "fast"
+escalation = { confirmations = 2 }
+"#;
+
+const CAPABILITY_KEYS: [&str; 5] = [
+ "base_threshold = 0.5",
+ "threshold_step = 0.1",
+ "fail_open = false",
+ "recent_turn_window = 4",
+ "message_hash_fallback = true",
+];
+
+#[test]
+fn implicit_escalation_mode_keeps_loading_with_capability_keys() {
+ for setting in CAPABILITY_KEYS {
+ let config = format!("{CONFIG}{setting}\n");
+ assert!(Runner::from_toml(&config).is_ok(), "{setting}");
+ }
+}
+
+#[test]
+fn explicit_escalation_mode_rejects_capability_keys() {
+ for setting in CAPABILITY_KEYS {
+ let config = format!("{CONFIG}mode = \"escalation\"\n{setting}\n");
+ let error = match Runner::from_toml(&config) {
+ Ok(_) => panic!("expected a load error for {setting}"),
+ Err(error) => error.to_string(),
+ };
+ assert!(
+ error.contains("mode escalation cannot use capability routing settings"),
+ "{setting}: {error}"
+ );
+ }
+}
diff --git a/crates/switchyard-runner/tests/stage_knobs.rs b/crates/switchyard-runner/tests/stage_knobs.rs
new file mode 100644
index 000000000..d660eab1c
--- /dev/null
+++ b/crates/switchyard-runner/tests/stage_knobs.rs
@@ -0,0 +1,60 @@
+// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+// SPDX-License-Identifier: Apache-2.0
+
+//! Stage fallback-judge knobs that cannot take effect are rejected at load.
+
+use switchyard_runner::Runner;
+
+const CONFIG: &str = r#"
+schema_version = 1
+[llm_clients.local]
+format = "openai_chat"
+base_url = "http://127.0.0.1:1/v1"
+[targets.fast]
+id = "fast-model"
+llm_client = "local"
+[targets.strong]
+id = "strong-model"
+llm_client = "local"
+[targets.judge]
+id = "judge-model"
+llm_client = "local"
+[routes.stage]
+id = "stage"
+type = "stage_router"
+capable_target = "strong"
+efficient_target = "fast"
+picker = "efficient_first"
+confidence_threshold = 0.5
+"#;
+
+const CLASSIFIER: &str = "[routes.stage.classifier]\ntarget = \"judge\"\nbase_threshold = 0.5\n";
+
+fn load_error(config: &str) -> String {
+ match Runner::from_toml(config) {
+ Ok(_) => panic!("expected a load error for:\n{config}"),
+ Err(error) => error.to_string(),
+ }
+}
+
+#[test]
+fn accepts_the_default_classifier_trigger() {
+ assert!(Runner::from_toml(&format!("{CONFIG}{CLASSIFIER}")).is_ok());
+ let explicit = format!("{CONFIG}{CLASSIFIER}classify_trigger = \"every_request\"\n");
+ assert!(Runner::from_toml(&explicit).is_ok());
+}
+
+#[test]
+fn rejects_classifier_knobs_that_have_no_effect() {
+ for setting in [
+ "classify_trigger = \"user_turn\"",
+ "classify_trigger = \"new_session\"",
+ "message_hash_fallback = true",
+ ] {
+ let error = load_error(&format!("{CONFIG}{CLASSIFIER}{setting}\n"));
+ assert!(
+ error.contains("have no effect on the stage fallback judge"),
+ "{setting}: {error}"
+ );
+ }
+}
diff --git a/docs/operations/troubleshooting.md b/docs/operations/troubleshooting.md
index bac4a1955..3d54ae117 100644
--- a/docs/operations/troubleshooting.md
+++ b/docs/operations/troubleshooting.md
@@ -17,7 +17,7 @@ log field, a log line, or a `--dry-run` message.
| HTTP 400 from the upstream when the client sends a thinking level | The upstream error body in the response | A Claude target on an OpenAI wire format behind a gateway. | [Thinking](../integrations/pi.md#thinking) and `omit_body_fields`. |
| HTTP 400 with code `context_length_exceeded` | The response body | Every target on the route overflowed. | [Context-Window Handling](context_window.md). |
| HTTP 400 on requests that contain images | The response body | The route sets `vision = false`. | Remove it, or declare `true` only when every target accepts images. See the [route keys](../reference/toml_schema.md#routesname). |
-| The server does not start: `unknown field` | The `--dry-run` message | An `auto` route with a tuning key; a typo; a key from another route type. | [Auto](../reference/toml_schema.md#auto) and the [validation errors](../reference/toml_schema.md#validation-errors). |
+| The server does not start: `unknown field`, or `have no effect on the stage fallback judge` | The `--dry-run` message | An `auto` route with a tuning key; a typo; a key from another route type; `classifier.classify_trigger` other than the default, or `classifier.message_hash_fallback`, on a `stage_router` route. | [Auto](../reference/toml_schema.md#auto), the [stage_router keys](../reference/toml_schema.md#stage_router), and the [validation errors](../reference/toml_schema.md#validation-errors). |
| The server does not start: two targets `both name model` | The `--dry-run` message | Two targets share a model ID on one client with different `reasoning_effort`, `extra_body`, or `omit_body_fields`. | [Reasoning effort](../routing_algorithms/model_selection.md#reasoning-effort). |
| The client shows zero input tokens on `/v1/messages` | `prompt_tokens` in the routing log | The streamed `message_start` event reports zero usage when the target is on an OpenAI format. The real counts arrive in `message_delta`. | [Streamed token counts](../routing_algorithms/model_selection.md#pin-the-provider-and-endpoint). |
| Cost per run is higher than the single capable model | `cached_tokens` in the routing log on the turn after a model change; cost per arm | Each switch loses the prompt cache; the judge adds a call per turn; the efficient model is too weak and most turns escalate. | [Counter-examples](../routing_algorithms/model_selection.md#counter-examples) and the [inverted-pair check](../routing_algorithms/model_selection.md#check-for-an-inverted-pair). |
@@ -43,7 +43,6 @@ rows that return an error or hold a request open say so.
| A tier that refuses connections or returns HTTP 408, 429, or 5xx, with the default `max_retries = 2` | Each request retries that tier, then falls through to the route's other target, so one model serves everything, slowly. After the retries fail, a 5-second cooldown sends later requests straight to the other target. A tier that hangs behaves differently: without `timeout_ms` the request waits, and with it the request that reaches the deadline returns `504`. Only requests inside the cooldown that follows fall through. | Runtime log lines for the failed attempts and the fallback. | Fix the endpoint and watch `.models` call counts in `/v1/stats`. Set `timeout_ms` so a hang becomes a `504` instead of a wait. See [`[llm_clients.<name>]`](../reference/toml_schema.md#llm_clientsname). |
| A judge client without `timeout_ms` | A judge that accepts the request and never finishes holds the request open. | None. | Give the judge its own `[llm_clients]` entry with `timeout_ms`. See [Tuning options](../routing_algorithms/llm_classifier_routing.md#tuning-options). |
| `auto` or `stage_router` serving chat-only traffic | No tool results, so the signals abstain and every turn falls open to the efficient tier. | None. | Use Task or Composite. See [When routing will not help](../routing_algorithms/overview.md#when-routing-will-not-help). |
-| `classifier.classify_trigger = "user_turn"` or `"new_session"`, with or without `classifier.message_hash_fallback`, on a `stage_router` route | The stage fallback judge runs on every ambiguous turn and retains nothing, so the trigger changes nothing. The keys are accepted for compatibility. `message_hash_fallback` with the default trigger is a load error instead. | None. | Use a [composite](../routing_algorithms/composite_routing.md) route for a judge that runs once per user turn. |
| `capable_first_threshold` on a standalone `stage_router` route | Only a composite classifier selects the tier that uses it. | None. | Use `confidence_threshold`, or a composite route. See the [knob reference](../routing_algorithms/stage_router_routing.md#knob-reference). |
| A custom `llm_classifier` with one routing target | Every judge call is paid for and no choice is made. | Startup: `custom classifier has only one routing target; judge calls add cost without a routing choice`. | Add a second target, or use `passthrough`. |
diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.md
index f5f0b8e54..9497adca6 100644
--- a/docs/reference/toml_schema.md
+++ b/docs/reference/toml_schema.md
@@ -424,8 +424,8 @@ so it is consulted on every ambiguous turn and never retains a verdict.
| `classifier.threshold_step` | No | `0.0` | Finite, non-negative amount added once for uncertain or unmatched verdicts and twice for unsupported verdicts. `base_threshold + 2 * threshold_step` must be at most `1`. |
| `classifier.fail_open` | No | `true` | When `true`, judge client failures and deadlines let the turn fall open to the picker default and record fail-open evidence. When `false`, they stop the request. |
| `classifier.decision` | No | unset | Use a decision model instead of an LLM judge. The classifier target must be in `decision_targets`. It cannot be combined with `base_threshold`, `threshold_step`, `prompt`, `response_format_type`, or `max_output_tokens`; the load error is `decision cannot use LLM judge settings`. See [Decision model classifier](../routing_algorithms/composite_routing.md#decision-model-classifier). |
-| `classifier.classify_trigger` | No | `every_request` | Accepted for compatibility with `llm_classifier`. No effect on the stage fallback judge: it runs on every ambiguous turn and retains no verdict, so `user_turn` and `new_session` change nothing here. Use a [composite](#composite) route to judge once per user turn. |
-| `classifier.message_hash_fallback` | No | `false` | Accepted for compatibility. No effect on the stage fallback judge. Setting it with the default trigger is a load error. |
+| `classifier.classify_trigger` | No | `every_request` | Only the default is accepted here. The stage fallback judge runs on every ambiguous turn and retains no verdict, so `user_turn` and `new_session` are rejected at load. Use a [composite](#composite) route to judge once per user turn. |
+| `classifier.message_hash_fallback` | No | `false` | Rejected at load on a `stage_router` route. The fallback judge retains nothing to key on. |
| `classifier.recent_turn_window` | No | unset | When unset, the judge sees the opening task and the latest user follow-up. When set, it also sees that many trailing messages. This counts messages, while the route's `recent_turn_window` counts tool results, so the two values are different units. |
| `classifier.prompt` | No | packaged prompt | Replaces the capability judge prompt. The verdict schema is unchanged. |
| `classifier.response_format_type` | No | `json_schema` | Structured-output mode for the LLM judge. Use `json_object` when the provider does not support JSON Schema; Switchyard adds the schema to the prompt and validates the verdict locally. Not with `classifier.decision`. |
diff --git a/docs/routing_algorithms/stage_router_routing.md b/docs/routing_algorithms/stage_router_routing.md
index 234b677fa..80edf1277 100644
--- a/docs/routing_algorithms/stage_router_routing.md
+++ b/docs/routing_algorithms/stage_router_routing.md
@@ -243,8 +243,8 @@ required columns.
| `classifier.threshold_step` | `0.0` | Added once for uncertain or unmatched verdicts, twice for unsupported. |
| `classifier.fail_open` | `true` | On judge failure, fall open to the picker default instead of failing the request. |
| `classifier.decision` | unset | Use a decision model instead of an LLM judge. Excludes the LLM judge keys below. |
-| `classifier.classify_trigger` | `every_request` | No effect here. The judge runs on every ambiguous turn. |
-| `classifier.message_hash_fallback` | `false` | No effect here. A load error with the default trigger. |
+| `classifier.classify_trigger` | `every_request` | Only the default is accepted. The judge runs on every ambiguous turn. |
+| `classifier.message_hash_fallback` | `false` | Rejected at load. The judge retains no verdict. |
| `classifier.recent_turn_window` | unset | Trailing messages the judge sees after the opening task. A different unit from the route's `recent_turn_window`, which counts tool results. |
| `classifier.prompt` | packaged | Replaces the judge prompt. |
| `classifier.response_format_type` | `json_schema` | `json_object` for providers without JSON Schema. |