Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
feat(stage): add efficient holds and pinned tier routing (#1160)
sabhatinas5f5a44f004bc
diff --git a/bindings/typescript/README.md b/bindings/typescript/README.mdindex 9c3306c0a..f142cb173 100644--- a/bindings/typescript/README.md+++ b/bindings/typescript/README.md@@ -4,6 +4,10 @@ TypeScript bindings for composite routing. Rust builds a Decision request and usthe caller's answer with local tool signals to select a model. The caller handlesclassifier calls, completion requests, and branch state.+Capable-first routing defaults to pinning the efficient tier after switching to+it. Set `stage.efficient_hold_turns: 0` to disable this hold. Efficient-first+defaults to no efficient hold. Persist the returned state to retain holds.+Requires Node.js 22.19 or newer and the repository's Rust toolchain. The packageis private. Its native addon is built for the local host. Linux is covered in CI.diff --git a/bindings/typescript/src/index.ts b/bindings/typescript/src/index.tsindex e79bd4740..66bcab4b9 100644--- a/bindings/typescript/src/index.ts+++ b/bindings/typescript/src/index.ts@@ -18,7 +18,12 @@ export interface CompositeOptions {instructions?: unknown;};};- stage: { efficient_first_threshold: number; capable_first_threshold: number };+ stage: {+ efficient_first_threshold: number;+ capable_first_threshold: number;+ /** Defaults to -1 for capable-first, otherwise 0. Set 0 to disable the hold. */+ efficient_hold_turns?: number;+ };}export interface Selection {diff --git a/bindings/typescript/test/composite.test.mjs b/bindings/typescript/test/composite.test.mjsindex 7d6a68252..b2c888b83 100644--- a/bindings/typescript/test/composite.test.mjs+++ b/bindings/typescript/test/composite.test.mjs@@ -64,7 +64,10 @@ test("decision request, cutoff, and branch-owned state", async () => {assert.deepEqual(await router.route(continuation, { state, decide: unexpectedDecision }), { model });assert.deepEqual(state, saved);}- assert.deepEqual(await router.route(continuation, { decide: unexpectedDecision }), { model: options.targets.efficient });+ assert.deepEqual(await router.route(continuation, { decide: unexpectedDecision }), {+ model: options.targets.efficient,+ state: { stage_previous_tier: { string: "weak" } },+ });const once = structuredClone(options);once.classifier.classify_trigger = "new_session";@@ -73,12 +76,59 @@ test("decision request, cutoff, and branch-owned state", async () => {assert.deepEqual(await sessionRouter.route(messages, { state, decide: unexpectedDecision }), { model: options.targets.capable });});+test("serialized state preserves capable holds until they expire", async () => {+ const router = CompositeRouter.create(options);+ let selected = await router.route(continuation, { decide: unexpectedDecision });+ assert.equal(selected.model, options.targets.efficient);++ const failed = structuredClone(continuation);+ failed.at(-1).content[0].is_error = true;+ failed.at(-1).content[0].content[0].text = "Out of memory: kill process 1234";+ const requests = [failed, continuation, continuation, continuation];+ const expected = [options.targets.capable, options.targets.capable, options.targets.capable, options.targets.efficient];+ for (const [i, request] of requests.entries()) {+ assert.ok(selected.state);+ const state = JSON.parse(JSON.stringify(selected.state));+ const saved = structuredClone(state);+ selected = await router.route(request, { state, decide: unexpectedDecision });+ assert.equal(selected.model, expected[i]);+ assert.deepEqual(state, saved);+ // A separate branch without the returned state has no hold.+ assert.equal((await router.route(continuation, { decide: unexpectedDecision })).model, options.targets.efficient);+ }+});++test("capable-first pins efficient by default and accepts an explicit opt-out", async () => {+ for (const turns of [undefined, 0]) {+ const config = structuredClone(options);+ config.stage.efficient_hold_turns = turns;+ const router = CompositeRouter.create(config);+ const initial = await router.route(messages, { decide: async () => response(0.41) });+ assert.equal(initial.model, options.targets.capable);+ const write = structuredClone(continuation);+ write[1].content[0].name = "write";+ let selected = await router.route(write, { state: initial.state, decide: unexpectedDecision });+ assert.equal(selected.model, options.targets.efficient);+ const failed = structuredClone(continuation);+ failed.at(-1).content[0].is_error = true;+ failed.at(-1).content[0].content[0].text = "Out of memory: kill process 1234";+ let state = selected.state;+ for (let i = 0; i < 3; i++) {+ assert.ok(state);+ selected = await router.route(failed, { state: JSON.parse(JSON.stringify(state)), decide: unexpectedDecision });+ assert.equal(selected.model, turns === 0 ? options.targets.capable : options.targets.efficient);+ state = selected.state ?? state;+ }+ }+});+test("validation and decision failure policy", async () => {for (const change of [(config) => { config.targets.efficient = config.targets.capable; },(config) => { config.classifier.decision.cutoff = 2; },(config) => { config.classifier.decision.extra = true; },(config) => { config.classifier.classify_trigger = "every_request"; },+ (config) => { config.stage.efficient_hold_turns = -2; },]) {const config = structuredClone(options);change(config);@@ -104,7 +154,10 @@ test("validation and decision failure policy", async () => {const fallback = CompositeRouter.create(config);const { state } = await fallback.route(messages, { decide: async () => response(0.41) });assert.deepEqual(await fallback.route(messages, { state, decide }), { model: options.targets.capable });- assert.deepEqual(await fallback.route(messages, { decide }), { model: options.targets.efficient });+ assert.deepEqual(await fallback.route(messages, { decide }), {+ model: options.targets.efficient,+ state: { stage_previous_tier: { string: "weak" } },+ });assert.deepEqual(await router.route(messages, { state, decide: async () => ({ answers: {} }) }), { model: options.targets.capable });});diff --git a/crates/libsy/src/algorithms/composite.rs b/crates/libsy/src/algorithms/composite.rsindex 1ce565dff..56d7faa85 100644--- a/crates/libsy/src/algorithms/composite.rs+++ b/crates/libsy/src/algorithms/composite.rs@@ -264,7 +264,77 @@ mod tests {router_with_trigger(ClassifyTrigger::UserTurn)}+ #[tokio::test]+ async fn capable_first_defaults_to_pinning_efficient_after_a_switch() -> Result<()> {+ for capable_first in [false, true] {+ for turns in [None, Some(0), Some(1), Some(-1)] {+ let mut stage = StageRouterConfig::new(PickerMode::EfficientFirst, 0.5);+ stage.efficient_hold_turns = turns;+ let router = router_with_stage(ClassifyTrigger::UserTurn, stage)?;+ let recorder = Arc::new(Recorder::default());+ *recorder.judge_p_solve.lock() = if capable_first { 0.0 } else { 1.0 };+ let mut initial = turn_request(false);+ initial.llm_request.messages.truncate(1);+ let mut write = turn_request(false);+ if let switchyard_protocol::ContentBlock::ToolCall(call) =+ &mut write.llm_request.messages[1].content[0]+ {+ call.name = "Write".into();+ }+ for request in [+ initial,+ turn_request(true),+ write.clone(),+ write.clone(),+ write,+ turn_request(true),+ turn_request(true),+ turn_request(true),+ ] {+ test_drive_with_models(+ router.clone(),+ request,+ runtime_models(),+ recorder.serve(),+ )+ .await?;+ }+ let actual: Vec<_> = recorder+ .routed()+ .into_iter()+ .map(|call| call.target)+ .collect();+ let initial = if capable_first { "strong" } else { "weak" };+ let mut expected = vec![initial, "strong", "strong", "strong", "weak"];+ let count = turns.unwrap_or(if capable_first { -1 } else { 0 });+ expected.extend((0..3).map(|i| {+ if count == -1 || i < count {+ "weak"+ } else {+ "strong"+ }+ }));+ assert_eq!(+ actual, expected,+ "capable_first={capable_first}, turns={turns:?}"+ );+ assert_eq!(recorder.judge_calls(), 1);+ }+ }+ Ok(())+ }+fn router_with_trigger(classify_trigger: ClassifyTrigger) -> Result<Arc<CompositeRouter>> {+ router_with_stage(+ classify_trigger,+ StageRouterConfig::new(PickerMode::EfficientFirst, 0.5),+ )+ }++ fn router_with_stage(+ classify_trigger: ClassifyTrigger,+ stage: StageRouterConfig,+ ) -> Result<Arc<CompositeRouter>> {Ok(Arc::new(CompositeRouter::new(CompositeRouterConfig {judge: CompositeJudgeConfig::Capability(TaskClassifierConfig {judge: CapabilityJudgeConfig::Llm(LlmCapabilityConfig {@@ -274,7 +344,7 @@ mod tests {classify_trigger,..Default::default()}),- stage: StageRouterConfig::new(PickerMode::EfficientFirst, 0.5),+ stage,})?))}diff --git a/crates/libsy/src/algorithms/stage.rs b/crates/libsy/src/algorithms/stage.rsindex ebfb4f374..df649dfad 100644--- a/crates/libsy/src/algorithms/stage.rs+++ b/crates/libsy/src/algorithms/stage.rs@@ -143,9 +143,13 @@ pub struct StageRouterConfig {pub recent_window: Option<usize>,/// Exact tool-name semantics added to the built-in coding vocabulary.pub tool_semantics: ToolSemantics,- /// Requests to keep on the capable tier after an escalation. A clean test- /// pass clears the hold early.- pub capable_hold_turns: u32,+ /// Requests to keep on capable after an escalation. `0` disables the hold;+ /// `-1` pins after switching to this tier.+ pub capable_hold_turns: i64,+ /// Requests to keep on efficient after a switch. `None` defaults to `-1`+ /// under composite's capable-first policy, otherwise `0`. `Some(0)` disables+ /// the hold; `Some(-1)` pins until the session state is cleared.+ pub efficient_hold_turns: Option<i64>,/// Note handed to the model on a signal-driven escalation, and on a/// hand-back to the efficient tier when a de-escalation note is configured.pub handoff_notes: Option<HandoffNoteConfig>,@@ -169,6 +173,7 @@ impl StageRouterConfig {recent_window: None,tool_semantics: ToolSemantics::default(),capable_hold_turns: 2,+ efficient_hold_turns: None,handoff_notes: None,capable_system_prompt: None,efficient_system_prompt: None,@@ -189,7 +194,8 @@ impl StageRouter {/// when configured, is called through the runtime `judge` model.////// Errors if either threshold in `config` is outside `[0.0, 1.0]`, or if the- /// fallback judge is asked to retain verdicts: it runs on every ambiguous+ /// hold counts are outside `-1..=u32::MAX`. Also rejects a+ /// fallback judge configured to retain verdicts: it runs on every ambiguous/// turn, so `classify_trigger` and `message_hash_fallback` would do nothing.pub fn new(config: StageRouterConfig) -> Result<Self> {if let Some(fallback) = &config.llm_fallback@@ -238,12 +244,29 @@ pub(crate) fn build_stage_route(config: StageRouterConfig) -> Result<FallThrough}}config.tool_semantics.validate()?;+ for (name, turns) in [+ ("capable_hold_turns", Some(config.capable_hold_turns)),+ ("efficient_hold_turns", config.efficient_hold_turns),+ ] {+ let Some(turns) = turns else { continue };+ if !(-1..=i64::from(u32::MAX)).contains(&turns) {+ return Err(LibsyError::AlgorithmError {+ message: format!(+ "{name} must be -1 or between 0 and {}, got {turns}",+ u32::MAX+ ),+ });+ }+ }let default_tier = config.mode.default_tier();let fall_open = FallOpen { default_tier };let mut classifier = StageClassifier::new(config.mode, config.confidence_threshold).with_capable_first_threshold(config.capable_first_threshold).with_capable_hold_turns(config.capable_hold_turns);+ if let Some(turns) = config.efficient_hold_turns {+ classifier = classifier.with_efficient_hold_turns(turns);+ }if let Some(notes) = config.handoff_notes {classifier = classifier.with_handoff_notes(notes);}@@ -251,10 +274,11 @@ pub(crate) fn build_stage_route(config: StageRouterConfig) -> Result<FallThroughrecent_window: config.recent_window.unwrap_or(DEFAULT_RECENT_WINDOW),tool_semantics: config.tool_semantics,};+ let classifier = Arc::new(classifier.with_signals(signals));let mut router = FallThrough::new_with_state().with_name(STAGE_ROUTER)- .with_processor(Arc::new(signals))- .with_classifier(Arc::new(classifier));+ .with_processor(classifier.clone())+ .with_classifier(classifier);if let Some(fallback) = config.llm_fallback {router = router.with_classifier(Arc::new(SourceStamp {inner: Arc::new(LlmTaskClassifier::new(LlmClassifierConfig::Capability {@@ -451,6 +475,213 @@ mod tests {Ok(())}+ #[test]+ fn hold_turns_accept_pins_and_reject_invalid_counts() {+ for capable in [true, false] {+ for turns in [-2, -1, 0, 1, i64::from(u32::MAX), i64::from(u32::MAX) + 1] {+ let mut config = config();+ if capable {+ config.capable_hold_turns = turns;+ } else {+ config.efficient_hold_turns = Some(turns);+ }+ assert_eq!(+ StageRouter::new(config).is_ok(),+ (-1..=i64::from(u32::MAX)).contains(&turns)+ );+ }+ }+ }++ /// A clean write selects efficient without ending finite capable holds early.+ fn production_request() -> Request {+ let mut request = turn_request(false);+ if let switchyard_protocol::ContentBlock::ToolCall(call) =+ &mut request.llm_request.messages[1].content[0]+ {+ call.name = "Write".to_string();+ }+ request+ }++ #[tokio::test]+ async fn standalone_pickers_default_to_no_efficient_hold() -> Result<()> {+ for mode in [PickerMode::EfficientFirst, PickerMode::CapableFirst] {+ let mut config = StageRouterConfig::new(mode, 0.5);+ config.capable_hold_turns = 0;+ let router = recording_router(config)?;+ let recorder = Arc::new(Recorder::default());+ for request in [turn_request(true), production_request(), turn_request(true)] {+ test_drive_with_models(router.clone(), request, runtime_models(), recorder.serve())+ .await?;+ }+ let actual: Vec<_> = recorder+ .routed()+ .into_iter()+ .map(|call| call.target)+ .collect();+ assert_eq!(actual, ["strong", "weak", "strong"]);+ }+ Ok(())+ }++ /// Both pickers honor both holds; the switch request is not one of the held turns.+ #[tokio::test]+ async fn holds_expire_or_pin_after_switching_in_both_pickers() -> Result<()> {+ for mode in [PickerMode::EfficientFirst, PickerMode::CapableFirst] {+ for capable in [true, false] {+ for turns in [0, 1, 2, -1] {+ let mut config = StageRouterConfig::new(mode, 0.5);+ config.capable_hold_turns = if capable { turns } else { 0 };+ config.efficient_hold_turns = Some(if capable { 0 } else { turns });+ let router = recording_router(config)?;+ let recorder = Arc::new(Recorder::default());+ let (switch, opposing, held_target, other_target) = if capable {+ (turn_request(true), production_request(), "strong", "weak")+ } else {+ (production_request(), turn_request(true), "weak", "strong")+ };+ let mut requests = vec![opposing.clone(), switch];+ requests.extend(std::iter::repeat_n(opposing, 4));+ for request in requests {+ test_drive_with_models(+ router.clone(),+ request,+ runtime_models(),+ recorder.serve(),+ )+ .await?;+ }+ let actual: Vec<_> = recorder+ .routed()+ .into_iter()+ .map(|call| call.target)+ .collect();+ let mut expected = vec![other_target, held_target];+ expected.extend((0..4).map(|i| {+ if turns == -1 || i < turns {+ held_target+ } else {+ other_target+ }+ }));+ assert_eq!(+ actual, expected,+ "{mode:?}, capable={capable}, turns={turns}"+ );+ }+ }+ }+ Ok(())+ }++ /// Initial selection does not pin. Once switched, pins survive missing signals,+ /// passing tests, and critical errors, and remain scoped to one session/agent.+ #[tokio::test]+ async fn pins_survive_signals_and_remain_isolated() -> Result<()> {+ for mode in [PickerMode::EfficientFirst, PickerMode::CapableFirst] {+ for capable in [true, false] {+ let mut config = StageRouterConfig::new(mode, 0.5);+ config.capable_hold_turns = -1;+ config.efficient_hold_turns = Some(-1);+ let router = recording_router(config)?;+ let recorder = Arc::new(Recorder::default());+ let (initial, switch, held, other) = if capable {+ (production_request(), turn_request(true), "strong", "weak")+ } else {+ (turn_request(true), production_request(), "weak", "strong")+ };+ let mut no_signals = turn_request(false);+ no_signals.llm_request.messages.truncate(1);+ let mut passing = turn_request(false);+ if let switchyard_protocol::ContentBlock::ToolResult(result) =+ &mut passing.llm_request.messages[2].content[0]+ {+ result.content = vec![switchyard_protocol::ContentBlock::Text {+ text: "test result: ok. 1 passed; 0 failed".into(),+ }];+ }+ let mut child = initial.clone();+ child.metadata.as_mut().unwrap().agent_id = Some("child".into());+ let mut other_session = initial.clone();+ other_session.metadata.as_mut().unwrap().session_id = Some("other".into());+ let mut final_request = initial.clone();+ final_request.metadata.as_mut().unwrap().session_final = Some(true);+ let requests = [+ initial.clone(),+ switch,+ no_signals,+ passing,+ initial.clone(),+ child,+ other_session,+ final_request,+ initial,+ ];+ for request in requests {+ test_drive_with_models(+ router.clone(),+ request,+ runtime_models(),+ recorder.serve(),+ )+ .await?;+ }+ let actual: Vec<_> = recorder+ .routed()+ .into_iter()+ .map(|call| call.target)+ .collect();+ assert_eq!(+ actual,+ [other, held, held, held, held, other, other, held, other]+ );+ }+ }+ Ok(())+ }++ /// Judge-driven switches must arm the same pin as signal-driven switches.+ #[tokio::test]+ async fn judge_switches_arm_pins_and_held_turns_skip_the_judge() -> Result<()> {+ for capable in [true, false] {+ let recorder = Arc::new(Recorder::default());+ let mut config = config_with_judge(&recorder, if capable { 1.0 } else { 0.0 });+ config.capable_hold_turns = -1;+ config.efficient_hold_turns = Some(-1);+ let router = recording_router(config)?;+ for p_solve in [+ if capable { 1.0 } else { 0.0 },+ if capable { 0.0 } else { 1.0 },+ if capable { 1.0 } else { 0.0 },+ ] {+ *recorder.judge_p_solve.lock() = p_solve;+ test_drive_with_models(+ router.clone(),+ turn_request(false),+ runtime_models(),+ recorder.serve(),+ )+ .await?;+ }+ let actual: Vec<_> = recorder+ .routed()+ .into_iter()+ .map(|call| call.target)+ .collect();+ assert_eq!(+ actual,+ if capable {+ ["weak", "strong", "strong"]+ } else {+ ["strong", "weak", "weak"]+ }+ );+ assert_eq!(recorder.judge_calls(), 2);+ }+ Ok(())+ }+// ── routing integration tests ────────────────────────────────────────────const ESCALATION: &str = "the previous model was stalling; pick up the diagnosis";diff --git a/crates/libsy/src/algorithms/util/stage.rs b/crates/libsy/src/algorithms/util/stage.rsindex 76d688c6f..8f0d18231 100644--- a/crates/libsy/src/algorithms/util/stage.rs+++ b/crates/libsy/src/algorithms/util/stage.rs@@ -22,10 +22,11 @@ use opentelemetry::KeyValue;use serde::Deserialize;use super::prompts;-use super::tool_signals::ToolSignals;+use super::tool_signals::{ToolSignalProcessor, ToolSignals};use crate::Result;use crate::core::algorithm::Driver;use crate::core::classifier::{Classification, Classifier, Score};+use crate::core::processor::{Event, Processor};use crate::core::state::{State, StateValue};use crate::observability::meter;use switchyard_protocol::{Category, Request};@@ -53,6 +54,8 @@ const CLEAN_PRODUCTION_WEIGHT: f64 = 1.2;const SEVERITY_CRITICAL: f32 = 1.0;/// Session-state key for the remaining capable-tier recovery turns.const CAPABLE_HOLD_KEY: &str = "capable_hold_turns";+const EFFICIENT_HOLD_KEY: &str = "efficient_hold_turns";+const PREVIOUS_TIER_KEY: &str = "stage_previous_tier";/// Counts final stage-router choices by decision source and semantic target.const ROUTING_DECISIONS_METRIC: &str = "switchyard.stage_router.routing_decisions";@@ -158,6 +161,8 @@ pub enum DecisionSource {Override,/// A recent escalation is being held on the capable tier.CapableHold,+ /// A recent switch is being held on the efficient tier.+ EfficientHold,/// Scorer crossed `confidence_threshold`.Dimensions,/// Scorer was not confident, so the signals did not decide this turn.@@ -174,6 +179,7 @@ impl DecisionSource {match self {Self::Override => "override",Self::CapableHold => "capable_hold",+ Self::EfficientHold => "efficient_hold",Self::Dimensions => "dimensions",Self::Ambiguous => "ambiguous",Self::LlmClassifier => "llm-classifier",@@ -522,7 +528,9 @@ pub struct StageClassifier {mode: PickerMode,confidence_threshold: f64,capable_first_threshold: f64,- capable_hold_turns: u32,+ capable_hold_turns: i64,+ efficient_hold_turns: Option<i64>,+ signals: ToolSignalProcessor,handoff_notes: Option<HandoffNoteConfig>,}@@ -535,6 +543,8 @@ impl StageClassifier {confidence_threshold,capable_first_threshold: 0.0,capable_hold_turns: 2,+ efficient_hold_turns: None,+ signals: ToolSignalProcessor::default(),handoff_notes: None,}}@@ -545,13 +555,26 @@ impl StageClassifier {self}- /// Keep the capable tier for this many requests after an escalation. A- /// clean passing test clears the hold early. Set to zero to disable.- pub fn with_capable_hold_turns(mut self, turns: u32) -> Self {+ /// Keep capable for this many requests after an escalation. Zero disables+ /// it; `-1` pins after a switch. Validated by the enclosing stage router.+ pub fn with_capable_hold_turns(mut self, turns: i64) -> Self {self.capable_hold_turns = turns;self}+ /// Hold the efficient tier after a switch. Zero disables it; `-1` pins.+ /// Validated by the enclosing stage router.+ pub fn with_efficient_hold_turns(mut self, turns: i64) -> Self {+ self.efficient_hold_turns = Some(turns);+ self+ }++ /// Extract signals only on requests that need a new classification.+ pub(crate) fn with_signals(mut self, signals: ToolSignalProcessor) -> Self {+ self.signals = signals;+ self+ }+/// Hand the routed model a note on a signal-driven escalation, and on a/// hand-back to the efficient tier when a de-escalation note is configured.pub fn with_handoff_notes(mut self, config: HandoffNoteConfig) -> Self {@@ -574,18 +597,19 @@ impl StageClassifier {}}- fn capable_hold_key(request: &Request) -> String {+ fn agent_key(request: &Request, key: &str) -> String {request.metadata.as_ref().and_then(|metadata| metadata.agent_id.as_deref())- .map_or_else(- || CAPABLE_HOLD_KEY.to_string(),- |agent_id| format!("{CAPABLE_HOLD_KEY}:{agent_id}"),- )+ .map_or_else(|| key.to_string(), |agent_id| format!("{key}:{agent_id}"))}- fn consume_capable_hold(state: &mut State, key: &str) -> bool {+ fn consume_hold(state: &mut State, key: &str) -> bool {+ // Reuse the existing state values: a true flag is an unbounded hold.+ if matches!(state.extra.get(key), Some(StateValue::Bool(true))) {+ return true;+ }let Some(StateValue::Count(remaining)) = state.extra.get_mut(key) else {return false;};@@ -600,14 +624,80 @@ impl StageClassifier {true}- fn set_capable_hold(&self, state: &mut State, key: &str) {- if self.capable_hold_turns == 0 {- state.extra.remove(key);- } else {- state+ fn set_hold(state: &mut State, key: &str, turns: i64) {+ match turns {+ -1 => {+ state.extra.insert(key.to_string(), StateValue::Bool(true));+ }+ 1..=0xffff_ffff => {+ state+ .extra+ .insert(key.to_string(), StateValue::Count(turns as u32));+ }+ _ => {+ state.extra.remove(key);+ }+ }+ }+}++/// Observe the final cascade choice, including judge and fall-open decisions.+#[async_trait]+impl Processor for StageClassifier {+ async fn process(&self, state: &mut State, event: Event<'_>) -> Result<()> {+ if let Event::Request { request, driver } = event {+ let held = [CAPABLE_HOLD_KEY, EFFICIENT_HOLD_KEY].iter().any(|key| {+ match state.extra.get(&Self::agent_key(request, key)) {+ Some(StateValue::Bool(true)) => true,+ Some(StateValue::Count(remaining)) => *remaining > 0,+ _ => false,+ }+ });+ if held {+ state.tool_signals = None;+ return Ok(());+ }+ return self+ .signals+ .process(state, Event::Request { request, driver })+ .await;+ }+ if let Event::Decision {+ request, category, ..+ } = event+ {+ let (tier, hold_key, other_key, turns) = match category {+ Some(Category::Capable) => (+ Tier::Capable,+ CAPABLE_HOLD_KEY,+ EFFICIENT_HOLD_KEY,+ self.capable_hold_turns,+ ),+ Some(Category::Efficient) => (+ Tier::Efficient,+ EFFICIENT_HOLD_KEY,+ CAPABLE_HOLD_KEY,+ self.efficient_hold_turns.unwrap_or_else(|| {+ if fall_open_tier(state) == Some(Tier::Capable) {+ -1+ } else {+ 0+ }+ }),+ ),+ _ => return Ok(()),+ };+ let previous_key = Self::agent_key(request, PREVIOUS_TIER_KEY);+ let previous = state.extra- .insert(key.to_string(), StateValue::Count(self.capable_hold_turns));+ .insert(previous_key, StateValue::String(tier.label().to_string()));+ // The initial choice is not a switch. Held turns must not rearm themselves.+ if matches!(previous, Some(StateValue::String(ref label)) if label != tier.label()) {+ state.extra.remove(&Self::agent_key(request, other_key));+ Self::set_hold(state, &Self::agent_key(request, hold_key), turns);+ }}+ Ok(())}}@@ -619,27 +709,44 @@ impl Classifier for StageClassifier {request: &mut Request,driver: &Driver,) -> Result<(Classification, Option<switchyard_protocol::Response>)> {+ let capable_hold_key = Self::agent_key(request, CAPABLE_HOLD_KEY);+ for (key, category, source) in [+ (+ CAPABLE_HOLD_KEY,+ Category::Capable,+ DecisionSource::CapableHold,+ ),+ (+ EFFICIENT_HOLD_KEY,+ Category::Efficient,+ DecisionSource::EfficientHold,+ ),+ ] {+ if Self::consume_hold(state, &Self::agent_key(request, key)) {+ let target = driver.first_model_for(&category)?.clone();+ record_routing_decision(source, &target);+ driver.set_evidence(serde_json::json!({"source": source.as_str()}));+ return Ok((+ Classification::Scores(vec![Score {+ target,+ confidence: 1.0,+ category: Some(category),+ }]),+ None,+ ));+ }+ }let Some(signal) = state.tool_signals.clone() else {// No tool activity yet — nothing to score, so the signals have no// opinion, same as a below-threshold turn.return Ok((Self::abstain(), None));};-- let capable_hold_key = Self::capable_hold_key(request);- let clean_test_pass = signal.tests_passed && signal.no_error_streak > 0;- if clean_test_pass {- state.extra.remove(&capable_hold_key);- }let (mode, threshold) = match fall_open_tier(state) {Some(Tier::Capable) => (PickerMode::CapableFirst, self.capable_first_threshold),Some(Tier::Efficient) => (PickerMode::EfficientFirst, self.confidence_threshold),None => (self.mode, self.confidence_threshold),};- let outcome = if !clean_test_pass && Self::consume_capable_hold(state, &capable_hold_key) {- resolved(Tier::Capable, DecisionSource::CapableHold, 0.5, Some(1.0))- } else {- pick_tier(&signal, mode, threshold)- };+ let outcome = pick_tier(&signal, mode, threshold);record_score_metrics(&signal, &outcome);match outcome {PickOutcome::Resolved {@@ -654,12 +761,13 @@ impl Classifier for StageClassifier {};let target = driver.first_model_for(&category)?;if tier == Tier::Capable+ && self.capable_hold_turns > 0&& matches!(source,DecisionSource::Override | DecisionSource::Dimensions){- self.set_capable_hold(state, &capable_hold_key);+ Self::set_hold(state, &capable_hold_key, self.capable_hold_turns);}record_routing_decision(source, target);if source == DecisionSource::Override@@ -1144,7 +1252,7 @@ mod tests {}#[tokio::test]- async fn passing_tests_clear_the_capable_hold() -> Result<()> {+ async fn passing_tests_do_not_interrupt_the_capable_hold() -> Result<()> {let classifier = StageClassifier::new(PickerMode::EfficientFirst, 0.5);let mut state = state_with(critical());let driver = driver();@@ -1161,8 +1269,11 @@ mod tests {let classification = classifier.score(&mut state, &mut Request::default(), &driver).await?;- assert!(classification.0.argmax(false)?.is_none());- assert!(!state.extra.contains_key(CAPABLE_HOLD_KEY));+ assert_eq!(classification.0.argmax(false)?.unwrap().target, "strong");+ assert_eq!(+ state.extra.get(CAPABLE_HOLD_KEY),+ Some(&StateValue::Count(1))+ );Ok(())}@@ -1216,6 +1327,49 @@ mod tests {Ok(())}+ /// Holds skip signal extraction and notes, then resolve at full confidence.+ #[tokio::test]+ async fn held_decisions_have_full_confidence_without_tool_signals() -> Result<()> {+ for (key, target, category) in [+ (CAPABLE_HOLD_KEY, "strong", Category::Capable),+ (EFFICIENT_HOLD_KEY, "weak", Category::Efficient),+ ] {+ for value in [StateValue::Count(1), StateValue::Bool(true)] {+ let mut state = State::default();+ state.extra.insert(key.to_string(), value);+ let classifier = StageClassifier::new(PickerMode::EfficientFirst, 0.5)+ .with_handoff_notes(config(false));+ let mut request = crate::algorithms::util::tier_fixtures::turn_request(true);+ let original = request.llm_request.clone();+ let driver = driver();+ classifier+ .process(+ &mut state,+ Event::Request {+ request: &mut request,+ driver: &driver,+ },+ )+ .await?;+ assert!(+ state.tool_signals.is_none(),+ "held turns must not extract signals"+ );+ let (classification, _) =+ classifier.score(&mut state, &mut request, &driver).await?;+ let winner = classification.argmax(false)?.expect("a hold must resolve");+ assert_eq!(winner.target, target);+ assert_eq!(winner.category, Some(category.clone()));+ assert_eq!(winner.confidence, 1.0);+ assert_eq!(+ request.llm_request, original,+ "held turns must not add handoff notes"+ );+ }+ }+ Ok(())+ }+#[tokio::test]async fn classifier_falls_open_to_default_and_records_it() -> Result<()> {// A quiet signal corroborates neither axis, so the scorer abstains.diff --git a/crates/switchyard-node/src/lib.rs b/crates/switchyard-node/src/lib.rsindex 77d8408fa..ed41761ac 100644--- a/crates/switchyard-node/src/lib.rs+++ b/crates/switchyard-node/src/lib.rs@@ -80,6 +80,7 @@ enum Candidate {struct StageOptions {efficient_first_threshold: f64,capable_first_threshold: f64,+ efficient_hold_turns: Option<i64>,}#[napi]@@ -118,6 +119,7 @@ impl NativeComposite {let mut stage_config =StageRouterConfig::new(PickerMode::EfficientFirst, stage.efficient_first_threshold);stage_config.capable_first_threshold = stage.capable_first_threshold;+ stage_config.efficient_hold_turns = stage.efficient_hold_turns;let router = CompositeRouter::new(CompositeRouterConfig {judge: CompositeJudgeConfig::Capability(TaskClassifierConfig {judge: CapabilityJudgeConfig::Decision(DecisionJudgeConfig {diff --git a/crates/switchyard-runner/src/algorithm.rs b/crates/switchyard-runner/src/algorithm.rsindex 378f1d11a..83551b2f2 100644--- a/crates/switchyard-runner/src/algorithm.rs+++ b/crates/switchyard-runner/src/algorithm.rs@@ -654,7 +654,11 @@ pub struct StageTierConfig {pub tool_semantics: ToolSemantics,/// Requests to keep on the capable tier after an escalation.#[serde(default)]- pub capable_hold_turns: Option<u32>,+ pub capable_hold_turns: Option<i64>,+ /// Requests to keep on efficient after a switch. Defaults to `-1` for+ /// composite's capable-first policy, otherwise `0`. `-1` pins it.+ #[serde(default)]+ pub efficient_hold_turns: Option<i64>,/// Notes handed to a tier when the router switches to it.#[serde(default)]pub handoff_notes: Option<HandoffNoteConfig>,@@ -1612,6 +1616,7 @@ fn build_algorithm(recent_turn_window,tool_semantics,capable_hold_turns,+ efficient_hold_turns,handoff_notes,..} = tiers;@@ -1632,6 +1637,7 @@ fn build_algorithm(if let Some(turns) = capable_hold_turns {config.capable_hold_turns = *turns;}+ config.efficient_hold_turns = *efficient_hold_turns;config.tool_semantics = tool_semantics.clone();config.handoff_notes = handoff_notes.clone();// The judge is called through its own target, so it is not a routing@@ -1682,6 +1688,7 @@ fn build_algorithm(if let Some(turns) = stage.capable_hold_turns {stage_config.capable_hold_turns = turns;}+ stage_config.efficient_hold_turns = stage.efficient_hold_turns;stage_config.tool_semantics = stage.tool_semantics.clone();stage_config.handoff_notes = stage.handoff_notes.clone();let judge = if matches!(classifier.mode, Some(ClassifierMode::Custom)) {diff --git a/crates/switchyard-runner/src/config.rs b/crates/switchyard-runner/src/config.rsindex ca79eb453..bd21e7110 100644--- a/crates/switchyard-runner/src/config.rs+++ b/crates/switchyard-runner/src/config.rs@@ -1357,6 +1357,34 @@ new = ["send_message"])}+ #[test]+ fn stage_and_composite_validate_hold_turns() {+ for source in [+ stage_config(),+ stage_config().replace("efficient_first", "capable_first"),+ composite_config(),+ ] {+ for field in ["capable_hold_turns", "efficient_hold_turns"] {+ for (turns, valid) in [+ (-2_i64, false),+ (-1, true),+ (0, true),+ (3, true),+ (i64::from(u32::MAX), true),+ (i64::from(u32::MAX) + 1, false),+ ] {+ let configured =+ source.replace("capable_hold_turns = 2", &format!("{field} = {turns}"));+ let result = runner_from_toml(&configured);+ assert_eq!(result.is_ok(), valid, "{field} = {turns}");+ if let Err(error) = result {+ assert!(error.to_string().contains(field), "{error}");+ }+ }+ }+ }+ }+#[test]fn composite_decision_judge_reuses_capability_validation() {let source = composite_config().replace(diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.mdindex 93f20d66e..eeeb68214 100644--- a/docs/reference/toml_schema.md+++ b/docs/reference/toml_schema.md@@ -413,7 +413,8 @@ the signals work and for tuning.| `confidence_threshold` | Yes | — | Corroboration a decisive pick needs. In `[0, 1]`. `efficient_first_threshold` is accepted as an alias. || `capable_first_threshold` | No | `0.0` | Threshold used after an outer classifier selects the capable tier. Only a [composite](#composite) route sets that tier, so this key has no effect on a standalone `stage_router` route. || `recent_turn_window` | No | `3` | Trailing tool results the signals are computed over. |-| `capable_hold_turns` | No | `2` | Requests kept on the capable tier after an escalation the signals or a hard override picked. A clean test pass clears the hold early; `0` disables it. |+| `capable_hold_turns` | No | `2` | Requests kept on capable after a switch. Signal-driven capable decisions also start finite holds. Active holds skip signal processing and classification; `0` disables; `-1` pins after a switch. |+| `efficient_hold_turns` | No | `0` | Requests kept on efficient after a switch, even when errors appear. `0` disables; `-1` pins after a switch. || `tool_semantics.observe` | No | `[]` | Exact ASCII case-insensitive domain tool names that count as read-only investigation. || `tool_semantics.mutate` | No | `[]` | Exact ASCII case-insensitive domain tool names that count as state-changing production. || `tool_semantics.plan` | No | `[]` | Exact ASCII case-insensitive domain tool names that count as planning or task decomposition. |@@ -428,7 +429,7 @@ the signals work and for tuning.|---|:---:|---|---|| `handoff_notes.escalation_note` | Yes, when the table is present | — | Note appended to the request when the signals or a hard override escalate to the capable tier. || `handoff_notes.deescalation_note` | No | unset | Note appended to the request whenever the signals select the efficient tier, whether or not a capable turn came before. |-| `handoff_notes.only_on_wrong_signal_escalation` | No | `true` | When `true`, only the escalating turn itself carries the escalation note. `false` also adds it to the capable-hold turns that follow. |+| `handoff_notes.only_on_wrong_signal_escalation` | No | `true` | When `true`, restricts escalation notes to signal-driven capable decisions. Held turns receive no notes with either setting. |#### `classifier` keys@@ -521,7 +522,8 @@ confidence threshold. Tool signals still drive escalation and recovery. See| `stage.efficient_first_threshold` | No | `0.5` | Signal confidence required after an Efficient classification. In `[0, 1]`. Also accepts `confidence_threshold`. || `stage.capable_first_threshold` | No | `0.0` | Signal confidence required after a Capable classification. In `[0, 1]`. || `stage.recent_turn_window` | No | `3` | Trailing tool results the signals are computed over. |-| `stage.capable_hold_turns` | No | `2` | Requests kept on the capable tier after escalation. A clean test pass clears the hold early; `0` disables it. |+| `stage.capable_hold_turns` | No | `2` | Same capable hold as `stage_router`: `0` disables; `-1` pins after a switch. |+| `stage.efficient_hold_turns` | No | CF: `-1`; EF: `0` | Defaults to pinning efficient after a switch when the classifier selected capable-first. Explicit `0` disables; positive values hold that many requests. || `stage.tool_semantics.observe` | No | `[]` | Additional exact ASCII case-insensitive tool names that count as observation. || `stage.tool_semantics.mutate` | No | `[]` | Additional exact ASCII case-insensitive tool names that count as mutation. || `stage.tool_semantics.plan` | No | `[]` | Additional exact ASCII case-insensitive tool names that count as planning. |diff --git a/docs/routing_algorithms/composite_routing.md b/docs/routing_algorithms/composite_routing.mdindex 1e5d7ee09..366f85801 100644--- a/docs/routing_algorithms/composite_routing.md+++ b/docs/routing_algorithms/composite_routing.md@@ -17,6 +17,11 @@ during execution. Both algorithms and therelationship between them are fixed, and the configuration below names themdirectly rather than composing arbitrary algorithms.+Under capable-first, `stage.efficient_hold_turns` defaults to `-1`: after switching+to efficient, it stays there until session state is cleared. Efficient-first+defaults to `0`. Set this field explicitly to override either default; `0`+disables the hold, and a positive value holds that many subsequent requests.+The classifier can use an LLM judge or a decision model. A decisionmodel is configured through a decision client and target, separately from thecapable and efficient completion targets.diff --git a/docs/routing_algorithms/overview.md b/docs/routing_algorithms/overview.mdindex 9074eb4a4..f2e8a0a6a 100644--- a/docs/routing_algorithms/overview.md+++ b/docs/routing_algorithms/overview.md@@ -154,7 +154,7 @@ stage router with these settings. It does not compare strategies at runtime.| `picker` | `efficient_first` | The efficient tier serves a turn when the tool signals are not confident. || `confidence_threshold` | `0.5` | How strongly the tool signals must agree before they pick a tier on their own. || `recent_turn_window` | `3` | How many recent tool results the signals are computed over. |-| `capable_hold_turns` | `2` | How many turns stay on the capable tier after an escalation. A clean test pass ends the hold early. |+| `capable_hold_turns` | `2` | How many turns stay on the capable tier after an escalation, skipping signal processing and classification. || Classifier | none | No LLM judge is called. |Auto accepts no tuning keys. To change a setting, write the equivalentdiff --git a/docs/routing_algorithms/stage_router_routing.md b/docs/routing_algorithms/stage_router_routing.mdindex eac940e1f..c2a3f2b17 100644--- a/docs/routing_algorithms/stage_router_routing.md+++ b/docs/routing_algorithms/stage_router_routing.md@@ -20,7 +20,7 @@ A tool-using agent's run moves through stages that call for different amounts ofmodel capability. The built-in vocabulary is calibrated for coding agents: itrecognizes file observation, mutation, planning, shell activity, and test results.-For each LLM call, stage-router estimates the agent's state from the+For each LLM call without an active hold, stage-router estimates the agent's state from theconversation's **tool-result history**, scoring signals in two directions:- **RECOVERY → capable**: `severity` (windowed error severity), `spinning` (deep@@ -37,7 +37,7 @@ The axes are **corroborative**: the signed score is `tanh`-squashed to aconfidence in `[0, 1]`. One maxed scoring dimension produces about `0.46`;corroborating evidence pushes confidence decisively past a `0.5` threshold.Repeated failures, critical-error severity, and context compaction are hard-overrides to the capable tier. An active capable hold also bypasses the scorer.+overrides to the capable tier. Active holds bypass signal extraction and scoring.Severity and test results come from tool output that ran something. The contentsreturned by the built-in read and search tools do not count, unless the toolreports a failure.@@ -53,8 +53,10 @@ The routing decision for one turn:%%{init: {"flowchart": {"curve": "linear", "nodeSpacing": 36, "rankSpacing": 38}}}%%flowchart TBstart(["New turn"])+ hold{"Active hold?"}+ heldRoute(["Route to held tier<br/>confidence = 1.0"])history[/"Read tool-result history"/]- recovery{"Hard recovery signal<br/>or capable hold?"}+ recovery{"Hard recovery signal?"}scorer["Compute signed tool-signal score"]band{"Outside the<br/>ambiguous band?"}classifier{"Usable classifier verdict?"}@@ -63,7 +65,9 @@ flowchart TBclassifierRoute(["Route by classifier verdict"])defaultRoute(["Route to picker default"])- start --> history+ start --> hold+ hold -->|Yes| heldRoute+ hold -->|No| historyhistory --> recoveryrecovery -->|Yes| capablerecovery -->|No| scorer@@ -81,14 +85,31 @@ flowchart TBclass start,history input;class scorer process;- class recovery,band,classifier decision;+ class hold,recovery,band,classifier decision;class capable capable;- class signalRoute,classifierRoute,defaultRoute outcome;+ class heldRoute,signalRoute,classifierRoute,defaultRoute outcome;```-After a signal-driven capable decision, `capable_hold_turns` keeps recovery turns-on that tier for up to two requests by default. A clean passing test clears the-hold early. Set the value to `0` to disable it.+Both pickers support `capable_hold_turns` (default `2`) and+`efficient_hold_turns` (default `0` on standalone routes; composite's+capable-first policy defaults to `-1`). A positive value holds that tier for the+specified number of requests after a switch, excluding the switch request.+Held requests do not restart the counter. A finite capable hold also starts or+restarts on a signal-driven capable decision. Active holds return the selected+tier at confidence `1.0` before extracting tool signals or scoring. The existing+cascade then skips the fallback judge. Passing tests and errors do not end holds+early, and held requests receive no handoff notes.++Set either value to `0` to disable that hold, or `-1` to pin to that tier after+switching into it. The initial selection does not activate a pin. Pins override+later signals, including passing tests and critical errors, and skip the fallback+judge. If both values are `-1`, the first tier switch pins its destination.+Switches chosen by signals, the fallback judge, or the picker default all count.++Holds use the existing session state and are separate for each agent ID. Supply+a stable session ID across requests. A pin ends when that session state is cleared+(including session completion, expiry, or a server restart). Values below `-1` or+above `4294967295` are rejected.## Pickers@@ -230,14 +251,15 @@ required columns.| `confidence_threshold` | required | Corroboration a decisive pick needs, `0` to `1`. Alias `efficient_first_threshold`. || `capable_first_threshold` | `0.0` | Used only after an outer `composite` classifier selects the capable tier. No effect on a standalone route. || `recent_turn_window` | `3` | Trailing tool results the signals are computed over. |-| `capable_hold_turns` | `2` | Requests kept on the capable tier after an escalation the signals or a hard override picked. `0` disables. |+| `capable_hold_turns` | `2` | Requests kept on capable after switching to it; signal-driven capable decisions also start finite holds. `0` disables; `-1` pins after a switch. |+| `efficient_hold_turns` | `0` | Requests kept on efficient after switching to it. `0` disables; `-1` pins after a switch. || `tool_semantics.observe` | `[]` | Domain tool names that count as investigation. || `tool_semantics.mutate` | `[]` | Domain tool names that count as production. || `tool_semantics.plan` | `[]` | Domain tool names that count as planning. || `tool_semantics.new` | `[]` | Domain tool names that count as neutral forward activity. || `handoff_notes.escalation_note` | required in the table | Note handed to the capable tier when the signals or a hard override escalate. || `handoff_notes.deescalation_note` | unset | Note handed to the efficient tier whenever the signals select it, with or without a capable turn before. |-| `handoff_notes.only_on_wrong_signal_escalation` | `true` | `false` also adds the escalation note to capable-hold turns. |+| `handoff_notes.only_on_wrong_signal_escalation` | `true` | When `true`, restricts escalation notes to signal-driven capable decisions. Held turns receive no notes with either setting. || `classifier.target` | required in the table | Judge target. Not a routing destination. || `classifier.base_threshold` | required for LLM judges | Lowest `p_solve` that still routes efficient. || `classifier.threshold_step` | `0.0` | Added once for uncertain or unmatched verdicts, twice for unsupported. |@@ -285,6 +307,7 @@ picker = "efficient_first"confidence_threshold = 0.5recent_turn_window = 3 # optional, defaults to 3capable_hold_turns = 2 # optional, defaults to 2+efficient_hold_turns = 0 # optional; -1 pins after switching to efficient```Save as `routes.toml` and start the server:@@ -342,7 +365,7 @@ note accompanies a signal-driven efficient decision.[routes.stage.handoff_notes]escalation_note = "the previous model was stalling; pick up the diagnosis"# deescalation_note = "..." # optional-# only_on_wrong_signal_escalation = true # default; false also includes capable holds+# only_on_wrong_signal_escalation = true # default; held turns never receive notes```Classifier and picker-default decisions do not add handoff notes.@@ -403,7 +426,8 @@ the `stage router override` log line carries it for overrides:| Source | When ||---|---|| `override` | A repeated failure, critical-error severity, or context-compaction marker forced the capable tier. Structured logs set `override_reason` to `repeated_failure`, `critical_error`, or `compaction`. |-| `capable_hold` | A recent escalation kept this recovery turn on the capable tier. |+| `capable_hold` | A hold or pin kept this turn on capable. Any switch to capable can start it; signal-driven capable decisions also start finite holds. |+| `efficient_hold` | A configured hold or pin kept this turn on the efficient tier. || `dimensions` | The signed score fell outside the ambiguous band and selected a tier. || `ambiguous` | The signed score stayed inside the band. This label never reaches `/v1/stats` or the logs: the turn continues to the classifier or the picker default, and that later source is recorded. || `llm-classifier` | The signals were ambiguous and the classifier returned a usable verdict. |