Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.
Rust
fix(escalation): require fresh consistent failure evidence (#637)
Alex Steiner47bba06c9c45
diff --git a/benchmark/SWE_ATLAS_ESCALATION_REPORT.md b/benchmark/SWE_ATLAS_ESCALATION_REPORT.mdnew file mode 100644index 000000000..92751180b--- /dev/null+++ b/benchmark/SWE_ATLAS_ESCALATION_REPORT.md@@ -0,0 +1,224 @@+# SWE-Atlas escalation-router findings++## Summary++The escalation router could interpret duplicate command serialization in a NeMo Gym Terminus+trajectory as repeated failed work. A single assistant turn may contain the same command both as+raw JSON text and as a structured tool call. The judge prompt previously treated a command shown+two or more times as loop evidence without requiring those attempts to occur in distinct turns.++This change makes escalation depend on fresh, consistent failure evidence across turns. It keeps+the existing one-way switch and latch: a session can move from the efficient model to the capable+model at most once and remains there afterward.++On a deterministic 40-task paired sample (20 SWE-Atlas RF and 20 SWE-Atlas TW), the patched router+matched the two single-model arms at 15/40 correct and improved over the original router's 10/40.+It switched 5 times instead of 9. Under the synthetic token prices defined below, it cost+$0.889/task versus $1.171/task for the original router. GLM-only also scored 15/40 and remained the+least expensive arm at $0.501/task, so the result supports the patch over the original router but+does not show that routing beats the best fixed model on this sample.++## Changes++- Normalize only the transcript sent to the escalation judge. When a raw Terminus command batch+ has a one-to-one match with structured tool calls in the same assistant turn, the redundant raw+ representation is omitted. The worker model still receives the original history.+- Require the judge to classify positive evidence as `repetition`, `false_progress`, `drift`,+ `desperation`, or `capability_gap`, and report whether the evidence is new.+- Advance the confirmation streak only for fresh positive evidence in the same category. A+ category change starts a new streak; a decline or stale evidence resets it. An unavailable judge+ preserves the prior streak rather than converting an infrastructure failure into a routing+ decision.+- Clarify in the judge prompt that duplicate serialization within one turn and an immediate,+ adaptive change of terminal strategy are not repeated failed attempts.+- Preserve the existing maximum-one-switch and post-switch latch behavior.++## Focused SWE-Atlas reproducer observations++The live checks used GLM 5.2 as the efficient model and Opus 4.8 as the capable model. The reward+column is the task's binary verifier result. "GLM only" means that the patched router did not+switch models during the trajectory.++| Dataset | Task | Patched behavior | Reward | Observation |+| --- | --- | --- | ---: | --- |+| RF | `697e7458be1623d850a88838` | 28 GLM calls, then 36 Opus calls; one switch | 1 | Two fresh `repetition` findings confirmed the escalation path and latch. |+| RF | `69391d8d1ce51c407be1e531` | GLM only; 93 logged episodes | 0 | Findings changed from `repetition` to `false_progress`, so they did not form one confirmation streak. The earlier router switched without improving the reward. |+| RF | `694b4b99829f00e24fd11889` | GLM only; 34 logged episodes | 0 | No positive escalation finding. |+| TW | `6902ef3ab97fe23e2ad271f3` | GLM only; 58 logged episodes | 0 | One isolated `repetition` finding did not trigger a switch. The earlier router switched without improving the reward. |+| RF | `696719205599a51110d4b45f` | Three GLM-only runs: 72, 93, and 70 logged episodes | 0, 0, 0 | One isolated positive finding across the replicas; no latch. Direct GLM and direct Opus also scored 0. |+| RF | `696719205599a51110d4b455` | Three GLM-only runs: 51, 60, and 48 logged episodes | 0, 0, 0 | No transition in any replica. Direct GLM and direct Opus also scored 0. |++A separate passing run of `697e7458be1623d850a88838` produced a positive finding followed by a+decline, stayed on GLM, and scored 1. This checks that non-consecutive findings do not accumulate.++The positive switched task also scored 1 in a direct-GLM run. It therefore demonstrates correct+switching, confirmation, and latching, but does not establish a causal accuracy improvement.+Similarly, the unsuccessful controls show improved escalation precision and avoided Opus calls;+they do not show that the efficient model could solve those tasks.++## Paired benchmark++### Design++The paired benchmark used the following fixed setup:++- 20 tasks from `scale-ai/swe-atlas-rf@1` and 20 from `scale-ai/swe-atlas-tw@1`.+ Tasks were selected deterministically by ranking the hash of+ `20260904:<dataset>:<task>`; they were not selected by outcome.+- Four arms per task: direct GLM, direct Opus, the original escalation router, and the patched+ escalation router. This produced 160 task-arm runs.+- GLM was `nvidia/zai-org/glm-5.2`; Opus was+ `aws/anthropic/bedrock-claude-opus-4-8` with medium effort. GLM also served as the escalation+ judge. The two escalation arms used two confirmations, maximum one switch, and a post-switch+ latch.+- The original router was built from upstream commit+ `9a743e89223a0d5b14011f1226d5b068f730a3b8`; the patched router was built from+ `507376b1d851b825378cf147d09a8b5594773298`.+- There was no agent-turn limit. Worker requests allowed up to 128,000 output tokens and a+ 900-second model-server timeout. Five initial runs reached Harbor's separate 3,600-second agent+ timeout. Only those five task-arm runs were repeated with a 3x agent-timeout multiplier; all five+ retries completed and replaced the invalid attempts. The final effective matrix is 160/160 valid.+- Each table entry is one trial per task and arm. Binary verifier reward is reported as correct.+ Because model sampling is stochastic, differences between unswitched GLM router runs and direct+ GLM runs are not necessarily routing effects.++### Accuracy and routing++| Dataset | Arm | Correct | Accuracy | Switches | Correct after switch |+| --- | --- | ---: | ---: | ---: | ---: |+| RF (20) | Direct GLM | 7 | 0.350 | -- | -- |+| RF (20) | Always Opus | 9 | 0.450 | -- | -- |+| RF (20) | Original escalation | 6 | 0.300 | 7 | 2 |+| RF (20) | **Patched escalation** | **9** | **0.450** | **4** | **2** |+| TW (20) | Direct GLM | 8 | 0.400 | -- | -- |+| TW (20) | Always Opus | 6 | 0.300 | -- | -- |+| TW (20) | Original escalation | 4 | 0.200 | 2 | 0 |+| TW (20) | **Patched escalation** | **6** | **0.300** | **1** | **0** |+| Combined (40) | Direct GLM | 15 | 0.375 | -- | -- |+| Combined (40) | Always Opus | 15 | 0.375 | -- | -- |+| Combined (40) | Original escalation | 10 | 0.250 | 9 | 2 |+| Combined (40) | **Patched escalation** | **15** | **0.375** | **5** | **2** |++Against the original router on the same 40 tasks, the patched router won 7 outcomes, lost 2, and+tied 31. Against direct GLM it won 5 and lost 5; against direct Opus it won 6 and lost 6. All 14+switched trajectories across the two router arms had exactly one GLM-to-Opus transition and no+hand-back, confirming the switch and latch invariants in live execution.++The patch traded earlier sensitivity for precision. Original-router switches occurred after a+median of 23 GLM calls; patched-router switches occurred after a median of 51. One patched RF retry+did not switch until 141 GLM calls, then made 57 latched Opus calls and still scored 0. Direct Opus+and the original router also scored 0 on that task. This avoided a premature switch but exposed a+cost problem: judging every turn can be expensive when the destination model is unlikely to help.++### Synthetic cost calculation++The inference service did not provide billable cost. The values below are synthetic estimates for+comparing routing behavior, not NVIDIA prices or charges. The assumed rates are:++| Model | Uncached input / 1M tokens | Cached input / 1M tokens | Output / 1M tokens |+| --- | ---: | ---: | ---: |+| GLM 5.2 | $0.50 | $0.05 | $2.00 |+| Opus 4.8 | $5.00 | $0.50 | $25.00 |++For each model in each trajectory:++```text+uncached_input_tokens = max(prompt_tokens - cached_tokens, 0)++model_cost = (+ uncached_input_tokens * uncached_input_rate+ + cached_tokens * cached_input_rate+ + output_tokens * output_rate+) / 1,000,000++trajectory_cost = sum(worker_model_costs) + sum(escalation_judge_costs)+```++Cache-creation tokens are part of the non-cached prompt-token remainder and therefore receive the+uncached-input rate. The calculation includes GLM judge calls for the two router arms. It excludes+the Harbor verifier, cluster resources, and other infrastructure because those calls are outside+Switchyard's per-trajectory statistics.++| Dataset | Arm | Synthetic cost | Cost/task | Cost/correct |+| --- | --- | ---: | ---: | ---: |+| RF (20) | Direct GLM | $12.953 | $0.648 | $1.850 |+| RF (20) | Always Opus | $32.196 | $1.610 | $3.577 |+| RF (20) | Original escalation | $35.169 | $1.758 | $5.862 |+| RF (20) | **Patched escalation** | **$27.200** | **$1.360** | **$3.022** |+| TW (20) | Direct GLM | $7.093 | $0.355 | $0.887 |+| TW (20) | Always Opus | $14.829 | $0.741 | $2.471 |+| TW (20) | Original escalation | $11.657 | $0.583 | $2.914 |+| TW (20) | **Patched escalation** | **$8.356** | **$0.418** | **$1.393** |+| Combined (40) | Direct GLM | $20.045 | $0.501 | $1.336 |+| Combined (40) | Always Opus | $47.025 | $1.176 | $3.135 |+| Combined (40) | Original escalation | $46.826 | $1.171 | $4.683 |+| Combined (40) | **Patched escalation** | **$35.556** | **$0.889** | **$2.370** |++The patched router was 24.1% cheaper than the original router and produced five additional correct+outcomes. It was 24.4% cheaper than always-Opus at the same aggregate accuracy. It was 77.4% more+expensive than direct GLM at the same aggregate accuracy. Of the patched router's $35.556 synthetic+total, $3.698 (10.4%) came from judge calls; this motivates reducing judge cadence after repeated+negative verdicts.++### TW model-order reversal++TW demonstrates a limitation that this patch does not solve: the escalation route assumes its+configured Opus target has positive expected gain over GLM. On this sample, direct GLM scored 8/20+while direct Opus scored 6/20, so that assumption is not valid at the workload level.++The patched router switched on only one TW task. Direct GLM, direct Opus, and both router arms all+scored 0 on that task, so the switch added cost without changing its outcome. The patched router's+two-correct deficit versus direct GLM was not caused by switching: it lost three independently+sampled GLM-only outcomes and gained one different GLM-only outcome. The original router provides+a clearer harmful-switch example on task `6902ef3ab97fe23e2ad2727e`: direct GLM scored 1, direct Opus+scored 0, and the original router switched and scored 0. The patch avoided that switch, although its+independent GLM-only run also scored 0.++The next routing change should therefore be separate from the trajectory-evidence fix: gate+escalation on a workload- or task-conditioned estimate of expected accuracy gain and cost. When+recent paired evidence says the nominal strong model is worse, the route should disable or reverse+that transition rather than asking only whether the current trajectory looks stuck. This requires+held-out calibration or online exploration; the router cannot infer relative model quality from one+model's failing trajectory alone.++## Validation evidence++The implementation was validated with Rust 1.96.1 on a Slurm compute node using:++```text+cargo fmt --all --check+cargo clippy --workspace --all-targets --all-features -- -D warnings+cargo test --workspace --all-features+uv run ruff check .+uv run mypy switchyard+uv run maturin develop --uv+uv run pytest tests/ -v+```++The validation completed successfully: the Rust workspace passed, including 289 `libsy` tests;+ruff and mypy passed; the native wheel built and installed; and pytest reported 115 passed,+2 deselected, and 2 subtests passed.+Focused regression coverage includes exact and multi-command transcript normalization, partial and+multiplicity mismatches, category changes, stale evidence, unavailable-judge behavior, and the+one-way latch.++After rebasing onto current `main` on 2026-09-22, the review fixes were revalidated with Rust+1.96.1 and Python 3.12 using `cargo fmt --all --check`, workspace clippy with all targets and+features, and workspace tests with all features. The full Rust gate passed, including 323 `libsy`+tests and all 56 server integration tests. This post-rebase run caught and fixed stale call+signatures and mock verdicts that still used the pre-rebase runtime interfaces and escalation+schema.++No live provider calls are made by the repository validation commands. The live SWE-Atlas checks+described above were separate, explicitly configured benchmark runs.++## Conclusion++The patch fixes the concrete duplicate-serialization false positive and materially improves the+tested router: it matches the single-model aggregate accuracy, gains 5 correct outcomes over the+original router, cuts switches from 9 to 5, and reduces synthetic cost by 24.1%. It does not make+the router the best cost/accuracy arm; direct GLM matches its aggregate accuracy at substantially+lower cost. The remaining work is not to loosen the new evidence rule globally. It is to add+model-order calibration for workloads such as TW and reduce judge overhead or stop judging when+escalation has low expected value. Replicated trials on held-out tasks are needed before treating+the point estimates as stable production gains.diff --git a/crates/libsy-llm-client/tests/observability.rs b/crates/libsy-llm-client/tests/observability.rsindex 32aae6feb..93c12adb5 100644--- a/crates/libsy-llm-client/tests/observability.rs+++ b/crates/libsy-llm-client/tests/observability.rs@@ -772,7 +772,9 @@ async fn stateful_escalation_warns_once_without_a_session_id() -> switchyard_lib})?) as Arc<dyn Algorithm>;let client = Arc::new(JudgeClient {judge_model: "warning-judge".into(),- outcome: JudgeOutcome::Reply(r#"{"escalate":false,"reason":"progressing"}"#),+ outcome: JudgeOutcome::Reply(+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#,+ ),}) as Arc<dyn RoutedLlmClient>;for _ in 0..2 {@@ -828,7 +830,9 @@ async fn deescalation_evidence_stays_pending_until_confirmed() -> switchyard_libswitchyard_llm_client::run(router.clone(),- ClientRouter::single(client(r#"{"escalate":true,"reason":"stuck"}"#)),+ ClientRouter::single(client(+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck"}"#,+ )),request.clone(),classifier_models("evidence-judge", "evidence-efficient", "evidence-capable"),None,@@ -836,7 +840,9 @@ async fn deescalation_evidence_stays_pending_until_confirmed() -> switchyard_lib.await?;let outcome = switchyard_llm_client::decide(router,- ClientRouter::single(client(r#"{"escalate":false,"reason":"recovered"}"#)),+ ClientRouter::single(client(+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"recovered"}"#,+ )),request,classifier_models("evidence-judge", "evidence-efficient", "evidence-capable"),)diff --git a/crates/libsy/src/algorithms/escalation.rs b/crates/libsy/src/algorithms/escalation.rsindex 534ef34a0..76c77d76b 100644--- a/crates/libsy/src/algorithms/escalation.rs+++ b/crates/libsy/src/algorithms/escalation.rs@@ -15,8 +15,8 @@ use super::util::buffered_response::buffer_response;use super::util::classifier_contract::ClassifierContractConfig;use super::util::decisive;use super::util::escalation::{- self, DeescalationConfig, EscalationJudge, EscalationJudgeConfig, EscalationPolicy,- EvaluationPhase,+ self, DeescalationConfig, EscalationCategory, EscalationJudge, EscalationJudgeConfig,+ EscalationPolicy, EvaluationPhase,};use super::util::llm_judge::JudgeClassifier;use crate::core::algorithm::Driver;@@ -25,6 +25,8 @@ use crate::core::state::{State, StateValue};use crate::{LibsyError, Result};const STREAK_KEY: &str = "escalation_streak";+/// Session-state key holding the category currently being confirmed.+const CATEGORY_KEY: &str = "escalation_category";const STRONG_CALLS_KEY: &str = "escalation_strong_calls";const RELEASE_STREAK_KEY: &str = "escalation_release_streak";const WEAK_COOLDOWN_KEY: &str = "escalation_weak_cooldown";@@ -42,6 +44,20 @@ fn set_count(state: &mut State, key: &str, value: u32) {.insert(key.to_string(), StateValue::Count(value));}+/// Reads the failure category whose confirmation streak is in progress, if any.+fn category(state: &State) -> Option<&str> {+ match state.extra.get(CATEGORY_KEY) {+ Some(StateValue::String(category)) => Some(category),+ _ => None,+ }+}++/// Clears the confirmation streak together with the category it was confirming.+fn clear_streak(state: &mut State) {+ set_count(state, STREAK_KEY, 0);+ state.extra.remove(CATEGORY_KEY);+}+fn assistant_message(response: &AggLlmResponse) -> Message {Message {role: Role::Assistant,@@ -171,7 +187,7 @@ impl EscalationClassifier {set_count(state, RELEASE_STREAK_KEY, release_streak);if release_streak >= deescalation.config.confirmations {- set_count(state, STREAK_KEY, 0);+ clear_streak(state);set_count(state, STRONG_CALLS_KEY, 0);set_count(state, RELEASE_STREAK_KEY, 0);tracing::debug!(@@ -225,7 +241,7 @@ impl Classifier<State> for EscalationClassifier {.strong_max_calls.is_some_and(|strong_max_calls| strong_calls >= strong_max_calls){- set_count(state, STREAK_KEY, 0);+ clear_streak(state);set_count(state, STRONG_CALLS_KEY, 0);set_count(state, RELEASE_STREAK_KEY, 0);set_count(@@ -336,19 +352,52 @@ impl Classifier<State> for EscalationClassifier {));}- let (classification, _) = self+ let judge_models = driver.models_for(&Category::Judge);+ if judge_models.is_empty() {+ return Err(LibsyError::AlgorithmError {+ message: "no models available for category Judge".to_string(),+ });+ }+ let verdict = self.escalation_judge- .score(state, &mut judge_request, driver)- .await?;+ .verdict(state, &judge_request, driver, judge_models)+ .await;let held = count(state, STREAK_KEY);- let best = classification.argmax(false)?;- let (escalate, pending) = match &best {- Some(score) if score.target == capable => (true, held.saturating_add(1)),- Some(_) => (false, 0),- None => (false, held),+ let held_category = category(state).map(str::to_string);+ let (escalate, pending, pending_category) = match verdict.as_ref() {+ Some(verdict) => {+ let category = verdict.category.label();+ tracing::info!(+ escalate = verdict.escalate,+ category,+ new_evidence = verdict.new_evidence,+ "escalation judge verdict"+ );+ if verdict.escalate+ && verdict.new_evidence+ && verdict.category != EscalationCategory::None+ {+ let next = if held_category.as_deref() == Some(category) {+ held.saturating_add(1)+ } else {+ 1+ };+ (true, next, Some(category.to_string()))+ } else {+ (false, 0, None)+ }+ }+ None => (false, held, held_category),};set_count(state, STREAK_KEY, pending);+ if let Some(category) = pending_category {+ state+ .extra+ .insert(CATEGORY_KEY.to_string(), StateValue::String(category));+ } else {+ state.extra.remove(CATEGORY_KEY);+ }if escalate && pending >= self.confirmations {driver.set_evidence(serde_json::json!({@@ -367,6 +416,11 @@ impl Classifier<State> for EscalationClassifier {"source": "escalation","verdict": "pending",}));+ } else if verdict.is_some() {+ driver.set_evidence(serde_json::json!({+ "source": "escalation",+ "verdict": "continue",+ }));}Ok((@@ -475,13 +529,13 @@ mod tests {}}- /// Builds a router with escalation enabled (`confirmations=1` latches immediately).- fn escalation_router() -> Result<Arc<LlmTaskClassifier>> {+ /// Builds a router with escalation enabled.+ fn escalation_router_with_confirmations(confirmations: u32) -> Result<Arc<LlmTaskClassifier>> {Ok(Arc::new(LlmTaskClassifier::new(LlmClassifierConfig::Escalation {contract: ClassifierContractConfig::default(),config: EscalationJudgeConfig {- confirmations: 1,+ confirmations,..EscalationJudgeConfig::default()},max_output_tokens: DEFAULT_JUDGE_MAX_OUTPUT_TOKENS,@@ -526,9 +580,16 @@ mod tests {Ok(selected)}+ /// Builds a router that latches on its first supported escalation verdict.+ fn escalation_router() -> Result<Arc<LlmTaskClassifier>> {+ escalation_router_with_confirmations(1)+ }+#[tokio::test]async fn serves_efficient_when_judge_declines() -> Result<()> {- let judge = Queue::new([r#"{"escalate":false,"reason":"progressing"}"#]);+ let judge = Queue::new([+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#,+ ]);let model = Queue::new(["efficient answer"]);let (selected_model, response) = test_drive_with_models(@@ -547,6 +608,44 @@ mod tests {Ok(())}+ /// A parsed decline records continue evidence on the routing outcome.+ #[tokio::test]+ async fn records_continue_evidence_when_judge_declines() -> Result<()> {+ let judge = Queue::new([+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#,+ ]);+ let model = Queue::new(["efficient answer"]);+ let serve = Arc::new(queued(model, judge));+ let routing_serve = Arc::clone(&serve);+ let outcome = crate::drive(+ escalation_router()?,+ classify_request(),+ Arc::new(crate::core::algorithm::RuntimeModels::new(runtime_models())),+ move |call| {+ let serve = Arc::clone(&routing_serve);+ async move {+ let target = call.models.first().cloned().ok_or(LibsyError::NoTargets)?;+ let request = call.request.clone();+ let response = serve+ .serve(target.clone(), request)+ .await+ .map_err(|source| LibsyError::client_call(target, source));+ call.respond(response)+ }+ },+ )+ .await?;++ assert_eq!(+ outcome.metadata.and_then(|metadata| metadata.evidence),+ Some(serde_json::json!({+ "source": "escalation",+ "verdict": "continue",+ }))+ );+ Ok(())+ }+#[tokio::test]async fn config_overrides_the_packaged_prompt() -> Result<()> {let prompts = Arc::new(Mutex::new(Vec::new()));@@ -564,7 +663,9 @@ mod tests {})});recorded.lock().extend(prompt);- std::future::ready(Ok(reply(r#"{"escalate":false,"reason":"progressing"}"#)))+ std::future::ready(Ok(reply(+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#,+ )))} else {std::future::ready(Ok(reply("efficient answer")))}@@ -586,7 +687,9 @@ mod tests {#[tokio::test]async fn upgrades_to_capable_when_judge_escalates() -> Result<()> {- let judge = Queue::new([r#"{"escalate":true,"reason":"stuck in a loop"}"#]);+ let judge = Queue::new([+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck in a loop"}"#,+ ]);let model = Queue::new(["efficient draft", "capable answer"]);let (selected_model, response) = test_drive_with_models(@@ -605,9 +708,106 @@ mod tests {Ok(())}+ /// A verdict in a different category restarts the confirmation streak at one.+ #[tokio::test]+ async fn confirmation_streak_requires_the_same_category() -> Result<()> {+ let judge = Queue::new([+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"repeated command"}"#,+ r#"{"escalate":true,"category":"drift","new_evidence":true,"reason":"off task"}"#,+ r#"{"escalate":true,"category":"drift","new_evidence":true,"reason":"still off task"}"#,+ ]);+ let model = Queue::new(["efficient t1", "efficient t2", "efficient t3", "capable t3"]);+ let router = escalation_router_with_confirmations(2)?;+ let request = classify_session_request();++ let (first, _) = test_drive_with_models(+ router.clone(),+ request.clone(),+ runtime_models(),+ queued(Arc::clone(&model), Arc::clone(&judge)),+ )+ .await?;+ let (second, _) = test_drive_with_models(+ router.clone(),+ request.clone(),+ runtime_models(),+ queued(Arc::clone(&model), Arc::clone(&judge)),+ )+ .await?;+ let (third, _) =+ test_drive_with_models(router, request, runtime_models(), queued(model, judge)).await?;++ assert_eq!(first, "efficient");+ assert_eq!(second, "efficient");+ assert_eq!(third, "capable");+ Ok(())+ }++ /// An escalate verdict without fresh evidence resets the streak instead of extending it.+ #[tokio::test]+ async fn verdict_without_new_evidence_resets_the_streak() -> Result<()> {+ let judge = Queue::new([+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"repeated command"}"#,+ r#"{"escalate":true,"category":"repetition","new_evidence":false,"reason":"only old evidence remains"}"#,+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"new repeated command"}"#,+ ]);+ let model = Queue::new(["efficient t1", "efficient t2", "efficient t3"]);+ let router = escalation_router_with_confirmations(2)?;+ let request = classify_session_request();++ for _ in 0..3 {+ let (selected, _) = test_drive_with_models(+ router.clone(),+ request.clone(),+ runtime_models(),+ queued(Arc::clone(&model), Arc::clone(&judge)),+ )+ .await?;+ assert_eq!(selected, "efficient");+ }+ Ok(())+ }++ /// An unparseable verdict keeps the streak and its category for the next turn.+ #[tokio::test]+ async fn unavailable_judge_preserves_the_category_streak() -> Result<()> {+ let judge = Queue::new([+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"repeated command"}"#,+ "not json",+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"another repeated command"}"#,+ ]);+ let model = Queue::new(["efficient t1", "efficient t2", "efficient t3", "capable t3"]);+ let router = escalation_router_with_confirmations(2)?;+ let request = classify_session_request();++ let (first, _) = test_drive_with_models(+ router.clone(),+ request.clone(),+ runtime_models(),+ queued(Arc::clone(&model), Arc::clone(&judge)),+ )+ .await?;+ let (second, _) = test_drive_with_models(+ router.clone(),+ request.clone(),+ runtime_models(),+ queued(Arc::clone(&model), Arc::clone(&judge)),+ )+ .await?;+ let (third, _) =+ test_drive_with_models(router, request, runtime_models(), queued(model, judge)).await?;++ assert_eq!(first, "efficient");+ assert_eq!(second, "efficient");+ assert_eq!(third, "capable");+ Ok(())+ }+#[tokio::test]async fn stays_capable_after_latch() -> Result<()> {- let judge = Queue::new([r#"{"escalate":true,"reason":"stuck"}"#]);+ let judge = Queue::new([+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck"}"#,+ ]);let model = Queue::new(["efficient draft", "capable t1", "capable t2"]);let router = escalation_router()?;let request = classify_session_request();@@ -629,10 +829,10 @@ mod tests {#[tokio::test]async fn deescalation_holds_then_returns_to_efficient() -> Result<()> {let judge = Queue::new([- r#"{"escalate":true,"reason":"stuck"}"#,- r#"{"escalate":false,"reason":"recovered"}"#,- r#"{"escalate":false,"reason":"routine"}"#,- r#"{"escalate":false,"reason":"progressing"}"#,+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck"}"#,+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"recovered"}"#,+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"routine"}"#,+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#,]);let model = Queue::new(["efficient draft",@@ -658,10 +858,10 @@ mod tests {#[tokio::test]async fn deescalation_hard_limit_forces_a_weak_cooldown() -> Result<()> {let judge = Queue::new([- r#"{"escalate":true,"reason":"stuck"}"#,- r#"{"escalate":true,"reason":"still hard"}"#,- r#"{"escalate":true,"reason":"still hard"}"#,- r#"{"escalate":false,"reason":"progressing"}"#,+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck"}"#,+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"still hard"}"#,+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"still hard"}"#,+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#,]);let model = Queue::new(["efficient draft",@@ -695,7 +895,9 @@ mod tests {#[tokio::test]async fn strong_review_returns_an_efficient_fallback_without_judging_it() -> Result<()> {- let judge = Queue::new([r#"{"escalate":true,"reason":"stuck"}"#]);+ let judge = Queue::new([+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck"}"#,+ ]);let model = Queue::new(["efficient draft", "capable t1", "efficient fallback"]);let router = deescalation_router(DeescalationConfig {strong_min_calls: 1,diff --git a/crates/libsy/src/algorithms/util/escalation.rs b/crates/libsy/src/algorithms/util/escalation.rsindex 0335b4d3f..2160cb7ec 100644--- a/crates/libsy/src/algorithms/util/escalation.rs+++ b/crates/libsy/src/algorithms/util/escalation.rs@@ -94,8 +94,9 @@ impl DeescalationConfig {#[derive(Clone, Debug, Deserialize)]#[serde(default, deny_unknown_fields)]pub struct EscalationJudgeConfig {- /// Consecutive escalate verdicts required before a turn moves to the capable tier, which- /// is also the turn that latches the session. Any decline clears the streak.+ /// Consecutive fresh-evidence verdicts in the same category required before a turn moves to+ /// the capable tier, which is also the turn that latches the session. Any decline or stale+ /// evidence clears the streak./// `1` escalates on the first verdict; the router's main cost dial./// `2` or higher needs a session id, since the streak is retained per session.pub confirmations: u32,@@ -157,15 +158,39 @@ impl EvaluationPhase {}}-/// The judge's verdict. The schema also requires a `reason`, which makes the judge state its-/// case and measurably sharpens the verdict. Routing reads only the boolean; the reason is-/// kept solely so an operator can see why the judge held or escalated when the-/// `switchyard_libsy::algorithms::util::escalation` target is enabled at `debug`.+/// Bounded trouble pattern used to correlate escalation confirmations across turns.+#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq)]+#[serde(rename_all = "snake_case")]+pub(crate) enum EscalationCategory {+ None,+ Repetition,+ FalseProgress,+ Drift,+ Desperation,+ CapabilityGap,+}++impl EscalationCategory {+ /// Stable state and telemetry label.+ pub(crate) const fn label(self) -> &'static str {+ match self {+ Self::None => "none",+ Self::Repetition => "repetition",+ Self::FalseProgress => "false_progress",+ Self::Drift => "drift",+ Self::Desperation => "desperation",+ Self::CapabilityGap => "capability_gap",+ }+ }+}++/// The judge's typed verdict, including the evidence needed to confirm a stable pattern.#[derive(Deserialize)]pub(crate) struct EscalationVerdict {- escalate: bool,- #[serde(default)]- reason: String,+ pub(crate) escalate: bool,+ pub(crate) category: EscalationCategory,+ pub(crate) new_evidence: bool,+ pub(crate) reason: String,}/// Builds the condensed trajectory presented to the escalation judge.@@ -314,21 +339,102 @@ pub(crate) fn conversation_turn(request: &Request) -> usize {/// relies on.fn message_text(message: &Message) -> String {let mut parts = Vec::new();- collect_text(&message.content, &mut parts);+ let terminus_commands = if message.role == Role::Assistant {+ message+ .content+ .iter()+ .filter_map(|block| match block {+ ContentBlock::ToolCall(call) if call.name == "bash_command" => call+ .arguments+ .get("keystrokes")+ .and_then(|value| value.as_str()),+ _ => None,+ })+ .collect::<Vec<_>>()+ } else {+ Vec::new()+ };+ collect_text(&message.content, &mut parts, &terminus_commands);parts.join(" ")}+/// Removes a Terminus command batch when a structured bash call carries the same action.+///+/// The model-facing request remains untouched. Only the judge's plain-text view is normalized,+/// so one action cannot look like two attempts while the agent still sees its native history.+fn without_duplicated_terminus_commands(text: &str, tool_commands: &[&str]) -> String {+ let mut normalized_text = String::with_capacity(text.len());+ let mut unmatched_tool_commands = tool_commands.to_vec();+ let mut copied_through = 0;+ let mut scan_from = 0;++ while let Some(relative_start) = text[scan_from..].find('{') {+ let start = scan_from + relative_start;++ let mut values =+ serde_json::Deserializer::from_str(&text[start..]).into_iter::<serde_json::Value>();+ let Some(Ok(mut value)) = values.next() else {+ scan_from = start + 1;+ continue;+ };+ let end = start + values.byte_offset();+ let Some(commands) = value+ .get("commands")+ .and_then(|commands| commands.as_array())+ else {+ scan_from = start + 1;+ continue;+ };+ let Some(command_batch) = commands+ .iter()+ .map(|command| command.get("keystrokes").and_then(|value| value.as_str()))+ .collect::<Option<Vec<_>>>()+ else {+ scan_from = start + 1;+ continue;+ };+ let mut remaining_tool_commands = unmatched_tool_commands.clone();+ let fully_encoded = command_batch.iter().all(|command| {+ let Some(index) = remaining_tool_commands+ .iter()+ .position(|candidate| candidate == command)+ else {+ return false;+ };+ remaining_tool_commands.swap_remove(index);+ true+ });+ if command_batch.is_empty() || !fully_encoded {+ scan_from = start + 1;+ continue;+ }++ value["commands"] = serde_json::Value::Array(Vec::new());+ let Ok(normalized) = serde_json::to_string(&value) else {+ scan_from = start + 1;+ continue;+ };+ normalized_text.push_str(&text[copied_through..start]);+ normalized_text.push_str(&normalized);+ copied_through = end;+ scan_from = end;+ unmatched_tool_commands = remaining_tool_commands;+ }+ normalized_text.push_str(&text[copied_through..]);+ normalized_text+}+/// Appends the judge-relevant text of each block, descending into tool results.-fn collect_text(content: &[ContentBlock], parts: &mut Vec<String>) {+fn collect_text(content: &[ContentBlock], parts: &mut Vec<String>, tool_commands: &[&str]) {for block in content {match block {ContentBlock::Text { text } | ContentBlock::Refusal { text } => {- parts.push(text.clone());+ parts.push(without_duplicated_terminus_commands(text, tool_commands));}ContentBlock::ToolCall(call) => {parts.push(format!("tool_call {}({})", call.name, call.arguments));}- ContentBlock::ToolResult(result) => collect_text(&result.content, parts),+ ContentBlock::ToolResult(result) => collect_text(&result.content, parts, &[]),_ => {}}}@@ -378,7 +484,7 @@ fn summarize_for_judge(for instruction in instructions {let mut parts = Vec::new();- collect_text(&instruction.content, &mut parts);+ collect_text(&instruction.content, &mut parts, &[]);instruction_anchors.push(format!("[{}] {}",role_label(instruction.role),@@ -648,6 +754,194 @@ mod tests {assert_eq!(message_text(&result), "no such file");}+ /// A raw command batch fully mirrored by structured tool calls is emptied in the judge view.+ #[test]+ fn message_text_deduplicates_terminus_commands_for_the_judge() {+ let first_command = "grep -n bug app.py\n";+ let second_command = "sed -n '1,80p' app.py\n";+ let message = Message {+ role: Role::Assistant,+ content: vec![+ ContentBlock::Text {+ text: format!(+ "Before\n```json\n{}\n```\nAfter",+ json!({+ "analysis": "inspect the reported file",+ "commands": [+ {"keystrokes": first_command, "duration": 0.1},+ {"keystrokes": second_command, "duration": 0.1},+ ],+ })+ ),+ },+ ContentBlock::ToolCall(ToolCall {+ id: "call-1".to_string(),+ name: "bash_command".to_string(),+ arguments: json!({"keystrokes": first_command, "duration": 0.1}),+ }),+ ContentBlock::ToolCall(ToolCall {+ id: "call-2".to_string(),+ name: "bash_command".to_string(),+ arguments: json!({"keystrokes": second_command, "duration": 0.1}),+ }),+ ],+ };++ let text = message_text(&message);++ assert!(text.contains("inspect the reported file"), "{text}");+ assert!(text.contains("Before"), "{text}");+ assert!(text.contains("After"), "{text}");+ assert!(text.contains(r#""commands":[]"#), "{text}");+ assert_eq!(text.matches("grep -n bug app.py").count(), 1, "{text}");+ assert_eq!(text.matches("sed -n '1,80p' app.py").count(), 1, "{text}");+ assert_eq!(text.matches("tool_call bash_command(").count(), 2, "{text}");+ }++ /// Each structured tool call can absorb only one rendered batch, so a repeated batch stays.+ #[test]+ fn message_text_deduplicates_multiple_batches_once_per_tool_call() {+ let first_command = "grep -n bug app.py\n";+ let second_command = "sed -n '1,80p' app.py\n";+ let batch = |command| {+ json!({+ "analysis": "inspect",+ "commands": [{"keystrokes": command}],+ })+ .to_string()+ };+ let message = Message {+ role: Role::Assistant,+ content: vec![+ ContentBlock::Text {+ text: format!(+ "First {} second {} repeated {}",+ batch(first_command),+ batch(second_command),+ batch(first_command)+ ),+ },+ ContentBlock::ToolCall(ToolCall {+ id: "call-1".to_string(),+ name: "bash_command".to_string(),+ arguments: json!({"keystrokes": first_command}),+ }),+ ContentBlock::ToolCall(ToolCall {+ id: "call-2".to_string(),+ name: "bash_command".to_string(),+ arguments: json!({"keystrokes": second_command}),+ }),+ ],+ };++ let text = message_text(&message);++ assert_eq!(text.matches(r#""commands":[]"#).count(), 2, "{text}");+ assert_eq!(text.matches("grep -n bug app.py").count(), 2, "{text}");+ assert_eq!(text.matches("sed -n '1,80p' app.py").count(), 1, "{text}");+ }++ /// A rendered batch stays when no structured tool call carries the same command.+ #[test]+ fn message_text_keeps_terminus_commands_without_matching_tool_call() {+ let command = "grep -n bug app.py\n";+ let text = json!({+ "analysis": "inspect the reported file",+ "commands": [{"keystrokes": command}],+ })+ .to_string();+ let without_tool_call = Message {+ role: Role::Assistant,+ content: vec![ContentBlock::Text { text: text.clone() }],+ };+ let mismatched_tool_call = Message {+ role: Role::Assistant,+ content: vec![+ ContentBlock::Text { text },+ ContentBlock::ToolCall(ToolCall {+ id: "call-1".to_string(),+ name: "bash_command".to_string(),+ arguments: json!({"keystrokes": "sed -n '1,20p' app.py\n", "duration": 0.1}),+ }),+ ],+ };++ assert_eq!(+ message_text(&without_tool_call)+ .matches("grep -n bug app.py")+ .count(),+ 1+ );+ assert_eq!(+ message_text(&mismatched_tool_call)+ .matches("grep -n bug app.py")+ .count(),+ 1+ );+ }++ /// A batch stays intact when only some of its commands have matching tool calls.+ #[test]+ fn message_text_keeps_a_partially_encoded_terminus_batch() {+ let first_command = "grep -n bug app.py\n";+ let second_command = "sed -n '1,80p' app.py\n";+ let message = Message {+ role: Role::Assistant,+ content: vec![+ ContentBlock::Text {+ text: json!({+ "commands": [+ {"keystrokes": first_command},+ {"keystrokes": second_command},+ ],+ })+ .to_string(),+ },+ ContentBlock::ToolCall(ToolCall {+ id: "call-1".to_string(),+ name: "bash_command".to_string(),+ arguments: json!({"keystrokes": first_command}),+ }),+ ],+ };++ let text = message_text(&message);++ assert_eq!(text.matches("grep -n bug app.py").count(), 2, "{text}");+ assert_eq!(text.matches("sed -n '1,80p' app.py").count(), 1, "{text}");+ assert!(!text.contains(r#""commands":[]"#), "{text}");+ }++ /// Duplicate commands in one batch need one tool call each before the batch is removed.+ #[test]+ fn message_text_keeps_duplicate_commands_without_one_tool_call_each() {+ let command = "grep -n bug app.py\n";+ let message = Message {+ role: Role::Assistant,+ content: vec![+ ContentBlock::Text {+ text: json!({+ "commands": [+ {"keystrokes": command},+ {"keystrokes": command},+ ],+ })+ .to_string(),+ },+ ContentBlock::ToolCall(ToolCall {+ id: "call-1".to_string(),+ name: "bash_command".to_string(),+ arguments: json!({"keystrokes": command}),+ }),+ ],+ };++ let text = message_text(&message);++ assert_eq!(text.matches("grep -n bug app.py").count(), 3, "{text}");+ assert!(!text.contains(r#""commands":[]"#), "{text}");+ }+#[test]fn truncate_middle_keeps_head_and_tail() {let text = "a".repeat(40) + &"z".repeat(40);diff --git a/crates/libsy/src/algorithms/util/llm_judge.rs b/crates/libsy/src/algorithms/util/llm_judge.rsindex 4798178e9..cb8277dae 100644--- a/crates/libsy/src/algorithms/util/llm_judge.rs+++ b/crates/libsy/src/algorithms/util/llm_judge.rs@@ -261,7 +261,7 @@ where/// mid-stream, or unparseable reply — is logged and folded into `None` for the policy's/// fallback branch. A closed driver stream is folded too; the algorithm's next driver/// call surfaces it, so nothing is masked.- async fn verdict(+ pub(crate) async fn verdict(&self,state: &mut State,request: &Request,diff --git a/crates/libsy/src/prompts/escalation/deescalation.md b/crates/libsy/src/prompts/escalation/deescalation.mdindex 3d1e3fe9c..4f9870829 100644--- a/crates/libsy/src/prompts/escalation/deescalation.md+++ b/crates/libsy/src/prompts/escalation/deescalation.md@@ -27,5 +27,12 @@ The routing input begins with one of these router-generated markers:success, or a context compaction. A strong-tier turn that merely readsfiles or plans is not by itself evidence that the trouble is resolved.+ In this phase the router reads only `escalate`. Fill the other fields+ consistently anyway: when retaining, set `category` to the trouble+ pattern that caused the escalation and `new_evidence` to true only when+ the newest strong-tier turn or its tool output shows that trouble is+ still active; when releasing, return `category: "none"` and+ `new_evidence: false`.+The router, not the judge, applies confirmation counts and decides whento change tiers. Judge only the phase named in the routing input.diff --git a/crates/libsy/src/prompts/escalation/prompt.md b/crates/libsy/src/prompts/escalation/prompt.mdindex 6d57369bb..d29a4017f 100644--- a/crates/libsy/src/prompts/escalation/prompt.md+++ b/crates/libsy/src/prompts/escalation/prompt.md@@ -9,11 +9,20 @@ messages and tool results). Judge the *trajectory* — is the agent makingreal progress toward the stated task — not the difficulty of the taskitself. Return exactly one JSON object:-{"escalate": boolean, "reason": "one short sentence naming the pattern"}+{"escalate": boolean, "category": "none|repetition|false_progress|drift|desperation|capability_gap", "new_evidence": boolean, "reason": "one short sentence naming the pattern"}++Use `category: "none"` whenever `escalate` is false. `new_evidence`+is true only when the NEWEST assistant turn or its resulting tool output+adds evidence for the named pattern. Older turns may establish context,+but do not repeat an escalation vote solely because old trouble remains+visible in the rolling transcript. If the newest turn recovered, adapted,+or made progress, return `escalate: false`, `category: "none"`, and+`new_evidence: false`.Escalation is one-way for the rest of the task and expensive. Escalateonly on a clear PATTERN of trouble, never on a single failed command.-When the evidence is thin or ambiguous, return {"escalate": false}.+When the evidence is thin or ambiguous, decline with `category: "none"`+and `new_evidence: false`.The bar is not "is there friction" — agentic coding is full of frictionthe efficient tier works through on its own. The bar is "is this run@@ -58,8 +67,10 @@ Hold weak — no model can fix these, so escalation is pure waste:# Trouble patterns — escalate when you see theseRepetition and loops (the most common way agent runs die):-- The same command or edit failing 2+ times with materially the same- error, especially with unrelated changes in between.+- The same command or edit failing across 2+ DISTINCT assistant turns+ with materially the same error, especially with unrelated changes in+ between. Count executed attempts across turns, not repeated renderings+ inside one message.- Near-identical tool calls repeated, or the same files re-read, withoutnew information gained — including longer cycles (A -> B -> C -> A).- Fighting the environment: repeatedly invoking a missing executable,@@ -105,6 +116,15 @@ Desperation:# Expected friction — do NOT escalate on theseAgentic coding is full of failures that are part of healthy work:+- A command appearing both in the assistant's JSON/text and as one or+ more structured tool-call blocks in that SAME turn. Agent harnesses+ commonly serialize one intended action more than once; this is one+ attempt, not a loop. Repetition is evidence only when separate turns+ show separate executions with materially the same failed result.+- Terminal-input serialization trouble (for example tabs triggering+ completion or a heredoc being mangled) when the next turn changes the+ write mechanism, quoting, or transport. That is adaptation, not+ repeated failure.- A test written to fail first (TDD) or a bug being reproduced onpurpose.- A compile, lint, or test error fixed or meaningfully acted on in the@@ -130,49 +150,56 @@ Agentic coding is full of failures that are part of healthy work:- A long-running command (build, install, test suite) that simply hasnot finished, or the agent waiting on information it asked for.-The distinguishing question: is each failure producing new information-that changes the next action? Failing forward is fine; failing in place-is trouble. Also weigh the session's own recovery record: if this same-session already shows friction the agent subsequently cleared (a failure-followed by a verified fix or passing check), lean toward holding — a-session that has recovered before will usually recover again.+The distinguishing question: across DISTINCT assistant turns, is each+failure producing new information that changes the next action? Failing+forward is fine; failing in place is trouble. Never infer a multi-turn+pattern from duplicated representations inside one turn. Also weigh the+session's own recovery record: if this same session already shows+friction the agent subsequently cleared (a failure followed by a verified+fix or passing check), lean toward holding — a session that has recovered+before will usually recover again.# Worked examples (none drawn from any benchmark task set)* Turn 3; the agent ran the test suite, 4 tests fail, and it is now- reading the first failing test. -> {"escalate": false} — reproducing- failures is the job.+ reading the first failing test. -> {"escalate": false, "category":+ "none", "new_evidence": false, "reason": "working through the first+ reproduced failure"}* The agent has run `pytest tests/test_api.py` 4 times with the sameImportError, editing an unrelated config file between attempts. ->- {"escalate": true, "reason": "same ImportError 4 times while editing- unrelated files"}+ {"escalate": true, "category": "repetition", "new_evidence": true,+ "reason": "same ImportError 4 times while editing unrelated files"}* `conda` is not installed; the agent has tried `conda install` fiveways instead of using the `pip` that earlier output showed present.- -> {"escalate": true, "reason": "fighting missing executable instead- of adapting"}+ -> {"escalate": true, "category": "repetition", "new_evidence":+ true, "reason": "fighting missing executable instead of adapting"}* Task: "make the provided integration tests pass." Recent turns:renaming variables and reformatting docstrings; tests not run in 8- turns. -> {"escalate": true, "reason": "drifted to cosmetic edits,- verification abandoned"}+ turns. -> {"escalate": true, "category": "drift", "new_evidence":+ true, "reason": "drifted to cosmetic edits, verification abandoned"}* The agent says "All tests pass, task complete" but the last visibletest output shows "2 failed, 11 passed". -> {"escalate": true,- "reason": "claims success contradicted by latest test output"}+ "category": "false_progress", "new_evidence": true, "reason":+ "claims success contradicted by latest test output"}* The agent wrote a reproduction script that exits 0 without invokingthe code path the issue describes, concluded "bug not reproducible",- and is wrapping up. -> {"escalate": true, "reason": "reproduction- never exercised the reported code path"}+ and is wrapping up. -> {"escalate": true, "category":+ "false_progress", "new_evidence": true, "reason": "reproduction never+ exercised the reported code path"}* Two turns of edits, one failed build, then a fixed build and a- passing test. -> {"escalate": false}+ passing test. -> {"escalate": false, "category": "none",+ "new_evidence": false, "reason": "latest build recovered and passed"}* `npm install` has been running for one turn with no output yet. ->- {"escalate": false} — slow command, not a stall.+ {"escalate": false, "category": "none", "new_evidence": false,+ "reason": "command is still running"}* Four different serialization libraries failed to import; the agentis now writing the converter with a fifth approach it has not tried- before. -> {"escalate": false} — sequential alternatives are- adaptation, even when none has succeeded yet.+ before. -> {"escalate": false, "category": "none", "new_evidence":+ false, "reason": "latest turn changed approach"}* Task: tune a slow batch pipeline. The agent is investigating whythe message broker fails to start, since the pipeline cannot be- measured without it. -> {"escalate": false} — unblocking- verification serves the task.+ measured without it. -> {"escalate": false, "category": "none",+ "new_evidence": false, "reason": "working to unblock verification"}Do not emit markdown, commentary, or chain-of-thought — only the JSONobject.diff --git a/crates/libsy/src/prompts/escalation/schema.json b/crates/libsy/src/prompts/escalation/schema.jsonindex 977c41882..e98b3c5fd 100644--- a/crates/libsy/src/prompts/escalation/schema.json+++ b/crates/libsy/src/prompts/escalation/schema.json@@ -10,12 +10,21 @@"type": "boolean","description": "True when the run is likely doomed without escalation to the strong tier."},+ "category": {+ "type": "string",+ "enum": ["none", "repetition", "false_progress", "drift", "desperation", "capability_gap"],+ "description": "The single trouble pattern supporting escalation, or none when escalation is false."+ },+ "new_evidence": {+ "type": "boolean",+ "description": "True only when the newest assistant turn or its resulting tool output adds evidence for the named trouble pattern."+ },"reason": {"type": "string","description": "One short sentence naming the trouble pattern, or stating why the run is progressing."}},- "required": ["escalate", "reason"],+ "required": ["escalate", "category", "new_evidence", "reason"],"additionalProperties": false}}diff --git a/crates/switchyard-server/tests/server.rs b/crates/switchyard-server/tests/server.rsindex cd21d3c91..3c2656257 100644--- a/crates/switchyard-server/tests/server.rs+++ b/crates/switchyard-server/tests/server.rs@@ -409,7 +409,8 @@ async fn upstream_chat(.pointer("/response_format/json_schema/schema/properties/escalate").is_some(){- r#"{"escalate":false,"reason":"making progress"}"#.to_string()+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"making progress"}"#+ .to_string()} else if model == "model/classifier" && requests_schema_invalid_verdict {r#"{"crux":"bounded task","primary_rule":"SUP-1","capability_boundary":"supported","p_solve":0.1,"unexpected":true}"#.to_string()} else if model == "model/classifier" {@@ -598,7 +599,12 @@ async fn upstream_responses_silo(.get("escalate").is_some(){- json!({"escalate": strong, "reason": "state probe"})+ json!({+ "escalate": strong,+ "category": if strong { "capability_gap" } else { "none" },+ "new_evidence": strong,+ "reason": "state probe",+ })} else {json!({"crux": "state probe", "primary_rule": if strong { "LIM-1" } else { "SUP-1" },diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.mdindex 8bf382a08..4b51a7918 100644--- a/docs/reference/toml_schema.md+++ b/docs/reference/toml_schema.md@@ -253,7 +253,7 @@ Escalation mode serves the weak target first and judges the completed turn. See| `strong_target` | Yes | — | Target used after the session latches. || `weak_target` | Yes | — | Target served before the latch. || `prompt` | No | packaged prompt | Replaces the trajectory-judge prompt. |-| `escalation.confirmations` | No | `2` | Consecutive escalate verdicts required to latch. Above `1` needs a stable session ID. |+| `escalation.confirmations` | No | `2` | Consecutive fresh-evidence verdicts for the same failure category required to latch. Above `1` needs a stable session ID. || `escalation.recent_turn_window` | No | `28` | Trailing messages shown to the judge. || `escalation.window_message_chars` | No | `500` | Per-message cap inside that window. || `escalation.deescalation` | No | unset | Enables phase-aware de-escalation. Requires a stable session ID. |diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.mdindex 411446478..994359a08 100644--- a/docs/routing_algorithms/escalation_router_routing.md+++ b/docs/routing_algorithms/escalation_router_routing.md@@ -61,8 +61,10 @@ The route-level `prompt` key replaces the packaged trajectory-judge prompt. Ituses the escalation verdict schema rather than the capability verdict schema.Switchyard supplies that schema according to the route's `response_format_type`:through the structured-output request in the default `json_schema` mode, or in-the prompt in `json_object` mode. When de-escalation is enabled, Switchyard-appends the phase-specific verdict contract to packaged and custom prompts.+the prompt in `json_object` mode. The verdict includes an `escalate` decision, a+bounded failure `category`, whether the newest turn adds `new_evidence`, and a+short `reason`. When de-escalation is enabled, Switchyard appends the+phase-specific verdict contract to packaged and custom prompts.## How the decision works@@ -72,8 +74,10 @@ For each turn on an unlatched session, Switchyard:2. Appends that reply to the transcript and asks the judge to rule on thecompleted turn. The judge therefore rates work the weak model actually did,not a prediction about work it might do.-3. Increments a consecutive-escalate streak on an escalate verdict, and resets it- to zero on a decline.+3. Increments a confirmation streak only when an escalate verdict cites fresh+ evidence for the same failure category as the preceding vote. A different+ category starts a new streak at one; a decline or stale evidence resets it to+ zero.4. Serves the buffered weak reply when the streak has not yet reached`confirmations` — so a judged turn that does not escalate costs one weak callplus one judge call, and no strong call.@@ -119,7 +123,7 @@ configuration, so a bare `escalation = {}` is a valid, tuned route:| Key | Default | Meaning ||---|---|---|-| `confirmations` | `2` | Consecutive escalate verdicts required before the session latches to strong. Must be at least `1`. |+| `confirmations` | `2` | Consecutive fresh-evidence verdicts for the same failure category required before the session latches to strong. Must be at least `1`. || `recent_turn_window` | `28` | Trailing messages shown to the judge on top of the anchors. Must be at least `1`. || `window_message_chars` | `500` | Per-message truncation cap inside that trailing window. Must be at least `50`. |@@ -146,11 +150,13 @@ weak_cooldown_calls = 8With this table present, Switchyard marks judge input as either`EFFICIENT_EVALUATION` or `STRONG_EVALUATION`. In the strong phase,-`escalate: true` keeps the strong tier. An `escalate: false` verdict can release-the next request only after `strong_min_calls` is reached and the configured-confirmation streak is complete. A timeout, error, or unparseable verdict-retains the strong tier. Omitting the table preserves the permanent latch and-does not add phase markers to judge input.+`escalate: true` keeps the strong tier; the router reads only `escalate` there,+so `category` and `new_evidence` are reported but do not affect release. An+`escalate: false` verdict can release the next request only after+`strong_min_calls` is reached and the configured confirmation streak is+complete. A timeout, error, or unparseable verdict retains the strong tier.+Omitting the table preserves the permanent latch and does not add phase markers+to judge input.The strong-phase verdict is judged against the trouble that caused theescalation: the packaged rules release only once the failure that triggered the@@ -179,8 +185,8 @@ route. The request still succeeds, but its temporary state cannot carry into thenext request.Anchor and transcript caps remain fixed. Set the route-level-`max_output_tokens` key to change the judge's reply budget. Any decline still-resets the streak to zero.+`max_output_tokens` key to change the judge's reply budget. Any decline or+verdict without new evidence resets the streak to zero.## Run the route@@ -226,6 +232,10 @@ per-session routing stats under the judge's model id, tagged with the`classifier` tier — so per-session token accounting includes judge overheadalongside the tiers the session was served by.+The server log records each parsed escalation verdict's `escalate` decision,+category, and `new_evidence` flag. The judge's free-form reason is neither+retained nor logged.+## When not to use escalation routing- **One-shot requests.** No trajectory to judge. Use