Skip to content

Switchyard lets LLM applications route traffic across models and providers while preserving native OpenAI and Anthropic API compatibility - enabling flexible model selection, benchmarking, and cost/performance optimization.

Rust

fix(escalation): require fresh consistent failure evidence (#637)

Alex Steiner47bba06c9c45
diff --git a/benchmark/SWE_ATLAS_ESCALATION_REPORT.md b/benchmark/SWE_ATLAS_ESCALATION_REPORT.md
new file mode 100644
index 000000000..92751180b
--- /dev/null
+++ b/benchmark/SWE_ATLAS_ESCALATION_REPORT.md
@@ -0,0 +1,224 @@
+# SWE-Atlas escalation-router findings
+
+## Summary
+
+The escalation router could interpret duplicate command serialization in a NeMo Gym Terminus
+trajectory as repeated failed work. A single assistant turn may contain the same command both as
+raw JSON text and as a structured tool call. The judge prompt previously treated a command shown
+two or more times as loop evidence without requiring those attempts to occur in distinct turns.
+
+This change makes escalation depend on fresh, consistent failure evidence across turns. It keeps
+the existing one-way switch and latch: a session can move from the efficient model to the capable
+model at most once and remains there afterward.
+
+On a deterministic 40-task paired sample (20 SWE-Atlas RF and 20 SWE-Atlas TW), the patched router
+matched the two single-model arms at 15/40 correct and improved over the original router's 10/40.
+It switched 5 times instead of 9. Under the synthetic token prices defined below, it cost
+$0.889/task versus $1.171/task for the original router. GLM-only also scored 15/40 and remained the
+least expensive arm at $0.501/task, so the result supports the patch over the original router but
+does not show that routing beats the best fixed model on this sample.
+
+## Changes
+
+- Normalize only the transcript sent to the escalation judge. When a raw Terminus command batch
+ has a one-to-one match with structured tool calls in the same assistant turn, the redundant raw
+ representation is omitted. The worker model still receives the original history.
+- Require the judge to classify positive evidence as `repetition`, `false_progress`, `drift`,
+ `desperation`, or `capability_gap`, and report whether the evidence is new.
+- Advance the confirmation streak only for fresh positive evidence in the same category. A
+ category change starts a new streak; a decline or stale evidence resets it. An unavailable judge
+ preserves the prior streak rather than converting an infrastructure failure into a routing
+ decision.
+- Clarify in the judge prompt that duplicate serialization within one turn and an immediate,
+ adaptive change of terminal strategy are not repeated failed attempts.
+- Preserve the existing maximum-one-switch and post-switch latch behavior.
+
+## Focused SWE-Atlas reproducer observations
+
+The live checks used GLM 5.2 as the efficient model and Opus 4.8 as the capable model. The reward
+column is the task's binary verifier result. "GLM only" means that the patched router did not
+switch models during the trajectory.
+
+| Dataset | Task | Patched behavior | Reward | Observation |
+| --- | --- | --- | ---: | --- |
+| RF | `697e7458be1623d850a88838` | 28 GLM calls, then 36 Opus calls; one switch | 1 | Two fresh `repetition` findings confirmed the escalation path and latch. |
+| RF | `69391d8d1ce51c407be1e531` | GLM only; 93 logged episodes | 0 | Findings changed from `repetition` to `false_progress`, so they did not form one confirmation streak. The earlier router switched without improving the reward. |
+| RF | `694b4b99829f00e24fd11889` | GLM only; 34 logged episodes | 0 | No positive escalation finding. |
+| TW | `6902ef3ab97fe23e2ad271f3` | GLM only; 58 logged episodes | 0 | One isolated `repetition` finding did not trigger a switch. The earlier router switched without improving the reward. |
+| RF | `696719205599a51110d4b45f` | Three GLM-only runs: 72, 93, and 70 logged episodes | 0, 0, 0 | One isolated positive finding across the replicas; no latch. Direct GLM and direct Opus also scored 0. |
+| RF | `696719205599a51110d4b455` | Three GLM-only runs: 51, 60, and 48 logged episodes | 0, 0, 0 | No transition in any replica. Direct GLM and direct Opus also scored 0. |
+
+A separate passing run of `697e7458be1623d850a88838` produced a positive finding followed by a
+decline, stayed on GLM, and scored 1. This checks that non-consecutive findings do not accumulate.
+
+The positive switched task also scored 1 in a direct-GLM run. It therefore demonstrates correct
+switching, confirmation, and latching, but does not establish a causal accuracy improvement.
+Similarly, the unsuccessful controls show improved escalation precision and avoided Opus calls;
+they do not show that the efficient model could solve those tasks.
+
+## Paired benchmark
+
+### Design
+
+The paired benchmark used the following fixed setup:
+
+- 20 tasks from `scale-ai/swe-atlas-rf@1` and 20 from `scale-ai/swe-atlas-tw@1`.
+ Tasks were selected deterministically by ranking the hash of
+ `20260904:<dataset>:<task>`; they were not selected by outcome.
+- Four arms per task: direct GLM, direct Opus, the original escalation router, and the patched
+ escalation router. This produced 160 task-arm runs.
+- GLM was `nvidia/zai-org/glm-5.2`; Opus was
+ `aws/anthropic/bedrock-claude-opus-4-8` with medium effort. GLM also served as the escalation
+ judge. The two escalation arms used two confirmations, maximum one switch, and a post-switch
+ latch.
+- The original router was built from upstream commit
+ `9a743e89223a0d5b14011f1226d5b068f730a3b8`; the patched router was built from
+ `507376b1d851b825378cf147d09a8b5594773298`.
+- There was no agent-turn limit. Worker requests allowed up to 128,000 output tokens and a
+ 900-second model-server timeout. Five initial runs reached Harbor's separate 3,600-second agent
+ timeout. Only those five task-arm runs were repeated with a 3x agent-timeout multiplier; all five
+ retries completed and replaced the invalid attempts. The final effective matrix is 160/160 valid.
+- Each table entry is one trial per task and arm. Binary verifier reward is reported as correct.
+ Because model sampling is stochastic, differences between unswitched GLM router runs and direct
+ GLM runs are not necessarily routing effects.
+
+### Accuracy and routing
+
+| Dataset | Arm | Correct | Accuracy | Switches | Correct after switch |
+| --- | --- | ---: | ---: | ---: | ---: |
+| RF (20) | Direct GLM | 7 | 0.350 | -- | -- |
+| RF (20) | Always Opus | 9 | 0.450 | -- | -- |
+| RF (20) | Original escalation | 6 | 0.300 | 7 | 2 |
+| RF (20) | **Patched escalation** | **9** | **0.450** | **4** | **2** |
+| TW (20) | Direct GLM | 8 | 0.400 | -- | -- |
+| TW (20) | Always Opus | 6 | 0.300 | -- | -- |
+| TW (20) | Original escalation | 4 | 0.200 | 2 | 0 |
+| TW (20) | **Patched escalation** | **6** | **0.300** | **1** | **0** |
+| Combined (40) | Direct GLM | 15 | 0.375 | -- | -- |
+| Combined (40) | Always Opus | 15 | 0.375 | -- | -- |
+| Combined (40) | Original escalation | 10 | 0.250 | 9 | 2 |
+| Combined (40) | **Patched escalation** | **15** | **0.375** | **5** | **2** |
+
+Against the original router on the same 40 tasks, the patched router won 7 outcomes, lost 2, and
+tied 31. Against direct GLM it won 5 and lost 5; against direct Opus it won 6 and lost 6. All 14
+switched trajectories across the two router arms had exactly one GLM-to-Opus transition and no
+hand-back, confirming the switch and latch invariants in live execution.
+
+The patch traded earlier sensitivity for precision. Original-router switches occurred after a
+median of 23 GLM calls; patched-router switches occurred after a median of 51. One patched RF retry
+did not switch until 141 GLM calls, then made 57 latched Opus calls and still scored 0. Direct Opus
+and the original router also scored 0 on that task. This avoided a premature switch but exposed a
+cost problem: judging every turn can be expensive when the destination model is unlikely to help.
+
+### Synthetic cost calculation
+
+The inference service did not provide billable cost. The values below are synthetic estimates for
+comparing routing behavior, not NVIDIA prices or charges. The assumed rates are:
+
+| Model | Uncached input / 1M tokens | Cached input / 1M tokens | Output / 1M tokens |
+| --- | ---: | ---: | ---: |
+| GLM 5.2 | $0.50 | $0.05 | $2.00 |
+| Opus 4.8 | $5.00 | $0.50 | $25.00 |
+
+For each model in each trajectory:
+
+```text
+uncached_input_tokens = max(prompt_tokens - cached_tokens, 0)
+
+model_cost = (
+ uncached_input_tokens * uncached_input_rate
+ + cached_tokens * cached_input_rate
+ + output_tokens * output_rate
+) / 1,000,000
+
+trajectory_cost = sum(worker_model_costs) + sum(escalation_judge_costs)
+```
+
+Cache-creation tokens are part of the non-cached prompt-token remainder and therefore receive the
+uncached-input rate. The calculation includes GLM judge calls for the two router arms. It excludes
+the Harbor verifier, cluster resources, and other infrastructure because those calls are outside
+Switchyard's per-trajectory statistics.
+
+| Dataset | Arm | Synthetic cost | Cost/task | Cost/correct |
+| --- | --- | ---: | ---: | ---: |
+| RF (20) | Direct GLM | $12.953 | $0.648 | $1.850 |
+| RF (20) | Always Opus | $32.196 | $1.610 | $3.577 |
+| RF (20) | Original escalation | $35.169 | $1.758 | $5.862 |
+| RF (20) | **Patched escalation** | **$27.200** | **$1.360** | **$3.022** |
+| TW (20) | Direct GLM | $7.093 | $0.355 | $0.887 |
+| TW (20) | Always Opus | $14.829 | $0.741 | $2.471 |
+| TW (20) | Original escalation | $11.657 | $0.583 | $2.914 |
+| TW (20) | **Patched escalation** | **$8.356** | **$0.418** | **$1.393** |
+| Combined (40) | Direct GLM | $20.045 | $0.501 | $1.336 |
+| Combined (40) | Always Opus | $47.025 | $1.176 | $3.135 |
+| Combined (40) | Original escalation | $46.826 | $1.171 | $4.683 |
+| Combined (40) | **Patched escalation** | **$35.556** | **$0.889** | **$2.370** |
+
+The patched router was 24.1% cheaper than the original router and produced five additional correct
+outcomes. It was 24.4% cheaper than always-Opus at the same aggregate accuracy. It was 77.4% more
+expensive than direct GLM at the same aggregate accuracy. Of the patched router's $35.556 synthetic
+total, $3.698 (10.4%) came from judge calls; this motivates reducing judge cadence after repeated
+negative verdicts.
+
+### TW model-order reversal
+
+TW demonstrates a limitation that this patch does not solve: the escalation route assumes its
+configured Opus target has positive expected gain over GLM. On this sample, direct GLM scored 8/20
+while direct Opus scored 6/20, so that assumption is not valid at the workload level.
+
+The patched router switched on only one TW task. Direct GLM, direct Opus, and both router arms all
+scored 0 on that task, so the switch added cost without changing its outcome. The patched router's
+two-correct deficit versus direct GLM was not caused by switching: it lost three independently
+sampled GLM-only outcomes and gained one different GLM-only outcome. The original router provides
+a clearer harmful-switch example on task `6902ef3ab97fe23e2ad2727e`: direct GLM scored 1, direct Opus
+scored 0, and the original router switched and scored 0. The patch avoided that switch, although its
+independent GLM-only run also scored 0.
+
+The next routing change should therefore be separate from the trajectory-evidence fix: gate
+escalation on a workload- or task-conditioned estimate of expected accuracy gain and cost. When
+recent paired evidence says the nominal strong model is worse, the route should disable or reverse
+that transition rather than asking only whether the current trajectory looks stuck. This requires
+held-out calibration or online exploration; the router cannot infer relative model quality from one
+model's failing trajectory alone.
+
+## Validation evidence
+
+The implementation was validated with Rust 1.96.1 on a Slurm compute node using:
+
+```text
+cargo fmt --all --check
+cargo clippy --workspace --all-targets --all-features -- -D warnings
+cargo test --workspace --all-features
+uv run ruff check .
+uv run mypy switchyard
+uv run maturin develop --uv
+uv run pytest tests/ -v
+```
+
+The validation completed successfully: the Rust workspace passed, including 289 `libsy` tests;
+ruff and mypy passed; the native wheel built and installed; and pytest reported 115 passed,
+2 deselected, and 2 subtests passed.
+Focused regression coverage includes exact and multi-command transcript normalization, partial and
+multiplicity mismatches, category changes, stale evidence, unavailable-judge behavior, and the
+one-way latch.
+
+After rebasing onto current `main` on 2026-09-22, the review fixes were revalidated with Rust
+1.96.1 and Python 3.12 using `cargo fmt --all --check`, workspace clippy with all targets and
+features, and workspace tests with all features. The full Rust gate passed, including 323 `libsy`
+tests and all 56 server integration tests. This post-rebase run caught and fixed stale call
+signatures and mock verdicts that still used the pre-rebase runtime interfaces and escalation
+schema.
+
+No live provider calls are made by the repository validation commands. The live SWE-Atlas checks
+described above were separate, explicitly configured benchmark runs.
+
+## Conclusion
+
+The patch fixes the concrete duplicate-serialization false positive and materially improves the
+tested router: it matches the single-model aggregate accuracy, gains 5 correct outcomes over the
+original router, cuts switches from 9 to 5, and reduces synthetic cost by 24.1%. It does not make
+the router the best cost/accuracy arm; direct GLM matches its aggregate accuracy at substantially
+lower cost. The remaining work is not to loosen the new evidence rule globally. It is to add
+model-order calibration for workloads such as TW and reduce judge overhead or stop judging when
+escalation has low expected value. Replicated trials on held-out tasks are needed before treating
+the point estimates as stable production gains.
diff --git a/crates/libsy-llm-client/tests/observability.rs b/crates/libsy-llm-client/tests/observability.rs
index 32aae6feb..93c12adb5 100644
--- a/crates/libsy-llm-client/tests/observability.rs
+++ b/crates/libsy-llm-client/tests/observability.rs
@@ -772,7 +772,9 @@ async fn stateful_escalation_warns_once_without_a_session_id() -> switchyard_lib
})?) as Arc<dyn Algorithm>;
let client = Arc::new(JudgeClient {
judge_model: "warning-judge".into(),
- outcome: JudgeOutcome::Reply(r#"{"escalate":false,"reason":"progressing"}"#),
+ outcome: JudgeOutcome::Reply(
+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#,
+ ),
}) as Arc<dyn RoutedLlmClient>;
for _ in 0..2 {
@@ -828,7 +830,9 @@ async fn deescalation_evidence_stays_pending_until_confirmed() -> switchyard_lib
switchyard_llm_client::run(
router.clone(),
- ClientRouter::single(client(r#"{"escalate":true,"reason":"stuck"}"#)),
+ ClientRouter::single(client(
+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck"}"#,
+ )),
request.clone(),
classifier_models("evidence-judge", "evidence-efficient", "evidence-capable"),
None,
@@ -836,7 +840,9 @@ async fn deescalation_evidence_stays_pending_until_confirmed() -> switchyard_lib
.await?;
let outcome = switchyard_llm_client::decide(
router,
- ClientRouter::single(client(r#"{"escalate":false,"reason":"recovered"}"#)),
+ ClientRouter::single(client(
+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"recovered"}"#,
+ )),
request,
classifier_models("evidence-judge", "evidence-efficient", "evidence-capable"),
)
diff --git a/crates/libsy/src/algorithms/escalation.rs b/crates/libsy/src/algorithms/escalation.rs
index 534ef34a0..76c77d76b 100644
--- a/crates/libsy/src/algorithms/escalation.rs
+++ b/crates/libsy/src/algorithms/escalation.rs
@@ -15,8 +15,8 @@ use super::util::buffered_response::buffer_response;
use super::util::classifier_contract::ClassifierContractConfig;
use super::util::decisive;
use super::util::escalation::{
- self, DeescalationConfig, EscalationJudge, EscalationJudgeConfig, EscalationPolicy,
- EvaluationPhase,
+ self, DeescalationConfig, EscalationCategory, EscalationJudge, EscalationJudgeConfig,
+ EscalationPolicy, EvaluationPhase,
};
use super::util::llm_judge::JudgeClassifier;
use crate::core::algorithm::Driver;
@@ -25,6 +25,8 @@ use crate::core::state::{State, StateValue};
use crate::{LibsyError, Result};
const STREAK_KEY: &str = "escalation_streak";
+/// Session-state key holding the category currently being confirmed.
+const CATEGORY_KEY: &str = "escalation_category";
const STRONG_CALLS_KEY: &str = "escalation_strong_calls";
const RELEASE_STREAK_KEY: &str = "escalation_release_streak";
const WEAK_COOLDOWN_KEY: &str = "escalation_weak_cooldown";
@@ -42,6 +44,20 @@ fn set_count(state: &mut State, key: &str, value: u32) {
.insert(key.to_string(), StateValue::Count(value));
}
+/// Reads the failure category whose confirmation streak is in progress, if any.
+fn category(state: &State) -> Option<&str> {
+ match state.extra.get(CATEGORY_KEY) {
+ Some(StateValue::String(category)) => Some(category),
+ _ => None,
+ }
+}
+
+/// Clears the confirmation streak together with the category it was confirming.
+fn clear_streak(state: &mut State) {
+ set_count(state, STREAK_KEY, 0);
+ state.extra.remove(CATEGORY_KEY);
+}
+
fn assistant_message(response: &AggLlmResponse) -> Message {
Message {
role: Role::Assistant,
@@ -171,7 +187,7 @@ impl EscalationClassifier {
set_count(state, RELEASE_STREAK_KEY, release_streak);
if release_streak >= deescalation.config.confirmations {
- set_count(state, STREAK_KEY, 0);
+ clear_streak(state);
set_count(state, STRONG_CALLS_KEY, 0);
set_count(state, RELEASE_STREAK_KEY, 0);
tracing::debug!(
@@ -225,7 +241,7 @@ impl Classifier<State> for EscalationClassifier {
.strong_max_calls
.is_some_and(|strong_max_calls| strong_calls >= strong_max_calls)
{
- set_count(state, STREAK_KEY, 0);
+ clear_streak(state);
set_count(state, STRONG_CALLS_KEY, 0);
set_count(state, RELEASE_STREAK_KEY, 0);
set_count(
@@ -336,19 +352,52 @@ impl Classifier<State> for EscalationClassifier {
));
}
- let (classification, _) = self
+ let judge_models = driver.models_for(&Category::Judge);
+ if judge_models.is_empty() {
+ return Err(LibsyError::AlgorithmError {
+ message: "no models available for category Judge".to_string(),
+ });
+ }
+ let verdict = self
.escalation_judge
- .score(state, &mut judge_request, driver)
- .await?;
+ .verdict(state, &judge_request, driver, judge_models)
+ .await;
let held = count(state, STREAK_KEY);
- let best = classification.argmax(false)?;
- let (escalate, pending) = match &best {
- Some(score) if score.target == capable => (true, held.saturating_add(1)),
- Some(_) => (false, 0),
- None => (false, held),
+ let held_category = category(state).map(str::to_string);
+ let (escalate, pending, pending_category) = match verdict.as_ref() {
+ Some(verdict) => {
+ let category = verdict.category.label();
+ tracing::info!(
+ escalate = verdict.escalate,
+ category,
+ new_evidence = verdict.new_evidence,
+ "escalation judge verdict"
+ );
+ if verdict.escalate
+ && verdict.new_evidence
+ && verdict.category != EscalationCategory::None
+ {
+ let next = if held_category.as_deref() == Some(category) {
+ held.saturating_add(1)
+ } else {
+ 1
+ };
+ (true, next, Some(category.to_string()))
+ } else {
+ (false, 0, None)
+ }
+ }
+ None => (false, held, held_category),
};
set_count(state, STREAK_KEY, pending);
+ if let Some(category) = pending_category {
+ state
+ .extra
+ .insert(CATEGORY_KEY.to_string(), StateValue::String(category));
+ } else {
+ state.extra.remove(CATEGORY_KEY);
+ }
if escalate && pending >= self.confirmations {
driver.set_evidence(serde_json::json!({
@@ -367,6 +416,11 @@ impl Classifier<State> for EscalationClassifier {
"source": "escalation",
"verdict": "pending",
}));
+ } else if verdict.is_some() {
+ driver.set_evidence(serde_json::json!({
+ "source": "escalation",
+ "verdict": "continue",
+ }));
}
Ok((
@@ -475,13 +529,13 @@ mod tests {
}
}
- /// Builds a router with escalation enabled (`confirmations=1` latches immediately).
- fn escalation_router() -> Result<Arc<LlmTaskClassifier>> {
+ /// Builds a router with escalation enabled.
+ fn escalation_router_with_confirmations(confirmations: u32) -> Result<Arc<LlmTaskClassifier>> {
Ok(Arc::new(LlmTaskClassifier::new(
LlmClassifierConfig::Escalation {
contract: ClassifierContractConfig::default(),
config: EscalationJudgeConfig {
- confirmations: 1,
+ confirmations,
..EscalationJudgeConfig::default()
},
max_output_tokens: DEFAULT_JUDGE_MAX_OUTPUT_TOKENS,
@@ -526,9 +580,16 @@ mod tests {
Ok(selected)
}
+ /// Builds a router that latches on its first supported escalation verdict.
+ fn escalation_router() -> Result<Arc<LlmTaskClassifier>> {
+ escalation_router_with_confirmations(1)
+ }
+
#[tokio::test]
async fn serves_efficient_when_judge_declines() -> Result<()> {
- let judge = Queue::new([r#"{"escalate":false,"reason":"progressing"}"#]);
+ let judge = Queue::new([
+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#,
+ ]);
let model = Queue::new(["efficient answer"]);
let (selected_model, response) = test_drive_with_models(
@@ -547,6 +608,44 @@ mod tests {
Ok(())
}
+ /// A parsed decline records continue evidence on the routing outcome.
+ #[tokio::test]
+ async fn records_continue_evidence_when_judge_declines() -> Result<()> {
+ let judge = Queue::new([
+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#,
+ ]);
+ let model = Queue::new(["efficient answer"]);
+ let serve = Arc::new(queued(model, judge));
+ let routing_serve = Arc::clone(&serve);
+ let outcome = crate::drive(
+ escalation_router()?,
+ classify_request(),
+ Arc::new(crate::core::algorithm::RuntimeModels::new(runtime_models())),
+ move |call| {
+ let serve = Arc::clone(&routing_serve);
+ async move {
+ let target = call.models.first().cloned().ok_or(LibsyError::NoTargets)?;
+ let request = call.request.clone();
+ let response = serve
+ .serve(target.clone(), request)
+ .await
+ .map_err(|source| LibsyError::client_call(target, source));
+ call.respond(response)
+ }
+ },
+ )
+ .await?;
+
+ assert_eq!(
+ outcome.metadata.and_then(|metadata| metadata.evidence),
+ Some(serde_json::json!({
+ "source": "escalation",
+ "verdict": "continue",
+ }))
+ );
+ Ok(())
+ }
+
#[tokio::test]
async fn config_overrides_the_packaged_prompt() -> Result<()> {
let prompts = Arc::new(Mutex::new(Vec::new()));
@@ -564,7 +663,9 @@ mod tests {
})
});
recorded.lock().extend(prompt);
- std::future::ready(Ok(reply(r#"{"escalate":false,"reason":"progressing"}"#)))
+ std::future::ready(Ok(reply(
+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#,
+ )))
} else {
std::future::ready(Ok(reply("efficient answer")))
}
@@ -586,7 +687,9 @@ mod tests {
#[tokio::test]
async fn upgrades_to_capable_when_judge_escalates() -> Result<()> {
- let judge = Queue::new([r#"{"escalate":true,"reason":"stuck in a loop"}"#]);
+ let judge = Queue::new([
+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck in a loop"}"#,
+ ]);
let model = Queue::new(["efficient draft", "capable answer"]);
let (selected_model, response) = test_drive_with_models(
@@ -605,9 +708,106 @@ mod tests {
Ok(())
}
+ /// A verdict in a different category restarts the confirmation streak at one.
+ #[tokio::test]
+ async fn confirmation_streak_requires_the_same_category() -> Result<()> {
+ let judge = Queue::new([
+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"repeated command"}"#,
+ r#"{"escalate":true,"category":"drift","new_evidence":true,"reason":"off task"}"#,
+ r#"{"escalate":true,"category":"drift","new_evidence":true,"reason":"still off task"}"#,
+ ]);
+ let model = Queue::new(["efficient t1", "efficient t2", "efficient t3", "capable t3"]);
+ let router = escalation_router_with_confirmations(2)?;
+ let request = classify_session_request();
+
+ let (first, _) = test_drive_with_models(
+ router.clone(),
+ request.clone(),
+ runtime_models(),
+ queued(Arc::clone(&model), Arc::clone(&judge)),
+ )
+ .await?;
+ let (second, _) = test_drive_with_models(
+ router.clone(),
+ request.clone(),
+ runtime_models(),
+ queued(Arc::clone(&model), Arc::clone(&judge)),
+ )
+ .await?;
+ let (third, _) =
+ test_drive_with_models(router, request, runtime_models(), queued(model, judge)).await?;
+
+ assert_eq!(first, "efficient");
+ assert_eq!(second, "efficient");
+ assert_eq!(third, "capable");
+ Ok(())
+ }
+
+ /// An escalate verdict without fresh evidence resets the streak instead of extending it.
+ #[tokio::test]
+ async fn verdict_without_new_evidence_resets_the_streak() -> Result<()> {
+ let judge = Queue::new([
+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"repeated command"}"#,
+ r#"{"escalate":true,"category":"repetition","new_evidence":false,"reason":"only old evidence remains"}"#,
+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"new repeated command"}"#,
+ ]);
+ let model = Queue::new(["efficient t1", "efficient t2", "efficient t3"]);
+ let router = escalation_router_with_confirmations(2)?;
+ let request = classify_session_request();
+
+ for _ in 0..3 {
+ let (selected, _) = test_drive_with_models(
+ router.clone(),
+ request.clone(),
+ runtime_models(),
+ queued(Arc::clone(&model), Arc::clone(&judge)),
+ )
+ .await?;
+ assert_eq!(selected, "efficient");
+ }
+ Ok(())
+ }
+
+ /// An unparseable verdict keeps the streak and its category for the next turn.
+ #[tokio::test]
+ async fn unavailable_judge_preserves_the_category_streak() -> Result<()> {
+ let judge = Queue::new([
+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"repeated command"}"#,
+ "not json",
+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"another repeated command"}"#,
+ ]);
+ let model = Queue::new(["efficient t1", "efficient t2", "efficient t3", "capable t3"]);
+ let router = escalation_router_with_confirmations(2)?;
+ let request = classify_session_request();
+
+ let (first, _) = test_drive_with_models(
+ router.clone(),
+ request.clone(),
+ runtime_models(),
+ queued(Arc::clone(&model), Arc::clone(&judge)),
+ )
+ .await?;
+ let (second, _) = test_drive_with_models(
+ router.clone(),
+ request.clone(),
+ runtime_models(),
+ queued(Arc::clone(&model), Arc::clone(&judge)),
+ )
+ .await?;
+ let (third, _) =
+ test_drive_with_models(router, request, runtime_models(), queued(model, judge)).await?;
+
+ assert_eq!(first, "efficient");
+ assert_eq!(second, "efficient");
+ assert_eq!(third, "capable");
+ Ok(())
+ }
+
#[tokio::test]
async fn stays_capable_after_latch() -> Result<()> {
- let judge = Queue::new([r#"{"escalate":true,"reason":"stuck"}"#]);
+ let judge = Queue::new([
+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck"}"#,
+ ]);
let model = Queue::new(["efficient draft", "capable t1", "capable t2"]);
let router = escalation_router()?;
let request = classify_session_request();
@@ -629,10 +829,10 @@ mod tests {
#[tokio::test]
async fn deescalation_holds_then_returns_to_efficient() -> Result<()> {
let judge = Queue::new([
- r#"{"escalate":true,"reason":"stuck"}"#,
- r#"{"escalate":false,"reason":"recovered"}"#,
- r#"{"escalate":false,"reason":"routine"}"#,
- r#"{"escalate":false,"reason":"progressing"}"#,
+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck"}"#,
+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"recovered"}"#,
+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"routine"}"#,
+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#,
]);
let model = Queue::new([
"efficient draft",
@@ -658,10 +858,10 @@ mod tests {
#[tokio::test]
async fn deescalation_hard_limit_forces_a_weak_cooldown() -> Result<()> {
let judge = Queue::new([
- r#"{"escalate":true,"reason":"stuck"}"#,
- r#"{"escalate":true,"reason":"still hard"}"#,
- r#"{"escalate":true,"reason":"still hard"}"#,
- r#"{"escalate":false,"reason":"progressing"}"#,
+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck"}"#,
+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"still hard"}"#,
+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"still hard"}"#,
+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"progressing"}"#,
]);
let model = Queue::new([
"efficient draft",
@@ -695,7 +895,9 @@ mod tests {
#[tokio::test]
async fn strong_review_returns_an_efficient_fallback_without_judging_it() -> Result<()> {
- let judge = Queue::new([r#"{"escalate":true,"reason":"stuck"}"#]);
+ let judge = Queue::new([
+ r#"{"escalate":true,"category":"repetition","new_evidence":true,"reason":"stuck"}"#,
+ ]);
let model = Queue::new(["efficient draft", "capable t1", "efficient fallback"]);
let router = deescalation_router(DeescalationConfig {
strong_min_calls: 1,
diff --git a/crates/libsy/src/algorithms/util/escalation.rs b/crates/libsy/src/algorithms/util/escalation.rs
index 0335b4d3f..2160cb7ec 100644
--- a/crates/libsy/src/algorithms/util/escalation.rs
+++ b/crates/libsy/src/algorithms/util/escalation.rs
@@ -94,8 +94,9 @@ impl DeescalationConfig {
#[derive(Clone, Debug, Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct EscalationJudgeConfig {
- /// Consecutive escalate verdicts required before a turn moves to the capable tier, which
- /// is also the turn that latches the session. Any decline clears the streak.
+ /// Consecutive fresh-evidence verdicts in the same category required before a turn moves to
+ /// the capable tier, which is also the turn that latches the session. Any decline or stale
+ /// evidence clears the streak.
/// `1` escalates on the first verdict; the router's main cost dial.
/// `2` or higher needs a session id, since the streak is retained per session.
pub confirmations: u32,
@@ -157,15 +158,39 @@ impl EvaluationPhase {
}
}
-/// The judge's verdict. The schema also requires a `reason`, which makes the judge state its
-/// case and measurably sharpens the verdict. Routing reads only the boolean; the reason is
-/// kept solely so an operator can see why the judge held or escalated when the
-/// `switchyard_libsy::algorithms::util::escalation` target is enabled at `debug`.
+/// Bounded trouble pattern used to correlate escalation confirmations across turns.
+#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq)]
+#[serde(rename_all = "snake_case")]
+pub(crate) enum EscalationCategory {
+ None,
+ Repetition,
+ FalseProgress,
+ Drift,
+ Desperation,
+ CapabilityGap,
+}
+
+impl EscalationCategory {
+ /// Stable state and telemetry label.
+ pub(crate) const fn label(self) -> &'static str {
+ match self {
+ Self::None => "none",
+ Self::Repetition => "repetition",
+ Self::FalseProgress => "false_progress",
+ Self::Drift => "drift",
+ Self::Desperation => "desperation",
+ Self::CapabilityGap => "capability_gap",
+ }
+ }
+}
+
+/// The judge's typed verdict, including the evidence needed to confirm a stable pattern.
#[derive(Deserialize)]
pub(crate) struct EscalationVerdict {
- escalate: bool,
- #[serde(default)]
- reason: String,
+ pub(crate) escalate: bool,
+ pub(crate) category: EscalationCategory,
+ pub(crate) new_evidence: bool,
+ pub(crate) reason: String,
}
/// Builds the condensed trajectory presented to the escalation judge.
@@ -314,21 +339,102 @@ pub(crate) fn conversation_turn(request: &Request) -> usize {
/// relies on.
fn message_text(message: &Message) -> String {
let mut parts = Vec::new();
- collect_text(&message.content, &mut parts);
+ let terminus_commands = if message.role == Role::Assistant {
+ message
+ .content
+ .iter()
+ .filter_map(|block| match block {
+ ContentBlock::ToolCall(call) if call.name == "bash_command" => call
+ .arguments
+ .get("keystrokes")
+ .and_then(|value| value.as_str()),
+ _ => None,
+ })
+ .collect::<Vec<_>>()
+ } else {
+ Vec::new()
+ };
+ collect_text(&message.content, &mut parts, &terminus_commands);
parts.join(" ")
}
+/// Removes a Terminus command batch when a structured bash call carries the same action.
+///
+/// The model-facing request remains untouched. Only the judge's plain-text view is normalized,
+/// so one action cannot look like two attempts while the agent still sees its native history.
+fn without_duplicated_terminus_commands(text: &str, tool_commands: &[&str]) -> String {
+ let mut normalized_text = String::with_capacity(text.len());
+ let mut unmatched_tool_commands = tool_commands.to_vec();
+ let mut copied_through = 0;
+ let mut scan_from = 0;
+
+ while let Some(relative_start) = text[scan_from..].find('{') {
+ let start = scan_from + relative_start;
+
+ let mut values =
+ serde_json::Deserializer::from_str(&text[start..]).into_iter::<serde_json::Value>();
+ let Some(Ok(mut value)) = values.next() else {
+ scan_from = start + 1;
+ continue;
+ };
+ let end = start + values.byte_offset();
+ let Some(commands) = value
+ .get("commands")
+ .and_then(|commands| commands.as_array())
+ else {
+ scan_from = start + 1;
+ continue;
+ };
+ let Some(command_batch) = commands
+ .iter()
+ .map(|command| command.get("keystrokes").and_then(|value| value.as_str()))
+ .collect::<Option<Vec<_>>>()
+ else {
+ scan_from = start + 1;
+ continue;
+ };
+ let mut remaining_tool_commands = unmatched_tool_commands.clone();
+ let fully_encoded = command_batch.iter().all(|command| {
+ let Some(index) = remaining_tool_commands
+ .iter()
+ .position(|candidate| candidate == command)
+ else {
+ return false;
+ };
+ remaining_tool_commands.swap_remove(index);
+ true
+ });
+ if command_batch.is_empty() || !fully_encoded {
+ scan_from = start + 1;
+ continue;
+ }
+
+ value["commands"] = serde_json::Value::Array(Vec::new());
+ let Ok(normalized) = serde_json::to_string(&value) else {
+ scan_from = start + 1;
+ continue;
+ };
+ normalized_text.push_str(&text[copied_through..start]);
+ normalized_text.push_str(&normalized);
+ copied_through = end;
+ scan_from = end;
+ unmatched_tool_commands = remaining_tool_commands;
+ }
+ normalized_text.push_str(&text[copied_through..]);
+ normalized_text
+}
+
/// Appends the judge-relevant text of each block, descending into tool results.
-fn collect_text(content: &[ContentBlock], parts: &mut Vec<String>) {
+fn collect_text(content: &[ContentBlock], parts: &mut Vec<String>, tool_commands: &[&str]) {
for block in content {
match block {
ContentBlock::Text { text } | ContentBlock::Refusal { text } => {
- parts.push(text.clone());
+ parts.push(without_duplicated_terminus_commands(text, tool_commands));
}
ContentBlock::ToolCall(call) => {
parts.push(format!("tool_call {}({})", call.name, call.arguments));
}
- ContentBlock::ToolResult(result) => collect_text(&result.content, parts),
+ ContentBlock::ToolResult(result) => collect_text(&result.content, parts, &[]),
_ => {}
}
}
@@ -378,7 +484,7 @@ fn summarize_for_judge(
for instruction in instructions {
let mut parts = Vec::new();
- collect_text(&instruction.content, &mut parts);
+ collect_text(&instruction.content, &mut parts, &[]);
instruction_anchors.push(format!(
"[{}] {}",
role_label(instruction.role),
@@ -648,6 +754,194 @@ mod tests {
assert_eq!(message_text(&result), "no such file");
}
+ /// A raw command batch fully mirrored by structured tool calls is emptied in the judge view.
+ #[test]
+ fn message_text_deduplicates_terminus_commands_for_the_judge() {
+ let first_command = "grep -n bug app.py\n";
+ let second_command = "sed -n '1,80p' app.py\n";
+ let message = Message {
+ role: Role::Assistant,
+ content: vec![
+ ContentBlock::Text {
+ text: format!(
+ "Before\n```json\n{}\n```\nAfter",
+ json!({
+ "analysis": "inspect the reported file",
+ "commands": [
+ {"keystrokes": first_command, "duration": 0.1},
+ {"keystrokes": second_command, "duration": 0.1},
+ ],
+ })
+ ),
+ },
+ ContentBlock::ToolCall(ToolCall {
+ id: "call-1".to_string(),
+ name: "bash_command".to_string(),
+ arguments: json!({"keystrokes": first_command, "duration": 0.1}),
+ }),
+ ContentBlock::ToolCall(ToolCall {
+ id: "call-2".to_string(),
+ name: "bash_command".to_string(),
+ arguments: json!({"keystrokes": second_command, "duration": 0.1}),
+ }),
+ ],
+ };
+
+ let text = message_text(&message);
+
+ assert!(text.contains("inspect the reported file"), "{text}");
+ assert!(text.contains("Before"), "{text}");
+ assert!(text.contains("After"), "{text}");
+ assert!(text.contains(r#""commands":[]"#), "{text}");
+ assert_eq!(text.matches("grep -n bug app.py").count(), 1, "{text}");
+ assert_eq!(text.matches("sed -n '1,80p' app.py").count(), 1, "{text}");
+ assert_eq!(text.matches("tool_call bash_command(").count(), 2, "{text}");
+ }
+
+ /// Each structured tool call can absorb only one rendered batch, so a repeated batch stays.
+ #[test]
+ fn message_text_deduplicates_multiple_batches_once_per_tool_call() {
+ let first_command = "grep -n bug app.py\n";
+ let second_command = "sed -n '1,80p' app.py\n";
+ let batch = |command| {
+ json!({
+ "analysis": "inspect",
+ "commands": [{"keystrokes": command}],
+ })
+ .to_string()
+ };
+ let message = Message {
+ role: Role::Assistant,
+ content: vec![
+ ContentBlock::Text {
+ text: format!(
+ "First {} second {} repeated {}",
+ batch(first_command),
+ batch(second_command),
+ batch(first_command)
+ ),
+ },
+ ContentBlock::ToolCall(ToolCall {
+ id: "call-1".to_string(),
+ name: "bash_command".to_string(),
+ arguments: json!({"keystrokes": first_command}),
+ }),
+ ContentBlock::ToolCall(ToolCall {
+ id: "call-2".to_string(),
+ name: "bash_command".to_string(),
+ arguments: json!({"keystrokes": second_command}),
+ }),
+ ],
+ };
+
+ let text = message_text(&message);
+
+ assert_eq!(text.matches(r#""commands":[]"#).count(), 2, "{text}");
+ assert_eq!(text.matches("grep -n bug app.py").count(), 2, "{text}");
+ assert_eq!(text.matches("sed -n '1,80p' app.py").count(), 1, "{text}");
+ }
+
+ /// A rendered batch stays when no structured tool call carries the same command.
+ #[test]
+ fn message_text_keeps_terminus_commands_without_matching_tool_call() {
+ let command = "grep -n bug app.py\n";
+ let text = json!({
+ "analysis": "inspect the reported file",
+ "commands": [{"keystrokes": command}],
+ })
+ .to_string();
+ let without_tool_call = Message {
+ role: Role::Assistant,
+ content: vec![ContentBlock::Text { text: text.clone() }],
+ };
+ let mismatched_tool_call = Message {
+ role: Role::Assistant,
+ content: vec![
+ ContentBlock::Text { text },
+ ContentBlock::ToolCall(ToolCall {
+ id: "call-1".to_string(),
+ name: "bash_command".to_string(),
+ arguments: json!({"keystrokes": "sed -n '1,20p' app.py\n", "duration": 0.1}),
+ }),
+ ],
+ };
+
+ assert_eq!(
+ message_text(&without_tool_call)
+ .matches("grep -n bug app.py")
+ .count(),
+ 1
+ );
+ assert_eq!(
+ message_text(&mismatched_tool_call)
+ .matches("grep -n bug app.py")
+ .count(),
+ 1
+ );
+ }
+
+ /// A batch stays intact when only some of its commands have matching tool calls.
+ #[test]
+ fn message_text_keeps_a_partially_encoded_terminus_batch() {
+ let first_command = "grep -n bug app.py\n";
+ let second_command = "sed -n '1,80p' app.py\n";
+ let message = Message {
+ role: Role::Assistant,
+ content: vec![
+ ContentBlock::Text {
+ text: json!({
+ "commands": [
+ {"keystrokes": first_command},
+ {"keystrokes": second_command},
+ ],
+ })
+ .to_string(),
+ },
+ ContentBlock::ToolCall(ToolCall {
+ id: "call-1".to_string(),
+ name: "bash_command".to_string(),
+ arguments: json!({"keystrokes": first_command}),
+ }),
+ ],
+ };
+
+ let text = message_text(&message);
+
+ assert_eq!(text.matches("grep -n bug app.py").count(), 2, "{text}");
+ assert_eq!(text.matches("sed -n '1,80p' app.py").count(), 1, "{text}");
+ assert!(!text.contains(r#""commands":[]"#), "{text}");
+ }
+
+ /// Duplicate commands in one batch need one tool call each before the batch is removed.
+ #[test]
+ fn message_text_keeps_duplicate_commands_without_one_tool_call_each() {
+ let command = "grep -n bug app.py\n";
+ let message = Message {
+ role: Role::Assistant,
+ content: vec![
+ ContentBlock::Text {
+ text: json!({
+ "commands": [
+ {"keystrokes": command},
+ {"keystrokes": command},
+ ],
+ })
+ .to_string(),
+ },
+ ContentBlock::ToolCall(ToolCall {
+ id: "call-1".to_string(),
+ name: "bash_command".to_string(),
+ arguments: json!({"keystrokes": command}),
+ }),
+ ],
+ };
+
+ let text = message_text(&message);
+
+ assert_eq!(text.matches("grep -n bug app.py").count(), 3, "{text}");
+ assert!(!text.contains(r#""commands":[]"#), "{text}");
+ }
+
#[test]
fn truncate_middle_keeps_head_and_tail() {
let text = "a".repeat(40) + &"z".repeat(40);
diff --git a/crates/libsy/src/algorithms/util/llm_judge.rs b/crates/libsy/src/algorithms/util/llm_judge.rs
index 4798178e9..cb8277dae 100644
--- a/crates/libsy/src/algorithms/util/llm_judge.rs
+++ b/crates/libsy/src/algorithms/util/llm_judge.rs
@@ -261,7 +261,7 @@ where
/// mid-stream, or unparseable reply — is logged and folded into `None` for the policy's
/// fallback branch. A closed driver stream is folded too; the algorithm's next driver
/// call surfaces it, so nothing is masked.
- async fn verdict(
+ pub(crate) async fn verdict(
&self,
state: &mut State,
request: &Request,
diff --git a/crates/libsy/src/prompts/escalation/deescalation.md b/crates/libsy/src/prompts/escalation/deescalation.md
index 3d1e3fe9c..4f9870829 100644
--- a/crates/libsy/src/prompts/escalation/deescalation.md
+++ b/crates/libsy/src/prompts/escalation/deescalation.md
@@ -27,5 +27,12 @@ The routing input begins with one of these router-generated markers:
success, or a context compaction. A strong-tier turn that merely reads
files or plans is not by itself evidence that the trouble is resolved.
+ In this phase the router reads only `escalate`. Fill the other fields
+ consistently anyway: when retaining, set `category` to the trouble
+ pattern that caused the escalation and `new_evidence` to true only when
+ the newest strong-tier turn or its tool output shows that trouble is
+ still active; when releasing, return `category: "none"` and
+ `new_evidence: false`.
+
The router, not the judge, applies confirmation counts and decides when
to change tiers. Judge only the phase named in the routing input.
diff --git a/crates/libsy/src/prompts/escalation/prompt.md b/crates/libsy/src/prompts/escalation/prompt.md
index 6d57369bb..d29a4017f 100644
--- a/crates/libsy/src/prompts/escalation/prompt.md
+++ b/crates/libsy/src/prompts/escalation/prompt.md
@@ -9,11 +9,20 @@ messages and tool results). Judge the *trajectory* — is the agent making
real progress toward the stated task — not the difficulty of the task
itself. Return exactly one JSON object:
-{"escalate": boolean, "reason": "one short sentence naming the pattern"}
+{"escalate": boolean, "category": "none|repetition|false_progress|drift|desperation|capability_gap", "new_evidence": boolean, "reason": "one short sentence naming the pattern"}
+
+Use `category: "none"` whenever `escalate` is false. `new_evidence`
+is true only when the NEWEST assistant turn or its resulting tool output
+adds evidence for the named pattern. Older turns may establish context,
+but do not repeat an escalation vote solely because old trouble remains
+visible in the rolling transcript. If the newest turn recovered, adapted,
+or made progress, return `escalate: false`, `category: "none"`, and
+`new_evidence: false`.
Escalation is one-way for the rest of the task and expensive. Escalate
only on a clear PATTERN of trouble, never on a single failed command.
-When the evidence is thin or ambiguous, return {"escalate": false}.
+When the evidence is thin or ambiguous, decline with `category: "none"`
+and `new_evidence: false`.
The bar is not "is there friction" — agentic coding is full of friction
the efficient tier works through on its own. The bar is "is this run
@@ -58,8 +67,10 @@ Hold weak — no model can fix these, so escalation is pure waste:
# Trouble patterns — escalate when you see these
Repetition and loops (the most common way agent runs die):
-- The same command or edit failing 2+ times with materially the same
- error, especially with unrelated changes in between.
+- The same command or edit failing across 2+ DISTINCT assistant turns
+ with materially the same error, especially with unrelated changes in
+ between. Count executed attempts across turns, not repeated renderings
+ inside one message.
- Near-identical tool calls repeated, or the same files re-read, without
new information gained — including longer cycles (A -> B -> C -> A).
- Fighting the environment: repeatedly invoking a missing executable,
@@ -105,6 +116,15 @@ Desperation:
# Expected friction — do NOT escalate on these
Agentic coding is full of failures that are part of healthy work:
+- A command appearing both in the assistant's JSON/text and as one or
+ more structured tool-call blocks in that SAME turn. Agent harnesses
+ commonly serialize one intended action more than once; this is one
+ attempt, not a loop. Repetition is evidence only when separate turns
+ show separate executions with materially the same failed result.
+- Terminal-input serialization trouble (for example tabs triggering
+ completion or a heredoc being mangled) when the next turn changes the
+ write mechanism, quoting, or transport. That is adaptation, not
+ repeated failure.
- A test written to fail first (TDD) or a bug being reproduced on
purpose.
- A compile, lint, or test error fixed or meaningfully acted on in the
@@ -130,49 +150,56 @@ Agentic coding is full of failures that are part of healthy work:
- A long-running command (build, install, test suite) that simply has
not finished, or the agent waiting on information it asked for.
-The distinguishing question: is each failure producing new information
-that changes the next action? Failing forward is fine; failing in place
-is trouble. Also weigh the session's own recovery record: if this same
-session already shows friction the agent subsequently cleared (a failure
-followed by a verified fix or passing check), lean toward holding — a
-session that has recovered before will usually recover again.
+The distinguishing question: across DISTINCT assistant turns, is each
+failure producing new information that changes the next action? Failing
+forward is fine; failing in place is trouble. Never infer a multi-turn
+pattern from duplicated representations inside one turn. Also weigh the
+session's own recovery record: if this same session already shows
+friction the agent subsequently cleared (a failure followed by a verified
+fix or passing check), lean toward holding — a session that has recovered
+before will usually recover again.
# Worked examples (none drawn from any benchmark task set)
* Turn 3; the agent ran the test suite, 4 tests fail, and it is now
- reading the first failing test. -> {"escalate": false} — reproducing
- failures is the job.
+ reading the first failing test. -> {"escalate": false, "category":
+ "none", "new_evidence": false, "reason": "working through the first
+ reproduced failure"}
* The agent has run `pytest tests/test_api.py` 4 times with the same
ImportError, editing an unrelated config file between attempts. ->
- {"escalate": true, "reason": "same ImportError 4 times while editing
- unrelated files"}
+ {"escalate": true, "category": "repetition", "new_evidence": true,
+ "reason": "same ImportError 4 times while editing unrelated files"}
* `conda` is not installed; the agent has tried `conda install` five
ways instead of using the `pip` that earlier output showed present.
- -> {"escalate": true, "reason": "fighting missing executable instead
- of adapting"}
+ -> {"escalate": true, "category": "repetition", "new_evidence":
+ true, "reason": "fighting missing executable instead of adapting"}
* Task: "make the provided integration tests pass." Recent turns:
renaming variables and reformatting docstrings; tests not run in 8
- turns. -> {"escalate": true, "reason": "drifted to cosmetic edits,
- verification abandoned"}
+ turns. -> {"escalate": true, "category": "drift", "new_evidence":
+ true, "reason": "drifted to cosmetic edits, verification abandoned"}
* The agent says "All tests pass, task complete" but the last visible
test output shows "2 failed, 11 passed". -> {"escalate": true,
- "reason": "claims success contradicted by latest test output"}
+ "category": "false_progress", "new_evidence": true, "reason":
+ "claims success contradicted by latest test output"}
* The agent wrote a reproduction script that exits 0 without invoking
the code path the issue describes, concluded "bug not reproducible",
- and is wrapping up. -> {"escalate": true, "reason": "reproduction
- never exercised the reported code path"}
+ and is wrapping up. -> {"escalate": true, "category":
+ "false_progress", "new_evidence": true, "reason": "reproduction never
+ exercised the reported code path"}
* Two turns of edits, one failed build, then a fixed build and a
- passing test. -> {"escalate": false}
+ passing test. -> {"escalate": false, "category": "none",
+ "new_evidence": false, "reason": "latest build recovered and passed"}
* `npm install` has been running for one turn with no output yet. ->
- {"escalate": false} — slow command, not a stall.
+ {"escalate": false, "category": "none", "new_evidence": false,
+ "reason": "command is still running"}
* Four different serialization libraries failed to import; the agent
is now writing the converter with a fifth approach it has not tried
- before. -> {"escalate": false} — sequential alternatives are
- adaptation, even when none has succeeded yet.
+ before. -> {"escalate": false, "category": "none", "new_evidence":
+ false, "reason": "latest turn changed approach"}
* Task: tune a slow batch pipeline. The agent is investigating why
the message broker fails to start, since the pipeline cannot be
- measured without it. -> {"escalate": false} — unblocking
- verification serves the task.
+ measured without it. -> {"escalate": false, "category": "none",
+ "new_evidence": false, "reason": "working to unblock verification"}
Do not emit markdown, commentary, or chain-of-thought — only the JSON
object.
diff --git a/crates/libsy/src/prompts/escalation/schema.json b/crates/libsy/src/prompts/escalation/schema.json
index 977c41882..e98b3c5fd 100644
--- a/crates/libsy/src/prompts/escalation/schema.json
+++ b/crates/libsy/src/prompts/escalation/schema.json
@@ -10,12 +10,21 @@
"type": "boolean",
"description": "True when the run is likely doomed without escalation to the strong tier."
},
+ "category": {
+ "type": "string",
+ "enum": ["none", "repetition", "false_progress", "drift", "desperation", "capability_gap"],
+ "description": "The single trouble pattern supporting escalation, or none when escalation is false."
+ },
+ "new_evidence": {
+ "type": "boolean",
+ "description": "True only when the newest assistant turn or its resulting tool output adds evidence for the named trouble pattern."
+ },
"reason": {
"type": "string",
"description": "One short sentence naming the trouble pattern, or stating why the run is progressing."
}
},
- "required": ["escalate", "reason"],
+ "required": ["escalate", "category", "new_evidence", "reason"],
"additionalProperties": false
}
}
diff --git a/crates/switchyard-server/tests/server.rs b/crates/switchyard-server/tests/server.rs
index cd21d3c91..3c2656257 100644
--- a/crates/switchyard-server/tests/server.rs
+++ b/crates/switchyard-server/tests/server.rs
@@ -409,7 +409,8 @@ async fn upstream_chat(
.pointer("/response_format/json_schema/schema/properties/escalate")
.is_some()
{
- r#"{"escalate":false,"reason":"making progress"}"#.to_string()
+ r#"{"escalate":false,"category":"none","new_evidence":false,"reason":"making progress"}"#
+ .to_string()
} else if model == "model/classifier" && requests_schema_invalid_verdict {
r#"{"crux":"bounded task","primary_rule":"SUP-1","capability_boundary":"supported","p_solve":0.1,"unexpected":true}"#.to_string()
} else if model == "model/classifier" {
@@ -598,7 +599,12 @@ async fn upstream_responses_silo(
.get("escalate")
.is_some()
{
- json!({"escalate": strong, "reason": "state probe"})
+ json!({
+ "escalate": strong,
+ "category": if strong { "capability_gap" } else { "none" },
+ "new_evidence": strong,
+ "reason": "state probe",
+ })
} else {
json!({
"crux": "state probe", "primary_rule": if strong { "LIM-1" } else { "SUP-1" },
diff --git a/docs/reference/toml_schema.md b/docs/reference/toml_schema.md
index 8bf382a08..4b51a7918 100644
--- a/docs/reference/toml_schema.md
+++ b/docs/reference/toml_schema.md
@@ -253,7 +253,7 @@ Escalation mode serves the weak target first and judges the completed turn. See
| `strong_target` | Yes | — | Target used after the session latches. |
| `weak_target` | Yes | — | Target served before the latch. |
| `prompt` | No | packaged prompt | Replaces the trajectory-judge prompt. |
-| `escalation.confirmations` | No | `2` | Consecutive escalate verdicts required to latch. Above `1` needs a stable session ID. |
+| `escalation.confirmations` | No | `2` | Consecutive fresh-evidence verdicts for the same failure category required to latch. Above `1` needs a stable session ID. |
| `escalation.recent_turn_window` | No | `28` | Trailing messages shown to the judge. |
| `escalation.window_message_chars` | No | `500` | Per-message cap inside that window. |
| `escalation.deescalation` | No | unset | Enables phase-aware de-escalation. Requires a stable session ID. |
diff --git a/docs/routing_algorithms/escalation_router_routing.md b/docs/routing_algorithms/escalation_router_routing.md
index 411446478..994359a08 100644
--- a/docs/routing_algorithms/escalation_router_routing.md
+++ b/docs/routing_algorithms/escalation_router_routing.md
@@ -61,8 +61,10 @@ The route-level `prompt` key replaces the packaged trajectory-judge prompt. It
uses the escalation verdict schema rather than the capability verdict schema.
Switchyard supplies that schema according to the route's `response_format_type`:
through the structured-output request in the default `json_schema` mode, or in
-the prompt in `json_object` mode. When de-escalation is enabled, Switchyard
-appends the phase-specific verdict contract to packaged and custom prompts.
+the prompt in `json_object` mode. The verdict includes an `escalate` decision, a
+bounded failure `category`, whether the newest turn adds `new_evidence`, and a
+short `reason`. When de-escalation is enabled, Switchyard appends the
+phase-specific verdict contract to packaged and custom prompts.
## How the decision works
@@ -72,8 +74,10 @@ For each turn on an unlatched session, Switchyard:
2. Appends that reply to the transcript and asks the judge to rule on the
completed turn. The judge therefore rates work the weak model actually did,
not a prediction about work it might do.
-3. Increments a consecutive-escalate streak on an escalate verdict, and resets it
- to zero on a decline.
+3. Increments a confirmation streak only when an escalate verdict cites fresh
+ evidence for the same failure category as the preceding vote. A different
+ category starts a new streak at one; a decline or stale evidence resets it to
+ zero.
4. Serves the buffered weak reply when the streak has not yet reached
`confirmations` — so a judged turn that does not escalate costs one weak call
plus one judge call, and no strong call.
@@ -119,7 +123,7 @@ configuration, so a bare `escalation = {}` is a valid, tuned route:
| Key | Default | Meaning |
|---|---|---|
-| `confirmations` | `2` | Consecutive escalate verdicts required before the session latches to strong. Must be at least `1`. |
+| `confirmations` | `2` | Consecutive fresh-evidence verdicts for the same failure category required before the session latches to strong. Must be at least `1`. |
| `recent_turn_window` | `28` | Trailing messages shown to the judge on top of the anchors. Must be at least `1`. |
| `window_message_chars` | `500` | Per-message truncation cap inside that trailing window. Must be at least `50`. |
@@ -146,11 +150,13 @@ weak_cooldown_calls = 8
With this table present, Switchyard marks judge input as either
`EFFICIENT_EVALUATION` or `STRONG_EVALUATION`. In the strong phase,
-`escalate: true` keeps the strong tier. An `escalate: false` verdict can release
-the next request only after `strong_min_calls` is reached and the configured
-confirmation streak is complete. A timeout, error, or unparseable verdict
-retains the strong tier. Omitting the table preserves the permanent latch and
-does not add phase markers to judge input.
+`escalate: true` keeps the strong tier; the router reads only `escalate` there,
+so `category` and `new_evidence` are reported but do not affect release. An
+`escalate: false` verdict can release the next request only after
+`strong_min_calls` is reached and the configured confirmation streak is
+complete. A timeout, error, or unparseable verdict retains the strong tier.
+Omitting the table preserves the permanent latch and does not add phase markers
+to judge input.
The strong-phase verdict is judged against the trouble that caused the
escalation: the packaged rules release only once the failure that triggered the
@@ -179,8 +185,8 @@ route. The request still succeeds, but its temporary state cannot carry into the
next request.
Anchor and transcript caps remain fixed. Set the route-level
-`max_output_tokens` key to change the judge's reply budget. Any decline still
-resets the streak to zero.
+`max_output_tokens` key to change the judge's reply budget. Any decline or
+verdict without new evidence resets the streak to zero.
## Run the route
@@ -226,6 +232,10 @@ per-session routing stats under the judge's model id, tagged with the
`classifier` tier — so per-session token accounting includes judge overhead
alongside the tiers the session was served by.
+The server log records each parsed escalation verdict's `escalate` decision,
+category, and `new_evidence` flag. The judge's free-form reason is neither
+retained nor logged.
+
## When not to use escalation routing
- **One-shot requests.** No trajectory to judge. Use